How not to design a survey
United Nations asks: ‘Would you rather be free, or well-fed?’
Imagine that I give you 16 attributes that your country might have. You have to choose six things from a list that includes:
- Freedom from violence
- Political freedom
- Nutritious food
- Better health care
- Equality between men and women
- Other good stuff…
Do I learn anything at all about your priorities from this type of survey?
Would I get to understand the aspirations you have for your children? Would you tell me which values you desperately want to keep, or would you highlight where you want your community go?
Giving you a list of characteristics that a good society might have is clearly a strange way to structure a survey. When companies do market research, they do not ask potential customers: Do you care about prices? Do you prefer durable products? Do you like non-poisonous products?
All that marketers would have learned from a silly survey would be: people like low prices and high quality. The “research” would tell them nothing about how people make trade-offs. It would be hard to argue that it would give us even suggestive evidence about what people value the most.
Did you ever have to choose between an honest government and job opportunities? No. Because these two things probably go together.
When voting, did you ever consider a candidate who wanted to “take action on climate change” but refused to “protect forests”? Probably not because, again, these goals go together.
The world does not work like a check list. And our desires do not work that way either.
Interpreting the “top choices” of a list of features is extremely problematic. It can be costly to combine some features together, while other attributes could automatically reinforce each other. Goals could interact in major or in subtle ways. Or there could be no obvious trade-off at all.
So what insights shall we gain if we run the following survey?
[there are 16 choices in total, some are omitted here for brevity]
If you live in the United States, you probably feel that you have political freedom. But you might still worry that the food you eat is not that safe.
You probably want better health care. You want your community to be safe, you want reliable power at home, and you probably also want a good education.
That’s already six priorities. Do you care about “equality between men and women”? Do you think people should be free from discrimination and persecution? Even if you are not particularly selfish, you do not have any choices left on the UN survey.
Some friends in my social networks reacted to this survey with disappointed “look how few people care about climate change” comments. If you superficially read the results, you might have the impression that climate change is barely a top 10 concern:
As it happens, the data do not actually tell us how many people care about climate change.
Some people were not able to check all things they valued, and probably did not treat the survey like a referendum in the first place. Worryingly, the website that showcases the results does not state the exact wording of the question. The reader will have to take the survey himself to learn that the wording was: “Which of these are most important for you and your family?”
If you give me a box of Belgian chocolates and tell me “pick the six pieces that you crave the most” then you should not accuse me of thinking that the remaining pieces are not delicious. Surveys on more important topics are no different.
Among international organizations, it is now fashionable to say “we are listening to you.” But legitimacy is not gained by enabling universal participation in questionable polls. People do not have greater voice just because you plot their opinions on a pretty chart.
The UN website claims that “we aim to capture people’s voices, priorities and views, so world leaders can be informed as they begin the process of defining the next set of global goals to end poverty.” Usually, we say that the ideal policy process is data-driven. This survey shows why that is not always the case.
I shudder to imagine that world leaders might actually take a flawed survey—a half-finished box of chocolates—and treat it as something that captures the public opinion.
