When we look at the correlation between SAT score and bias, we find that not only do smarter people (as determined by SAT score) assume that everyone else is more susceptible to thinking errors than they are, but also smarter people are in fact more vulnerable to common mental mistakes than the average person is. This “intelligence bias” has serious ramifications for designing a survey for experts – the bread and butter of the NewtonX survey team. Highly intelligent people tend to have strong beliefs. And in the case of expert surveys, where you are interviewing people with extremely niche knowledge (so niche that only 50 people in the entire world possess it), this bias can become even more exaggerated.
This guide to designing expert surveys for some of the smartest people out there is an extract from the NewtonX survey design center playbook. The advice in this article is sourced from leading quantitative and qualitative survey experts from leading consulting firms, survey institutes, and other prominent firms.
1. To begin your expert survey, start from the end
Ask yourself what results you are trying to get and what you wish to prove with these results. Your survey questions should be a means to an end. Don’t think “it would be interesting to ask this” — instead, you should always ask yourself, “what will I be able to demonstrate with the results of this question.”
2. Negative formulations are confusing
Jane Austen might do it, but that doesn’t mean you should. Negative formulations, such as asking “How unlikely are you to invest in a third-party cloud data center” can easily be misread as their positive counterparts — thus skewing your results. Ask questions with a positive sentence construction for more uniform and accurate results.
3. Test your questions – can you find a tester in the target audience?
Before deploying your expert survey, give your sample questions to a few testers who will try out your survey. This will accomplish two things: it will test the clarity of the questions and test the overall length of the survey. We cannot emphasize this enough — even the most advanced survey designers systematically use testers, and get extremely useful feedback out of it! Ideally, try to find testers who are as close as possible to your target audience. Even one tester is vastly better than relying only on yourself to test your survey.
4. Give specific temporal parameters
Don’t just ask, “How much does your organization spend on customer support.” Open ended questions like this leave room for answers like, “Far too much by my standards.” Instead, give parameters around your question, such as “What was your dollar spend in 2017 on Customer Support employees and tools. Was this higher or lower than 2016.”
5. Use response scales when possible to guide respondents through your expert survey
If you’re looking for specific answers within a range on your expert survey, offer a response scale. For instance, to use the customer support example, you could give options for “over 1 million,” “between 800,000- 1M”; etc. When you do this, ensure that your ranges aren’t so large that they seem misleading (for instance, 500,000- 1M), that there isn’t a gap between options, and that the range of options is well distributed to capture responses in an even manner and avoid getting more than 20% of the responses in the extremes (ideally your responses should follow a normal distribution within the ranges provided). Rephrase Yes/No questions when possible to include phrases such as “How much,” “How often,” or “How likely” so that you can include a response scale for richer data.
6. Long questions create bad answers. When in doubt, split the question
Avoid garden path or run-on sentences. Splitting the question into two separate queries is the best path to clarification in 90% of use cases. Don’t risk confusing your audience with questions like “What are the biggest barriers to entry that you see in blockchain today, and what are the hottest trends?” Instead, split the question into two separate ones so that you prompt the most robust answer possible to each of your questions.
7. Indicate desired answer length on your expert survey
If you want an answer that has a certain about of detail or elaboration, give your survey respondents an idea of how many sentences you want their responses to be. This is a more specific direction than asking for “in-depth responses,” for example, which can result in differing interpretations of what constitutes in-depth.
8. Be careful what you wish for. Avoid leading questions!
Questions like “How beneficial do you think blockchain technology can be to your business” leads the respondent to think Blockchain is beneficial or should be beneficial to the business. If you are trying to assess what the respondent thinks of Blockchain and whether or not it can be beneficial to its business, this is not the best way to ask. Instead, opt for a neutral question such as “What is your perception of Blockchain?”
9. Jargon is good – if it’s the jargon of your respondents
Do your due diligence before writing your expert survey. Make sure you understand key terms and are employing them correctly — this will help you meet your respondents on their level, and will elicit more in-depth, insightful responses.
10. Make it clear what you want: an opinion is not a datapoint
People tend to want to share their opinions over facts — unless that fact backs up an opinion. Because of this human bias, you want to make sure to specify when you want a number instead of an opinion. Asking for a number and a unit usually removes all room for doubt. For instance, you could ask “In dollars, what is the total addressable market for IoT devices in the manufacturing industry?”
11. Even complex topics can have simple questions on your expert survey
This is the most important point to remember when designing a survey for highly technical experts. These people are experts because they understand extremely complex technologies, and you will get the most out of them by allowing them to respond in-depth to a question that they interpret correctly. If a respondent struggles to answer, this means that the question is not clear and the response of the respondent will lose weight as a consequence. A good rule of thumb for this is ensuring that the respondent can answer your question after a single read. In other words, how “painful” is it to understand your question. A question should never be painful to understand. As our senior survey designers say: “All else being equal, answering your survey questions should be enjoyable.”
