Surveys bridge the gap between customers and brands. The questions asked through the survey serve as an outlet for customers to express their emotions and opinions.
At the same time, brands can leverage survey questions to learn more about the customers and what they feel about their products and services. However, asking biased survey questions can send all your effort and time down the drain.
In this extensive guide, we take a deep dive into the world of survey questions and address the impact of biased questions on their effectiveness.
We will discuss different types of biased survey questions and learn about them through examples. Moreover, we will share tips you can implement to avoid using biased survey questions once and for all.
Let’s get started!
Collect feedback with JustFeedback
JustFeedback helps your business increase profits and reduce risk by improving your customer experience
What is a biased survey question?
Biased survey questions are ones that are not neutral.
As the name suggests, they drive respondents toward a specific response and hinder their ability to provide honest feedback. Biased survey questions are generally vague and unclear, and lead to confusion and chaos.
As we discuss the different types of biased survey questions and their examples, you will realize how these biases put customers in a tough spot and impact their responses.
A biased survey question can affect the effectiveness of the survey and you might even end up collecting incorrect responses unintentionally.
Such unreliable survey responses may lead to bad decision-making, especially if the brand is using surveys to identify cracks in its customer experience strategy.
How does survey bias impact business?
A company’s reputation is attached to the customer surveys. The worse the survey questions, the more the reputation declines.
As a brand, you should prioritize effective and bias-free survey question writing as modern-day customers tend to opt out of surveys if the questions are vague, confusing, and lack context.
Since answers to biased survey questions are inaccurate and often skewed, you might even end up jeopardizing any data-driven strategy running in your company.
Besides, respondents might render your brand as unethical and unprofessional if you ask them biased survey questions.
10 types & examples of biased survey questions
Let’s move forward and discuss the various types of biased survey questions. We have also included examples of biased questions to facilitate better understanding.
1. Double-barreled questions
As the name suggests, these are biased survey questions that are essentially two questions disguised as one. At the same time, the respondent can only give one answer to both the disguised questions. For example,
Double-barreled question — “Was the product sturdy and functional?”
Now this is a question that is inquiring the respondent about two different aspects of the product — material strength and functionality.
The respondent has only one opportunity to rate both metrics. The product might have got the job done, but that doesn’t necessarily mean it is sturdy, and vice versa.
2. Leading questions
Manipulation takes a whole new level with leading questions. These are biased survey questions that have only one motive — to lead the respondent to a certain answer.
Leading questions are framed as a statement seeking confirmation or denial from the respondent. For example,
Leading question — “The product helped solve your problem. Correct?”
This is quite an obvious leading question as it explicitly expresses the preferences of the interviewer.
The only way to avoid leading questions in a survey is by providing respondents with adequate freedom and refraining from adding personal opinions to the question.
3. Dichotomous questions
As the name suggests, dichotomous questions are biased survey questions that offer respondents only two answer choices.
Brands using dichotomous questions find it easier to interpret results and make analysis simpler. Here is an example of a dichotomous question,
Dichotomous question — “Do you plan to purchase our product?”
Again, the question pushes respondents into a corner. Dichotomous questions make respondents give a yes/no answer. Therefore, they will either choose one of the two answer choices or simply opt out of the survey.
The best way to deal with dichotomous question bias is by adding a “maybe” or “don’t know” option for respondents who wish to express uncertainty.
4. Social desirability bias questions
The respondents to these survey questions are humans. We are social beings and a significant percentage of our decisions are based on social approval.
Brands can leverage such social desirability bias by making respondents choose a socially approved answer to their survey questions. For example,
Social desirability bias question — “Do you plan to move into a more comfortable accommodation?”
For most respondents, the answer to the question would be “yes” because we all seek growth as individuals.
Now, there will be respondents who are happy with their present accommodation. Thus, social desirability bias in a survey question makes it difficult to solicit honest answers.
5. Assimilation bias
The way you structure a survey can also impact the answers respondents leave.
While a change in words can impact survey results, a change in structure can also deform the meaning of the questions, especially when multiple questions are stacked together. For example,
Assimilation bias — “Did the product function properly?” followed by “How would rate the product on functionality?”
Functionality can be quite a broad topic. When these two questions are stacked together, respondents might rate the product on functionality thinking that the product functions properly, as asked in the first question.
Moreover, respondents might ignore other metrics of product success since the question focuses only on functionality.
6. Absolute questions
The biased survey questions containing absoluteness are often presented using words like “all, never, none, always.” The reason behind this is the ability of these terms to push the respondent to a corner and retrieve a desired response. For example,
Absolute questions — “Have you never shopped at our store?”
These biased survey questions are framed in a way the respondent feels accused of not being part of the brand. The “pushiness” of the questions can make people leave extreme responses, without even considering their options.
However, brands can always make a conscious choice of omitting such words and sentences and rephrasing the question to avoid absoluteness.
7. Loaded questions
Another type of biased survey question worth mentioning in the post is a loaded question. Now, these are survey questions that put forward an assumption and push respondents to consider it.
One should note that the assumptions made in the question can be false for some respondents. Regardless, a loaded question functions similarly to a leading question — directing people to a desired response. For example,
Loaded question — “How much did you enjoy using our product?”
The question comes with a presumption that customers enjoyed using the product. The only thing the surveyor wants to know is how much the customers enjoyed it. This survey question falls flat for people who didn’t enjoy using the product at all.
8. Vague questions
This biased survey question type is a classic. While the other biased questions mentioned in the post only tend to limit respondents’ freedom, vague survey questions add a touch of confusion and chaos.
A vague question, as the name suggests, includes words that have no definite meaning and are open to different interpretations. For example,
Vague question — “Was the marketing campaign a success?”
Now, “success” can have different meanings for different people. Therefore, it is best to avoid vague biased survey questions. It can be achieved by focusing more on definite parameters and qualities and less on subjective interpretations.
9. Demand characteristics bias questions
Most customers have a sense of why you are running surveys and soliciting feedback. The very realization that companies are running surveys to get better can make respondents answer the surveys in a certain way.
Demand characteristics bias makes respondents answer surveys assuming what you need or what is the most favorable answer. For example,
Demand characteristics bias — “We are considering diversifying our product line-up. Should we go ahead with the decision?”
Many respondents will say “yes” because they assume diversifying is the way to go or something you need. The best way to avoid demand characteristics bias is by removing assumptions in the question.
10. Contrast effect
The contrast effect is quite similar to the assimilation effect discussed earlier. Here, respondents will have to consider two or more questions when responding to the survey.
However, respondents might assume that since you have already covered an issue in the first question, they don’t have to consider it when answering the second question, and so on. For example,
Contrast effect — “Did you purchase the product easily on our website?” followed by “How would you rate your customer experience?”
Even if the respondents found it easy to purchase the product on your website, they might not have liked the overall customer experience, and vice versa. In other words, you will only end up consolidating confusing data.
Factors contributing to biased survey questions
What are the factors that can lead you to write biased survey questions? Since there are several elements involved in survey questions, you will have to consider the following factors to avoid writing biased questions —
1. Words
The choice of words and phrases in a survey question can have a major impact on how participants respond. The question can trigger biases if the words are used to imply a certain response. Use words/phrases that support the question’s intent.
2. Order
We have already discussed how multiple questions stacked together can impact participants’ responses. Assimilation and contrast biases are prime examples of how the order of questions can drastically change responses.
Make sure the survey questions are in the correct order before you hit send.
3. Response
Answer choices depend on the type of survey questions.
However, the answer options you provide with the questions can lead to bias if they don’t provide adequate freedom to the respondents. We suggest you include all possible responses to help respondents provide honest customer feedback.
4. Context
As a brand or surveyor, you have got to keep things neutral. The context can be simplified by providing the participants with adequate information regarding the survey’s purpose. Use neutral language and tone to not trigger assumptions and biases.
How do we avoid biased survey questions?
Since biased survey questions can significantly impact the effectiveness of responses and lead to confusion and misinformation, we highly recommend avoiding biased survey questions by following these tips.
The very first thing you’d want to ensure is that you understand the different types of biased survey questions. We have discussed some of the most important ones in the post. Read them thoroughly to make bias identification easier.
Always go for multiple question structures when drafting survey questions. This makes it easier to avoid biases as you are deliberately randomizing the question order.
Besides, you can leverage different types of questions with different rating scales. It can help participants share feedback in unique ways.
Another effective way to avoid biased survey questions is to let fresh eyes have a look at the questions. Gather feedback from your colleagues about the questions and identify any potential source of bias. Make changes accordingly.
We highly recommend you pre-test the survey to evaluate the effectiveness of the questions and survey flow. Create a small group of respondents and test the questions on them.
Lastly, you’d want to avoid any assumptions, overloading, confusing language, and jargon when writing survey questions. Make sure you write clearly and consider user experience as well.
Wrapping Up
Survey bias, as discussed in the post, can quickly flush all your hard work down the toilet. Therefore, you should stay alert and ensure that the questions are well-tested before pushing the “send” button.
We highly recommend you look at the survey questions from the perspective of respondents and see if you can answer the questions without bias.
Moreover, keep things simple and your questions neutral when drafting a customer survey. Did you find this post helpful? Stay tuned for more informative posts in the future.