How to get honest answers in a survey

People answer honestly when questions are neutral, options are balanced, earlier questions do not steer later ones, and respondents trust that their answers are private. Most survey bias comes from wording, order and context, so small changes to how you ask can change what you learn.

Sibr is a polling, survey and paid-study app on the Internet Computer, open to Pi Network users and everyone else. The principles below come from survey research and apply to any tool.

Why do people give answers that are not true?

Usually not on purpose. Common reasons:

• The question pushes them. Wording signals the "right" answer.
• They want to look good. People understate behaviour seen as bad and overstate behaviour seen as good. This is called social desirability bias. Pew Research Center notes it is stronger when an interviewer is present than in self-completed surveys.
• They take shortcuts. When a question is hard or the survey is long, people pick a good enough answer, agree with a statement, or choose the first option. Survey researchers call this satisficing.
• They do not have an answer. Without a "Not sure" option, people guess.

How does wording change answers?

Small word changes can move results a lot. Pew Research Center reports that 51% of people favoured letting doctors "give terminally ill patients the means to end their lives", but only 44% favoured letting doctors "assist terminally ill patients in committing suicide". Same policy, different words.

To keep wording neutral:

• Remove loaded words like "great", "unfair" or "obviously".
• Name both sides. "Do you favour or oppose..." rather than "Do you favour...".
• Ask one thing at a time. Split double-barreled questions.
• Avoid agree or disagree statements where you can. Pew notes some people tend to agree with statements whatever they say. Offer a choice between two clear alternatives instead.

How do answer options create bias?

Options can be as leading as the question.

Pew reports that in self-completed surveys, people lean toward options near the top of a list. Pew handles this by randomising option order where the list has no natural order.

Does question order matter?

Yes. An earlier question can set a frame for the next one. In one Pew example, support for legal agreements for same-sex couples was 45% when asked right after a question on same-sex marriage, and 37% without that question before it.

Good practice:

• Ask general questions before specific ones.
• Put sensitive or personal questions near the end.
• Read your survey as a respondent and ask whether any question hints at how to answer the next.

How do privacy and trust affect honesty?

People are more candid when they believe their answers will not be tied to them. Tell people who will see their answers and how they will be used, and only ask for personal details you need. Read How Sibr protects respondent privacy for what Sibr shows creators.

How do I spot careless answers?

Three signals are worth checking:

• Speed. Answers far faster than a careful reader could manage.
• Failed attention checks. A simple instruction, such as "Select Blue for this question", that a reader cannot miss.
• Straight lines and empty text. The same score for every rating, or text like "asdf".

Use these as reasons to look closer, not as automatic proof. Prolific, a research panel, sets clear rules for rejecting people over attention checks: in studies of 5 minutes or more, a person must fail at least two checks before they can be rejected.

Rewards can raise response rates, but they also give some people a reason to rush. That is why quality checks matter most when you pay. See Paid surveys with Pi.

How to do it on Sibr

1. Write neutral questions in the survey builder and use the preview to read them as a respondent would.
2. Add "Other", "None of these" or "Not sure" options where they fit. Choice questions take up to 10 options.
3. Order questions from general to specific, and keep personal questions near the end.
4. Make sensitive questions optional so nobody is forced to guess.
5. For paid studies, add up to three attention checks on choice questions and set an estimated time. Answers that fail checks or come in far too fast are held for your review instead of being paid at once. One failed check flags a study under 5 minutes; two failed checks flag longer studies.
6. Respondents on Sibr appear in results by response number, never by username, which supports candid answers.

Sources

• Pew Research Center, Writing Survey Questions: https://www.pewresearch.org/writing-survey-questions/
• Prolific, attention and comprehension check policy: https://researcher-help.prolific.com/en/articles/445153-prolific-s-attention-and-comprehension-check-policy
• Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213 to 236.