Crafting Questionnaires to Unlock the Full Power of Technological Fraud Detection

Learn how to enhance fraud detection in online surveys by strategically designing questionnaires with open-ended, grid, and trap questions that maximize the effectiveness of advanced technological tools.

The rise in sophisticated online survey fraud, driven by factors like the COVID-19 pandemic, the professionalization of fraud through social media, and modern technologies like AI, has created significant challenges for market researchers. As discussed in our previous article, traditional fraud detection methods are no longer sufficient.

To effectively combat this issue, it's crucial to pair advanced technological tools with well-crafted questionnaires. The design of your survey plays a vital role in enabling these tools to identify and filter out fraudulent responses effectively. In this article, we'll explore strategies for structuring open-ended questions, grid questions, and trap questions to enhance fraud detection, ensuring the integrity of your data in an increasingly complex environment.

Crafting Fraud-Sensitive Questionnaires

Open-Ended Questions:

In the past, open-ended questions in online surveys were often avoided or restricted to specific cases, such as brand recall, due to the increased time and cost of evaluation and concerns that mandatory open-ended questions might lead to higher dropout rates. However, with modern AI solutions that enable automated analysis, the effort required for evaluating open-ended responses has significantly decreased.

Additionally, open-ended questions can be included in questionnaires for quality control purposes only. They have proven to be highly effective in combating new fraud tactics, as they allow for both substantive checks (e.g., assessing the meaningfulness of responses) and analysis of input patterns (e.g., detecting copy-paste behavior).

• Mandatory: Fraudsters often avoid answering open-ended questions if they can, and inattentive participants tend to skip them too. Therefore, making these questions mandatory is crucial. If designed appropriately (e.g., not too many or overly difficult questions), the dropouts caused by mandatory open-ended questions can actually improve data quality by filtering out disengaged participants.
• AI-Friendly: Open-ended questions should be well suited for AI-assisted quality checks. They need to provide a frame of reference for the AI to assess the meaningfulness of responses. Questions like "Would you like to tell us anything else?" are less effective because they lack a content framework, making it difficult for AI to evaluate answer quality.
• Detecting AI-Generated Responses: To identify AI-generated responses, questions that are emotional or opinion-based are particularly useful, as chatbots struggle to authentically replicate personal opinions or emotional expressions.
• Strategic Placement: For thorough quality assessment, it is recommended to include at least two open-ended questions in a questionnaire, strategically placed, such as one at the beginning and one at the end. In longer surveys, a respondent may momentarily lose focus and provide a nonsensical answer, but if this occurs more than once, it significantly increases the likelihood of poor data quality. This distribution also helps evaluate participant engagement throughout the survey. Additionally, analyzing duplicates or partial duplicates within a single interview, as well as across multiple interviews, can further aid in quality assessment.

Grid Questions

Grid questions are favored by market researchers for their efficiency in both answering and evaluating. They also provide a useful tool for in-survey quality checks by allowing the analysis of participant click behavior for signs of inattentiveness or fraud. Their effectiveness is maximized under the following conditions:

• Sufficient Number of Statements: To accurately determine whether responses are genuine or arbitrary, a grid question should include a sufficient number of statements. Our experience suggests a minimum of seven statements for this purpose.
• Appropriate Number of Options: The number of response options should be inversely related to the number of statements in the grid. As the number of statements increases, fewer options are needed. We recommend at least three options to ensure reliable quality checks.
• Inverted Statements: Statements should be phrased to create potential inconsistencies or contradictions if answered uniformly. This can be achieved by including both positive and negative statements about the subject being examined, ensuring that the responses require thoughtful consideration.

Trap Questions

Incorporating trap questions into a questionnaire can be an effective way to verify whether participants are paying attention and responding honestly. Trap questions can take various forms, such as differently worded repeat questions, prompts to select a specific answer option, or questions about non-existent options. The latter, for example, can expose over claiming, where respondents—often fraudsters—select as many options as possible to increase their chances of qualifying for the survey.

The primary purpose of trap questions is to assess the quality of responses rather than to gather content for analysis. Careful implementation is essential. Consider the following:

• Use Sparingly: Incorporate only a maximum of two trap questions to avoid overwhelming or irritating participants.
• Design Subtly: Ensure that trap questions are not easily recognized as such to maintain their effectiveness, as revealing their intent could lead participants to adjust their responses to match the study's criteria.
• Avoid Sole Reliance: Use trap questions as part of a broader quality assurance strategy rather than the sole criterion for evaluating interview quality. Over reliance on trap questions could unfairly penalize otherwise careful participants who make a mistake, leading to their exclusion from the survey and the loss of any incentives they might have earned.

Conclusion

In today's environment of sophisticated online survey fraud, traditional detection methods alone are no longer enough. To ensure data quality, it's crucial to combine advanced technology with smart questionnaire design. By incorporating well-structured open-ended questions, grid questions, and trap questions, you can significantly boost the effectiveness of fraud detection.

A carefully crafted questionnaire not only filters out fraudulent responses but also enhances the accuracy of your data. As fraud tactics evolve, staying ahead requires a strategic blend of technology and design to maintain the integrity of your research.