Questionnaire Design: The Trick of Keeping it Short (But Making it Big)

When writing a survey questionnaire there is a strong temptation to keep thinking of more and more information that might be useful, or worse, might be interesting.

The questionnaire gets increasingly longer as we succumb to the idea that it is important to gather all this data.

The result?

The questionnaire becomes too long, boring and tedious, and consequently, the survey participants zone out and either log off or adopt strategies to get through the survey as quickly as possible, without having to give it more thought than is absolutely necessary. 

An interviewer may be able to keep the participant going for longer, offering a reason for them to pay attention, but when conducted online, a questionnaire must do all the work. Participant fatigue of this nature means that although you may collect lots of data, much of it (if not most), is highly dubious or misleading.

Sometimes you can spot it, through flatlining or speeding, where the speed with which a page or the whole survey is completed means that the question and responses cannot possibly have been read.

But fatigue answering, giving answers at random, or making pretty patterns on answer grids (my favourite technique to avoid having to read reams of attitude statements), can be difficult to identify, at least until you come to analyse the data and realise that it doesn’t make sense. We tend to post-rationalise the results we get, trying to convince ourselves or our clients that we have meaningful information, but all too often with that nagging doubt that the data is not very useful.

Ask Yourself: How do we keep participants engaged in the process by using a short and fun questionnaire, but still retain all the data we need? 

Do you really need all that data? How much of the data you collect is interesting, but not part of your core objectives?

Wherever that is the case, it should go. 

If it really is interesting but not part of your core objectives for this project, then consider whether it should become a separate project in itself. Trying to effectively put two projects together for cost efficiency will only damage them both. The cost of conducting them separately is rarely significantly more and means that each gets the attention it deserves, rather than one being an ill-thought through adjunct to the main survey. 

Ask Yourself: What data can be obtained elsewhere? (Avoid asking it again)

If you are using an online panel to conduct your research, then a lot of demographic information will already be known. Don’t ask it again.

Panel providers may be keen for you to re-ask to ensure the information is up to date, but someone’s age being a year out because of a recent a birthday, unlogged by the panel provider, is not likely to destroy your analysis.

There may be opportunities to infer data, particularly behavioural information because you know from other studies that it is highly correlated with core data. For example, if attitudes to household cleanliness are part of your core data, then how often the participant washes the kitchen floor can be modelled from other data you already hold, saving the need to ask the question. Again, inferences may not be 100% accurate but are likely to be better than data given by a bored participant.

Using what you already know or can infer, and bringing in data from other sources, should be a key part of the questionnaire design.

Cambridge Analytica and the 2016 Presidential Campaign

This way of thinking will have been used by Cambridge Analytica in the promotion of the Donald Trump 2016 presidential campaign.

Researchers took a relatively short survey hosted on Facebook, which was designed as a personality quiz to tell participants what kind of person they were. Cambridge Analytica were then able to use this short quiz to create personality profiles, combining these with information collected from elsewhere on Facebook and other sources, to create political profiles. This model was then applied to millions of other people whose data was known to Facebook, to understand which messages to promote to them for the Trump campaign.

What they did is questionable in terms of ownership of the data, highlighted by the recent investigations into if/where data protection laws were broken.

Many of us can relate to the difficulty of accurately predicting membership of a group when it is not based the full data that was used to create the group. In this case, the political profile. Finding the ‘golden questions’ to predict membership of a group in any segmentation is never completely accurate.

There may be questions about the efficacy of the outcome, while the claims regarding the impact that the work actually had on the result have been inconsistent. Nevertheless, this approach is an example of keeping the primary data collection questionnaire short and interesting, and then importing information from elsewhere to fill in the gaps.

In many studies, you won’t have the breadth of information that these analysts had to fill in those gaps, but if your survey is for your customers and you have a customer database that provides transactional or behavioural data, then you probably don’t need to ask it in the survey.

With some good correlations, you may be able to predict attitudinal segmentation membership from the behavioural, transactional and demographic details you already hold.

By keeping the questionnaire short, you will avoid asking unnecessary questions, make the data more reliable, and avoid turning a willing participant into a bored one. And bored survey participants are what we need to avoid.

Conclusion

To make sure you get the best quality data for the core information that you really need keep the questionnaire as short as possible by:

• Identifying the core questions that you need to ask to address the key business decisions
  
  
• Avoid asking other questions simply because you can or because they are nice to know
  
  
• Combining data from other sources where you can, such as customer transaction data, rather than asking for it again
  
  
• Inferring or modelling data based on the core questions using work you have already done

By keeping the questionnaire short and avoiding asking unnecessary questions you avoid turning a willing participant into a bored one who will give you poor quality, unconsidered data.