Open-ended questions usually provide much less value than structured questions in online surveys. What could be done to improve this?
Peter Winters 
19 Feb, 2007
Getting more value out of open-ended questions asked in online surveys
It is interesting to discover what people really think when you discuss things informally! At last year’s EphMRA Athens Conference, I was chatting to one pharmaceutical company researcher who, as the sun set over the blue waters of the Aegean, said to me “you know, Internet Research may be good for many things, but it is no good for open-ended questions”.
She had a point – online surveys, or indeed quantitative research surveys generally, rarely seem to provide much insight from open-ended questions; by far the greatest value from online surveys usually comes from the structured questions. This is a shame as Internet research does present an opportunity to generate verbatim responses directly from respondents. What could be done to improve the quality and usefulness of this type of data?
Making open-ended questions reasonable for respondents to answer
The first principle is that the questionnaire should be designed in such a way that any open-ended question be reasonable for respondents to answer. This means, inter alia, that it should be made clear to respondents what is expected of them – and often it is appropriate to advise respondents to summarise their immediate thoughts briefly, in one or two sentences.
Open-ended questions are very prone to “order-bias”. If a number of concepts are being tested, then be very careful about asking respondents to provide an open-ended response one after another. Subsequent responses can be less detailed or include “same as above” type comments. Indeed, if a questionnaire has a number of open-ended questions, it is likely that respondents will make comments in the “wrong” place – such as describing a disadvantage to a product in question asking about “advantages”. Consequently, it is often worth doing an analysis on an holistic basis, respondent-by-respondent, as well as question-by-question.
Try and avoid situations where some respondents are asked to write about situations which are of little or no interest to them – such as comment on a concept which they liked “a little”. Concentrate on those who really liked it; or really disliked it!
Use theoretical models to categorise the data
It is often valuable to categorise responses according to a theoretical model. In cases where the online study follows-on from a qualitative study, it is usually possible to use constructs developed at this prior stage. We used such an approach for a packaging study we conducted in 2006. The qualitative research had identified attributes of the packs which could be categorised into functional needs, secondary benefits and style characteristics – and the code frame was designed around this taxonomy.
For certain types of study, it is appropriate to look to psychology for guidance in how to employ and interpret open-ended questions. One area of particular interest for brand research is how emotions are dual-coded to associations, and how these combinations vary by individual. Psychiatrists Greenspan and Shanker believe that “this dual-coding of experience is key to understanding how emotions organize intellectual abilities and indeed create the sense of self”1. For example, “a voice is loud and inviting or jarring2”. Open-ended questions can be designed to give another level of insight compared to what is possible with structured lists.
I should mention that there is also an entire academic disciple relating to analysing language use, termed Discourse Analysis, though I personally cannot see a role for this type of approach in quantitative research surveys.
Another area where open-ended questions can be useful is in understanding the language of your target market. For example, it could be that the marketing of your drug involves certain concepts to be expressed in a particular way – such as the terminology to describe the condition, or a unique benefit that your drug can provide. Open-ended questions can explore whether particular language is being used, and in what context.
Analytical tools
So much has changed in market research since I started my career at the British Market Research Bureau in 1985 – but until recently, the analysis of open-ended responses had hardly improved at all since those days. In using the open-ended responses, the analyst could decide whether to have verbatim responses, or have the data coded.
If the data was to be coded, the process of taking involved taking the first (say) 50 respondents and using this to create a code frame; and then having the entire sample coded against this code frame. Although this provided an overall sense of the data, there were a number of shortcomings, which include:
- The coding could not usually handle anything more than simple mentions; more complicated discussions and reasons were lost in the deconstructed ways that code frames were usually designed.
- The analyst could not link the verbatim to the coded data – i.e. drill-down to see what type of things people were really saying within a category.
- The analyst would have great difficulty in quality-checking the coding beyond spot-checking. There is often a subjective element in the coding of data, and consistency of coding approach is often very challenging. Re-working coded data can be an arduous and time-consuming task.
Yet there are now new tools to address many of these issues. One that we use is called Ascribe, from Language Logic, which is a web-based system which allows a high-level of transparency in the way the code-frame is created; an ability to drill-down from codes through to verbatim responses; an ability to compare original and translated verbatim responses in international studies; an ability to re-arrange the code frame, such as rearrange and/or combine codes; and re-arrange nets, once the coding has been completed. Whilst it is still not as easy to manipulate as structured data, it is a vast improvement on the way open-ended data used to be handled.
A complimentary system to Ascribe is the SPSS Text Analysis for Surveys (STAfS) system which is designed to replace the need for manual coding. This system is only available in English, so for international studies it would be necessary to translate all the open-ended responses into English. I asked Michael Vallely, my SPSS trainer at a recent course, to describe how this system works, and he wrote the following paragraphs:
“STAfS takes an iterative approach to categorizing open ended text responses. It extracts words and phrases expressing key opinions and ideas. The extracted words are then used as the basis for categorizing responses. The first step of an analysis is to run the text responses through STAfS extraction engine. STAfS uses sophisticated algorithms to identify recurring words and phrases and assigns these words a ‘type’. Types are higher level concepts that contain more than one term. For example we could have a type called ‘pharmaceutical companies’, terms contained within this type could be Pfizer, Johnson & Johnson, Novartis, etc.
The STAfS engine can be ‘tuned’ to enhance its capabilities. This can involve making words and phrases synonymous with other words, identifying ‘excluded’ words, i.e. words that STAfS can ignore, creating news ‘types’. Tuning the engine entails refining the dictionaries which contain our types, synonyms and excluded terms. Next we generate categories based on the extract terms & types. There are linguistics & frequency based categorization methods. The frequency method generated categories based on the occurrence or terms or types. We can preset a threshold so that a category is created if a term or type has occurred at least ‘x’ number of times.
There are three linguistic based categorization methods using advanced text processing algorithms that incorporate semantic linkage of terms, uni-term repetition and multi-term component recognition. We then modify the created categories by examining the components within each, merging categories or deleting unusable categories. When we are satisfied with our categories we can easily export the results into an appropriate statistics package ready for further analysis.”
End comment
Online research provides a different opportunity to interviewer-led research for collecting and analysing open-ended questions. Online does not really offer the chance to probe and prompt in the same way as interviewer-led research – yet it does provide a consistent platform for generating verbatim open-ended responses which is not influenced by social interaction with the interviewer. As I have argued, I think we could get more value out of open-ended questions used with online surveys.
1 The First Idea: How Symbols, Language and Intelligence evolved from our Primate Ancestors to Modern Humans, (2004), p.56 Stanley I. Greenspan & Stuart G. Shanker
2 ibid
Your response:
Login to respond to this article: