|
It has happened to every researcher at one time or another: We write a survey that just doesn't perform up to our expectations. Somehow, key questions going in are left unanswered going out.
Inaccurate and irrelevant results are caused by rushing through the survey design in order to get into field on schedule. Maybe an ad campaign is scheduled, and the pre-wave absolutely has to be out of field by a certain day. Or maybe project approval takes longer than expected, and the time to design and write the survey is crunched into an impossibly short time frame. These scenarios are all too familiar. But by investing some extra time and effort in survey design, many field and analysis problems can be greatly reduced or even totally eliminated.
Things to think about before writing the survey
Effective survey design starts before a single question is ever written. It may sound obvious, but it can't be stressed enough that you must be clear on the study's objectives. Research objectives should be specific. The only way to really know what questions to ask in a survey is to know exactly what you are trying to learn. An example of a well-defined objective might be "To identify and quantify the importance of factors in our customers' purchase decisions." A much less well-developed objective might be "To see why our customers buy our products."
The objectives should also be focused, meaning you must resist the temptation to get additional data "since we've got them on the phone already." The most common way that objectives become fuzzy is by attempting to get buy-in from as many people in the organization as possible. This is often done by including a question on the survey from every department.
The problem is each department has different informational priorities. As the research manager, it is your responsibility to deliver relevant, useful information. You can do this only if your objective is well defined and focused. In the long run, the reputation you develop for delivering insightful research will get you all the buy-in you need.
This is also the time to consult the statistician who will be responsible for the analysis. The most efficient question formats elicit data that lends itself to analysis with as little back-end manipulation as possible. A statistician can suggest what kinds of questions will fulfill these needs.
A basic example is the rating-scale dilemma, 3-point, 5-point, 10-point, or more? If the objectives require that respondents be divided into one of two groups, then large scales are probably not necessary. However, if you plan to incorporate cluster analysis, which can segment respondents into numerous sub-groups, then scales with more widely dispersed points may be required.
Things to keep in mind when writing the survey
Make sure every question is appropriate for the interviewing method. For example, you can't rely on visual props in a phone survey, and you can't do aided/unaided recall in a self-administered survey.
Make sure that the respondent is able and willing to answer the question. If a respondent is unlikely to have the information you are looking for, at best you will get "don't knows," and at worst you will get misleading data. Similarly, questions that are intimate or sensitive in nature may result in a high percentage of non-responses. In both cases, you run the risk of alienating the respondents, either by making them feel stupid for not knowing the answer or by making them feel that their privacy has been invaded.
Another way to alienate the respondent is by using jargon. As marketers, we have developed a particular vocabulary to describe marketing concepts. For us, they are clear and concise terms that make communicating about our field more efficient. But for the respondent, it sounds like highfalutin marketing-speak.
For example, an "attribute" is something we are all familiar with, but it is unlikely that the typical survey respondent will know what we mean. Rather than ask them to "rate attributes," it is better to ask them to "agree or disagree with statements" and tease out the attribute ranking in the analysis.
Always pre-test
No one thinks of everything. This is why you should always pre-test. When the survey is administered to real respondents, problems with wording, skip patterns, and question relevancy will quickly come to light. Think of the few interviews that may have to be thrown out as a cheap insurance policy against winding up with 1,000 useless responses.
When choosing a supplier to conduct your research, take time to make sure they understand the objectives, not just the specs. Experienced researchers can suggest alternative approaches or point out potential problems before the study goes to the field. Use your supplier's statisticians to think through the analysis up front so that all of your bases are covered. If your supplier doesn't understand analysis issues, find one that does. Your research projects are too important to take chances.
|
