Questionnaire Design Risks: The Challenge of Staying Focused

(part of Chapter 1: Know Your Goals)

One of the main reasons MR projects fail is due to lack of an agenda, or the inability to stay focused on an originally projected design. As more individuals become involved with a project, they bring with them the risk of transforming a once precise questionnaire into an excessively long one with illogical flow.

The most important aspects to keep in mind when designing a questionnaire are fit, flow and length. Without these, the questionnaire becomes a burden on the respondents, which directly affects project success. Read on to find out how to avoid the pitfalls of questionnaire design.

Very risky business

One of the most common reasons MR projects fail is because there is no specific agenda. Perhaps it started with one, but then the goals were diluted as different decision-makers started to contribute to the instrument design.

It is a painfully common scenario: You start with a very precise MR study topic. Perhaps it’s a project to test potential messages for a new advertising campaign. However, once word gets out that you’re planning to do a custom MR project, it seems everybody wants to add “just” two or three questions to the questionnaire: “As long as you’re talking to our customers, can you please ask them about distribution channels?” or, “Can you please find out if they’ve seen this recent ad we ran?”

Before you know it, a very precise, focused questionnaire that can be done with excellence has now become a questionnaire with an illogical flow that doesn’t make any sense to the people who are taking the survey. Now, instead of being a nice 10-minute questionnaire, it’s a 30-minute monster. Even worse, the people screened for the primary objective may be unqualified to answer these tangential questions—so the resulting data end up weak—and now all of those unwelcome survey vampires are unhappy internal clients!

Alas, it is time to be the survey cop. Someone has to define, protect, and uphold the agreed upon project priorities. Does that mean you should just say no to these requests? Of course not. Sometimes the requestor can be shown that the answers to their questions already exist. In their excitement to add on to a new project, perhaps they have forgotten to look for existing information.

A little guidance about where the answers may possibly exist could solve the conflict. And if truly no readily available answers exist, it may simply be time to scope a new project to address their specific needs. Once you select the agency for your primary study, you may find some economies of scale by having them bid on this new project as well.

Why Is This So Risky?

If you decide to add questions to your instrument that are not specifically aligned with your primary goals, you have to keep in mind three things: fit, flow, and length.

Fit. Are the people you’re going to recruit for the primary goals also the right people to answer these new questions? If not, you might need to revamp the original sampling plan, which can drive up costs and introduce delays.

• Most studies require very specific samples (the people who will be qualified to take your questionnaire or participate in your focus group).
  
  For example, let’s say the primary goal of this project is to measure awareness of your brand. You are likely interested in measuring awareness only among people who actually buy your product category (you might not want to measure awareness of your frozen pizza brand among people who never buy frozen pizza). You’re going after a very specific audience.


• If you want to add questions outside of the original scope, you have to decide if the same population can answer the additional questions. Perhaps, instead, they would be best answered by a differently qualified group. Maybe now you would have to change your recruiting parameters to include both people who already buy your product category and those who do not.
  
  In the pizza example, perhaps your company is now thinking about extending its brand into frozen hamburgers; so now you want to find out if your brand has permission to enter that category. It may be valid to test your brand’s permissions among people who do and do not currently buy frozen pizza.


• Unless you have a large budget for extra sampling, it’s probably not going to be a good idea. If a project requires dramatically different lines of questioning for two separately qualified groups, it is almost always best to simply conduct two separate studies.

Flow. Another thing to consider before adding extra questions to an instrument is this: Do the extra questions in any way bias participants’ responses to the rest of the questionnaire? That is, are we going to be introducing a topic or a concept that might somehow change how people would respond to subsequent questions? It is important to be careful about this. Will new questions need to be sequenced in a way that is awkward to respondents to avoid biasing responses?

• Of course, you also want to avoid asking questions that are too cognitively difficult or intrusive—you don’t want to cause dropouts over secondary objectives.

Length. For surveys, shorter durations mean higher response and completion rates. Unnecessarily long or complicated questionnaires can drive up costs and drive down data quality—and do so very quickly.

This is an excerpt from the book, "How to Hire & Manage Market Research Agencies," which is available on Amazon. Published by Research Rockstar LLC. Copyright © by Kathryn Korostoff. All rights reserved.

Tweet

Browse "How to Hire + Manage Market Research Agencies" by section

Intro: Maximize Success: Know The Risks
     Understanding Market Research Challenges

Chapter 1: Know Your Goals
     Planning Your Market Research Project
     Defining Appropriate Goals For Your Market Research Project
     Practical Market Research Planning Considerations
     Questionnaire Design Risks: The Challenge of Staying Focused
     Defining Project Goals Through Questionnaire Real Estate

Chapter 2: Defining Your Project's Scope
     Project Scope: Product Concept Testing and Message Testing
     Project Scope: Market Segmentation
     Project Scope: Customer Loyalty and Win-Loss Research
     Project Scope: Brand Awareness and Needs Discovery
     Revealing Sponsorship of Your Market Research Project
     Planning For Your Target Population: B2B or B2C?

Chapter 3: Considering Internal Audience
     Understanding Your Internal Audience: Research Receptiveness
     Dealing With Hard-Core Market Research Skeptics

Chapter 4: Leveraging Existing Research
     Leverage Existing Research

Chapter 5: Planning Your Agency Relationship
     Building a Relationship With Your Agency
     Hiring a Market Research Agency: Managing Your Budget

Chapter 6: RFIs and RFPs
     The Benefits of RFIs
     The RFP: Finding Great Research Partners
     A Few Helpful Reminders on RFPs

Chapter 7: Market Research Agency Selection
     Tips For Reading Market Research Proposals
     Checking Agency References and Follow-Up
     How to Check Market Research Agency Credentials
     Choosing the Right Market Research Agency

Chapter 8: Managing the Agency
     Project Management Planning
     Keeping Your Market Research Project On Track

Chapter 9: A Successful Deliverables Strategy
     Planning Your Market Research Deliverables
     Planning a Great Presentation of Research Results
     Presentation Tactics and Options

Understanding Market Research Challenges

by Kathryn Korostoff, Research Rockstar LLC

Understanding the risks associated with MR is the key to project success. By familiarizing yourself with the risks associated with MR before beginning any projects, you will know what to avoid, resulting in higher success rates and more accurate research results. Read on to learn about the most common market research challenges. Read Article »

Planning Your Market Research Project

by Kathryn Korostoff, Research Rockstar LLC

While planning a market research project may appear to be a fairly straightforward task, in reality, it takes some real time and effort. Agency selection is one of the key factors to project success, so don’t rush the process. Read on to learn about the timeline logistics of hiring the perfect firm and setting things in the right direction from the start. Read Article »

Defining Appropriate Goals For Your Market Research Project

by Kathryn Korostoff, Research Rockstar LLC

When it comes to defining appropriate project goals, it is best to think in terms of questions or hypotheses. Your golden ticket lies in formulating fewer, “high-level” goals rather than multiple ones in need of conquering. Too many goals dilute a project and increase the risk of failure. Read on to learn how to formulate your goals. Read Article »

Practical Market Research Planning Considerations

by Kathryn Korostoff, Research Rockstar LLC

Prior to conducting any project, it’s necessary to decide if custom market research is truly the best choice. If custom MR turns out to be the best option for your project, the next step is to think precisely about defining project goals. Market research is truly a garbage in/garbage out process; start with clear goals, and your chances for market research success are excellent. Read Article »

Defining Project Goals Through Questionnaire Real Estate

by Kathryn Korostoff, Research Rockstar LLC

It is important that everyone on a team understand the primary goals of a market research project. While getting a group of individuals to agree on priorities may be a daunting task, certain exercises such as Chartstorming™ may help you reach a consensus. Read on to learn how to manage your questionnaire real estate through better group collaboration. Read Article »

Project Scope: Product Concept Testing and Message Testing

by Kathryn Korostoff, Research Rockstar LLC

While defining a project scope may seem like a daunting task, it really comes down to two things; knowing you want to get out of your project, and deciding how you’re going to get it. It also helps to know some of the common language used to define market research studies. Read on to learn about product concept testing and message testing. Read Article »

Project Scope: Market Segmentation

by Kathryn Korostoff, Research Rockstar LLC

While defining a project scope may seem like a daunting task, it really comes down to two things; knowing you want to get out of your project, and deciding how you’re going to get it. It also helps to know some of the common language used to define market research studies. Read on to learn about a common project type: market segmentation. Read Article »

Project Scope: Customer Loyalty and Win-Loss Research

by Kathryn Korostoff, Research Rockstar LLC

While defining a project scope may seem like a daunting task, it really comes down to two things; knowing you want to get out of your project, and deciding how you’re going to get it. It also helps to know some of the common language used to define market research studies. Read on to learn about customer loyalty and win-loss research. Read Article »

Project Scope: Brand Awareness and Needs Discovery

by Kathryn Korostoff, Research Rockstar LLC

While defining a project scope may seem like a daunting task, it really comes down to two things; knowing you want to get out of your project, and deciding how you’re going to get it. It also helps to know some of the common language used to define market research studies. Read on to learn about brand awareness and needs discovery. Read Article »

Revealing Sponsorship of Your Market Research Project

by By Kathryn Korostoff, Research Rockstar LLC

Defining your project scope also includes whether or not to reveal the sponsor of your market research study. Read these guidelines for help making this decision based on best practices and common exceptions. Read Article »

Planning For Your Target Population: B2B or B2C?

by By Kathryn Korostoff, Research Rockstar LLC

B2B and B2C research are two very different animals. Which one are you doing? The answer has implications for partner selection, questionnaire design, and sample source options. Read Article »

Understanding Your Internal Audience: Research Receptiveness

by By Kathryn Korostoff, Research Rockstar LLC

There are multiple facets to consider about an internal audience’s readiness for market research. These items range from an audience’s receptiveness to specific techniques, all the way to presentation of data. It is important to take note of the your internal audience’s abilities and preferences to make sure they will be able to receive and apply research results successfully. Read on to learn about aspects to consider when assessing your internal audience’s readiness for MR. Read Article »

Dealing With Hard-Core Market Research Skeptics

by By Kathryn Korostoff, Research Rockstar LLC

While we all hope for an accepting reception from colleagues when it comes to research findings, it’s a rare occurrence for everyone to be on board. Dealing with internal skeptics is no easy task, but a few precautions taken prior to delivering your research findings can greatly reduce disruption risk. Read on to see what you can do to get skeptics on your side. Read Article »

Leverage Existing Research

by Kathryn Korostoff, Research Rockstar LLC

Before you jump into your primary market research head first, it’s important that you first seek out existing, relevant research. This not only saves your organization time, but also prevents wasted funding on a project that would have duplicated existing results. If finding relevant research seems like a daunting task, don’t worry, there are multiple sources you can turn to. Read on to learn how to get started. Read Article »

Building a Relationship With Your Agency

by By Kathryn Korostoff, Research Rockstar LLC

Before you select a market research agency, think carefully about the type of relationship you want. It’s important that the agency you choose can work with your team in a way that will truly meet your needs. There are hundreds of agencies out there; they do not all have the same styles or collaboration approaches. Read on to learn how to build a great relationship with your chosen agency. Read Article »

Hiring a Market Research Agency: Managing Your Budget

by By Kathryn Korostoff, Research Rockstar LLC

Before you select a market research agency, think carefully about the type of relationship you want, especially when planning your budget. It’s important that the agency you choose can work with your team in a way that will truly meet your needs. There are hundreds of agencies out there; they do not all have the same styles or collaboration approaches. Read on to learn how to build a great relationship with your chosen agency. Read Article »

The Benefits of RFIs

by By Kathryn Korostoff, Research Rockstar LLC

Taking time to write either an RFI or RFP may seem like a burden, but in the long run as a form of damage control that saves you both time and money. For market research, it is more common to go straight to an RFP. But in some cases starting with an RFI can “filter” a long list of possible companies down to a more manageable set. Read on to learn how to write RFIs and RFPs for market research. Read Article »

The RFP: Finding Great Research Partners

by By Kathryn Korostoff, Research Rockstar LLC

In many cases, skipping an RFI and going straight to an RFP is an appropriate choice. If you’ve already narrowed your choices to a manageable number of potential research partners (no more than 5), it’s time to issue an RFP. A well-written RFP will result in proposals that will help you choose the best research partner for your needs. Read on to learn about the elements of a successful RFP. Read Article »

A Few Helpful Reminders on RFPs

by By Kathryn Korostoff, Research Rockstar LLC

Documenting your market research in an RFP saves time, energy and money. Some aspects to consider include legalese and deliverables. When looking at the specifics of your market research needs, it is important to consider your internal audience, too. Read on for some practial RFP tips. Read Article »

Tips For Reading Market Research Proposals

by By Kathryn Korostoff, Research Rockstar LLC

Reading proposals can be tricky. Obviously, a market research agency is trying to sell itself to you, so they’re going to represent their abilities in the best way possible. So what can you do? Learn how to read proposals carefully and look out for details that are too good to be true. Be realistic when considering your needs versus an agency’s abilities, and trust your gut. Read on to learn some practical proposal-reading tips. Read Article »

Checking Agency References and Follow-Up

by By Kathryn Korostoff, Research Rockstar LLC

While you should always check a research agency’s provided references, it’s also smart to look for more objective sources who can attest to a firm’s capabilities. In addition, the way in which a market research firm follows-up their proposal submission says a lot about their communication skills and style. Read on to learn about checking market research agency references, and other steps you can take in choosing an agency. Read Article »

How to Check Market Research Agency Credentials

by By Kathryn Korostoff, Research Rockstar LLC

There are literally hundreds of market research agencies from which you can choose. Of course, finding ones that have the right qualifications can be tricky. There is no professional certification required in the field, so you need to do a little work. Read on for some tips on checking potential research partners for appropriate credentials. Read Article »

Choosing the Right Market Research Agency

by By Kathryn Korostoff, Research Rockstar LLC

Once you’ve narrowed your agency choices down to a short-list of finalists, choosing between them can be a daunting task. While it may be tempting to go with the agency offering the lowest price, it’s important to consider other factors, too. Read on to learn some different strategies your organization can use to make the right agency choice. Read Article »

Project Management Planning

by By Kathryn Korostoff, Research Rockstar LLC

Once you’re hired an agency, then what? Well, if you’re looking to reap the benefits of a successful project, your active involvement is necessary. By working closely with your agency, you’ll be able to represent your internal clients’ needs while providing the agency with important direction. Indeed, your involvement can help the agency be more successful. Read on to learn what to expect from working with an agency once you start the project. Read Article »

Keeping Your Market Research Project On Track

by By Kathryn Korostoff, Research Rockstar LLC

Though it is easy to jump to conclusions at any sign of trouble, the important thing to remember is that both you and your agency are working towards the ultimate success of your project. That said, there are legitimate issues that may arise. Luckily, most issues have reasonable solutions—and an experienced researcher will help you navigate them. Read on to learn about legitimate red flags versus project paranoia. Read Article »

Planning Your Market Research Deliverables

by By Kathryn Korostoff, Research Rockstar LLC

When delivering market research project findings, the most effective approach is one that combines multiple forms of delivery. After all, this information is the kind you want retained by your colleagues so they can really apply it. And we all know that delivering a 100-page slide deck just puts them to sleep. Read on to learn more. Read Article »

Planning a Great Presentation of Research Results

by By Kathryn Korostoff, Research Rockstar LLC

Think of your project’s final presentation as more of a performance. Assigning roles and specific duties to individuals keeps the presentation organized and engaging. You want the audience to be attentive, so it’s important to include them. Don’t be afraid to ask questions or start dialogue. Read on to learn about the basics of a great presentation. Read Article »

Presentation Tactics and Options

by By Kathryn Korostoff, Research Rockstar LLC

Imagine it is the end of a market research project, and you are getting ready to present the final results. Are you ready? With a little planning, you will be. The following Q & A answers key questions about presentation tactics to make sure your presentation goes smoothly. Also offered are some alternatives to traditional presentation tactics. Read Article »