Who’s Responsible for Lousy Questionnaire Design?

 

By Edward Appleton

I just finished reading a great blog post by The Service Witch – http://bit.ly/wTcpOI – on the topic of questionnaire design.

The blog contrasts traditional survey design –  long, repetitive, irritating – with a short mobile survey texted to shoppers making a purchase at a given retail outlet. Negative responses are fed immediately to local operational staff who then get back to customers on whatever issue they have.

The Service Witch touches on an important point – “What’s In It For Respondents?”. This is less of an issue for the shorter mobile survey quoted, but certainly more of a challenge for the longer traditional questionnaire.  It’s a question that we Researchers (continue to) ignore at our peril.

My question is related but different one – who’s really responsible for poor questionnaire design?

Clients – or Agencies? Maybe both?

Do we enter into some kind of Faustian pact whereby Agencies over-promise (for example) on questionnaire length to make us Clients happy – and Clients are lulled into a sense of “it’s OK if the Agency says so”?

Here’s my take:

1. Agencies need to highlight the risks of surveys that are too long more clearly. In my experience, this doesn’t happen enough. Critiquing a brief diplomatically is an important supplier-side skill – if the objectives are overly onerous within a defined framework, then Agencies need to get this across, pushing to establish priorities.

2. Clients need to be aware that there is a danger respondents will give lip-service particularly in terms of online quant. survey engagement. Critically, we won’t be able to quantify this effect (I’ll call it “Disengagement Bias”) once the “Survey Complete” button is pressed.

3. Measuring disengagement rates is a useful way of highlighting survey problems – but it’s not probably not enough to judge overall whether a questionnaire is good or bad. Whether you’re client-side or Agency side: be honest with yourself – when you click-through the pre-live version, see how long your interest level remains high, and how often and when you struggle to give a meaningful answer.

4. Keeping questions very focussed, easy to understand and answer is extremely useful. How often have you written a question where the honest answer would be: “Don’t know”?

5. Relying on grids is dangerous – as has been stated often elsewhere. Alas: there seem to be still plenty of grids out there.

6. Agencies that overpromise at a pitch phase in terms of what’s realistic in interview length may win short-term, but a savvy Client will value honest feedback, and that includes being able to disagree.

7. Surveys should be as engaging/ fun as we can make them without introducing additional unquantifiable bias – meaning it’s probably worth seeing what degree of (dare I say it) Gamification can be included – even it’s just by looking at including emoticons, using Flash (to name just a couple)….. and seeing what the extra cost is over html.

Finally, I believe that if we share more involving forms of MR Surveys with Marketing folk, show them how MR tools are moving on, becoming more intuitive, they might well become more engaged too.

If we really wish to pull the Voice of the Customer  into the Corporation in a way that gets greater attention, we’re going to need to get better at engaging a whole range of stakeholders, throwing off our dusty data-drenched masks and bringing things to life.

Curious, as ever, as to others’ views.