Complex Screeners - Could Less Be More?

One of the problems in highly specific recruiting is that all too often it merely verifies the assumptions people make about the nature of the problem they are researching.

 

Recruiting and incenting respondents for qualitative research projects has become exceedingly complex over the past several years. Each new layer of screening dilutes the incidence of qualified respondents in a geometric manner. From a statistical point of view, many of today's screeners make the proverbial needle in a haystack a relatively modest task.

"Fortunately," thanks to the vast reach and temporal efficiencies of the Internet, we have entered an era in which levels of recruiting specificity heretofore not considered even remotely practical can (and are) becoming commonplace.

But is it truly "fortunate" that we now have synchronous access to multitudes of individuals? Are we improving our research or are we simply succumbing to the "law of the instrument," which Abraham Kaplan formulated as:

"Give a small boy a hammer, and he will find that everything he encounters needs pounding."

My contention is that all too frequently we are detracting from the power of our research or, at least, adding no substantive value but greater cost, when we impose exceedingly fine grained recruiting specifications.

What follows are some philosophical arguments along this line – Given an opportunity, I'll follow this article with some "Rules of Thumb" in pursuit of a more reasonable approach to qualitative recruiting.

 

1. Validity vs Reliability

Qualitative research has strengths in the arena of validity because the process gives people the opportunity to tell you you're asking the wrong question or correct you when you summarize your interpretation of their answers in a way that misrepresents their intended meaning.

Qualitative's weakness is in the arena of statistical reliability. The sample is generally very small and frequently drawn non-randomly from a less-than-universal database. It is for this reason most researchers are careful not to project qualitative findings into grand conclusions applicable to large scale decisions.

For example, a good piece of quantitative may shoot down a product concept as being flawed because it is not compelling to a sufficient percentage of target consumers, while a good piece of qualitative may help with a solution by why individuals reject the concept. A quick example:

Client makes canned stew that doesn't sell well…

Quantitative tells us most triers find it's not dense enough to be a good stew. Focus groups tell us that while it's just not viscous enough to be a good stew, the product tastes good - it would be a fine soup. Rather than reformulating the product (an expensive proposition) – client repositions product as a hearty soup. By exploring the source of peoples' disenchantment via good qualitative research, we find the successful solution of managing expectations.

As Saul Ben-Zeev (Founder of Creative Research) used to say, "Cut a thousand people noting they bleed and you will arrive at the conclusion that cutting leads to bleeding. Dissect a single human so that you understand the circulatory system and you will arrive at the same conclusion with similar certainty but with a much more useful and pervasive insight."

The point is that the thoroughness with which you conduct and analyze qualitative research is frequently more important than whom you choose to address as the subject of your inquiry.

But that doesn't go far enough. Being excruciatingly careful about whom you recruit for a qualitative inquiry can be and often is counterproductive.

 

2. The Issue of Tautology

One of the problems in highly specific recruiting is that all too often it merely verifies the assumptions people make about the nature of the problem they are researching – an issue made particularly dangerous because it is done with the conviction that one is doing more precise, and therefore, better research.

Here's another example: Early in the evolution of the diet soft drink business, I was asked to do a study for a beverage alcohol company on diet soft drinks - an arena of interest since the client company was anxious to extend a line of mixers that might be positioned as diet soda as well. We recruited people who regularly drank diet sodas.

Fifteen minutes into the first group the client angrily summoned me out of the session and insisted we abort the group because several respondents … aren't on diets!.. although they exclusively used diet soda. I suggested that some people who drink diet soda don't describe themselves as being actively "on" diets, but are practicing the common process of banking caloric expenditures for use on more important occasions (generally involving chocolate).

She demurred and we continued with some very successful groups – chock full of insights about diet drinks from people who weren't on diets. Had this client been more directive in the design of the recruiting specifications, she would have insisted on talking only to people "on" diets, and in hindsight, would have inadvertently lost a truly critical level of insight into her chosen topic. In the name of a "better, more precise" inquiry, she would have narrowed her field of vision so as to preserve her assumption and thereby weakened her company's ability to successfully implement a line extension.

The point is that rigorously narrowing your field of vision may yield truths that are sound but far from optimal in their utility.

 

3. Statistical Issues

Fine-grain recruiting raises a number of statistical questions as well.

a. Pure Types

Most complex recruiting specs are designed from a template gleaned from the analysis of very large databases. Large databases, which incidentally, lend themselves to extraordinary levels of parsing. The organizing principles used are based on identifying sub-segments of individuals with manipulations ranging from simple cross-tabulations to more ornate procedures using multivariate algorithms.

But, the truth is statistical representations are theoretical constructs always approximations of reality – imposed for the sake of communicating similarities and differences and not necessarily an exact reflection of any one individual found within a given category.

In the real world, the definition of one's character can be less clear-cut than what emerges from even the most basic template. Classify Barak Obama as an African American and you would be, perhaps, half right. Classify a child as a 10 year old and you may be chronologically correct, but mentally way off base. As Roy Stout, my boss at Coca-Cola used to say, the average American has one breast and one testicle.

Complex approaches using multiple regressions to clump people into groupings by maximizing variance between and minimizing variance within groups tend to form groups that include individuals who manifest a "best fit" profile, but may include no one who totally matches all of that group's criteria for inclusion.

To be sure, there are specs in research that cannot be violated, but demanding that focus-group recruiting specs be imposed with great exactitude– even when the specs define a fuzzy reality gives the appearance of precision whether or not it's helpful or necessary –like excavating a building foundation with a dental pick.

The business of negotiating people into one's group who meet the basic intent of the law even if they don't embody its exact letter, frequently reflects the nature of reality – it doesn't violate it – and, perhaps more importantly, it can facilitate an especially rich source of important information regarding the boundaries of your target definition.

b. Dynamic Reality

Rigorously classifying individuals on the basis of criteria that may vary significantly from day to day makes the mercurial nature of peoples' attitudes and momentary behavior appear to be stable, but it does not necessarily reflect reality.

Furthermore, as Leon Festinger pointed out many years ago, people are quite capable of (indeed require) holding seemingly conflicting points of view at the same time. How often have we:

• Listened to people grouse about the depravities of TV only to find out they are six hour-a-day addicts?
• Talked to fast food junkies who descry its nutritional destitution? 
• Heard that advertising is "an insult to my intelligence" from someone who can cite dozens of ads word-for-word?

There is a tendency to denigrate focus groups as too easily swayed by such posturing. I disagree – understanding both sides of such apparently opposing points of view is an important part of good qualitative analysis.

A very strong case can be made that such anomalies are not misrepresentations so much as reflections of two types of truth – a presentation of social posture (how I would like to be seen) and empirical experience (how I actually behave) – both of which can accurately reflect different elements in the individual's total persona.

When we insist that respondents conform to rigid criteria and stay that way, we are not taking into account that a "pure" and "static" vision of the world is an artifact of statistics, not, necessarily reality.

c. The Incidence Mirage

Finally, it seems that some clients have an all but delusional sense of "incidence." A few simple calculations may illustrate the point.

Let's say our target market is comprised of people who consume any product in a given category once a month or more – a user population the client assures us represents 20% of the sampling universe. Let's say we have a screener with only five questions beyond category usage to determine eligibility. Let's say further that half of all respondents reaching each question from the previous one are "eligible" and, accordingly, allowed to continue toward an invitation to join the group. By today's standards, I think everyone would call this a pretty simple screening questionnaire.

This screener should yield people representing about a 0.625% incidence. Add one more question and you're down to about 0.3% - that's approximately three people in 1,000 – which means a group of 10 requires a data base of more than 3,000 to complete (assuming everyone agrees to participate).

In this context, a 10 page screener with two or three batteries of bipolar rating scales, a psychometric algorithm and three or four nested levels of demographics borders on the preposterous from a mathematical point of view.

And, of course, clients add a few "get a good mix" directives – a practice which suggests an intuitive realization that the incidence numbers are perhaps too small to define but require adherence nonetheless.

My intent here is not to bemoan the work involved. As I noted above, the Internet has made such screening feasible (albeit highly complex). I simply want to caution my clients to think about what they are doing to the nature of their findings.

We are told that we are in the era of long-tail marketing. Today's connection revolution allows us to direct appeals to ever narrowing slices of our market. So, we can in fact, talk directly to "Republican women who go hunting with their husbands" and "Calvinest personalities who roll-their-own cigarettes." But can we afford to devote our dear resources to a better understanding and targeted pitches to such rarified characters?

My sense is that we're getting carried away with our abilities to perform complex tasks and in the process, we're walking away from some of the powerful, serendipitous findings that good qualitative work should yield.

 

This content was provided by Fieldwork, Inc. Visit their website at www.fieldwork.com.