The 3Fs of AI Moderated Interviews: A Framework for Good Questions

The Promise and Limits of AI-Moderated Interviews

The rise of Large Language Models has also led to a corresponding development of AI moderated interviews. Often referred to as Quick Qual, many new solutions promise that AI moderators can replace human moderated interviews, matching the depth of an expert interviewer while offering the scale and speed of quantitative surveys. Obviously, this is not the case, and it is unlikely that human moderators need fear for their jobs any time soon. 

Why Quick Qual Still Requires Human Expertise

But Quick Qual solutions can gather consumer data in more depth than a traditional survey while offering cost and speed savings that, when coupled with their ability to scale, create an entirely new opportunity for insights professionals. Unfortunately, many researchers continue to approach question development for AI moderated interviews from a quantitative perspective, resulting in inefficiencies and lost opportunities.

We should not hesitate to make use of a new method of gathering and analyzing data safe in the knowledge that the true value behind an AI moderated project is the guidance and expertise that a human researcher brings. But even the best AI moderator will fall flat if positioned to fail through poorly thought out questions.  And this brings us to the crux of the matter: most researchers experimenting with AI moderators continue to approach Quick Qual as if it were a quantitative tool. 

The Most Common Mistakes in AI-Moderated Question Design

Common mistakes that we see repeated across the industry are an insistence on asking too many questions during an interview, on having repetitive questions, or asking Likert Scale questions that fail to leverage the power of an AI moderator. Other researchers ask questions that are too broad to incite anything other than the most general of responses from a respondent, or fail to provide the AI moderator with sufficient context to guide its follow-ups.

Introducing the 3F Framework: Flow, Focus, Follow-up

Instead of continuing to make mistakes that fail to get the most of our Quick Qual, we propose the 3F Framework for question development: Flow, Focus, and Follow-up.

Flow: Make AI Interviews Feel Like Real Conversations

Flow means avoiding quantitative questions and instead encourages the researcher to think about natural conversations and developing questions to mimic the way we talk to each other.  This approach not only leans into the strengths of AI moderators, but also allows for exploration with open ended questions designed to get respondents talking. 

Skip the Likert Scale questions (unless paired with qualitative probing) and instead ask why respondents do, like, or think something. Keep the questions tight enough to avoid a general, vague response. Avoid questions like “Do you like Disneyland?” and instead ask “What are your feelings about Disneyland? What do you like or dislike?” An AI moderator may not be able to match a human moderator, but well-designed questions will enable a platform to get the most out of the AI and the respondent by encouraging conversation and more complex answers.

Focus: Ask Less, Learn More

Focus is our second recommendation, with researchers asking fewer questions and instead using context to guide the AI’s follow ups. Respondents will lose interest if forced to answer too many direct questions, but often feel more engaged if the AI moderator uses smart follow-ups to build a conversational flow. At the same time, too many researchers fall into the trap of asking complex, multi-step questions that result in the respondent only answering the last part of the question. These multi-step questions are inefficient, forcing the AI to circle back to gather more information to answer each step of the question instead of doing what it should be doing: gathering more detail and nuance.   

Instead, researchers should ask a clear, direct question that allows for smart follow ups to gather more details. A handful of big picture questions that inspire profound respondent answers and that give the AI room to work will generate far better results than a paint-by-numbers quantitative approach. Finally, avoid using the AI moderator to ask filtering or screening questions. The main interview is not the place to ask basic demographic or audience identification questions that should have been handled during the recruiting process. Researchers should spend their limited time in front of respondents by focusing on the goals of their projects and ensuring that they are extracting as much data and value from each participant as possible.

Follow-Up: Where AI Moderators Truly Shine

The final F of our framework is for Follow-up. The ability to ask smart probing questions without losing sight of the goals of the research project is what separates low quality AI moderators from the best on the market, and researchers should leverage follow-up questions to get the most potential insights from each respondent. Just as we use Flow to inspire a more natural conversation, we need to give enough context so that the AI can go into more depth to round out the answer or to explore new avenues. To return to the previous Disneyland example, smart follow-up context will guide the AI on how to approach both positive and negative answers. “Find out why the respondent likes or dislikes Disneyland, what other theme parks they enjoy, what alternatives they prefer to Disneyland or theme parks, and the driving factors behind their preferences.”   

Mastering Quick Qual Through Experimentation

It’s not easy to adapt to a new technology, particularly one that bridges the gap between human moderated qualitative interviews and quantitative surveys. Doing so will require trial and error, and a willingness to experiment to find what works best for each AI moderated platform.  But researchers that embrace these challenges will find themselves well-placed to fully leverage the power and potential of AI moderators to conduct research at scale and speed while also providing deeper, more nuanced insights than traditional surveys.

The best way to master the potential of Quick Qual is to simply jump in and run internal tests or small projects to find what works and where question improvements can be made. And when in doubt, find a platform that will let you test the conversation flow before you launch expensive projects!