Inside Qualtrics Edge: Ali Henriques on AI, Insights & Innovation

In this episode, host Karen Lynch welcomes Ali Henriques, Global Director at Qualtrics Edge, for a deep dive into the transformation of research services at Qualtrics and how AI is shaping the future of insights. Ali shares her journey from cruise industry research to leading a $120M insights division, revealing how Qualtrics Edge is democratizing access to data through Instant Insights and synthetic audiences.

The conversation explores how synthetic data enhances agility, why continuous model hydration is critical, and how companies like Booking.com are already testing these capabilities. With humor, passion, and deep expertise, Ali offers a clear-eyed look at the future of market research and how teams can start exploring AI-driven solutions—today.

Key Discussion Points:

• What Qualtrics Edge is and how it reimagines traditional research services
• Instant Insights: Merging syndicated data with real-time behavioral signals
• Synthetic audiences: Training proprietary LLMs to scale insight generation
• Booking.com’s pilot study and lessons learned from synthetic data comparisons
• The future of research: fast-twitch insights, agentic AI, and democratized decision-making

Resources & Links:

• Qualtrics Edge Overview
• Booking.com
• Qualtrics X4 Experience Summit
• Greenbook IIEX Events

You can reach out to Ali Henriques on LinkedIn.

Many thanks to Ali Henriques for being our guest. Thanks also to our production team and our editor at Big Bad Audio.

Transcript

Karen: Hello, everybody. Welcome to another episode of the Greenbook Podcast. I’m your host, Karen Lynch. It’s such a pleasure to be with you all again today and to be with our guest. Not my first time conversing with Ali Henriques, but I’m really excited to have you here, Ali. For a little context, everybody, Ali is now the Global Director of Edge at Qualtrics. She started as a market researcher and she’s been on the client side and on the agency side, but now she’s leading a very large group, a million-dollar business for sure, $120 million, I believe, in research services. So, kudos to you, Ali. Can’t wait to hear more about it, but first and foremost, I want to welcome you to the show.

Ali: Thanks so much, Karen. It is great to be with you again. 

Karen: It’s such a pleasure. And Ali and I have met—I have been to the Qualtrics X4 Experience Management Summit, which we’re recording this on the tail end of that. But first, Ali, before we even get into, kind of, how we’ve met and talk about Qualtrics and Qualtrics Edge, tell the audience a little bit about you and your role, kind of bring yourself to life in a way that I can’t possibly.  

Ali: Absolutely, happy to. You said it really well. Research is all I’ve ever done. I’ll add to that, kind of, as we parley into Qualtrics, I joke with candidates and folks on my team that Qualtrics has been a key part of my entire adult life, right? As designed and planned by Ryan Smith and his family back in the day, they got Qualtrics into the hands of college students so that they brought the platform with them wherever they went, and that is absolutely true in my case. So, I cut my teeth in market research in the cruise line industry in South Florida. So, I spent the majority of my time early in career, client side, which was fascinating. I absolutely loved it. And the cruise industry is just so multifaceted. I felt like I was working with food and beverage brands, operations, then sales organizations, product teams at different companies almost every day. So, that was an absolutely enjoyable experience. I was also a certified moderator, so I did tons of work onboard interviewing guests, doing focus groups. We tested everything and it was a ton of fun. So anyways, that’s how I actually got my hands on my very first paid Qualtrics license, and I brought it with me everywhere, into ad agencies, research firms within agencies over the years, and it’s been a ton of fun. So, to kind of fast-forward more years than I’m willing to admit, [laugh] where we are today at Qualtrics is really our team’s rebranded over the course of the past year [audio break 00:02:52] Qualtrics Edge. It’s the team formerly known as Research Services, and the simplest way to describe that is an in-house research agency. We’ve got the ex-Kantar Millward Brown, Ipsos, Nielsen, client side researchers. We’ve got project managers that come to us from panel providers. So, we really are an in-house agency. And what’s changed is not only the name, but a lot of our investment in the products and solutions to meet the researcher’s needs, meet us where we are, but also the market where it is, and where AI is taking us maybe is better said. And so, that’s really the genesis of Qualtrics Edge. It’s an in-house incubation agency in many ways, where we’re still doing the stuff. We are a traditional research agency at our core, but we’re also imagining the future. And so, I’m really happy to be at this crossroads of hanging onto what I know and love, so dearly, but also imagining a better way.

Karen: Yeah. You know, just this morning, I was, you know, looking at LinkedIn and I was catching up on a few things, and somebody had shared—and now I’m not remembering who it was—but somebody had shared kind of a lengthy post, but in it they said, “We’re not in a future of work moment anymore. We’re in a present of rapid transformation, and AI is at the center of it.” I’m like, “That this is such a great quote.”

Ali: Oh, I feel that. 

Karen: Right? Don’t you feel that? Like, nope, this is it. We are in it. We are living it every day. It is the pinnacle of everything that’s transforming our work.

Ali: I love that so much. The way I tell my team is, this is happening to us. And the reason I bring that up is really nothing to do with Edge, but what I’m begging them to do for themselves and the future of their careers. I’m not telling you that you have to go and listen to the AI daily brief and read all of these things. I strongly recommend you do this for yourself and in your future. You want to know where this is going, even if I only understand every third word in that podcast, right, I’m taking in as much as I can so that I can connect dots, right, to my world and where we’re at. Well, said. I love that, Karen. 

Karen: Well, it’s a cultural immersion, right? If you think about it, like, the best way to learn a language or understand a culture is to put yourself in it. And that’s what we—people like yourself and myself—are recommending to people is put yourself in an AI culture mindset. Like, I am immersing myself in all things AI, listen to the podcasts, read the articles, experiment with the products, use it at your kind of consumer-ish level, just immerse yourself in it because it is transforming everything that we do. So anyway, we can probably talk about that for a long time, [laugh] but let’s stay a little bit to what we really want to do. And talking about this launch of Qualtrics Edge, so give people the kind of the headline on what it is. When I read the press release and all that, and I take it all in, I think like, yes, this is absolutely spot-on to what we would expect and want from a large research organization right now. But in your words, how are you guys changing the game, in your minds?

Ali: In its simplest form, what I like to say is that we’re democratizing insights. And a lot of people talk about that, but when I really break it down, research services, any kind of agency out there is working with an insights buyer. What we’re talking about with Edge is really the insights consumer. So, think about the stakeholders on the other end of all of the work that we do. How can we put some of this power in their hands? And I can feel some of the audience out there saying, “Ugh, that really just—I’m not sure that I feel good about that,” but we’ll do it in a controlled fashion that we all feel really confident in so that we can reduce the amount of ad hoc requests and omnibus studies and all of that kind of ad hoc budget that we hold on to for those dozens of stakeholders that we know we’re going to have to deliver insights to. What if there was just a shortcut to getting them what they need, maybe not the entirety of the need. So, that’s really kind of where it was born from. And the idea being, accelerating speed to insight, which again, is not unique to Qualtrics or Edge and what we’re doing; it’s happening to us, right? The market is telling us, I can’t wait eight weeks to make that decision. I’ve got to move more quickly. And when we do that, we are resource constrained: time, budget, only so many hours in the day, then it compromises quality. And so, the idea is, what if we could put some of this power into the hands of the market or the operator, the product team, so that they can accelerate, they’re not held up with us in our own constraints—us being the researchers—and they’re able to progress their work, again, within whatever kind of your risk profile and appetite is. We’re not betting millions of dollars on this insight that we’re extracting from this solution, no, but it’s better than a gut check. So.

Karen: Yeah. And there’s a way you phrased something, which was the what if, right? It sounds to me, from—again, I want to get into some of the details about the capabilities in just a moment—but it sounds to me that if you’re like, what if we could get some of these capabilities into the hands of our stakeholders? Just the idea of what if, the kind of openness and problem-solving language, like, that to me is the real spirit of what we’re doing in this time of disruption, is seeing what is now possible, what used to be impossible because of the way it was always done. And now these tools that have come up, yours included in a larger set of people leaning into AI functionality, are making what used to be impossible quite possible, actually, and not only possible, but on some levels, preferable. So, it’s actually, if your mindset can embrace where we’re at, it’s like suddenly you open up doors that you didn’t even know were possible. So.

Ali: I love that. We talk often inside of our little world in Qualtrics about counterfactuals, and that’s exactly what you’re describing. We might not have time, budget, appetite, resources to do these things, but with AI, we’re unlocking that. And I know we’ll get to synthetic, but that’s really the power of this, again, not betting multi-million-dollar decisions, but helping us explore some of these counterfactuals in a low-risk environment.

Karen: Yeah. And I’m anxious just to talk about that too because that’s really key. But first walk us through, again, kind of the core capabilities, and then—

Ali: Absolutely.

Karen: You know, kind of some of the spin-off stuff that you have. This is your time to kind of say, “Here’s what we’ve got.” [laugh].

Ali: I love it, thank you. Good thing my breath is back after X4 because [laugh] we really talked about it. We talked about it a lot then. So, we have two solutions as Qualtrics Edge. One is called Instant Insights, the other is called Audiences, and they’re actually beautifully complimentary. So, I’ll give you an example of that. I’ll start with Instant Insights because it’s the one that I think is just very immediately easily understood. So, at its core, it’s got consumer preference behaviors that are captured via syndicated study, right? So, there’s a syndicated study at its core, but what it’s doing is it’s ingesting other data sources that we license or acquire, things like search, news, trend headlines, transaction data, and then behavioral, in person, physical, as well as web app digital. And those additional signals are giving us something really, really special to, again, put in the hands of the marketers, and the product teams, and the operators to say, what if, right? And so, in a way, too—and researchers will understand this well—we’re also real time comparing the stated and the observed, not only from a visitation perspective, but even if you just think about search terms, there’s so much value we extract from those for the marketer related to offers and promotions. You see Chipotle BOGO, right? You see Subway, you know, Free Six, and all of these things surfacing that are just really, really powerful. So, in a way, our studies are a bit of a lagging indicator of what’s happening in markets. So, if we see these search terms, what is it doing to our then syndicated portion of the study, in terms of interest and visitation and all of those other, kind of, brand attributes. So, this is industry specific. We started with restaurants, so we’ve got QSR fast casual, casual, dining. We just launched at X4 a couple of weeks ago, hospitality. We’re starting with hotels, and that will expand to all the other verticals of travel, and we’ll keep going with a lot of these consumer high-touch type of industries, think retail. We’ll do healthcare, banking, and insurance. And many of these are also built in partnership with some of Qualtrics partners, think, like, Bain, Ipsos, Kantar. So, that’s Instant Insights. I’ll pause there before I explain Audiences.

Karen: Yeah, well, I think that what’s interesting is that to me as you’re talking, I’m like, yes, this all seems like, I don’t want to say traditional, but, like, yes, if I’m imagining pure democratization, as you mentioned before, years ago when we were talking about the democratization of insights, we’re picturing what is that going to look like? And I’m like, yes, this is the manifestation of all those conversations about an effective and efficient way to democratize insights for anybody within a client organization, but in this case, in tandem with you. So, that all makes logical sense. So, I’m curious to know about Edge Audiences and how that is different from, kind of, Instant Insights. 

Ali: Absolutely. So, Audiences is our branding for synthetic. And oh, God, this could probably take the rest of our time, [laugh] but I’ll try to keep it brief. I think it’s important to start with defining synthetic, and I’ll use our terms and how we’re describing it, and I’ll compare that to, kind of, what I see in market, and in a way, also explain what I see as client confusion. Synthetic, I wish I had—and we’ve got to pull it, right—the Google trend line for the amount of time synthetic has been searched, right, over the course of even just the past year, I think it would be really staggering. And so, we’re building for what I consider to be the hardest thing first, and that is a fully foundational proprietary LLM. Why are we doing that? Well, because we want to not create something that’s use-case market or any other kind of parameter specific, but rather make it as robust as possible before we start to kind of break it down into its components. And so, we’re doing that very stepwise logically. So, starting with high IR GenPop US type of audiences, use cases, and screening criteria quotas that you’d expect, and then we’ll expand into, kind of, tier-one markets with those same audiences before we get to niche B2B, right? So, we’ll step into it, but we’re not limiting it by use case like concept test, ad test, you know, anything like that. There will be great applications and there will be poor applications, and we will discover those together, but ultimately, it’s up to the clients and their appetite. So, this fully foundational model is where we’re starting. It’s a combination of a lot of what we’ve just talked about when you think about all those other data sources that we’re considering, proprietary research that Qualtrics has conducted. It’s a fun fact that within research services, Qualtrics is one of our top ten clients. We run so much research on behalf of Qualtrics research on research. We have all of that data, as well as other sources that we’re licensing for training purposes. So, that’s our kind of corpus of, I’ll call it more proprietary or licensed data, combined with what you see in publicly available LLMs like OpenAI, Claude, you name it. So, the power of those two together is really great. And again, we’re taking our validation kind of step by step, and I can certainly get into that a bit later.

Karen: What you’re doing is ensuring that it’s well trained, that the large language model that you are creating and building is well trained, which will ensure greater efficacy down the road, higher quality and integrity. So, just for those of you who are listening, who are not as up to speed, just that’s what’s happening here. All of that training is essential for quality input, right, so there’s quality output. Anyway, continue, Ali.

Ali: Oh, I so appreciate that point, yes. And I think I might even build on that with where I was going next. So, if I think about what’s in market, I kind of break down synthetic offerings into three categories. The first of its kind is—my words—a GPT wrapper, right? So, some kind of really slick interface that sits on top of a publicly available LLM. And what it’s doing is helping maybe some of insights consumers craft a prompt really effectively, right? What we’re doing is we’re asking it to understand all of our screeners and quotas. So, who do you need to consider before I ask you this question? But it will still give you record level data, and it will be wildly accurate, 80% by some accounts. But it’s—but—I use but—it is only considering publicly available information. So, it’ll do a great job at understanding, you know, ice cream behaviors and, you know, things that we do regularly that are just again, kind of higher incidence rate. So, that’s kind of the first layer of what I hear as synthetic. The next is one of the number one things I get from our sophisticated clients is, tell me how synthetic is different from weighting, extrapolation, or imputation, right? Well, this one will start to kind of explain because some of the offerings out there are predominantly machine learning-based, which is a lot of the roots in those other things that we’ve been doing for decades, the RAG approach. And so, with this one, it’s a lot lower risk because what we’re doing is, Karen has collected 400 responses already from humans, exactly, you know, replicating the quotas and specs that you desire. You want that to be 800. I’ve run out of time, I can’t stay in field for another two weeks. Well, RAG will take your 400 and make it 800, generating net-new records of data. That’s where it’s dissimilar to weighting an imputation. With weighting an imputation, we’re working within the confines of that original 400. We’re filling in blanks and columns, and we are increasing or decreasing the power of some of those rows. And so, this is where synthetic is truly generating net-new records that represent these demographic profiles usage behaviors that we screen for with our human participants, and creating responses to survey questions. And so, what Qualtrics is building towards is really a combination of those three because we’re considering the publicly available alongside our own data, but we will build towards this using the foundational LLM, the ability to also take your 400 and make it 800. And so, it’s really a combination of the three types of synthetic I’ve seen in market.

Karen: You know, it’s funny because I was just having a conversation, again this morning over coffee, the same person I shared that quote with earlier today. We were discussing synthetic data, but in the framework or the wrappings of synthetic respondents or synthetic personas or, you know, kind of this whole idea that there are individuals that are considered your digital twin or, you know, these kind of artificial entities representing, you know, participant number 13058 or something like that just to, you know, kind of throw it out there. So, we were discussing this and talking about how the data—and you can correct me if I’m wrong, Ali—but these models will have to continuously be updated as our population also ages, right? Because the behaviors that we understand right now are going to be very different from, you know, Gen Alpha who comes in and starts behaving very differently, whether it’s in the workplace, or at the quick serve restaurant, or at whatever retail [establishments 00:19:02], they’re already behaving differently and we may not have that kind of data. So, how are we, how are you, how is our industry, like, making sure that we are updating all these LLMs that we’re training on today, when really tomorrow we might have a whole new set of people that we need to learn from?

Ali: That’s such a great point. You bolded and underlined the point on training earlier. Where we’re shifting to now is hydration. And in part, our expectation is that the publicly available models will do that for themselves. They will be letting us know about external factors, events, traffic, weather, you name it, that might be happening in the world that could be influencing some of this behavior that we’re interested in, but then it’s on us as the owners and architects of this model to be hydrating it. So, this is a bit of the insurance and security clause for all of us out there that we will never shrink to zero in human data collection because this model will fall apart. And you hear that that’s happening, even with some of the big guys. And so, we will subscribe to training data that is a consistent external source of refreshed, always-on data and we will feed it with our own versions of—you know, you think about Instant Insights, all of those human-collected responses are feeding, and that will, as we expand the verticals, right, that will only get richer by market and industry size. So, hydration is absolutely key. We will continue to depend upon human panelists [laugh] to keep this model fresh and alive.

Karen: By the way, I also just want to, like, take a moment and honor the fact that you’re using ‘hydration,’ and that is so new and novel for me today, and I might be borrowing that so I will remember this moment. But lots of people talk about you have to feed it, you have to feed it. But I love the metaphor of hydration. You have to keep it hydrated, which feels so much more nourishing, as opposed to, like, contributing to the machine. There is something very, very, like, wonderful about the hydration analogy. Really well done [laugh].

Ali: What you don’t see is all of the green life around me, Karen. I have so many plants [laugh]. It is not mine; I will not take credit for that. But it’s a great point because what we risk doing here is completely dehumanizing research and that would be an epic failure. Humans remain at the core, and that is part of what I love about Qualtrics is that our whole ethos is to make business more human. And while it may feel like we’re veering from that, that’s not true. We still need the guest and the customer experience data. We need the review data. We need the sentiment data. So, I love that you brought that up. It keeps it, kind of, more organic and alive. 

Karen: Yeah, and it suddenly brings to mind health as opposed to something else. It just certainly felt like a very robust way to keep it well. So, anyway. So, cool. A little sidebar into hydration as a metaphor there. Let’s talk about some use cases. So, in your press releases and things, you’re very openly talking about Booking.com. So, let’s just talk a little bit about how Booking.com is already using Edge, at least what you can share, what’s not too proprietary either in their approach and the results. What can you tell us about them? 

Ali: Absolutely. I had the great pleasure of speaking on stage with Elina from Booking.com at X for a couple of weeks ago. We partnered on a pilot with synthetic and what we’re finding is, but no surprise to you or anybody listening, there are some haters out there. I started my presentation with a show of hands. I just want to know what we’re working with in the room. Who’s used synthetic? Who’s considering it, right, and who’s a hater? And so, what Elina came to us with was, I’ve got a hundred haters in my research organization. Let’s pilot something super risk-free and bring it to them and just be super—approach this eyes wide open, fully transparent, vulnerable on both sides. We’re going to find what we find and we are going to publish it and accept it. And so, we shared a lot of that a couple of weeks ago. So, I can absolutely share. I’d say the greatest learnings—and I will also be vulnerable as I share this—I didn’t expect that I would find myself being so introspective about the findings from synthetic data. It really came back to question design, study design, and a little bit of methodology. And we found ourselves kind of questioning, I wish we could grab those thousand humans that took this survey and ask them what they were thinking as they answered this question because what it really did was just unpack a lot of semantic debate around question structure, even on the most basic semi-screeners. So, for example, we took her travel trend study that’s conducted every January, and we ran synthetic at the very end of the month, and compared the humans against the AI-generated responses. And two basic question types: what type of trips did you take last year, and who did you travel with? Those are, like, basic questions. I think you know how to answer those. Well, we found some crazy different distributions depending on how we framed the question. So, with synthetic, the respondents don’t get tired, we can ask them endlessly, different framing, phrasing of the questions, and so we did that. We had a lot of fun experimenting. And when you break down a multi-select question type, the scientists would agree the truest form of that would be a yes-no for every single response option. And so, that’s what we did with synthetic, and it told us on the travel type question, well, beach trips. That was the highest proportion of trips taken in 2024. When the humans said no, family and friends. And you know, and I’ve been in Elina’s shoes. I was in travel and hospitality for so long and we’ve accepted that family travel, that makes sense, right? And what it really did was expose what was most likely a recency bias. If we’re fielding this study in January, what did you do in December, Karen, right? And if it was visiting grandma who happens to live by the beach, what do I choose? And so, the takeaway for Elina was that, well, I kind of believe the AI more. We know there’s tons of data to support that beach trips really are the most prevalent. Why do we keep seeing friends and family, you know, rising to the top? So, it was little things like this. And the same kind of pattern with travel parties, solo travel. Think about the last time you walked through an airport, how many people were families, and how many people were on a mission, just traveling by themselves. So, her distribution was very different from what the synthetic responded, at a couple of those junctures, right? Some of them were almost perfectly aligned. So, we picked on a couple of these and we unpacked why we think this might be happening. And the last example I’ll share from that pilot was—this one’s fun; it’s a bit of a word scramble—how humans report that they’re using AI in travel planning was very different from how AI reported humans are using AI in travel planning. You see what I did there? So—[laugh] and she has operational data on this. So, she was able to say, actually, no, these four attributes, synthetic is right. The humans are, again, maybe misremembering or recency bias, right? They’re saying I used it for restaurant recommendations. And she’s like, no, we know that the number one thing they’re doing with AI is itinerary planning, and destination planning, and what sites do I have to see, and what can’t I miss in my eight hours in this city, right? So, it was so, so fun. And such a great topic to, you know, draw any audience into. We all travel [laugh].

Karen: Yeah, I know. Well, [laugh] and what you may or may not know about me is, you know, beaches… beach is pretty high up there for me [laugh] in all of my work. Are we near a beach? Anyway, like, tell me where we’re going, and if we’re near a beach, it’s happening. Anyway, a question I have for you, which it almost sounds like as you’re talking and describing this, it almost sounds though, like, there’s, like, this rejuvenation, like, this joy and this curiosity that’s being piqued while you’re doing this work, like, that curiosity that drives researchers when they’re really engaged with the category or when they’re really engaged with a project or a brand. Like, I have had the pleasure of working on some of those projects myself where I’m like, I’m really interested in this, for whatever reason. This topic, I’m into it, I’m jazzed by it, I want to learn more, and my natural, kind of, innate curiosity gets brought into it. Which I always felt that those were some of the best projects I ever worked on, and hopefully had the best results as well. So, it almost sounds like you’re curious because you’re thinking like a researcher really leaning into this kind of new methodology and are intrigued by it. And then you’re having the, huh, this is interesting, what does this mean? Why is this happening, and you’re naturally asking researcher questions as you’re going about it. Am I right in kind of capturing what you’re describing? 

Ali: You’re spot on, and God, you’re even, like, surfacing memories of those types of projects for me as well. That’s what it kind of became. And again, eyes wide open, right, with the whole thing, and we weren’t really—her very first slide was this was not an A-B test, right? We were not trying to prove anything. This was a journey, and she said it so beautifully. And that’s kind of what we’re encouraging all of our clients to do is embark on this journey. I can tell you, and I’ve got plenty of research to suggest what types of use cases are a good fit for synthetic. I will not ask my synthetic model to tell me how dinner at, you know, Chili’s was last night. I’m just not going to do it, right [laugh]? But what this also exposed in one of our, kind of, takeaways, leave-behinds was, think about the just pretesting, cognitive testing, semantic testing, right, think of these applications for synthetic as well. You don’t have to take output of these models and use it for your insights, but use it to test before you take it to humans, right? And think about, too, rounding out response options and making them exhaustive. AI is a great tool to help with that. So, there are just some really fun, creative ways that we’ve kind of brainstormed, for the haters in our organization who aren’t ready to adopt it for answers yet, well, why don’t you start with this?

Karen: Again, we could talk on and on about, you know, just kind of the different use cases for all of these methods in general, but I do kind of want to recognize, like, we are talking about, kind of, now one very specific use case for edge and for edge products, but without going too far down the, oh my gosh, AI is so cool, which I will do any day, [laugh] I want to, like, kind of, you know, stay focused on, like, these early successes that you’re having now as you are bringing this to the marketplace. So like, other than Booking.com, do you have other, kind of, indicators, you know, that it’s working, other objectives or metrics or KPIs that you are meeting? Like, what can you share in terms of how you’re evaluating how well it’s doing?

Ali: Absolutely. And it’s super important to us. So, we have a white paper that publishes our just general kind of standards of excellence and framework for how we’re thinking about validation. To give you a little bit of a sneak peek under the hood, there’s a committee inside of Qualtrics that consists of cross-functional leaders. I am one. Somebody from our machine learning team, somebody from our data science organization, somebody from our AI Innovation Center are the approvers of any model for release. And so, we have a strict set of criteria that include everything from mean divergence to coverage to deviations, deviations of means, deviations of top-box. And we have scores against each of those that a model has to meet in order for it to be ready for public consumption. So that, yes, we take it very seriously. And what we did with Elina was pre-model release. That model was not ready to our standards yet, but we still put it to the test, right, and allowed for this kind of experimentation because what she also agreed to was the ability for us to share that back—to our earlier point on training—to let it know where it needs to be improved and where it did well. A couple of other just general points on how we think about validation and the quality of data. There are, kind of, four factors that we’re looking at. I’ll rattle them off: generalization, data shape, diversity, and transferability. And what we’re looking for within each of those is making sure that models can do a really great job at memorizing and regurgitating, and so, you know, we’re throwing new scenarios and new question types and new categories to make sure that it’s able to adapt to those. We’re making sure that there’s representation so that we’re never finding ourselves in a bias situation or too few populations to draw upon to come to that inference that also ties to diversity. So, those are more broad applications of how we think about this model and what it needs to perform for us. And then we’ve got our very technical, I think there are 12 or 15 items on that list that we all have to [laugh] set off on.

Karen: Looking at your journey with Edge, we’re talking a lot about kind of quantitative measures and survey questions. And I’m wondering about kind of qual and the ability for qual, not just qualitative—maybe qualitative open-ends are a part of your processing already, but just qualitative questioning. Like, how is that factoring into this mix that you have in either of these two products? What are you doing to get some qualitative insight? 

Ali: Such a great question. Later in the roadmap, Karen, and this might just be selfishly because of Qualtrics, and who we are and what we do, right, we are very much, quant heavy—we do have tons of qual capabilities and features, of course—but we will get there. And there are so many great companies that are—I’ll say it—far ahead of us with a hyper-focus on qual. You’ve got AI, moderators. You’ve got synthetically generated qual participants. All of these things, I think we will, we will do that too, right? You’re spot on, open-ends, right, with the types of qualitative questions that make their way into surveys is where we’re starting, absolutely. But I think what you’re kind of forcing is a question around the future of market research, and when, not if, will these worlds collide—if everyone could see my hands doing that just now [laugh]—because the two will become one. I’ve realized, I completely neglected to mention—if I can just take us back for a second to Instant Insights—that is powered with Qualtrics Assist, right? So, we’ve got the chatbot feature and function sitting on top of that because there’s a lot of different data sources and information. And so, as you said qual, I was like, oh well, questions, even open-ended questions or prompts are, in a form, qual. And so, we can ask Instant Insights, why is KFC’s satisfaction down relative to Popeyes? And it’s going to tell you the seven, eight things that it sees. And then, if I can for a second, talk about this kind of flywheel effect of Instant Insights and Audiences as a use-case application that some of our brands are trialing, I’ve picked up on this trend that visitation is down—I don’t know why I always pick on the Pacific Northwest—but it’s down in the Pacific Northwest [laugh] with 18 to 34-year-old males, okay? And—

Karen: So interesting.

Ali: [laugh]. I know, I don’t know… I don’t know why, but I’ll make it right. I’ll balance it out with my next one [laugh]. And so, Instant Insights has surfaced that trend for me, right? Like, we have a problem here. We can quickly flip into Audiences and say, what type of offer can I put into market to reactivate this segment of the population? And what will happen when these two start talking to each other all the time is that Instant Insights will also tell me, back to the points we were talking about earlier, well, you know, somebody already has a BOGO live right now, and these guys have that. And we know where else that core buying cohort is also shopping or spending their restaurant dollar, and what promos to avoid. So, it’s more than just asking synthetic, “Will this promo work?” It’s also considering what’s live and active in market in a way that, you know, you play that out, it would probably take five of us across the marketing team to pull together all of those different pieces to come to the same conclusion. So, it’s really a beautiful, kind of, better together story. And qual tipped me off to that. Because pretty soon—you talked about digital twins, right—I want to have a conversation with all of, you know, my segments, and know, you know, let’s keep it up at night and what we need to be, you know, doing better as a brand. That’s, you know, that’s qual. And [laugh]—

Karen: I know. There’s so many—I get very excited—this space does not scare me. I’ve been saying for a long time, like, you know, however m—two years ago, now, two-and-a-half years ago, like, when ChatGPT hit, I was like, oh, it is going to be my new best friend. And I still call it my friend, Chat. Like, I recognized right away that it’s basically having, like, the smartest per—expecially as it gets smarter and smarter, I’m like getting the smartest person, like, on your team. But not just on your team; like, you’re co-worker, you know, at the desk next to you, like, you just have access to this really intelligent entity, and I think that’s just a gift, right, across the board. But as you’re talking about that, I’m like, I want your different segments to have a discussion with each other. 

Ali: That’s right.

Karen: I want one segment to debate with another segment. Hey, you two talk amongst yourselves, get back to me. Like, [laugh]—

Ali: I love that. 

Karen: There are things that I want our language models to do with all of this synthetic data that I just, I think the possibilities are endless. If we can lean into our creative thinking and say, what else can we do to change the way we think or to broaden our minds or to spark that aha that drives innovation, I mean, to me, that’s where the sweet spot always is, is how our insights in forming innovation, that’s kind of where I lived for the entirety of that part of my career. How can we just get that? All you need is a something, right? So… good stuff, good stuff. Let’s go there because now we’re coming up at the top of the hour, and here it is. I told you this is what happens to me all the time. But we’ve already addressed the fact at the beginning of this episode that, like, we know we’re kind of in this now moment. Like, don’t worry about the future right now. Like, we are in this now, this digital transformation, this huge momentous, not just disruption, but powerful change. Where do you think research is headed in the next two to three years for real? Like, what does that new future look like? Because we’re in the future we thought was the future, and it’s the now. What’s then [laugh]?

Ali: I think in short, I’ve got some data and I’ll reference that in just a moment of where Global researchers, what their answer would be to this. But in short, I think we will move away from record-level data. I don’t know how much longer we’re going to need back to Karen’s 400 making it 800. I don’t know how much longer we’re going to need that. I think, if I could answer my own question, it would be, you know, until we are confident in what we’re getting back as answer to questions. So just, like, you know, chat being your best friend, in whatever prompt I give it, I’m trusting this answer. It’s cited with sources. We’ll have to do the same, right, with any kind of synthetically generated response. So, right now we’re confined and constrained and stuck in 20-minute long questionnaires that explores many different avenues as we could imagine as the architects of that design, right? And that’s kind of being unblocked, and we are now free to explore, just as we did with Booking’s synthetic experiment, iterations of questions and tug on threads that we might not have enough human population, it’s so niche, the IR is so low, we might not be able to even recruit them to ask these follow-up questions. And so, I see it as a shift away from record level. We will soon accept an answer to a question. But it depends on so many of these things falling into place, right, back to the 18 to 24-year-old males of the Pacific Northwest, right? We will get to what do I need to do to get these guys back? That’s the only thing I need to ask. And if I can dig it a little bit further out, the agent will be doing that for me soon, right? Ali, overnight I noticed that you’re struggling with these guys, so I activated the, you know, the 50% off—because that’s a different way of saying buy one get one free [laugh]—and look at what’s happening this morning as you wake up. So, I think it’ll be more of that fast twitch insight kind of generation to get us closer to action.

Karen: It’s funny because I remember last, it was the 2024 Qualtrics X-Force Summit where—and I think it was probably Brad—was talking on stage about agents and agentic AI. And explaining kind of the world of possibilities where in your CX ecosystem, you know, imagine the day when there’s a customer complaint, and then the agents like, “Yeah, let’s just send them a gift card,” or something like that. It’s like, “I got the complaint. I registered the complaint. I talked to your gift card vendor, and we’ve got one shipped out to them.” And you kind of show up one day and you’re like, “Oh, cool. Thanks for doing that for me.” [laugh]. I remember hearing that, thinking, I can’t wait till that’s really the reality because I think that’s, you know, that’s obviously what’s coming with the agentic actions in, you know, these kinds of organizations. So, very cool, very cool. So, anything else—again, I really want to be mindful of your time as well and our listeners’ time—anything else that you wish we had gotten to that we haven’t gotten to yet?

Ali: Well, I think it’s a mutual plug, Karen. I am so excited about IIEX. I’ll share, last year was the first year I have attended as, you know, representing a supplier, if you will, and I got so much out of it. I’m not kidding when I tell you I have a doc of 20-plus notes, and photos, and we still reference so much of what we’re seeing. It’s a finger on the pulse of just this heartbeat of innovation in this insight space. So, I would be doing myself a disservice not to talk about, first of all, how much value I see in this conference, but also what Qualtrics Edge will be doing this year. So, we will be there. We will be tugging on the threads of each of the things that we’ve started to talk about today, Karen. Will have four breakouts that go into great length and detail. We’ll have clients with us for a panel-style conversation for one of them, and we’ll go deep on validation and what it means, and what you need to be looking out for, not only from us, but so many others. And we’ll share some of that research on research that I just didn’t get time to talk about today. A study of over 3,000 global market researchers, so that we all feel good that it’s not just what I think what you think, and here’s the collective kind of perspective on where this is going. So, we’re super excited. And it’s coming up pretty quickly here, so I hope to see everyone there.

Karen: I’m excited, too. So, first of all, thank you for joining us last year, but also really thank you for this year, in advance. I’m excited. And to those who are listening, I think, like, this is probably it’s probably safe to say look up. So, in our exhibit hall that we’re going to have at DC, like, there’s going to be literally a Qualtrics Edge, so [laugh] that’ll be visible by looking up and traveling to their space specifically. I’m really excited for that. And I’ve looked at the topics—obviously, I’m deeply embedded in our agendas—and have seen the topics you’ve shared with Brigette. And, you know, I’m like, well, anybody who wants to learn a little bit more about, you know, just AI methods in particular, you have one that’s about agentic AI, I believe, so it’s like, you know, there’s some great stuff coming up that you’ll be bringing. So, thank you for bringing such stellar content as well. I’m excited for the whole event.

Ali: I am too. I am too.

Karen: I just can’t thank you enough for your time today. Thank you so much, Ali. What a pleasure.

Ali: Oh, likewise. It was fun. You talked about, you know, passion projects. You can hear it and you can feel it, right, in our voices. This was super fun. So, I really appreciate your time.

Karen: Yes. My pleasure, my pleasure. So many, many thanks to you again. And thanks to our editor, Big Bad Audio. Thank you for everything that you do to clean us up, to our production team at Greenbook, and also to all of you, our listeners, thank you so much for joining us week after week and especially for this episode. We’ll see you soon on another episode of the Greenbook Podcast. Bye-bye now.