The Governance Gap: Is Trust in AI Outpacing the Evidence?

Editor’s Note: In this takeaway from IIEX.AI, Ashley examines a defining challenge for today's research leaders: closing the gap between rapid AI adoption and real governance. A special thank you to Lisa Courtade (Organon), Tracy Adams (The ARF), Melissa Ramey (Salesforce), Liam Hickey (Knit), and Jenny Nicholson (WALDO.fyi) for sharing the real-world validation frameworks, governance principles, and accountability practices featured in this piece. (Missed part one on moving synthetic research from rapid generation to real-world decision confidence? Read the previous article here.)

---

Lisa Courtade, AVP of Global Insights & Analytics at Organon, was reviewing an AI-generated answer about an opportunity for a new product when something felt wrong. Trained as a statistician, she questioned whether the conclusion made sense and went back to the evidence.

The system, she explained during the IIEX.AI town hall “AI Is Commoditizing Research. Now What?”, had drawn on marketing content and exaggerated its claims into an unsupported answer.

The problem surfaced because someone with relevant expertise recognized a reason to investigate. Without that intervention, a plausible answer could have become the basis for a business decision.

That possibility ran through several IIEX.AI discussions. Alongside questions about automation and job displacement, speakers raised a more immediate concern: whether confidence in AI output is advancing faster than the practices needed to validate it.

For research teams, governance means establishing how AI outputs are checked, when expert review is required, and who remains responsible for the decisions they inform. As AI makes research outputs easier to produce and distribute, those responsibilities need to be clear before the results reach a decision-maker.

Confidence is Moving Faster Than Validation

In “Marketers Trust AI. Is the Evidence Keeping Up?”, Tracy Adams, Sr. Director of Research & Insights at The ARF, described a gap between growing comfort with AI and the processes needed to check its outputs.

She explained:

“...confidence is moving faster than validation, and marketers are becoming more comfortable with AI outputs, but the processes that would allow them to actually verify those outputs are not developing at the same rate,” 

Adams framed the issue as “institutional infrastructure that is lagging behind individual adoption and confidence.”

That distinction matters. An organization can encourage AI adoption without giving employees clear standards for evaluating the results. People may be expected to use the tools, exercise judgment, and move quickly, while still lacking a shared definition of what sufficient validation looks like.

Adams also cautioned that internal checks do not always establish whether an output is independently valid. A system may behave in line with an organization’s expectations while still producing conclusions that need further scrutiny.

For research teams, the practical question is what a test demonstrates. Does it show that a system produces consistent answers? That its findings match a credible benchmark? That its conclusions remain useful across different audiences and business contexts? Those are different claims, and each requires appropriate evidence.

Accountability can disappear while oversight appears intact

Melissa Ramey, Director of AI Transformation - Data Insights and User Experience at Salesforce,
 gave the problem a name in “Volume isn’t Value: Truth and Governance at the AI Frontier.”

“Accountability offloading is the delegation of judgment to the AI tool, the uncritical acceptance of the output of that delegated work, and then the transfer of blame to the AI for the faulty work,” she said.

The sequence is important. The failure begins before anyone blames the technology. It starts when a person hands over a task that requires judgment, then accepts the result without adequately exercising that judgment themselves.

Ramey added:

“In practice, accountability offloading lets us keep the feeling of oversight while we quietly lose the functions." 

A human can be present in a workflow without providing meaningful scrutiny. Approving a report is useful only if the reviewer has the expertise, evidence, time, and authority to challenge it.

Ramey also questioned whether adding another review at the end addresses the underlying problem. Describing an experience in which correct input became incorrect output, she explained that catching errors did not tell her why the product was creating them.

Her example suggests that governance needs to reach into the workflow itself. Teams need to investigate where information changes, what the tool is being asked to do, and how its behavior is evaluated. Repeatedly correcting the finished product leaves the source of the errors unresolved.

Researchers are worried about decisions made without them

The concern becomes more acute as AI makes research outputs accessible to people outside insights teams.

In “Market Research in 2028: Seeing Around the Corner with AI,” Liam Hickey, Head of Research at Knit, raised the concern that non-researchers could make decisions from AI outputs without research involvement.

The concern shifts attention from whether AI will replace researchers to how research expertise stays connected to the decisions the work supports. If an answer reaches a stakeholder without its context or limitations, the researcher may have little opportunity to explain what the evidence can sustain.

Broader access can help colleagues explore questions and use existing knowledge. It also requires a way to distinguish an exploratory answer from a finding ready to guide a consequential choice.

Hickey highlighted business context, transparency into how conclusions were reached, traceability to source data, and researcher involvement as foundations for trust. As he put it, insights professionals want to be able to “verify but trust, if you like, rather than trust but verify.”

Those requirements suggest a practical design principle: make the basis of an answer available alongside the answer itself. Users need to understand where it came from, which question it addresses, and when additional research involvement is required. Otherwise, access to an answer can be mistaken for sufficient evidence to act.

Expert judgment has to remain visible

Courtade’s experience with the product-opportunity answer illustrates the value of questioning an output before accepting it. It also raises a longer-term issue: how colleagues learn to make those judgments themselves.

Courtade acknowledged the value of using AI to reduce cognitive workload, particularly for teams under pressure. Her boundary was clear: “What I don’t recommend anybody do is cognitive surrender, which is have the AI do all the thinking.”

That boundary also has implications for how researchers learn. If experienced professionals silently correct AI output, colleagues may see the finished answer without seeing the reasoning that made it dependable.

Karen Lynch, Editor-in-Chief and Chief Programming Officer at Greenbook, emphasized the need to make that scrutiny visible:

“They have to see their seniors in the industry using these critical thinking skills and challenging and pushing back against AI. Otherwise, they will not learn the skill of pushing back on AI.”

Governance therefore includes professional practice alongside formal processes. Teams need opportunities to explain why a source is weak, why a comparison is misleading, or why a plausible finding needs further investigation. Those judgments become more valuable when others can learn how to make them.

Agentic workflows need identifiable owners

Agentic systems extend the accountability question across tasks and intermediate outputs. When work passes through several steps, reviewing the final answer may reveal little about where an error entered or how it spread.

In “AGENTIC AI: The Good, the Bad, and the Ugly,” Jenny Nicholson, Head of Creativity at WALDO.fyi, urged people building or buying these systems to identify the responsible human and understand the decisions that person makes.

She also offered a practical way to assess vendors: ask about difficult cases, failures, and testing.

Nicholson explained: 

“So if you’re evaluating a vendor or somebody building these systems, ask them, what’s the hardest edge case they run into? Where has it failed the most? What’s their testing protocol? How do they know?” 

Those questions move evaluation beyond a successful demonstration. They ask whether the supplier understands the system’s limitations and can explain the evidence behind its performance.

Nicholson also recommended breaking work into subtasks and starting where agents can provide value with lower risk and effort. That approach creates a manageable place to define success, test performance, and determine when a person needs to intervene.

For insights teams, ownership needs to be specific enough to function. Someone must be able to investigate a problem, pause the workflow, and decide what happens next. Responsibility assigned in principle has little value if the owner cannot see or influence the work.

Give confidence a basis in evidence

Taken together, these sessions make a case for treating governance as part of how AI-enabled research is designed and used.

A workable starting point is to ask five questions:

• Evidence: Can we trace the conclusion to the information supporting it?
• Validation: What checks establish that the output is suitable for this particular use?
• Review: Who has the expertise and authority to challenge it?
• Accountability: Who owns the decision and any correction?
• Limits: What uncertainty requires additional research or human intervention?

The answers should reflect the consequences of getting something wrong. Exploring possible discussion topics and recommending a major investment call for different levels of assurance.

The challenge for insights teams is to make validation and accountability as routine as AI use itself. That means putting the conditions for trust into the workflow, so the people using an answer can understand why it deserves to influence their decision.

As Ramey put it:

“The AI will help you build the tool, but only you have the expertise to build the governance.”