I’ve spent the final a number of years watching enterprise collaboration instruments get smarter. Be a part of a video name right this moment, and there’s a superb probability 5 – 6 AI brokers are working concurrently: transcription, speaker identification, captions, summarization, process extraction. On the product facet of it, every agent will get evaluated in isolation. Separate dashboards, separate metrics. Transcription accuracy? Test. Response latency? Test. Error charges? All inexperienced.
However here’s what I constantly observe as a UX Researcher: customers are annoyed, adoption stalls, and groups are attempting to establish the foundation trigger. Per the metrics, the dashboards look advantageous. Each particular person part passes its exams. So, the place are customers really struggling?
The reply, virtually each time, is orchestration. The brokers work advantageous alone. They crumble collectively. And the one means I’ve discovered to catch these failures is thru consumer expertise analysis strategies that engineering dashboards had been by no means designed to seize.
The Orchestration Visibility Hole
Right here’s an instance of gaps that want a deeper understanding by consumer analysis: a transcription agent stories 94% accuracy and 200-millisecond response instances. However what the dashboard doesn’t present is that customers are abandoning the function as a result of two brokers gave them conflicting details about who mentioned what in a gathering. The transcription agent and the speaker identification agent disagreed, and the consumer misplaced belief in the entire system.
This downside is about to get a lot larger. Proper now, fewer than 5% of enterprise apps have task-specific AI brokers in-built. Gartner thinks that’ll bounce to 40% by the tip of 2026. We’re headed towards a world the place a number of brokers coordinate on virtually all the pieces. If we can not determine learn how to consider orchestration high quality now, we might be scaling damaged experiences.
UX Analysis Strategies Tailored for Agent Analysis
Customary UX strategies want some tweaking when you’re coping with AI that behaves otherwise every time. I’ve landed on three approaches that really work for catching orchestration issues.
1. Suppose-Aloud Protocols for Agent Handoffs
In conventional think-aloud research, you ask folks to relate what they’re doing. For AI orchestration, I layer in what I name system attribution probes at key handoff factors. I pause and ask members to explain what they consider simply occurred behind the scenes, then map their responses towards the precise agent structure. Most customers are unaware that separate brokers deal with transcription, summarization, and process extraction. When one thing goes incorrect: a transcription error, as an illustration, they blame “the AI” as a monolith, even when the summarization and routing labored completely. Consumer suggestions alone received’t get you there. What I’ve discovered works is mapping what folks suppose the system simply did towards what truly occurred. The place these two diverge, that’s the place orchestration is failing. That’s the place the design work must occur
2. Journey Mapping Throughout Agent Touchpoints
Contemplate a single video name. The consumer clicks to hitch, and a calendar agent handles authentication. A speech-to-text agent transcribes, a show agent renders captions, and when the decision ends, a summarization agent writes up the assembly whereas a process extraction agent pulls out motion gadgets. A scheduling agent may then guide follow-ups. That’s six brokers in a single workflow and 6 potential failure factors.
I construct dual-layer journey maps: the consumer’s expertise on prime, the accountable agent beneath. When these layers fall out of sync – when customers anticipate continuity however the system has handed off to a brand new agent; that’s the place confusion units in, and the place I focus my analysis to unpack deeper points.
3. Heuristic Analysis for Agent Transparency
Nielsen Norman’s traditional heuristics stay foundational, however multi-agent programs require us to increase them. “Visibility of system standing” has a unique that means when six brokers are working concurrently; not as a result of customers want to know the underlying structure, however as a result of they want sufficient readability to recuperate when one thing goes incorrect. The aim isn’t architectural transparency; it’s actionable transparency. Can customers inform what the system simply did? Can they right or undo it? Do they know the place the system’s limitations are? These standards reframe orchestration as a UX downside, not simply an infrastructure concern.
I’ve run heuristic evaluations the place the interface was polished and interplay patterns felt acquainted, but customers nonetheless struggled. The floor design handed each conventional test, however when the system failed, customers had no technique to diagnose what went incorrect or learn how to repair it. They didn’t must know which agent brought on the problem. They wanted a transparent path to restoration.
Case Research: Enterprise Calling AI
Right here’s an actual scenario I labored on that illustrates why orchestration high quality can matter as a lot as particular person agent efficiency.
An enterprise calling platform had deployed AI for transcription, speaker identification, translation, summarization, and process extraction. Each part hit its efficiency targets. Transcription accuracy was above 95%. Speaker identification ran at 89% precision. Process extraction caught motion gadgets in 78% of conferences. Nonetheless, consumer satisfaction was at 3.2 out of 5, and solely 34% of eligible customers had adopted the AI options. The product staff’s intuition was to enhance the fashions. I suspected the issue was in how the brokers labored collectively.
We ran think-aloud periods and found one thing the dashboards by no means confirmed: customers assumed that edits they made to dwell captions would carry over to the ultimate transcript. They didn’t. The programs had been fully separate. Once I constructed out the journey map, plotting consumer actions on one layer and agent duty on one other, I observed the timing misalignment instantly. Motion gadgets had been arriving in customers’ process lists earlier than the assembly abstract was even prepared. On the consumer layer, this appeared like duties showing out of nowhere. On the agent layer, it was merely the duty extraction agent ending earlier than the summarization agent. Each had been performing accurately in isolation. The orchestration made them really feel damaged.
Heuristic analysis surfaced a subtler concern: when the interpretation and transcription brokers disagreed about speaker id, the system silently picked one. No indication, no confidence sign, no means for customers to intervene.
This pointed us towards a design speculation: the issue wasn’t agent accuracy, it was coordination and recoverability. Relatively than foyer for mannequin enhancements, we centered on three orchestration-level adjustments. First, we synchronized timing so summaries and duties arrived collectively, restoring context. Second, we constructed unified suggestions mechanisms that permit customers right outputs as soon as quite than per-agent. Third, we added standing indicators exhibiting when handoffs had been occurring.
Three months later, adoption had jumped from 34% to 58%. Satisfaction scores considerably improved with rankings of 4.1 out of 5. Help tickets about AI options dropped by 41%. We hadn’t improved a single mannequin. The engineering staff didn’t suppose UX adjustments alone may transfer these numbers. Truthful sufficient, truthfully. However three months of knowledge made it exhausting to argue. Agent coordination isn’t simply an infrastructure downside. It’s a UX downside, and it deserves that degree of consideration.
A Three-Layer Analysis Framework
Primarily based on what I’ve seen throughout a number of deployments, I now suggest evaluating orchestration on three ranges. Layer one is technical metrics: latency, accuracy, and error charges for every agent. You continue to want these. They catch component-level failures. However they can not see coordination issues.
Layer two is behavioral alerts. Observe the place customers abandon workflows, how usually they revise AI-generated outputs, and whether or not they come again after their first expertise. These patterns trace at orchestration points with out requiring direct consumer suggestions.
Layer three is qualitative analysis. Do customers perceive what the brokers are doing and why are they doing it? Do they belief the outputs? Does the entire system really feel coherent and accessible or disjointed? McKinsey’s 2025 AI survey discovered that 88% of organizations use AI someplace, however most haven’t moved previous pilots with restricted enterprise influence (McKinsey, 2025). I believe an enormous a part of that hole comes from orchestration high quality that no one is measuring correctly.
What This Means for Product Groups
In most organizations I’ve labored with, UX researchers and AI engineers have restricted collaboration. Engineers tune particular person brokers towards benchmarks. UX researchers take a look at interfaces. No person owns the house between brokers the place coordination occurs. That hole is precisely the place these failures dwell.
Deloitte estimates {that a} quarter of corporations utilizing generative AI will launch agentic pilots this 12 months, with that quantity doubling by 2027 (Deloitte, 2025). Groups that implement orchestration analysis early could have an actual benefit. Groups that don’t will maintain questioning why their AI options will not be touchdown with customers. The funding required just isn’t huge. It contains UX researchers in orchestration design discussions, constructing telemetry that captures agent transitions, and working common research centered particularly on multi-agent workflows.
Conclusion
As AI merchandise evolve from single assistants to coordinated agent programs, the definition of “working” has to evolve with them. A set of brokers that every go their particular person benchmarks can nonetheless ship a damaged consumer expertise. Efficiency dashboards received’t catch it as a result of they’re measuring the incorrect layer. Consumer complaints received’t make clear it as a result of folks blame “the AI” with out realizing which part failed or why.
That is precisely the place UX analysis earns its seat on the desk. Not as a ultimate test earlier than launch, however as a self-discipline woven all through the product lifecycle. UXR helps groups reply the earliest questions: Are we fixing the appropriate downside? Who’re we fixing it for? It shapes success metrics that mirror actual consumer outcomes, not simply mannequin efficiency. It evaluates how brokers behave collectively, not simply in isolation.
UX analysis exhibits you what earns belief and what chips away at it. It makes certain accessibility will get in-built from the beginning, not bolted on later when the system is simply too tangled to repair correctly. None of that is separate work. It’s all linked, every layer feeding into the following. And as AI programs get extra autonomous, extra opaque, this type of rigor isn’t non-compulsory. The issue is, when groups are shifting quick, analysis looks like a pace bump. One thing to circle again to after launch.
However the price of skipping it compounds rapidly. The orchestration issues I’ve described don’t floor in QA. They floor when actual customers encounter actual complexity, and by then, belief is already broken.
AI programs are solely getting extra advanced, extra autonomous, and extra embedded in how folks work. UX analysis is how we maintain these programs accountable to the folks they’re meant to serve.
Incessantly Requested Questions
This is likely one of the commonest frustrations I see in enterprise AI. Particular person brokers go their benchmarks in isolation, however the true issues present up when a number of brokers should work collectively. Orchestration failures occur on the handoffs, like when a transcription agent and speaker identification agent disagree about who mentioned what, or when process extraction finishes earlier than summarization, and customers obtain motion gadgets with no context.
These coordination points by no means seem on component-level dashboards as a result of every agent is technically doing its job. That’s exactly why consumer analysis strategies are important. They floor the place the expertise truly breaks down in ways in which engineering metrics weren’t designed to catch.
Acquainted strategies like think-aloud protocols and journey mapping nonetheless work, however they want some changes for AI programs. In think-aloud research, I’ve discovered it useful to incorporate what I name system attribution probes, moments the place you pause and ask customers to explain what they consider simply occurred behind the scenes. Journey maps profit from a dual-layer method: the consumer expertise on prime and the accountable agent beneath.
Orchestration issues lie the place these layers are out of sync, and analysis ought to give attention to figuring out and evaluating these points.
Longitudinal and ethnographic analysis are essential to know AI agent efficiency over time. Strategies like diary research and ethnography allow researchers to guage how customers work together with the AI and shift their utilization patterns throughout days or even weeks, how that impacts belief, and establish new points that will emerge.
Preliminary impressions of an AI system usually differ from a consumer’s expertise after steady utilization. Longitudinal research reveal behaviors and workarounds that customers develop, and touchpoints that contribute to customers abandoning the function completely.
Primarily based on what I’ve noticed throughout a number of deployments, I like to recommend evaluating orchestration on three ranges. Layer one covers the technical metrics akin to latency, accuracy, and error charges for every agent.
Layer two focuses on behavioral alerts akin to workflow abandonment charges, how usually customers revise AI-generated outputs, and if they’re returning customers. These patterns trace at orchestration points with out requiring direct consumer suggestions.
Layer three is qualitative analysis that evaluates if customers truly belief the outputs, perceive what the brokers are doing, and understand the system as coherent quite than disjointed. All three layers working collectively reveal issues that any single layer would miss.
Actionable transparency just isn’t about educating customers the underlying structure of each agent. Customers want readability and the power to know what the system simply did, right or recuperate from errors when one thing seems to be incorrect, and perceive the place the system’s limitations are.
Actionable transparency offers customers clear paths to recuperate from errors.
When errors happen, customers should be knowledgeable about what their choices are for resolving the problem and learn how to transfer ahead. In observe, this may very well be unified suggestions mechanisms to let customers right outputs as soon as, quite than individually for every agent. It is also standing indicators that floor when handoffs are occurring, or undo performance that works throughout the whole system. The aim is to design for recoverability. When orchestration breaks down, customers can regain management and belief.
An important shift is recognizing that the house between brokers, the place coordination occurs, wants an proprietor. In most organizations I’ve labored with, engineers tune particular person brokers towards benchmarks whereas UX researchers take a look at interfaces. No person owns that hole, and that’s precisely the place orchestration failures are likely to dwell.
To shut this hole, groups ought to carry UX researchers into orchestration design discussions early, not simply on the finish for interface testing. They need to construct telemetry that captures agent transitions and handoff factors, not simply particular person agent efficiency. They need to run common research centered particularly on multi-agent workflows quite than treating AI as a single monolithic function. This does require intentional cross-functional collaboration to construct higher AI-products.
Priyanka Kuvalekar is a Senior UX Researcher at Microsoft, main mixed-method analysis for Microsoft Groups Calling and agentic AI collaboration experiences
Login to proceed studying and revel in expert-curated content material.
