Many teams that built a customer support chatbot as a single agent with a large stack of prompts are now considering a restructure into specialised “knowledge experts” and “action takers”. The idea is intuitive, and the word “multi-agent” is everywhere. This article gives product managers and engineering leads three things before they sit down to agree on such a structure: the vocabulary to understand what is being proposed, the evidence on what actually works in production, and the questions that let you steer a design review rather than just receive an architecture.

Why this topic is contested

In June 2025, two influential engineering posts were published in the same week with seemingly opposite conclusions. Cognition (the company behind the Devin coding agent) published Don’t Build Multi-Agents, arguing that multi-agent architectures are fragile because context gets lost between agents and their independent decisions conflict. Anthropic published How we built our multi-agent research system, reporting that a multi-agent setup dramatically outperformed a single agent on research tasks.

Both turned out to be right, for different shapes of task. The reconciliation, well summarised in LangChain’s How and when to build multi-agent systems, comes down to one question: can the work be split into independent parts, or does everything depend on shared state? Research queries split cleanly (“investigate market A” and “investigate market B” can run in parallel and never need to talk to each other). A customer support conversation does not split: it is one continuous, stateful thread where turn seven depends on what was said in turn two, and where the topic can shift mid-conversation from an invoice question to a payment arrangement.

This matters because “multi-agent” is not one architecture. It is a spectrum, and the useful debate is not “one bot or many” but two more precise questions:

  1. Who owns the conversation state? One component end to end, or is state passed between agents?
  2. Who is allowed to talk to the customer? One voice, or several agents taking turns?

Keep these two questions in mind throughout. Most architecture disagreements are really disagreements about these two things.

The pattern vocabulary

Five patterns cover almost everything you will hear proposed, ordered from least to most distributed.

Pattern 1: Single agent, many prompts (the monolith). One agent, one large prompt (or a stack of prompts), all tools attached. This is the common starting point. Fits when: the domain is narrow, tool count is low, and the team is small. Fails when: the prompt grows so large that instructions conflict, and the tool list grows past what the model can reliably choose from. Practitioner experience suggests that once an agent has access to 15 to 20 tools, tool selection accuracy degrades noticeably, and the fix is narrower scopes, not bigger prompts.

Pattern 2: Router (triage). A lightweight classifier reads the incoming message and routes it to exactly one specialised handler (a scoped prompt plus a small tool set). No collaboration between handlers. This is one of the five composable patterns in Anthropic’s Building Effective Agents, which remains the best single primer on agent design. Fits when: queries fall into distinct categories needing different expertise. Fails when: conversations cross category boundaries mid-thread and the router has no mechanism to carry context across the switch.

Pattern 3: Orchestrator with specialists-as-tools (the manager pattern). One central agent owns the conversation from start to finish. It can call specialist agents the same way it calls any tool: the specialist does a bounded piece of work (look up a policy, diagnose a technical issue, draft a calculation) and returns a result. The specialist never talks to the customer and holds no state of its own. OpenAI’s A Practical Guide to Building Agents calls this the manager pattern; the Agents SDK documentation calls it “agents as tools”. Fits when: you need specialisation and one coherent customer experience at the same time. Costs: every specialist call is an extra model call, so latency and token cost rise with each hop.

Pattern 4: Peer handoffs (decentralised). Agents are peers. A triage agent recognises the topic and hands the entire conversation to a specialist, which takes over talking to the customer and may hand off again later. Fits when: domains are genuinely disjoint (a billing conversation and a network diagnosis rarely blend) and each specialist needs deep, distinct tooling. Fails when: handoffs pass a summary instead of the full conversation. Cognition’s core principle applies here: share full agent traces, not just individual messages, because any small detail from an earlier turn can change how the task should be interpreted. Summarised handoffs are the “telephone game” that produces most multi-agent failures.

Pattern 5: Group chat / swarm. Multiple agents discuss in a shared thread, negotiate, and converge on an answer. Verdict: impressive in demos, poor in production. Cognition’s one-year follow-up, Multi-Agents: What’s Actually Working, concludes that unstructured swarms of agents negotiating with each other are mostly a distraction, and that the setups that work in practice keep one main loop carrying state while subagents are stateless workers with narrow scope. Treat any swarm proposal for a support bot with heavy scepticism.

The popular framing of “knowledge experts and action takers” maps naturally onto pattern 3, with elements of pattern 2. That instinct is sound; the rest of this article sharpens it.

What the evidence says

There is no clean head-to-head benchmark proving “multi-agent support bots beat single-agent support bots”. What exists instead is a consistent body of evidence about where distributed designs fail and where they earn their cost.

Naive multi-agent systems underdeliver, measurably. The most rigorous study is the Berkeley MAST paper, Why Do Multi-Agent LLM Systems Fail? (NeurIPS 2025). The authors analysed 1,600+ execution traces across seven popular multi-agent frameworks and opened with a blunt observation: despite the enthusiasm, performance gains on benchmarks are often minimal. They catalogued 14 recurring failure modes in three clusters: system design issues, inter-agent misalignment, and task verification failures. Roughly a third of failures came from inter-agent misalignment alone: agents losing context, repeating steps, or ignoring their role. Crucially, the failures stemmed from system design, not model weakness. Better prompts do not fix a bad topology.

The real bottleneck in support bots is consistency, not intelligence. The tau-bench benchmark (built by Sierra, a leading customer service AI company, with an actively maintained successor) simulates realistic support conversations where an agent must follow written policy while using tools and talking to a customer. Even state-of-the-art models succeeded on fewer than half the tasks, and reliability across repeated attempts was far worse (below 25 percent on pass^8 in the retail domain, meaning the agent rarely got the same task right eight times in a row). The lesson: what limits support automation is reliable policy-following across many attempts. Adding agents does not address that bottleneck; it adds new seams where policy can be dropped.

Multi-agent wins exist, but only for a specific task shape. Anthropic’s research system beat a single agent by 90.2 percent on internal evaluations, but consumed roughly 15 times the tokens, and the authors drew the boundary themselves: domains that require all agents to share the same context, or that involve many dependencies between agents, are a poor fit. Their gains came from breadth-first, read-only research where subagents explore independently. A support conversation is the opposite shape: sequential, stateful, and write-heavy (it changes contracts, appointments, payments). Anthropic’s later guidance, When to use multi-agent systems (and when not to), is explicit that splitting agents by problem type creates a telephone game where each handoff degrades fidelity, and describes an experiment where role-specialised agents spent more tokens coordinating with each other than doing actual work. Their rule: split only where context can be truly isolated.

The latency cost is real and appears immediately. A production case study from Microsoft’s engineering group, Orchestration Patterns for Multi-Agent Systems: Performance and Trade-offs, documents a large retailer moving from a deterministic router (described as fast and predictable) to a richer coordinator that could invoke several agents per request. Smoke testing showed notable latency overhead even in simple two-agent cases, driven not just by the extra network calls but by the coordinator’s planning and response synthesis steps. No single mitigation eliminated it. Notably, the retailer’s reason for distributing was organisational (letting domain teams own their own agents and reuse them across channels), not accuracy. That is a legitimate reason, but it is a different reason, and it should be argued honestly as one.

Production support vendors converged on one brain, not a federation. The companies whose entire business is AI customer support did not build department-bot networks. Sierra promotes a single-agent mindset where the customer interacts with one consistent entity. Decagon structures specialisation as Agent Operating Procedures: modular bundles of instructions, logic, and actions that are loaded into the agent per situation. Intercom’s Fin is a pipeline of specialised stages (query refinement, retrieval, reranking, generation, validation) rather than a set of conversing agents. In all three, specialisation lives inside or behind one conversational brain.

A good short synthesis of the whole debate is Philipp Schmid’s Single vs Multi-Agent Systems: the choice is not ideological, it follows from the task shape.

The winning shape for a support bot

Putting the evidence together, the pattern that has won for customer support is what could be called a supervised single thread with routed specialisation. It has four rules.

Rule 1: One orchestrator owns the conversation end to end. A single component holds the conversation state, decides what happens next, and produces every customer-facing message. Nothing else ever talks to the customer. This eliminates the largest documented failure class (context loss between agents) by construction, and it guarantees one consistent tone and one consistent memory.

Rule 2: Knowledge experts are stateless workers. Knowledge experts should be read-only specialists: each has a narrow scope, a small tool set (retrieval over its slice of the knowledge base, perhaps a diagnostic lookup), and a contract to return a distilled, sourced answer to the orchestrator. They hold no conversation state and make no decisions about the customer interaction. This is exactly the configuration Cognition found to work: intelligence contributed from the side, while one loop carries the state and all writes stay single-threaded.

Rule 3: Action takers sit behind deterministic flows. Anything that changes the real world (modifying a contract, scheduling an engineer, setting up a payment arrangement) involves sequentially dependent steps where a wrong turn has a real cost. Here, do not give a model freedom to improvise the sequence. Encode the sequence as an explicit flow or state machine: the orchestrator enters the flow, the flow dictates the steps and validations, and the model’s job inside it is limited to conversation (eliciting the right information, confirming, explaining). This mirrors the broader principle from Anthropic’s guidance that workflows (predefined paths) beat autonomous agents wherever predictability matters more than flexibility. Move fast where failure is recoverable; be strict at the seam where the bot touches money and systems of record.

Rule 4: Full handoffs are the exception, and they carry full context. Reserve peer handoffs for domains that are genuinely disjoint and heavy enough to justify a separate agent. When you do hand off, transfer the complete conversation history, never a summary.

Drawn out, it looks like this:

                    Customer
                       |
              [ Orchestrator ]  <- owns state, owns every reply
              /        |        \
   [ Knowledge ]  [ Knowledge ]  [ Deterministic flows ]
   [ expert A  ]  [ expert B  ]  [ (action taking)     ]
   read-only,      read-only,     state machines wrapping
   stateless,      stateless,     tools that change things
   returns facts   returns facts

Underneath this structure sits a three-layer content model that keeps the system maintainable as it grows:

  • Knowledge layer (facts): what is true. Product details, tariffs, procedures. Owned like content, updated without touching code.
  • Behaviour layer (rules): how the bot conducts itself. Tone, escalation rules, what it may and may not promise. Owned like policy.
  • Flow layer (sequences): what must happen in what order for consequential actions. Owned like software.

Most quality problems in a support bot can be traced to these three concerns being tangled together in one giant prompt. A migration away from the monolithic prompt is, at its core, the act of untangling them, and that is valuable even before any “multi-agent” topology enters the picture.

Agent and prompt design principles

On agent and prompt design, these are the principles with the strongest backing.

Split by context boundaries, not by org chart. The test for whether something deserves to be a separate agent is: can it do its job well without the rest of the conversation’s context? A knowledge lookup passes that test. “The retention team’s agent” does not automatically pass it. Splits that mirror team structure without a context rationale recreate the telephone game.

Keep tool sets small. Aim for roughly three to five tools per specialist, each well described. If a proposed agent needs fifteen tools, it is probably two or three agents, or its scope is wrong.

Scoped loading beats permanent splitting. An underrated alternative to separate agents: keep one agent, but load only the relevant procedure and knowledge into its context per topic (the router selects which bundle to load). This gets most of the benefit of specialisation (small, focused context) with none of the coordination cost. Decagon’s Agent Operating Procedures work this way. Ask whether a proposed specialist truly needs to be a separate running agent, or whether it is really a content bundle.

Share full traces on any delegation that carries the conversation. Summaries drop the detail that turns out to matter. This is the single most repeated lesson in the practitioner literature.

Writes stay single-threaded. Only one component, following one flow, executes actions that change customer data. Parallelism is for reading, never for writing.

Prompts: separate instructions, knowledge, and examples. Within any single agent’s prompt, keep behavioural rules, injected knowledge, and few-shot examples in clearly separated sections. Prompts degrade when facts are buried inside instructions, because updating a fact then risks changing behaviour. OpenAI’s guide names high-quality instructions as the top success factor, ahead of framework or topology choices.

Start from the single agent’s limits. The healthiest design conversation starts by asking what is actually breaking today: prompt too long, instructions conflicting, tool selection failing, or teams blocked on each other. Each symptom points to a specific, minimal split. A design that starts from “we want a multi-agent architecture” and works backwards is the wrong direction of reasoning. Both the OpenAI guide and Anthropic’s patterns post give the same advice: add complexity only when a simpler structure demonstrably falls short.

Organisational ownership is a valid reason to split, if named. Wanting different teams to own, deploy, and test their part independently is the reason large organisations distribute agents, as the Microsoft case shows. That is fine. But it should appear in the design rationale as an organisational argument with its latency and evaluation cost priced in, not disguised as an accuracy argument.

Evaluating the migration

A restructure like this can quietly make things worse, and without measurement nobody will be able to tell. Four things make the migration safe.

Build the evaluation set before the new architecture. Collect a few hundred real conversations covering every intent, including the awkward cross-topic ones. This becomes the regression suite. The current bot’s scores on it are the baseline the new structure must beat.

Measure the router as its own component. In any routed design, the router is the single highest-leverage failure point: a perfect specialist receiving the wrong query produces a confident wrong answer. Router accuracy (including “correctly recognising a mid-conversation topic switch”) deserves its own metric and its own test set.

Measure consistency, not just success. Following tau-bench’s lead, run the same test conversation multiple times and track how often the bot gets it right every time (pass^k), not just whether it can get it right once. Support is a reliability business; a bot that is right 9 times out of 10 on the same question is a liability.

Set a latency budget per hop. Every orchestrator decision, specialist call, and synthesis step is a model call the customer waits for. Agree the end-to-end latency target first, then check the proposed architecture against it hop by hop. If the architecture cannot meet the budget on paper, it will not meet it in production.

Finally, insist on a shadow or parallel run: the new structure answering real traffic invisibly alongside the old one, scored on the same conversations, before any cutover. And agree the rollback path in advance.

Questions to ask in an architecture review

These are the questions that surface whether a proposal is well founded. They are reasonable to ask in any design review, and the pattern of answers matters more than any single one.

State and context

  1. Where does conversation state live, and is there exactly one owner of it?
  2. When work is delegated to a specialist, what exactly does it receive: the full conversation, or a summary? (Summaries are the classic failure mode.)
  3. What happens when a customer switches topic mid-conversation? Walk through a concrete example across the proposed boundaries.

Decomposition rationale

  1. For each proposed agent: what breaks today that this split fixes? (Look for concrete symptoms: prompt length, conflicting instructions, tool confusion.)
  2. Could this specialist be a loadable content bundle (prompt plus knowledge plus tools) inside one agent instead of a separately running agent? What does being separate buy us?
  3. Which splits are for accuracy and which are for team ownership? Both are fine, but they should be labelled honestly.

Actions and safety

  1. Which agent or flow is allowed to execute actions that change customer data, and is that path deterministic?
  2. Can two components ever act on the same customer record in one conversation? (The answer should be no.)

Latency and cost

  1. What is the end-to-end latency budget, and how many model calls does the worst-case path make?
  2. What is the expected token cost per conversation versus today?

Testing and migration

  1. How is the router tested, separately from the specialists?
  2. How do we run the new structure in shadow against real traffic before cutover, and what is the rollback plan?

Signals worth probing further. None of these is disqualifying on its own, but each deserves a follow-up question: a proposal that leads with framework names rather than with the state and context model; agent boundaries that exactly mirror the team structure of the delivery organisation; agents that pass summaries to each other; a design with no stated latency budget; an evaluation plan that amounts to “we will test it manually”; and a big-bang rewrite with no parallel-run phase. Conversely, good signs are: the proposal starts from what is breaking today, the first milestone is small (one routed specialist, measured), and evaluation infrastructure appears early in the plan rather than at the end.

Reading list

In recommended order:

  1. Building Effective Agents (Anthropic, Dec 2024). The foundational vocabulary: workflows vs agents, and the five composable patterns including routing and orchestrator-workers. Read this first; 20 minutes.
  2. A Practical Guide to Building Agents (OpenAI, 2025). PM-friendly walkthrough of the manager and handoff patterns, guardrails, and the advice to start single-agent.
  3. Don’t Build Multi-Agents (Cognition, Jun 2025). The sharpest statement of why context loss kills distributed designs, and the origin of “context engineering”.
  4. Multi-Agents: What’s Actually Working (Cognition, 2026). The follow-up that defines the narrow class of multi-agent setups that do work: one loop carries state, subagents are stateless, writes stay single-threaded.
  5. How and when to build multi-agent systems (LangChain, Jun 2025). Short reconciliation of the Cognition and Anthropic positions.
  6. Why Do Multi-Agent LLM Systems Fail? (Berkeley, NeurIPS 2025). The empirical failure taxonomy. Skim the abstract and Figure 1 for the 14 failure modes.
  7. Orchestration Patterns for Multi-Agent Systems (Microsoft ISE, Jun 2026). The most honest production account of what a router-to-coordinator migration costs, and why organisations do it anyway.
  8. tau-bench and tau2-bench (Sierra). The benchmark that defines what “good” means for support agents: policy-following consistency across repeated attempts.
  9. When to use multi-agent systems (and when not to) (Anthropic/Claude blog, 2026). Practical rules for where splitting helps: context isolation and independent parallel work.

The one-line summary

The orchestrator-plus-experts idea is the right one, but only in the constrained form where one component owns the conversation and every customer-facing word, knowledge experts are narrow stateless workers, actions run through deterministic flows, and any handoff carries the full conversation. The free-form version where expert bots collaborate as peers has been tried at scale, measured, and abandoned.