Agent architecture · field notes
Multi agent orchestration

Your multi-agent system works. But do you understand how your agents communicate?

Most builds get the agents right and leave the coordination layer undefined. Here are the five communication patterns, the four orchestration topologies, and the questions that tell you which one you actually built.

Field notes~18 min read 5 patterns · 4 topologiesInteractive

Today, I witnessed an interesting discussion at my workplace.

Our company founder and a friend from Deloitte were evaluating candidates for an engineering role. The initial screening involved qualifications and resumes, with Hermes being used in the process. Shortlisted candidates were then given a practical assignment:

Build a multi-agent orchestration system within two days.

The requirements included proper error handling, guardrails, constraints, and other elements expected in a reliable agentic system.

But what caught my attention wasn't the assignment itself. It was the way candidates were being evaluated.

As I listened to the discussion, I started thinking about something that I believe is frequently overlooked in agentic AI engineering.

How do the agents actually communicate and coordinate with one another?

Not just whether they can call tools. Not just whether they have skills. Not just whether they have guardrails.

But what kind of orchestration architecture are they using? How do they delegate tasks? How do they exchange information? What happens when two agents disagree? How does the system recover when an agent fails?

A lot of people focus on building agents. Far fewer focus on engineering the coordination layer that makes multiple agents work together effectively. And that's where the real complexity begins.

Section 01

Building an agent is not the same as engineering a multi-agent system

The current agentic AI ecosystem offers an enormous number of tools and frameworks. Agent frameworks, tool-calling systems, skills and reusable capabilities, MCP servers, guardrails, memory systems, agent harnesses, observability platforms, evaluation frameworks, model routing systems.

All of these can be valuable. But assembling these components does not automatically result in a well-engineered multi-agent system.

Imagine you build three agents: a research agent, a coding agent, and a verification agent. You connect them together and give each of them a set of tools. The system appears to work. But here are the questions that matter.

How does the research agent pass its findings to the coding agent?

Does it send a structured message? Does it write to shared state? Does an orchestrator relay the information? Does the coding agent directly request additional information?

What happens when the verification agent rejects the generated code?

Does the system retry? Does the creator revise its output? Does a supervisor intervene? Or does the entire workflow simply fail?

These are not merely implementation details. They are fundamental architectural decisions.

A multi-agent system is not just a collection of LLM calls. It is a coordinated system of decision-making components with defined responsibilities, communication paths, and execution rules.
Section 02

The missing layer: multi-agent communication architecture

When designing a multi-agent system, one of the first questions should be: how are the agents going to communicate and coordinate?

There isn't a single universal answer. Different tasks require different coordination patterns. Here are the five fundamental approaches, each with a diagram of the mechanism, what it buys you, and what it costs you.

Pattern 01

Delegation

A central orchestrator assigns the work. A central orchestrator or supervisor decides which agent handles a task, keeps control of the workflow, and collects the results.

OorchestratorA1A2A3ASSIGN · RETURN

A simplified flow looks like this. User request goes to the orchestrator, which routes to the research agent, the coding agent, or the verification agent. The orchestrator collects the results and determines what happens next.

Take an agentic software development system. The user asks:

"Build a REST API with authentication, database integration, and automated tests."

The orchestrator breaks the request into smaller tasks:

  1. Assign API design to the architecture agent.
  2. Assign database modeling to the database agent.
  3. Assign implementation to the coding agent.
  4. Assign testing to the verification agent.
  5. Assign security checks to the security agent.

Each agent has a specific responsibility. The orchestrator coordinates the overall process.

Advantages

  • Clear task ownership
  • Centralized control
  • Easier workflow monitoring
  • Straightforward task routing
  • Easier enforcement of global policies

Challenges

  • The orchestrator can become a bottleneck
  • Poor routing affects the entire workflow
  • A central coordinator may be a single point of failure
  • Complex tasks can overload the supervisor with context
When it makes sense

When the task has a clear hierarchy, distinct specialist roles, and a need for centralized coordination.

Pattern 02

Peer to peer

Agents collaborate directly. Agents communicate directly with one another rather than requiring every message to pass through a central orchestrator.

A1A2A3A4DIRECT · NO HUB

A planning agent designs a deployment strategy. It asks a security agent to review the proposed architecture. The security agent identifies a vulnerability and communicates its concerns directly to the planning agent. The planning agent revises the design. A separate infrastructure agent then evaluates whether the revised plan can be deployed.

The interaction looks more like a collaborative network than a rigid hierarchy.

Worth saying plainly

Peer to peer does not mean every agent should be allowed to communicate with every other agent. Unrestricted communication creates unnecessary complexity.

Advantages

  • Flexible collaboration
  • Less dependence on a central coordinator for every exchange
  • Useful for dynamic workflows
  • Agents interact according to the needs of the task

Challenges

  • Communication paths become harder to manage
  • Agents may create unnecessary interaction loops
  • Shared state and ownership get complicated
  • Debugging distributed decisions is difficult
  • Access control must prevent unauthorized interactions
When it makes sense

When agents need to collaborate dynamically and the system can support well-defined communication contracts and interaction boundaries.

Pattern 03

Creator and verifier

One agent generates, another validates. One agent creates an artifact. Another agent independently evaluates it. This is one of the most useful patterns for reliability-oriented systems.

CVREJECTDRAFTPASScreatorverifierships

The artifact could be code, a database migration, a research report, a business proposal, a security configuration, a structured data transformation, or an API specification.

If the verifier approves, the workflow proceeds. If it rejects, the system sends structured feedback back to the creator, which creates an iterative refinement loop.

In a coding agent that looks like this:

  1. The coding agent implements a feature.
  2. The testing agent executes automated tests.
  3. The security agent checks for vulnerabilities.
  4. The verifier reports failures and actionable feedback.
  5. The coding agent revises the implementation.
  6. The system runs the tests again.

This is much more meaningful than simply asking another LLM "is this code good?"

A robust creator and verifier architecture should define what exactly is being verified, what evidence is required for approval, which tests must pass, what counts as a failure, how many retries are allowed, when the system escalates to a human, and who has authority to approve the final artifact.

The principle

Verification should rely on appropriate evidence wherever possible, not merely on another model's opinion. For code that means automated tests, static analysis, type checking, and security scanning. For factual research it means source validation and evidence checks.

Advantages

  • Separates generation from evaluation
  • Encourages measurable quality checks
  • Can detect errors before deployment
  • Supports iterative improvement

Challenges

  • A verifier can also make mistakes
  • Two LLMs agreeing does not prove correctness
  • Repeated retries increase cost and latency
  • Poorly defined criteria create false confidence
When it makes sense

When correctness matters more than speed, and when there is real evidence available to verify against.

Pattern 04

Negotiation and debate

Agents exchange proposals and critiques. Some tasks involve competing objectives, conflicting constraints, or several plausible solutions. Agents exchange proposals, challenge assumptions, and revise their positions.

A1A2SPROPOSECRITIQUEDECIDESscorer

Imagine a system designing a cloud infrastructure architecture with three specialists: a cost optimization agent, a security agent, and a performance engineering agent.

The cost agent proposes a low-cost architecture. The security agent identifies compliance concerns. The performance agent argues the proposal creates latency bottlenecks. The agents exchange proposals and critiques, and the system produces a revised architecture that attempts to satisfy the relevant constraints.

A well-designed negotiation system needs defined objectives, explicit constraints, a termination condition, a conflict-resolution mechanism, an evaluation or scoring procedure, and a final decision authority.

Important caveat

Debate does not automatically improve correctness. A group of agents can confidently agree on an incorrect answer. The system still needs reliable evaluation.

Advantages

  • Makes trade-offs explicit
  • Encourages consideration of alternative solutions
  • Useful when objectives conflict
  • Supports collaborative planning

Challenges

  • Agents may endlessly debate
  • More discussion does not mean better decisions
  • Models can reinforce one another's mistakes
  • Something must resolve the disagreement
  • Token usage and latency grow rapidly
When it makes sense

When the task has genuine trade-offs between competing objectives, and there is a scoring mechanism to settle them.

Pattern 05

Broadcast and publish–subscribe

Distribute information to many agents. An agent or event bus publishes an event that multiple interested agents can consume. The publisher makes information available rather than addressing a single recipient.

A1A2A3A4PpublisherEVENT

Consider a developer-focused service monitoring platform. A monitoring agent detects that an external API is down and publishes an event:

"Payment API unavailable."

Multiple agents react. An incident analysis agent investigates. A notification agent alerts users. A logging agent records the incident. A recovery agent checks whether service has been restored. An analytics agent updates reliability metrics. The monitoring agent does not need to coordinate each downstream response individually.

A production implementation may require durable message delivery, idempotent event handlers, correlation IDs, event versioning, retry and dead-letter mechanisms, and access controls for event subscriptions.

Advantages

  • Loose coupling between components
  • Multiple agents react to the same event
  • Suits event-driven architectures
  • Enables asynchronous processing
  • New subscribers can be added without changing the publisher

Challenges

  • Event ordering may matter
  • Duplicate messages must be handled
  • Delivery failures need recovery mechanisms
  • Shared context may be incomplete
  • Event schemas must stay compatible
  • Tracing a distributed chain is difficult
When it makes sense

When multiple independent components need to react to the same event, particularly in monitoring, automation, and event-driven systems.

Section 03

Important distinction: these patterns are not mutually exclusive

This is where architectural thinking becomes important. Delegation, peer to peer communication, creator and verifier workflows, negotiation, and broadcast describe different aspects of coordination. They are not necessarily competing architectural choices.

A single multi-agent system can combine several of them:

  1. A supervisor delegates tasks to specialist agents.
  2. Two research agents communicate directly to resolve a missing dependency.
  3. Multiple research agents work in parallel.
  4. A creator agent generates a solution.
  5. A verifier agent validates the solution.
  6. A monitoring component broadcasts an event when a task fails.

This is a hybrid multi-agent architecture. The right question is not "which communication pattern is the best?"

Which communication pattern is appropriate for this particular interaction, and why?
Section 04

The second missing dimension: orchestration topology

Communication patterns explain how agents interact. Orchestration topology explains how the overall workflow is structured. These two concepts are related, but they are not the same.

Switch between the four below. The nodes stay the same, only the shape changes.

Sequential: A pipeline of agents

Agents execute in a predefined order. The output of one agent becomes the input to the next. A research workflow might run collect, then analyse, then write, then verify. Each stage depends on the one before it.

Advantages

Simple to understand. Easy to trace. Predictable execution order. Straightforward state passing.

Challenges

Latency accumulates across stages. A failed stage blocks the pipeline. Independent tasks cannot use parallelism. A rigid sequence handles change badly.

Read this before choosing it

A sequential workflow is not inherently inferior to a more complex architecture. If the task is naturally sequential, a pipeline may be the most appropriate design.

Section 05

The architecture I would want candidates to explain

Suppose a candidate submits a multi-agent system for a two-day engineering assignment. It has a planner, a researcher, a coding agent, a verifier, a tool layer, guardrails, and error handling. At first glance it might look impressive. But I would want to understand the architecture behind it.

Question 1

Why did you choose this topology?

Why sequential and not parallel. Why a supervisor is necessary. Why there is an iterative feedback loop. Was the architecture selected because the task required it, or because the framework made it easy to implement?
Question 2

How do agents exchange information?

Structured messages, shared state, an event bus, direct agent-to-agent requests, tool-mediated communication, a centralized broker. What is the message schema. How are intermediate artifacts represented. How is context passed between agents.
Question 3

Who owns the decisions?

If a research agent recommends one approach and the security agent recommends another, who decides. A supervisor, a deterministic policy, a dedicated decision agent, a human, or an evidence-based evaluation mechanism. The system should have a clear decision authority.
Question 4

What happens when agents disagree?

The creator says the implementation is ready. The verifier says it fails security checks. Does the creator receive structured feedback. Is the task retried. Is a different agent consulted. Is there a maximum retry limit. When does the system stop and ask a human.
Question 5

What happens when an agent fails?

The research agent times out. The coding agent returns invalid structured output. A tool call fails. An external API goes down. An agent burns its token budget. Does the system retry, fall back, skip, roll back, escalate, or terminate safely? A system that handles only the happy path is not ready for production.

These questions help distinguish someone who has assembled a demo from someone who understands agentic systems engineering.

Section 06

The evaluation dimensions that matter

For a two-day multi-agent engineering challenge, the quality of the final demo should not be the only consideration. The architecture, reliability, and engineering decisions should also be examined. Open any row for what to listen for.

Look for a decision, not a default. The candidate should be able to say what the task required and which shape serves it.
Named patterns and a message schema, not 'they call each other'.
There should be one answer, and the candidate should know it without thinking.
Ask where an artifact physically lives between two agents.
The most revealing dimension. Happy-path-only systems fall over here.
A guardrail only at the prompt layer is a suggestion, not a constraint.
If you cannot reconstruct a run after the fact, you cannot debug it.
Evidence beats a second model's opinion.
Retry budgets, caching, and knowing which calls are actually needed.
Irreversible actions should have a person in the path.

This framework evaluates whether the candidate understands the system as an engineered product rather than simply a collection of framework components.

Section 07

Tools, skills, harnesses, and guardrails are not the entire system

Modern agentic systems often emphasize tools, skills, harnesses, and guardrails. All of these are important. But they answer different questions.

ComponentCore question it answers
ToolsWhat can the agent do?
SkillsWhat capabilities can the agent reuse?
HarnessHow is the agent executed and controlled?
GuardrailsWhat is the agent allowed or forbidden to do?
OrchestrationHow do multiple components coordinate?
Communication protocolHow do agents exchange information?
State managementWhat does the system remember and share?
EvaluationHow do we know the result is acceptable?
ObservabilityWhat happened during execution?

You can have a sophisticated tool ecosystem and still build a poorly coordinated multi-agent system. You can have excellent guardrails and still have an inefficient orchestration design. You can have multiple powerful LLMs and still end up with a system that cannot recover from a simple failure.

The engineering challenge is integrating these components into a coherent, reliable architecture.
Section 08

A practical example: a reliable multi-agent coding system

Imagine we are building an agentic software development platform. The user submits:

"Build a REST API for a task management application with authentication, database integration, and automated tests."

We have five agents: planner, researcher, coder, verifier, security reviewer. Instead of simply connecting them, we define an architecture. Step through it.

Step 1 · owned by the planner

Planning

The planner decomposes the request into structured tasks. It produces a task specification containing functional requirements, technical constraints, expected artifacts, acceptance criteria, and dependencies.

Constraint

The planner does not directly modify the codebase.

Step 2 · owned by the researcher

Research

The researcher investigates relevant implementation details and produces a structured research artifact holding findings, sources, assumptions, and relevant technical constraints.

Constraint

The planner or orchestrator decides whether the research is sufficient.

Step 3 · owned by the coder

Implementation

The coding agent receives the approved task specification and the relevant research artifacts, then implements the feature in a controlled workspace.

Constraint

Tool permissions restrict it to the operations it actually needs.

Step 4 · owned by the verifier

Verification

The verifier checks the implementation using appropriate evidence: automated tests, type checking, linting, API contract validation, and relevant security checks.

Constraint

It produces a structured result: pass or fail, failed checks, evidence, and recommended corrections.

Step 5 · owned by the orchestrator

Feedback loop

If the implementation fails, the verifier reports to the orchestrator, which returns the failure report to the coding agent for correction.

Constraint

The system enforces a retry budget. If it keeps failing, the workflow stops or escalates to a human.

Step 6 · owned by the security reviewer

Security review

The security reviewer evaluates the implementation against the relevant security requirements.

Constraint

Its output is treated as an additional review artifact, not as an unquestionable source of truth.

Step 7 · owned by the orchestrator

Final decision

The orchestrator applies the acceptance criteria and determines whether the task is ready for approval.

Constraint

For high-impact actions, human approval may be required before deployment.

What communication methods are being used here?

This one system combines several patterns:

This is the difference between simply having five agents and designing a multi-agent system with explicit coordination logic.

Section 09

An important correction: more complex orchestration does not mean better engineering

A system does not need to use every communication pattern to be well engineered.

A sequential pipeline with clear state schemas, robust error handling, deterministic routing, strict tool permissions, comprehensive testing, and strong observability may be better engineered than a complex peer to peer system with dozens of agents and poorly defined responsibilities.

Similarly, a single agent with well-designed tools and a reliable control loop may outperform a multi-agent system when the task does not require multiple specialized roles.

The goal is not to maximize the number of agents. It is to choose the simplest architecture that reliably solves the problem.

Architectural complexity should be justified by task complexity. A candidate who uses a single agent with a well-designed workflow should not automatically be considered weaker than someone who deploys ten agents. The real question is whether the candidate can explain their decisions and demonstrate that the system meets its requirements.

Section 10

What interviewers should look for

If a candidate builds a multi-agent orchestration system, the interviewer should not focus exclusively on the final output. Start here:

"Walk me through a single request from the moment it enters your system to the moment the final output is produced."

Then follow up:

Section 11

The principle I took away from this discussion

Agentic engineering is not just about giving agents tools, skills, and guardrails. It is about designing the coordination layer. That means understanding how agents communicate, how tasks are delegated, how workflows are structured, how agents synchronize, how outputs are verified, how conflicts are resolved, how failures are handled, how state is shared, how decisions are evaluated, and how the system terminates safely.

Without a deliberate orchestration architecture, a multi-agent system can become little more than a collection of LLM calls connected by ad hoc logic. And that may be enough for a prototype. But building reliable agentic systems requires a deeper level of engineering.

The real question is not "how many agents did you build?" It is: what coordination mechanism does your task require, why did you choose it, and how do you know your system works reliably?

That's the difference between building agents and engineering agentic systems.

One final distinction

Agent communication is not the same as agent intelligence. An agent may communicate through a sophisticated protocol and still produce poor results. Likewise, a highly capable model may perform well in a simple sequential workflow.

Architecture must be evaluated alongside task success, reliability, cost, latency, observability, quality of decisions, and safety and operational constraints.

The takeaway

The architecture should serve the task, not the other way around.

Contents

  1. Building an agent is not engineering a system1 min
  2. The missing layer: communication architecture5 min
  3. The patterns are not mutually exclusive1 min
  4. The second missing dimension: topology2 min
  5. The architecture I would want explained1 min
  6. The evaluation dimensions that matter1 min
  7. Tools and guardrails are not the whole system1 min
  8. A practical example, end to end2 min
  9. More complexity does not mean better engineering1 min
  10. What interviewers should look for1 min
  11. The principle I took away1 min