Today, I witnessed an interesting discussion at my workplace.
Our company founder and a friend from Deloitte were evaluating candidates for an engineering role. The initial screening involved qualifications and resumes, with Hermes being used in the process. Shortlisted candidates were then given a practical assignment:
The requirements included proper error handling, guardrails, constraints, and other elements expected in a reliable agentic system.
But what caught my attention wasn't the assignment itself. It was the way candidates were being evaluated.
As I listened to the discussion, I started thinking about something that I believe is frequently overlooked in agentic AI engineering.
Not just whether they can call tools. Not just whether they have skills. Not just whether they have guardrails.
But what kind of orchestration architecture are they using? How do they delegate tasks? How do they exchange information? What happens when two agents disagree? How does the system recover when an agent fails?
A lot of people focus on building agents. Far fewer focus on engineering the coordination layer that makes multiple agents work together effectively. And that's where the real complexity begins.
Building an agent is not the same as engineering a multi-agent system
The current agentic AI ecosystem offers an enormous number of tools and frameworks. Agent frameworks, tool-calling systems, skills and reusable capabilities, MCP servers, guardrails, memory systems, agent harnesses, observability platforms, evaluation frameworks, model routing systems.
All of these can be valuable. But assembling these components does not automatically result in a well-engineered multi-agent system.
Imagine you build three agents: a research agent, a coding agent, and a verification agent. You connect them together and give each of them a set of tools. The system appears to work. But here are the questions that matter.
Does it send a structured message? Does it write to shared state? Does an orchestrator relay the information? Does the coding agent directly request additional information?
Does the system retry? Does the creator revise its output? Does a supervisor intervene? Or does the entire workflow simply fail?
These are not merely implementation details. They are fundamental architectural decisions.
The missing layer: multi-agent communication architecture
When designing a multi-agent system, one of the first questions should be: how are the agents going to communicate and coordinate?
There isn't a single universal answer. Different tasks require different coordination patterns. Here are the five fundamental approaches, each with a diagram of the mechanism, what it buys you, and what it costs you.
Delegation
A central orchestrator assigns the work. A central orchestrator or supervisor decides which agent handles a task, keeps control of the workflow, and collects the results.
A simplified flow looks like this. User request goes to the orchestrator, which routes to the research agent, the coding agent, or the verification agent. The orchestrator collects the results and determines what happens next.
Take an agentic software development system. The user asks:
The orchestrator breaks the request into smaller tasks:
- Assign API design to the architecture agent.
- Assign database modeling to the database agent.
- Assign implementation to the coding agent.
- Assign testing to the verification agent.
- Assign security checks to the security agent.
Each agent has a specific responsibility. The orchestrator coordinates the overall process.
Advantages
- Clear task ownership
- Centralized control
- Easier workflow monitoring
- Straightforward task routing
- Easier enforcement of global policies
Challenges
- The orchestrator can become a bottleneck
- Poor routing affects the entire workflow
- A central coordinator may be a single point of failure
- Complex tasks can overload the supervisor with context
When the task has a clear hierarchy, distinct specialist roles, and a need for centralized coordination.
Peer to peer
Agents collaborate directly. Agents communicate directly with one another rather than requiring every message to pass through a central orchestrator.
A planning agent designs a deployment strategy. It asks a security agent to review the proposed architecture. The security agent identifies a vulnerability and communicates its concerns directly to the planning agent. The planning agent revises the design. A separate infrastructure agent then evaluates whether the revised plan can be deployed.
The interaction looks more like a collaborative network than a rigid hierarchy.
Peer to peer does not mean every agent should be allowed to communicate with every other agent. Unrestricted communication creates unnecessary complexity.
Advantages
- Flexible collaboration
- Less dependence on a central coordinator for every exchange
- Useful for dynamic workflows
- Agents interact according to the needs of the task
Challenges
- Communication paths become harder to manage
- Agents may create unnecessary interaction loops
- Shared state and ownership get complicated
- Debugging distributed decisions is difficult
- Access control must prevent unauthorized interactions
When agents need to collaborate dynamically and the system can support well-defined communication contracts and interaction boundaries.
Creator and verifier
One agent generates, another validates. One agent creates an artifact. Another agent independently evaluates it. This is one of the most useful patterns for reliability-oriented systems.
The artifact could be code, a database migration, a research report, a business proposal, a security configuration, a structured data transformation, or an API specification.
If the verifier approves, the workflow proceeds. If it rejects, the system sends structured feedback back to the creator, which creates an iterative refinement loop.
In a coding agent that looks like this:
- The coding agent implements a feature.
- The testing agent executes automated tests.
- The security agent checks for vulnerabilities.
- The verifier reports failures and actionable feedback.
- The coding agent revises the implementation.
- The system runs the tests again.
This is much more meaningful than simply asking another LLM "is this code good?"
A robust creator and verifier architecture should define what exactly is being verified, what evidence is required for approval, which tests must pass, what counts as a failure, how many retries are allowed, when the system escalates to a human, and who has authority to approve the final artifact.
Verification should rely on appropriate evidence wherever possible, not merely on another model's opinion. For code that means automated tests, static analysis, type checking, and security scanning. For factual research it means source validation and evidence checks.
Advantages
- Separates generation from evaluation
- Encourages measurable quality checks
- Can detect errors before deployment
- Supports iterative improvement
Challenges
- A verifier can also make mistakes
- Two LLMs agreeing does not prove correctness
- Repeated retries increase cost and latency
- Poorly defined criteria create false confidence
When correctness matters more than speed, and when there is real evidence available to verify against.
Negotiation and debate
Agents exchange proposals and critiques. Some tasks involve competing objectives, conflicting constraints, or several plausible solutions. Agents exchange proposals, challenge assumptions, and revise their positions.
Imagine a system designing a cloud infrastructure architecture with three specialists: a cost optimization agent, a security agent, and a performance engineering agent.
The cost agent proposes a low-cost architecture. The security agent identifies compliance concerns. The performance agent argues the proposal creates latency bottlenecks. The agents exchange proposals and critiques, and the system produces a revised architecture that attempts to satisfy the relevant constraints.
A well-designed negotiation system needs defined objectives, explicit constraints, a termination condition, a conflict-resolution mechanism, an evaluation or scoring procedure, and a final decision authority.
Debate does not automatically improve correctness. A group of agents can confidently agree on an incorrect answer. The system still needs reliable evaluation.
Advantages
- Makes trade-offs explicit
- Encourages consideration of alternative solutions
- Useful when objectives conflict
- Supports collaborative planning
Challenges
- Agents may endlessly debate
- More discussion does not mean better decisions
- Models can reinforce one another's mistakes
- Something must resolve the disagreement
- Token usage and latency grow rapidly
When the task has genuine trade-offs between competing objectives, and there is a scoring mechanism to settle them.
Broadcast and publish–subscribe
Distribute information to many agents. An agent or event bus publishes an event that multiple interested agents can consume. The publisher makes information available rather than addressing a single recipient.
Consider a developer-focused service monitoring platform. A monitoring agent detects that an external API is down and publishes an event:
Multiple agents react. An incident analysis agent investigates. A notification agent alerts users. A logging agent records the incident. A recovery agent checks whether service has been restored. An analytics agent updates reliability metrics. The monitoring agent does not need to coordinate each downstream response individually.
A production implementation may require durable message delivery, idempotent event handlers, correlation IDs, event versioning, retry and dead-letter mechanisms, and access controls for event subscriptions.
Advantages
- Loose coupling between components
- Multiple agents react to the same event
- Suits event-driven architectures
- Enables asynchronous processing
- New subscribers can be added without changing the publisher
Challenges
- Event ordering may matter
- Duplicate messages must be handled
- Delivery failures need recovery mechanisms
- Shared context may be incomplete
- Event schemas must stay compatible
- Tracing a distributed chain is difficult
When multiple independent components need to react to the same event, particularly in monitoring, automation, and event-driven systems.
Important distinction: these patterns are not mutually exclusive
This is where architectural thinking becomes important. Delegation, peer to peer communication, creator and verifier workflows, negotiation, and broadcast describe different aspects of coordination. They are not necessarily competing architectural choices.
A single multi-agent system can combine several of them:
- A supervisor delegates tasks to specialist agents.
- Two research agents communicate directly to resolve a missing dependency.
- Multiple research agents work in parallel.
- A creator agent generates a solution.
- A verifier agent validates the solution.
- A monitoring component broadcasts an event when a task fails.
This is a hybrid multi-agent architecture. The right question is not "which communication pattern is the best?"
The second missing dimension: orchestration topology
Communication patterns explain how agents interact. Orchestration topology explains how the overall workflow is structured. These two concepts are related, but they are not the same.
Switch between the four below. The nodes stay the same, only the shape changes.
Sequential: A pipeline of agents
Agents execute in a predefined order. The output of one agent becomes the input to the next. A research workflow might run collect, then analyse, then write, then verify. Each stage depends on the one before it.
Simple to understand. Easy to trace. Predictable execution order. Straightforward state passing.
Latency accumulates across stages. A failed stage blocks the pipeline. Independent tasks cannot use parallelism. A rigid sequence handles change badly.
A sequential workflow is not inherently inferior to a more complex architecture. If the task is naturally sequential, a pipeline may be the most appropriate design.
Parallel: Many agents work simultaneously
Independent tasks execute concurrently and an aggregator combines the results. Analysing a software project, one agent examines the codebase, one checks dependencies, one analyses security risks, one reviews documentation.
Reduces wall-clock latency for independent tasks. Better use of available resources. Specialized analysis happens simultaneously. Scales well for certain workloads.
Requires synchronization. Shared resources create race conditions. Results arrive at different times. The aggregator must reconcile conflicts. Running many agents raises infrastructure cost.
The critical question is whether the tasks are genuinely independent. If one agent depends on another's output, parallelizing them creates invalid results or extra coordination.
Hierarchical: Supervisors and sub-agents
A supervisor delegates work to sub-orchestrators, which coordinate their own agents. Useful when the problem is too complex for a single orchestrator to manage directly, with each sub-orchestrator owning a domain.
Clear separation of responsibilities. Supports modular workflows. Allows domain-specific coordination. Reduces complexity in a single orchestration layer.
Decisions become hard to trace across levels. Context gets duplicated between supervisors. A supervisor can become a bottleneck. Failures propagate through layers. Nested orchestration increases cost and latency.
The presence of multiple supervisory layers does not automatically make a system more scalable or intelligent.
Graph based: Conditional workflows and feedback loops
Agents are nodes and transitions define the workflow. Unlike a fixed pipeline, the system decides based on current state. Generate, test, and on failure fix and test again. The graph may hold conditional branches, retry paths, loops, parallel branches, human approval steps and error-handling nodes.
Supports complex workflows. Makes conditional execution explicit. Enables iterative refinement. Can model retries and recovery paths. More control over state transitions.
Loops run indefinitely if poorly designed. State management becomes critical. Debugging complex transitions is hard. Retry policies must be carefully controlled.
A production-grade graph system needs explicit termination conditions, maximum iteration limits, state validation, retry budgets, timeout controls, failure escalation, and traceable state transitions. An agent can repeatedly attempt the same action without making progress.
The architecture I would want candidates to explain
Suppose a candidate submits a multi-agent system for a two-day engineering assignment. It has a planner, a researcher, a coding agent, a verifier, a tool layer, guardrails, and error handling. At first glance it might look impressive. But I would want to understand the architecture behind it.
Why did you choose this topology?
Why sequential and not parallel. Why a supervisor is necessary. Why there is an iterative feedback loop. Was the architecture selected because the task required it, or because the framework made it easy to implement?How do agents exchange information?
Structured messages, shared state, an event bus, direct agent-to-agent requests, tool-mediated communication, a centralized broker. What is the message schema. How are intermediate artifacts represented. How is context passed between agents.Who owns the decisions?
If a research agent recommends one approach and the security agent recommends another, who decides. A supervisor, a deterministic policy, a dedicated decision agent, a human, or an evidence-based evaluation mechanism. The system should have a clear decision authority.What happens when agents disagree?
The creator says the implementation is ready. The verifier says it fails security checks. Does the creator receive structured feedback. Is the task retried. Is a different agent consulted. Is there a maximum retry limit. When does the system stop and ask a human.What happens when an agent fails?
The research agent times out. The coding agent returns invalid structured output. A tool call fails. An external API goes down. An agent burns its token budget. Does the system retry, fall back, skip, roll back, escalate, or terminate safely? A system that handles only the happy path is not ready for production.These questions help distinguish someone who has assembled a demo from someone who understands agentic systems engineering.
The evaluation dimensions that matter
For a two-day multi-agent engineering challenge, the quality of the final demo should not be the only consideration. The architecture, reliability, and engineering decisions should also be examined. Open any row for what to listen for.
This framework evaluates whether the candidate understands the system as an engineered product rather than simply a collection of framework components.
Tools, skills, harnesses, and guardrails are not the entire system
Modern agentic systems often emphasize tools, skills, harnesses, and guardrails. All of these are important. But they answer different questions.
| Component | Core question it answers |
|---|---|
| Tools | What can the agent do? |
| Skills | What capabilities can the agent reuse? |
| Harness | How is the agent executed and controlled? |
| Guardrails | What is the agent allowed or forbidden to do? |
| Orchestration | How do multiple components coordinate? |
| Communication protocol | How do agents exchange information? |
| State management | What does the system remember and share? |
| Evaluation | How do we know the result is acceptable? |
| Observability | What happened during execution? |
You can have a sophisticated tool ecosystem and still build a poorly coordinated multi-agent system. You can have excellent guardrails and still have an inefficient orchestration design. You can have multiple powerful LLMs and still end up with a system that cannot recover from a simple failure.
A practical example: a reliable multi-agent coding system
Imagine we are building an agentic software development platform. The user submits:
We have five agents: planner, researcher, coder, verifier, security reviewer. Instead of simply connecting them, we define an architecture. Step through it.
Planning
The planner decomposes the request into structured tasks. It produces a task specification containing functional requirements, technical constraints, expected artifacts, acceptance criteria, and dependencies.
The planner does not directly modify the codebase.
Research
The researcher investigates relevant implementation details and produces a structured research artifact holding findings, sources, assumptions, and relevant technical constraints.
The planner or orchestrator decides whether the research is sufficient.
Implementation
The coding agent receives the approved task specification and the relevant research artifacts, then implements the feature in a controlled workspace.
Tool permissions restrict it to the operations it actually needs.
Verification
The verifier checks the implementation using appropriate evidence: automated tests, type checking, linting, API contract validation, and relevant security checks.
It produces a structured result: pass or fail, failed checks, evidence, and recommended corrections.
Feedback loop
If the implementation fails, the verifier reports to the orchestrator, which returns the failure report to the coding agent for correction.
The system enforces a retry budget. If it keeps failing, the workflow stops or escalates to a human.
Security review
The security reviewer evaluates the implementation against the relevant security requirements.
Its output is treated as an additional review artifact, not as an unquestionable source of truth.
Final decision
The orchestrator applies the acceptance criteria and determines whether the task is ready for approval.
For high-impact actions, human approval may be required before deployment.
What communication methods are being used here?
This one system combines several patterns:
- Delegation. The orchestrator assigns work to specialized agents.
- Sequential execution. Planning precedes implementation, and implementation precedes verification.
- Creator and verifier. The coding agent produces an artifact that the verifier evaluates.
- Iterative refinement. Failed checks send the workflow back to the coding agent.
- Potential parallelism. Independent research or security analysis may run concurrently when dependencies permit.
- Structured communication. Agents exchange task specifications, artifacts, and verification reports.
- Guardrails. Tools and permissions restrict what agents can do.
This is the difference between simply having five agents and designing a multi-agent system with explicit coordination logic.
An important correction: more complex orchestration does not mean better engineering
A system does not need to use every communication pattern to be well engineered.
A sequential pipeline with clear state schemas, robust error handling, deterministic routing, strict tool permissions, comprehensive testing, and strong observability may be better engineered than a complex peer to peer system with dozens of agents and poorly defined responsibilities.
Similarly, a single agent with well-designed tools and a reliable control loop may outperform a multi-agent system when the task does not require multiple specialized roles.
Architectural complexity should be justified by task complexity. A candidate who uses a single agent with a well-designed workflow should not automatically be considered weaker than someone who deploys ten agents. The real question is whether the candidate can explain their decisions and demonstrate that the system meets its requirements.
What interviewers should look for
If a candidate builds a multi-agent orchestration system, the interviewer should not focus exclusively on the final output. Start here:
Then follow up:
- Which agents communicate directly?
- Which messages are structured?
- Where is the shared state stored?
- What happens when the verifier rejects the creator's output?
- How do you prevent infinite loops?
- What is your retry policy?
- What is the system's termination condition?
- How do you measure whether your architecture is actually improving quality?
The principle I took away from this discussion
Agentic engineering is not just about giving agents tools, skills, and guardrails. It is about designing the coordination layer. That means understanding how agents communicate, how tasks are delegated, how workflows are structured, how agents synchronize, how outputs are verified, how conflicts are resolved, how failures are handled, how state is shared, how decisions are evaluated, and how the system terminates safely.
Without a deliberate orchestration architecture, a multi-agent system can become little more than a collection of LLM calls connected by ad hoc logic. And that may be enough for a prototype. But building reliable agentic systems requires a deeper level of engineering.
That's the difference between building agents and engineering agentic systems.
One final distinction
Agent communication is not the same as agent intelligence. An agent may communicate through a sophisticated protocol and still produce poor results. Likewise, a highly capable model may perform well in a simple sequential workflow.
Architecture must be evaluated alongside task success, reliability, cost, latency, observability, quality of decisions, and safety and operational constraints.
The architecture should serve the task, not the other way around.