Before you build an AI agent, learn how AI systems are actually built.

Something in AI engineering has become increasingly interesting to me.

People are jumping directly into AI agents without first understanding the engineering foundations that made agents possible. The path usually looks like this. A framework is released. A model gets connected to a tool, a vector database is added, a few prompts are written, and an orchestration layer ties it together.

The system works, at least on the inputs it has seen so far, and the builder starts calling themselves an AI engineer.

To be clear, I am in favor of agents. They are becoming an important abstraction for building software around increasingly capable models. An agent is still a system, though. It has inputs, outputs, dependencies, state, failure modes, latency constraints, security boundaries, evaluation requirements, infrastructure requirements, and operational costs.

Those concerns did not appear because agents became popular. They existed long before agents, and most of them are general engineering problems that AI systems inherit.

Before today's agent frameworks, there was machine learning. Before modern machine learning systems, there was statistical learning. Before that came decades of research in artificial intelligence, optimization, probability, search, computer science, distributed systems, databases, software engineering, and mathematics.

The modern AI stack is the result of all of those layers accumulating over time. That history matters because every abstraction hides complexity.

Modern AI development is often presented as if the central problem were "How do I make the model more intelligent?" In production, that is only one part of the problem. A more important question is:

That question covers fifteen failure modes. I group them by where they originate, because the origin usually determines the fix. Open any row for the failures that start there.

This is one of the most important things I have learned while working through the rapid evolution of AI systems: a surprisingly large portion of AI engineering is still conventional software engineering.

• Design and code. Architecture, abstraction, APIs, documentation, and code quality.
• Data and state. Databases, caching, and concurrency.
• Systems. Networking, distributed systems, infrastructure, and performance engineering.
• Delivery. Version control, testing, CI/CD, and containerization.
• Operations. Observability, logging, error handling, security, and incident response.

The AI component changes how one part of the system behaves. Everything around it still has to be engineered.

The probabilistic nature of modern AI often makes these disciplines more important. Traditional software generally tries to produce deterministic behavior from deterministic inputs. AI systems frequently operate under uncertainty, so the surrounding engineering has to compensate for it.

There is another layer that is increasingly easy to skip. If you want to work seriously on AI systems, understanding how models learn is still valuable. You should be able to reason with concepts such as these:

• Learning and inference. Training and inference, supervised and unsupervised learning, optimization, loss functions, fine-tuning, and inference trade-offs.
• Generalization and measurement. Overfitting, generalization, bias and variance, evaluation, calibration, and model uncertainty.
• Data and representation. Data distribution, distribution shift, feature representation, and embeddings.

The goal is enough understanding to reason about what the model is doing, which is well short of deriving every optimization algorithm from first principles.

Without it, you are operating the system from the outside. You can call the API, configure the framework, and connect the tools, but when the system behaves unexpectedly, you may not know why. That is where the difference between framework usage and engineering becomes obvious.

A production agent can be described as a stack. The exact architecture varies by system, but the principle holds.

This is why learning only an agent framework can create a dangerous illusion of competence.

I have nothing against AI-assisted coding. It is one of the most powerful productivity shifts software engineering has seen. The problem begins when generated code becomes a substitute for understanding. Compare two statements:

The first shows that you can produce software. The second shows that you can engineer a system. AI coding tools can dramatically reduce the cost of implementation, but they do not automatically transfer architectural judgment to the developer. That gap matters more as systems become more autonomous.

A practical test for any agent you have shipped is whether you can answer these nine questions.

If you cannot, the generated code may be running, but you are not necessarily in control of the system.

One idea I particularly appreciate in Thomas R. Caldwell's The AI Engineering Bible is its framing of AI development as a complete engineering lifecycle, in which building the model is one part of one stage.

The ordering matters, because the work does not end when the model responds correctly. Step through it.

That lifecycle is a much better mental model for AI engineering than Prompt → Agent → Ship. The book's running Customer Support AI example makes this concrete by carrying one production system through the entire lifecycle, showing how decisions made early in the architecture affect deployment, scaling, and maintenance later.

Individual components are often not the hardest part. You can learn an embedding model, RAG, tool calling, an agent framework, a vector database, and an LLM API one at a time. Production complexity emerges from the interactions between them.

Take a simple workflow: user, agent, retrieval, LLM, tool, database, response. Every arrow is an interface, and each one raises its own engineering questions. Switch between them below. The nodes stay the same, only the failing interface changes.

There is an interesting paradox here: the more autonomous AI systems become, the more traditional engineering matters.

The mapping from capability to control is fairly direct.

The pace of research is another reason fundamentals matter. The field is moving too quickly to treat today's frameworks as permanent knowledge.

If your understanding is tied exclusively to one framework, it becomes obsolete when that framework changes. If it is grounded in ML fundamentals, systems engineering, research literacy, and software engineering, you can adapt.

A quick note on where I am coming from: I'm Haroon, and I entered the AI field at the end of 2023, during the GPT era. I started from the fundamentals of machine learning and its core principles rather than from today's agent abstractions.

Since then, I have watched the industry move through a fast sequence of changes:

1. LLMs became mainstream.
2. RAG became mainstream.
3. AI coding accelerated.
4. Tool use became practical.
5. Agentic workflows emerged.
6. Multi-agent systems became a major research and engineering direction.
7. Inference infrastructure matured.
8. We are now moving toward increasingly autonomous AI systems.

The deeper I go into AI engineering and research, the more convinced I am that there is no point at which you finish the fundamentals and move on to being an AI engineer. They are the foundation you keep working from for your entire career.

None of this means every AI engineer needs to be a mathematician, a distributed-systems researcher, a deep-learning scientist, a compiler engineer, a security researcher, and an infrastructure expert at the same time. That is unrealistic, and specialization is necessary.

Specialization and foundational ignorance are different things, though. A strong AI engineer can move across layers when necessary, which takes:

• Enough ML to reason about model behavior.
• Enough software engineering to build maintainable systems.
• Enough distributed systems to understand scale.
• Enough infrastructure to deploy reliably.
• Enough security to understand the attack surface.
• Enough evaluation to measure whether the system actually works.
• Enough research literacy to understand what is changing.
• Enough systems thinking to connect all of those pieces.

Knowing how to call the newest model will be a common skill. The next generation of AI engineers will be the people who can answer questions like these.

I love the current agent ecosystem. I experiment and build with these systems, and I study new architectures as they appear. I believe agents will become an important interface for software.

I would still encourage anyone entering AI today to resist one temptation.

Learn the foundations. Understand ML, and learn software engineering deeply. Study systems, including distributed systems. Understand data and inference, and learn how models are evaluated and deployed. Learn security. Read research. Build things, then break them, measure them, debug them, deploy them, and operate them.

Then build agents.

Once you understand the layers underneath the abstraction, an agent framework stops looking like magic. It becomes what it actually is: another engineering tool for building intelligent systems. That shift in perspective is, in my opinion, one of the most important transitions an aspiring AI engineer can make.

The bar worth aiming for

Impressive AI demos are cheap to build now. The goal is to become someone who can build an AI system that survives reality.