Agent Engineering Is Not a Layer

Andrew Ng's AI-engineering skills map names four top-level skills: building and deploying AI applications, software engineering fundamentals, using coding agents, and shaping the build.

The nine posts before this one are all skill number two.

They cover how this system persists a vector, scopes a query, streams an event, prices a token, and boots on a spot instance. A reader could finish all nine and still have learned nothing about the part that makes it an AI product rather than a CRUD app with a model attached.

That part is skill number one. It has no box in the architecture diagram, because it is not a layer. It is a set of concerns that cut across every layer. A thing with no box is a thing nobody is assigned to, which is how you end up with a system that looks like it does agent engineering and does not.

That bottom rail is one grep -oiE over the nine posts for eight terms: forced tool, tool_choice, context, eval, blackboard, revision, token, orchestration. The agent-layer post scores 48. Testing scores 32, observability 19, the campaign and data posts one each.

The instrument is crude, and its two worst readings are the useful ones. The frontend post scores 15, and every single hit is the word "token" in the CSS design-token sense, which has nothing to do with any of this. The memory post scores 4 while using the word "embedding" 44 times, because its grounding vocabulary is embedding and retrieval and nothing on the list.

Keyword presence is a bad proxy for whether a concern is being handled. Hold that thought until the end of the post.

Each dot in the grid above is a file. The campaign layer carries control flow as the 409 that refuses a mix built from an unlocked strategy. The API layer carries output contracts as the forced-tool schemas its campaign endpoints define. The data layer carries grounding as a vector column. The deploy layer carries it as 310 MB of embedding model resident on a 2 GB box. The frontend carries failure semantics as a rule that a failed load can never render as an empty list.

Context engineering is mostly a decision about what an agent cannot see

The first instinct after the models got good was to collapse the crew into one long prompt. Four agents fit in one model's head now.

I tried it and the output got worse in a specific, reproducible way. Strategy reasoning bled into the copy. A model that had just worked through a campaign's messaging pillars would write drafts that sounded like they were still arguing for the strategy rather than executing it. In the other direction, once it was deep in copywriting, its strategic thinking got noticeably more decorative.

Separate contexts fixed it. The strategist sees the brief and the organization's memory. The writer sees the locked strategy and the piece it is writing, not the deliberation that produced the strategy. The editor sees the draft and the standard, not the writer's reasoning about the draft.

So the context is assembled per call rather than accumulated. build_system_prompt in backend/app_new/crew/base_agent.py returns a list of blocks:

        return [
            {"type": "text", "text": base},
            {
                "type": "text",
                "text": org_context,
                "cache_control": {"type": "ephemeral"},
            },
        ]

Ordering in that list is load-bearing. The cache breakpoint sits on the stable org context, and _relevant_examples_block appends per-query retrieval last, after every breakpoint, so a fresh retrieval never invalidates the cached prefix above it. The writer inserts a ghostwriting block in between, with its own breakpoint.

The part that is easiest to get wrong is where that per-call state lives. It travels as a kwarg:

        system = await self.build_system_prompt(query=query, **(system_extras or {}))

Nothing gets assigned to self. The orchestrator holds one writer instance for a whole run so its prompt cache survives across delegations, which means the instance is shared, which means anything request-scoped stored on it belongs to whichever request wrote last.

The failure mode is invisible until you have two concurrent runs: one overwrites the other's executive, and someone's CEO post ships in someone else's voice. There is no error, no log line, and no test that fails, because with one user at a time the code is correct.

An output contract is a tool_choice, not a request in a prompt

Everywhere code consumes model output, the model is pinned to exactly one tool. respond_with_forced_tool passes tools=[tool] and tool_choice={"type": "tool", "name": tool["name"]}, and nothing anywhere parses prose.

The interesting half is what happens when the contract is not met:

        self._record_run(
            api_kind="forced_tool",
            status="error",
            latency_ms=latency_ms,
            usage=usage,
            stop_reason=response.stop_reason,
            error_class="schema_mismatch",
        )
        raise ValueError(f"{self.AGENT_NAME} did not return the {tool['name']} tool call")

It raises, so nothing empty is persisted, and it writes a telemetry row classified schema_mismatch rather than provider_error. Contract drift after a model upgrade becomes a number on a chart instead of scattered reports of bad output, and backend/tests/test_agent_contracts.py asserts that exact row rather than just the raise.

Field-level parsing gets the same treatment. backend/app_new/crew/orchestrator_agent.py reads schema-required fields through a helper, because a bare payload[key] raises a KeyError whose message is just the key, which reaches the run transcript as assign_writer failed: KeyError:

def _require(payload: Dict[str, Any], key: str, tool: str) -> Any:
    if key not in payload:
        raise ValueError(f"{tool}: model omitted required field {key!r}")
    return payload[key]

One field in the editor's schema carries a comment that is really a bug report:

                        # Index, not title: title matching fails silently on any
                        # paraphrase and picks the wrong draft. See state.Review.
                        "draft_index": {"type": "integer", "minimum": 0},

Draft selection used to match the editor's model-generated title against the writer's model-generated title. Any paraphrase broke equality and the fallback took the first draft. Reproduced against the real code, a run scoring 55 and 95 returned the 55.

The failure mode is a model-generated string used as a join key. It fails silently, in the flattering direction, and it looks like the model just wrote something mediocre.

The model picks the move; the dispatcher decides which moves exist

The orchestrator is an agent whose five tools are the specialists. There is no if/elif choosing the order. It emits one tool call per turn, gets a compact JSON result back, and decides again.

That orchestrator answers at /orchestrate, where tests and evals reach it and the product's own screens do not. The chat routes by pattern matching and Compose runs a fixed relay, so the if/elif this section is about is absent from one controller rather than from the whole system. The enforcement point below holds wherever a model is allowed to pick the next move, which is the reason it was worth building before anything shipped on top of it.

max_revisions used to be a sentence in that orchestrator's system prompt. Then it was measured against a live model that kept asking for one more pass: ten revisions against a cap of two.

A prompt is a request. Requests get declined.

So _dispatch became the enforcement point, and every delegation now passes through a precondition check first:

        refusal = self._refusal(name, state)
        if refusal is not None:
            self.logger.info("delegation_refused", tool=name, reason=refusal)
            state.transcript.append(
                Turn("orchestrator", f"refused:{name}", refusal)
            )
            return {"refused": refusal, "phase": state.phase.value}

        return await handler(state)

The refusal is a return value shaped like every other tool result, and its text names the move to make instead. The model reads it on the next turn and corrects itself.

This is the distinction that separates agentic from a pipeline with a model bolted on. Nothing hardcodes the sequence. The state machine only removes options that cannot be coherent, and it explains each removal in the same channel the model is already reading.

The failure mode is a system where every invariant lives in a prompt, which means every invariant is advisory, which means the system is exactly as reliable as the model is agreeable on a given day.

Grounding is measured or it is asserted

Replacing the old keyword search with real embeddings moved three numbers, measured over a labelled corpus of 34 documents and 17 queries:

recall@5          0.28 -> 0.87
MRR               0.20 -> 0.69
score separation  0.00 -> 0.15

That 0.00 is not rounding. The old implementation returned a hardcoded score: 0.5 for every hit, so its separation was zero by construction and any caller thresholding on that score was thresholding on a constant. Separation is in the suite specifically to fail a retriever with perfect recall whose scores carry no information.

The other half of grounding is the part a prompt cannot do. Walking the product on an organization with zero ExecutiveProfile rows on file, the strategist assigned quotes to "Sarah Chen, CEO" and the writer printed By [CTO Name], Co-Founder and CTO into a draft that was one approval away from publishing. Both people were invented whole.

The prompt asks for the right thing. _exec_block in backend/app_new/api/v1/campaigns/mix.py puts the real roster in the context and, when the roster is empty, says so: omit exec on every piece, do not invent an executive, a name, or a role label.

That instruction is necessary and it is not sufficient, so the name is checked server-side in backend/app_new/api/v1/campaigns/_shared.py before anything is persisted:

def _resolve_target_exec(raw, execs) -> Optional[str]:
    if not raw or not execs:
        return None
    candidate = str(raw).strip()
    head = candidate.split(",")[0].strip().lower()
    if not head:
        return None
    for e in execs:
        name = (e.name or "").strip().lower()
        if name and head == name:
            return f"{e.name}, {e.title}" if e.title else e.name
    return None

Anything that does not resolve to a stored profile is dropped. Bracketed placeholders are detected in backend/app_new/services/placeholders.py and surfaced as a warning rather than stripped, because a warning tells the human something is missing and a silent strip just leaves a smoother lie.

An eval is the regression suite for behavior, and it needs a control of its own

Unit tests answer whether a function returned the right value. They cannot answer whether a crew of four routed correctly, revised only when the editor asked, and stopped.

Ten cases in backend/evals/ gate CI against a deterministic mock provider, keyless and reproducible. Seven drive scenarios through a well-behaved mock, and those seven have a ceiling: a scenario can only make the specialists misbehave, so what it proves about the orchestrator is capped by how badly the mock is willing to act. The case named editor_never_satisfied_bounded passed happily during the entire period the production code allowed ten revisions against a cap of two, because the mock politely stopped asking.

The three that earn their keep replace the orchestrator's decision function outright with a policy that spams request_revision forever, one that skips straight to writing with no strategy, and one that dies at the decision step. Those test our enforcement rather than the mock's manners.

And then there is the guard almost nobody builds, in backend/evals/run_retrieval_evals.py. Running with EMBEDDING_PROVIDER=deterministic embeds every document with hash vectors that carry no semantics, so retrieval should be near random. Under that provider the gate inverts and the run fails if the score is high:

        recall = agg["recall"] or 0.0
        ok = recall < ceiling

The ceiling is not a constant, because random top-k recall on a single-relevant query is k over N, and a constant crosses its own defect value as the corpus grows:

def control_ceiling(corpus_size: int) -> float:
    """The recall a hash-embedding control must stay under, scaled to chance."""
    return CONTROL_CHANCE_MULTIPLE * (TOP_K / corpus_size)

At 34 documents that is 0.368. The control scored 0.143 against a chance value of 0.147.

The keyword baseline is kept as a positive control with a floor of its own, checked before anything else is judged. If the tokenizer ever returns an empty list, the baseline returns nothing, the headline goes from 0.28 to 0.87 up to 0.00 to 0.87, and semantic retrieval appears to have improved.

The failure mode is a harness that rewards any output at all, because it produces a green table.

A failure has to be able to look like a failure

respond_stream yields an error event rather than raising, which is correct for a streaming API. The orchestrator's decision loop only looked for done.

So a provider outage read as "the model chose to stop", which routed to finalize, which set the phase to DONE with no error and zero content. A successful run, by every field the system recorded.

Two fixes, in backend/app_new/crew/orchestrator_agent.py. The decision step now raises:

        async for event in self.respond_stream(conversation, use_thinking=False):
            if event["type"] == "error":
                raise RuntimeError(event.get("error", "provider error"))
            if event["type"] == "done":
                final_content = event["content"]

And no path can reach a terminal success without a draft:

    async def _do_finalize(self, state: CrewRunState) -> Dict[str, Any]:
        best = state.best_draft()
        if best is None:
            # No draft means the run produced nothing. Finishing "successfully"
            # here is how a provider outage used to be reported as DONE.
            state.record_failure("finalize called with no draft to return")
            return {"phase": "failed", "reason": "no_draft"}
        state.record_finalized(f"Finalized: {best.title}")
        return {"phase": "done"}

The terminal SSE event is then derived from the terminal state rather than chosen separately, so run_complete cannot be emitted for a failed run.

The same discipline runs through cost accounting. backend/app_new/crew/pricing.py is a hand-maintained table of four model prefixes, and its most important lines are the ones that do nothing:

    match = None
    for prefix in PRICING_PER_MTOK:
        if model.startswith(prefix) and (match is None or len(prefix) > len(match)):
            match = prefix
    if match is None:
        return None

An unknown model records NULL in a nullable column and surfaces as unpriced_runs, never as zero. Zero is a number someone will sum.

The word was in the README before it was in the code

Before any of the above was true, an adversarial review of this repository landed a finding that reframed every other finding in it. The word "multi-agent" described a keyword router with f-string handoffs.

Not a lie exactly. There were four agents and there was code that picked which one to call. There was nothing in it the word was supposed to point at: no controller that chose, no typed object crossing the seams, no move that could be refused.

Commit bf0e402 is titled "genuine multi-agent orchestration" and its first line says it is answering that finding by earning the word. Which is the right framing, and it is not a confession.

Nothing complained. The product worked. Demos ran. Tests passed. The README was accurate about the architecture in the sense that every noun in it existed somewhere. An agent asked to build a multi-agent system will build something that answers to that description, and a description is the only specification it was given.

An agent implements the claim as convincingly as it implements the thing, so most of agent engineering is the discipline of checking which one you have.

Every concern in this post is a version of that check. Context engineering asks whether a boundary you drew on a diagram exists in the assembled prompt. Output contracts ask whether a schema is enforced or merely described. Control flow asks whether an invariant is code or a sentence. Grounding asks for a number where a claim was. Evaluation asks whether the thing measuring can fail. Failure semantics asks whether a bad run is distinguishable from a good one by any field the system actually writes.

None of those questions have a layer. All of them have a file.

What is still a claim here

Here is the counted deflation. _resolve_target_exec is the code that stops a fabricated executive from reaching a draft, and it is referenced in three application files and zero test files. Grep the suite for it and you get nothing. The prompt-side guard has tests in backend/tests/test_executive_voice.py. The server-side drop, which is the part that holds when the prompt does not, is protected by nothing but the fact that I remember writing it.

Two more limits worth naming. Of the three eval entry points in backend/evals/, one runs in CI: run_evals.py, ten cases, control flow only. The retrieval evals have an exit code and a docstring saying "so CI can gate on it" and nothing invokes them from .github/workflows/ci.yml. And zero of the ten CI-gated cases call a live model, so the suite proves the enforcement holds against controllers I wrote to misbehave in ways I anticipated.

That last sentence is the honest shape of all of this. Every guard in this post was built after something got past. The revision cap moved into the state machine after a model blew through it ten times. The name check went server-side after a draft was signed by a person who does not exist. The decision loop learned to raise after an outage was recorded as a success. None of them were foresight.

Which is the argument for treating this as a discipline rather than a design phase. You do not get these right at the start. You get them by looking at what your system actually did, in a form specific enough to be wrong.

The one-liner: agent engineering has no box in the diagram because it is the habit of checking whether the system is the one you described, and that check belongs in every box you already drew.

Next in the series: the security layer, where authorization fails closed, tenant scoping is checked at the query rather than in the handler, and egress is validated against resolved IPs rather than the hostname you were handed.

The views expressed here are my own and are not related to or reflective of my work or any organization I am affiliated with.