The Eval Has to Be Able to Embarrass You

A crew of four agents can route perfectly and write from irrelevant context.

That sentence is why backend/evals/ holds two suites. They answer different questions and fail independently.

The orchestrator evals ask whether the machine around the model keeps its own rules. Did the run route strategist to writer to editor, did it revise only when the editor asked, did it stop, did it refuse the moves that are illegal in order. Ten cases, gated in CI, all passing.

The retrieval evals ask whether search got better, as a number. Recall@5 from 0.281 to 0.871 against the keyword search it replaced, MRR from 0.198 to 0.693, over a labelled corpus of 34 documents and 17 queries.

Neither number is worth anything on its own. What makes them evidence is the set of things built specifically to make them come out wrong.

A mock you choose, never a mock you fall into

Both suites run keyless. CI has no API key, the numbers have to be reproducible run to run, and a suite that costs money per invocation gets run rarely and then never.

The orchestrator suite drives a deterministic mock client, and run_suite in backend/evals/harness.py turns it on by setting ANTHROPIC_MOCK to 1 and then clearing the settings cache, because the flag is read at call time through an lru_cached get_settings() that would otherwise hand back a config built before the flag was set.

Activation is explicit by design in backend/app_new/crew/mock_provider.py: the mock never engages as a fallback when a key is missing. The failure mode there is production serving canned content after a secret fails to load, with every health check green and every request answered.

Each case then gets its own throwaway in-memory SQLite database, telemetry included. Two of the metrics collected are counts of instrumented model calls and schema mismatches, and a shared database would report the suite's running total rather than the case's own.

A crash inside a case is recorded as a failed case, never a suite abort. A harness that dies on case three tells you nothing about cases four through ten.

The cases that earn their keep make the orchestrator misbehave

Seven of the ten cases leave the orchestrator's own decision policy alone and drive the mock through scenarios: the editor passes on the first read, the editor never passes, the writer errors, the strategist returns no angles.

Those seven have a ceiling. A scenario can only make the specialists misbehave, so what any of them proves about the orchestrator is capped by how badly the mock is willing to act.

editor_never_satisfied_bounded passed while the production code allowed ten revisions against a cap of two. The scenario stops asking after a while, so the cap was never reached. A live model has no such manners.

So EvalCase carries a decision_policy hook that replaces the orchestrator's decision function outright. The specialists stay honest and the orchestrator goes rogue, which is the direction a real model fails in. From backend/evals/cases.py:

def _rogue_skips_steps(agent):
    """Simulates a model that jumps straight to writing with no strategy."""
    calls = {"n": 0}

    async def policy(conversation):
        calls["n"] += 1
        if calls["n"] == 1:
            return ("id", "assign_writer")          # illegal: no angles yet
        if calls["n"] == 2:
            return ("id", "request_editor_review")  # illegal: no drafts yet
        if calls["n"] == 3:
            return ("id", "consult_strategist")     # recovers
        if calls["n"] == 4:
            return ("id", "assign_writer")
        return ("id", "finalize")

    return policy

Its verification does not settle for the run finishing. It asserts that refused:assign_writer and refused:request_editor_review both appear in the transcript, and that the run still reached DONE. Refusing an illegal move and surviving it are separate guarantees, and a machine that hard-fails on the first bad tool call has only one.

The other two adversarial policies demand revisions forever, which the cap has to refuse, and raise at the decision step, which is what a provider outage looks like from the orchestrator's seat.

The subtler lesson is in a case with no rogue policy at all. empty_strategy_still_finalizes gives the writer nothing to work from, and its assertion used to be that the run reached DONE. _verify_empty_strategy now requires the opposite: phase FAILED, no draft, and a run_failed event on the stream.

The old assertion blessed the exact bug it should have caught. A run with no angles produced no content, reported success anyway, and the eval agreed. Write the assertion against the outcome a user would call correct, not the state the code happens to reach.

Retrieval needed a number, not an adjective

"The crew sometimes retrieves irrelevant memories" is close to unfalsifiable. That is how the previous implementation shipped an ILIKE over the query's first five keywords, ordered by created_at, returning a hardcoded confidence, and stayed that way.

backend/evals/retrieval_cases.py is the ground truth that makes the claim checkable: 34 documents shaped like production rows, and 17 queries that each record which document ids genuinely answer them. Paraphrase queries share no significant token with their own answer, so "how did we speed up shipping?" has to find a document about release cadence going from fortnightly to daily. Distractors do share the tokens and answer nothing.

Top-k is 5 because get_relevant_examples passes a real query to search_similar(limit=5). Measuring at any other k reports a number nothing in the product consumes.

Both retrievers run over the same seeded corpus in the same process, so the comparison is controlled rather than two numbers from two runs. The old one is reimplemented in backend/evals/run_retrieval_evals.py rather than left in git history, every defect preserved, this one included:

    return [
        {"text": row.text_content, "source_type": row.content_type, "score": 0.5}
        for row in rows
    ]

That is why score separation goes from 0.00 to 0.15. The zero is not rounding. Every row scored the same constant, so the mean score of relevant documents minus the mean of irrelevant ones is zero by construction, and any caller thresholding on that confidence was thresholding on a constant. Recall@5 moves 0.281 to 0.871, precision@5 0.06 to 0.21, MRR 0.198 to 0.693. Head to head, semantic wins 12, loses 1, ties 3.

Four metrics, not one, because each is blind to what the next one catches. Recall asks whether the answer came back and not where. Precision stops "return the whole corpus" from being a winning strategy. MRR asks how high the first correct answer landed, which matches how the block is consumed, in rank order in a system prompt where the top item does most of the work. Separation asks whether the scores mean anything, and it reads the full ranking, because computing it among the documents that already won flatters a retriever most.

Grade the metrics with arithmetic done on paper

Every gate in the retrieval suite reads a number that backend/evals/retrieval_metrics.py produced, so none of them can disagree with it. The gate grades the metrics using the metrics.

_selftest_metrics is the independent measurement, its expectations arithmetic done on paper rather than output captured from a run:

        # precision: same retrieved list, two denominators. 1/5 vs 1/2.
        ("precision@5, two of five", precision_at_k(["a", "b", "c", "d", "e"], {"a", "c"}, 5), 0.4),
        ("precision@5, short return", precision_at_k(["a", "b"], {"a", "c"}, 5), 0.2),
        ("precision of returned, short return", precision_of_returned(["a", "b"], {"a", "c"}, 5), 0.5),

Three of the fifteen checks are load-bearing beyond arithmetic. The precision pair above must differ on the same input, because if they ever agree one is computing the other. A constant-score ranking must separate to exactly 0.0, the shape of the bug this suite replaced. An empty relevant set must return None rather than 0.0, or the no-answer case drags the headline down.

The self-test was validated by breaking the thing it grades. Stubbing recall_at_k to return 1.0 whenever any relevant document was found made the headline read 0.94 instead of 0.87, and the gate passed, because nothing downstream was computed a different way.

A threshold is only valid at the inputs it was measured at

Every gate here is calibrated from a measurement. None were picked in advance.

The MRR floor is the clearest case. The number chosen before measuring was 0.70, and it fails the current healthy implementation by 0.007. A gate set by ambition turns CI red for no reason, which trains everyone to ignore CI. The floor sits at 0.60, roughly three times the keyword baseline.

The part that took a second pass is that a calibrated number means nothing without the inputs it was calibrated over, and recall rises mechanically with k. Swept against the real corpus, from run_retrieval_evals.py (the note between table and constant trimmed):

#       k    sem recall    sem MRR    gate
#       1         0.478      0.562    FAIL   <- false alarm; retrieval is fine
#       3         0.799      0.677    pass
#       5         0.871      0.693    pass   <- calibrated here
#      10         0.888      0.693    pass
#      20         1.000      0.696    pass   <- asserts nothing
#      34         1.000      0.696    pass   <- asserts nothing
CALIBRATED_AT = {"top_k": 5, "documents": 34, "queries": 17}

The failure mode is a gate nobody can fail. At k of 20 every query returns every answer, the 0.75 recall floor is unreachable from below, and the suite passes no matter how badly ranking degrades. Raising top-k is a plausible edit, because it tracks the product.

So the gate refuses to render a verdict outside its calibration:

    if TOP_K != CALIBRATED_AT["top_k"]:
        return False, [
            f"NO VERDICT  TOP_K is {TOP_K}, but every threshold here was calibrated at "
            f"k={CALIBRATED_AT['top_k']}. Recall rises mechanically with k (measured "
            f"0.87 at k=5, 1.00 at k=20), so the {SEMANTIC_GATE.recall_at_k:.2f} floor "
            "stops asserting anything as k grows and false-alarms as it shrinks. "
            "Re-measure and update MEASURED, SEMANTIC_GATE, KEYWORD_BASELINE_FLOOR and "
            "CALIBRATED_AT together. MRR is the k-stable metric if you need one number "
            "across a budget change."
        ]

A threshold is a measurement with an expiry date. It is valid only over the inputs it was sampled at, and nothing records those inputs unless you write them down.

Two controls, and one of them is a floor

The first control is the familiar one. EMBEDDING_PROVIDER=deterministic embeds every document with hash vectors that carry no semantics, so retrieval should be near random, and under that provider the gate inverts: the run fails if the score is high. A harness that rewards any output at all is worse than no harness, because it produces a green table.

The ceiling that control has to stay under used to be the constant 0.40. It is now a multiple of chance, because random top-k recall on a single-relevant query is k over N, and a constant crosses its own defect value as the corpus moves. At 34 documents, 0.40 is 2.7 times chance. At 200 it would be 16 times chance and would bless a competent retriever as random. At 12 it sits below chance and fails a control that is working perfectly.

The multiple is 2.5, which evaluates to 0.368 at 34 documents, so today's verdict is unchanged. What changes is that it stays correct when the corpus does. And when the corpus gets small enough that the ceiling rises above the semantic gate's floor, the suite says so instead of reporting the reassuring answer:

        if ceiling > SEMANTIC_GATE.recall_at_k:
            return False, [
                f"NO VERDICT  the corpus is too small for this control: {len(CORPUS)} "
                f"documents puts chance recall@{TOP_K} at {chance:.3f}, so the ceiling "
                f"({CONTROL_CHANCE_MULTIPLE}x chance = {ceiling:.3f}) sits above the "
                f"semantic gate's floor ({SEMANTIC_GATE.recall_at_k:.2f}). Random and "
                "competent are indistinguishable here. Add documents rather than "
                "lowering the multiple."
            ]

The measured control scores 0.143 against a chance value of 5 over 34, or 0.147.

The second control is a floor rather than a ceiling, and it is the one most suites are missing. Every other check in the file asserts that something is absent: recall is low, no rows are unreadable, the no-answer query is not confident. An assertion that something is absent passes perfectly when the machinery producing it is dead.

So the keyword baseline is a positive control with a floor of 0.15, checked before anything else is judged. The failure mode is precise. If the tokenizer ever returns an empty list for every query, the baseline returns nothing, and the headline goes from "0.28 to 0.87" to "0.00 to 0.87". Semantic retrieval appears to have improved. Verified by stubbing the baseline to return nothing: the gate passed, and the table reported a larger win than the honest one.

Nobody investigates a number that went up.

The same argument runs one level down. _assert_corpus_visible refuses to measure at all unless a SQL count, the length of the ranking search_similar returns, and a probe document retrieving itself at rank one all agree, because a silently smaller corpus has fewer distractors and scores higher.

What the evals found that reading did not

One product bug, immediately. get_relevant_examples had a default query, the literal string "content examples brand voice writing style", used on every Writer call that carried no query of its own. It is a bag of keywords resembling no natural question, and embedding models rank that badly. Measured recall@5 of 0.43: two of the five slots injected into every Writer call held documents that were not brand examples.

With no query there is nothing to be similar to, so that path is now a filter on content_type and makes no embedding call at all. The eval case is kept unchanged as a regression guard against anyone reintroducing a keyword bag as a stand-in for a real query.

Two harness bugs, in a third suite. Commit 6445289 rebuilt the Claude-as-judge eval that scores writing quality, and found long-form outputs truncated by a token limit and variant diversity measured across variants generated from the same prompt. Both defects belonged to the measuring apparatus, and publishing those scores would have been reporting the method as the product. A generator cannot grade itself, and neither can a harness.

The losses are recorded rather than smoothed. q-funding fails on both retrievers, because "how much did we raise and from whom" does not retrieve "closed a $40M round led by Kestrel Partners", and it stays in as a known-failing case, because a suite where everything passes has stopped measuring. The no-answer query scores 0.54 against a mean irrelevant score of about 0.47, so its gate sits at 0.70 to catch a genuine collapse. Raw cosine is not a usable abstain threshold, and the suite does not claim it is.

Where the number is thinner than the headline

Here is the counted version.

Seventeen queries. The headline recall and MRR are means over sixteen of them, because the no-answer query has undefined recall and is graded on its top score instead. Of those sixteen, one fails on both retrievers by design. One measures a default query that no longer exists in the product. So fourteen queries grade live retrieval in the headline number, over 34 documents written alongside the retriever they grade, and the recall gate absorbs two full regressions before it fires: the measured 0.871 can fall to 0.75 and stay green. The suite has an exit code so CI can gate on it, and CI does not call it.

What the design earns is narrower and real. The orchestrator suite runs on every pull request and every push to main, keyless and deterministic, and fails when the machine breaks its own rules rather than when a model has an off day. The retrieval suite turns a claim into two columns of numbers with the controls needed to believe them.

The one-liner: build the thing that makes the eval fail before you believe the number it prints.

Sibling post: the memory layer, where the retriever these numbers grade is built, and where roughly 2,500 memories per org is the measured point at which scoring in Python crosses a 100 ms budget.

The views expressed here are my own and are not related to or reflective of my work or any organization I am affiliated with.