Four Kinds of Test, Because There Are Four Kinds of Wrong

The first time backend/tests/test_agent_contracts.py ran, it found a bug that had been in the streaming path the whole time.

The error handler logged e.status_code. Connection errors and timeouts do not have one. Every HTTP error the product had ever seen carried a status, so the handler had never been asked the question, and the one case where a network blip should degrade into a clean error event would instead have raised an AttributeError inside the exception handler. The fix in backend/app_new/crew/base_agent.py is a getattr with a default:

        except anthropic.APIError as e:
            self.logger.error(
                "Anthropic API error", error=str(e), status=getattr(e, "status_code", None)
            )

Nothing about that bug is exotic. What earns it the opening is the instrument that caught it. No unit test would have. The code was correct against every response the provider had actually returned, and the only way to reach the broken branch was to build a fake provider that fails in a way the real one rarely does.

That is the shape of this whole layer. CommsCrew has 436 passing backend tests, 4 skipped and 1 xfailed on the run I did while writing this, and they are not all the same kind of thing. Code invariants, provider contracts, agent behavior, retrieval quality, and the documented onboarding command each fail differently, so each gets a different instrument.

Step one was making CI able to fail at all

Before any of it meant anything, the pipeline had to be capable of turning red.

Commit 01fb74a is titled "make CI able to fail," and two of its changes are why. The first was a || echo "Tests skipped or no tests found" appended to the backend test command, which meant a failing suite exited zero and the job went green. The second was subtler and worse: backend/tests/conftest.py hardcoded an in-memory SQLite URL, so the CI job that spun up a Postgres service and pointed DATABASE_URL at it ran the entire suite against SQLite anyway. The Postgres container booted, passed its health check, and was never touched.

The fix turns on one lookup:

TEST_DB_URL = os.environ.get("TEST_DATABASE_URL", "sqlite+aiosqlite://")

Fast in-memory SQLite stays the default for local runs, on a StaticPool. Anything else gets a NullPool, so asyncpg connections are not reused across event loops. CI's integration job sets TEST_DATABASE_URL and the same suite runs against a real database, which is what catches the UUID, JSON and timestamp dialect differences SQLite quietly forgives.

The rule that came out of it sits at the top of .github/workflows/ci.yml as a comment, above six jobs:

# Honesty rule: every step here must be able to fail. No `|| echo` fallbacks,
# no test commands that report success after running zero tests.

The failure mode being guarded against is a green check mark that means nothing. It is the most expensive bug in a repository, because it does not cost you a broken build, it costs you every future broken build you now will not notice.

The coverage floor lives in the same job and is deliberately unambitious:

      - name: Run tests (coverage floor 62%, ratchet-only)
        run: |
          cd backend
          python -m pytest tests/ -x --tb=short -q --cov=app_new --cov-fail-under=62

Ratchet-only means the number goes up when real coverage goes up and never comes back down to accommodate a change. It started at 55 in commit fc99d2d, when the suite was 227 tests at 56.9%. A floor you can lower is a suggestion.

Contract tests check the boundary you do not own

backend/tests/test_agent_contracts.py is 199 lines and it never touches the network. It builds a ProbeAgent on top of the real BaseCrewAgent, replaces _client with a mock, and drives all three API paths the crew uses: a plain response, a forced tool call, and a stream.

What makes them contract tests rather than unit tests is the assertion target. Each one checks the returned value and the telemetry row the call was supposed to leave behind:

    @pytest.mark.asyncio
    async def test_schema_mismatch_fails_loudly_and_is_counted(self, agent):
        # Model answers with prose instead of the required tool call.
        agent._client.messages.create = AsyncMock(
            return_value=SimpleNamespace(
                content=[SimpleNamespace(type="text", text="I refuse")],
                stop_reason="end_turn",
                usage=_usage(),
            )
        )
        with pytest.raises(ValueError):
            await agent.respond_with_forced_tool([{"role": "user", "content": "x"}], self.TOOL)
        await flush_pending()
        (row,) = await _rows()
        assert row.status == "error"
        assert row.error_class == "schema_mismatch"
        assert row.api_kind == "forced_tool"

CommsCrew pins tool_choice whenever code is going to consume model output, so a schema miss is a real event with a name. This test asserts three separate things about that event: it raises rather than returning something plausible, it is counted, and it is counted under the right label. A miss recorded as a generic error is a miss you cannot chart.

The stream tests build a _FakeStream class, an async context manager and iterator that mimics the SDK's surface and takes a raise_after index. That is the entire mechanism for reproducing a provider dying mid-response:

        agent._client.messages.stream = MagicMock(
            return_value=_FakeStream([_text_delta("partial")], raise_after=1)
        )
        events = [e async for e in agent.respond_stream([{"role": "user", "content": "go"}])]
        await flush_pending()

        assert events[-1]["type"] == "error", "interrupted stream must surface an error event"

The failure mode is a stream that stops producing tokens and never says why, leaving the browser holding an open connection and a half-written draft. Half of the value here is that the test is cheap to write once the fake stream exists, which is the argument for building the fake at all.

Evals are the regression suite for behavior

Unit tests answer whether a function returns the right value. They cannot answer whether a crew of four agents routed correctly, revised only when the editor asked, and stopped.

backend/evals/harness.py is that second suite. Every case runs in its own throwaway in-memory SQLite database, telemetry included, so the numbers a case reports belong to that case alone:

    metrics = {
        "steps": sum(1 for e in events if e["event"] == "orchestrator_decision"),
        "revisions": state.revision_count,
        "final_phase": state.phase.value,
        "agent_runs": total_runs,
        "schema_misses": schema_misses,
        "duration_ms": round((time.perf_counter() - start) * 1000, 1),
    }
    passed, note = case.verify(state, events, metrics)

Verification is a function, not a string match. Each case in backend/evals/cases.py inspects the final run state, the emitted event stream and the metrics, and returns a pass or fail plus a human-readable note. Ten cases run in CI and all ten currently pass.

Three of the ten are the interesting ones, and they exist because of a failure the other seven could not see. EvalCase carries an optional decision_policy hook that replaces the orchestrator's decision function outright. Without it, the suite drives a deterministic mock that behaves well, so an eval named editor_never_satisfied_bounded passed happily while the production code allowed ten revisions against a cap of two. The mock politely stopped asking. A live model has no such manners.

The adversarial cases stub the orchestrator itself:

def _rogue_always_revise(agent):
    """Simulates a model that keeps demanding revisions forever."""
    calls = {"n": 0}

    async def policy(conversation):
        calls["n"] += 1
        if calls["n"] == 1:
            return ("id", "consult_strategist")
        if calls["n"] == 2:
            return ("id", "assign_writer")
        return ("id", "request_revision")

    return policy

And the matching verify does not settle for the run finishing:

def _verify_cap_enforced(state, events, m):
    if state.revision_count > state.max_revisions:
        return False, f"revision cap BREACHED: {state.revision_count} > {state.max_revisions}"
    refusals = [t for t in state.transcript if t.action.startswith("refused:")]
    if not refusals:
        return False, "expected the orchestrator to refuse an over-cap revision"
    return True, f"cap held at {state.revision_count}; {len(refusals)} illegal move(s) refused"

The distinction that earns its keep: testing a well-behaved mock proves the mock is well behaved. Only a rogue decision policy tests our enforcement, and enforcement is the thing a live model will actually attack. The other two adversarial cases send the orchestrator moves that are illegal in order (writing with no strategy, reviewing with no drafts) and kill the provider at the decision step.

A generator cannot grade itself

The retrieval evals in backend/evals/run_retrieval_evals.py turn "our semantic memory is better than the keyword search it replaced" into two columns of numbers over a labelled corpus of 34 documents and 17 queries, at the same top-k of 5 the product actually uses. Recorded measurements: recall@5 goes from 0.279 to 0.871, MRR from 0.198 to 0.693.

The thresholds are calibrated from that measurement rather than picked in advance. The MRR gate sits at 0.60 with a comment explaining that 0.70 was the number chosen before measuring, and it fails the current healthy implementation by 0.007. A gate set by ambition trains everyone to ignore CI.

But scoring a retriever with a harness you also wrote has an obvious hole, and this file spends most of its length on it. Two guards close it.

The first is a control whose gate inverts. Running with EMBEDDING_PROVIDER=deterministic embeds every document with hash vectors that carry no semantics, so retrieval should be near random, and under that provider the run fails if the score is high. The ceiling is not a constant, because random top-k recall on a single-relevant query is k over N and a constant crosses its own defect value as the corpus grows:

def control_ceiling(corpus_size: int) -> float:
    """The recall a hash-embedding control must stay under, scaled to chance."""
    return CONTROL_CHANCE_MULTIPLE * (TOP_K / corpus_size)

At 34 documents that evaluates to 0.368. The control I ran scored 0.143 over 85 retrieved rows, against a chance value of 0.147.

The second guard is the one most suites are missing. Every other check in that file asserts something is absent: recall is low, no rows are unreadable, the no-answer query is not confident. An assertion that something is absent passes perfectly when the machinery producing it is dead. So the keyword baseline is kept as a positive control with a floor of its own, and the gate checks it before it judges anything else:

    kw_recall = _aggregate(scores["keyword"])["recall"]
    if kw_recall is None or kw_recall < KEYWORD_BASELINE_FLOOR:
        return False, [
            f"FAIL  keyword baseline recall@5 is {kw_recall if kw_recall is None else f'{kw_recall:.3f}'}, "
            f"under the floor of {KEYWORD_BASELINE_FLOOR:.2f}. The baseline is a positive "
            "control: it is provider-independent and must score its known value in every "
            "run. A collapsed baseline does not mean semantic retrieval improved, it means "
            "the comparison is measuring nothing and reporting a LARGER win than the truth. "
            "Check _legacy_keyword_search and memory_service._extract_keywords."
        ]

If the tokenizer ever returns an empty list, the baseline returns nothing, and the headline goes from 0.28 to 0.87 up to 0.00 to 0.87. Semantic retrieval appears to have improved.

A broken control that flatters the thing it controls for is the worst shape a check can have, because nobody investigates a number that went up.

This is not theory in this repository. Commit 6445289 rebuilt the Claude-as-judge eval that scores actual writing quality, and its message records that two harness artifacts were found and fixed before the numbers were published: long-form outputs were being truncated by a token limit, and variant diversity was being measured across variants generated from the same prompt. Both bugs belonged to the measuring apparatus. Reporting them as product scores would have been reporting the method.

The documented command, run in a subprocess

backend/tests/test_fresh_clone_smoke.py guards a different claim: that the README's Quick Start works.

Rather than importing and calling the setup function, it shells out to the exact command the docs print, against a database file that does not exist yet:

def run_dev_init(db_path: Path) -> subprocess.CompletedProcess:
    """Run the documented init command in a clean subprocess."""
    env = {**os.environ, "DATABASE_URL": f"sqlite:///{db_path}"}
    return subprocess.run(
        [sys.executable, "-m", "app_new.dev_init"],
        cwd=BACKEND_DIR,
        env=env,
        capture_output=True,
        text=True,
        timeout=120,
    )

Then it registers a user through the API, calls a protected endpoint with the returned token, and logs in as the seeded demo account. Running it twice checks that the command is idempotent.

The same file holds the test I like most in this repository, and its docstring explains itself better than I can paraphrase. from app_new.main import app succeeds even when a runtime dependency is missing, because the scheduler is imported inside lifespan. That is not hypothetical: apscheduler was declared in requirements and absent from a working environment, so 365 tests passed while uvicorn app_new.main:app died on startup.

@pytest.mark.asyncio
async def test_the_app_can_actually_start_not_merely_import():
    from app_new.main import app as real_app

    async with real_app.router.lifespan_context(real_app):
        pass  # startup and shutdown both have to survive

Importing a module proves it parses. Entering the lifespan proves it boots.

Above that layer sits e2e/golden-path.spec.ts, two Playwright tests through real servers on dedicated ports with a fresh SQLite database and no AI key, with retries set to 0 on purpose. A flaky test gets fixed or quarantined, never retried into green.

Even the prose gets a guard

backend/tests/test_documented_claims.py exists because every other check in the repository protects code, and the reasoning lives in the prose.

Its docstring lists four documentation claims that went stale in a single afternoon, three of which were caught only because someone happened to mention a commit. Raising the coverage floor from 55% to 62% left three documents citing 55. Eliminating every any in the frontend left CONTRIBUTING advertising 28 of them, in the same paragraph that says an understated document is as much a defect as an overstated one.

The floor check reads the number out of CI and compares it to whatever the docs claim:

@pytest.mark.parametrize("doc", DOCS, ids=lambda p: p.name)
def test_documented_coverage_floor_matches_ci(doc):
    text = _read(doc)
    cited = {int(n) for n in re.findall(r"(\d+)%\s*(?:CI-enforced\s*)?coverage floor", text)}
    cited |= {int(n) for n in re.findall(r"coverage floor[^.\n]*?(\d+)%", text)}
    if not cited:
        pytest.skip(f"{doc.name} cites no coverage floor")

    actual = _ci_coverage_floor()
    assert cited == {actual}, (
        f"{doc.name} cites coverage floor(s) {sorted(cited)} but ci.yml enforces {actual}%"
    )

Only claims a machine can settle get a guard here. Judgement calls do not, and are not attempted. When one fails, the document is usually what is wrong.

Where this bar is lower than it sounds

Here is the counted deflation. Three eval entry points live in backend/evals/. One of them runs in CI.

run_evals.py gates every push to main and every pull request, all ten cases, keyless and deterministic. run_retrieval_evals.py has an exit code and a docstring that says "so CI can gate on it," and nothing calls it from .github/workflows/ci.yml. eval_crew.py, the Claude-as-judge suite that scores actual writing quality across 19 cases, costs roughly two to three dollars a run and is invoked by hand from a shell script. So the only agent behavior CI protects is control flow. Retrieval quality and writing quality are both measured well and both regress silently.

Three more limits worth naming plainly. The Postgres integration job runs 4 of the 41 test files in backend/tests/; the other 37 have only ever seen SQLite, which means dialect bugs outside models, auth, retrieval and migration would reach production before a test saw them. The coverage floor of 62% is honest but low: the suite currently measures 66%, which leaves 2,029 of 6,051 backend statements that nothing has ever executed. And the retrieval gate is calibrated at exactly k equals 5, verified by a check that refuses to render a verdict at any other budget, because at k equals 20 that same healthy retriever scores 1.000 and the floor stops asserting anything at all.

The one-liner: a test suite is only as honest as its ability to fail, so build the thing that makes it fail, then build the control that tells you whether the failure was real.

Next in the series: the closing post, and the finding underneath all of it, that a coding agent implements your claim as convincingly as it implements the real thing, and fundamentals are the instrument that tells the difference.

The views expressed here are my own and are not related to or reflective of my work or any organization I am affiliated with.