CommsCrew
Try the demo →

Engineering notes

Blog.

Fourteen deep dives on building CommsCrew, one per architectural layer plus the concerns that cut across all of them. Each one is a real account of one decision: what was chosen, what it cost, and the bill it is still paying. Every diagram is drawn from the actual code, and nothing here claims a capability that has not been measured.

THE TOOLS four good products Writing assistants Social schedulers Brand voice tools Analytics WHERE IT STOPS WHERE IT STOPS WHERE IT STOPS WHERE IT STOPS start after the decision about what to say assume the content already exists operate on a draft you bring them speak after everything is over THE HUMAN carries the context SEAM SEAM SEAM re-explain re-explain re-explain the strategy the audience what success meant most of the days go here nothing that learns from the last campaign reaches the tool that starts the next one THE LINE nobody owns it 1 2 3 4 Strategy Mix Produce Measure what it is for channel by message pieces get drafted against the strategy what the last campaign learned Nothing in the top half connects without passing through a person. The dashed path is the one nobody carries.
01 / 14Aug 29, 2026

It Takes Them Days to Prepare a Campaign

The market is full of tools that write posts. I went looking for the one that runs the whole line, and could not find it.

1 STRATEGY 2 MIX 3 PRODUCE 4 MEASURE CREW ACTS HUMAN DECIDES locked else 409 draft, critique recommend write bodies retrospective revise adjust materialise approve strategy_unchanged zero content status: draft _campaign_performance strategy_status drop or add creates rows ALLOWED_TRANSITIONS the crew stops here the human reads issues, not edits proposal planned rows review publish undelivered messages
02 / 14Aug 29, 2026

Nothing Exists Yet

The campaign pipeline is four stages, and every decision worth explaining lives in the boundaries between them.

CHOOSES ENFORCES EXECUTES SHARED STATE tool result {"phase": "review_done", "needs_revision": true} Orchestrator orchestrator_agent.py one tool call per turn _dispatch _refusal consult_strategist assign_writer request_revision request_editor_review finalize max_revisions = 2 Strategist Writer Editor Final draft strategist.py writer.py editor.py best_draft() no prose passes between specialists CrewRunState crew/state.py brief angles drafts reviews revision_count transcript WRITE METHODS record_strategy record_drafts record_reviews record_revision_request angles drafts reviews done REFUSED {"refused": ...} revision cap reached call finalize never executes
03 / 14Aug 29, 2026

The Orchestrator's Tools Are the Other Agents

A typed blackboard, five delegation tools, and a dispatcher that refuses illegal moves. How the crew layer decides who works next.

WRITE STORE READ WHAT THE MODEL SEES EXPLICIT typed by a human IMPLICIT inferred from human edits SEMANTIC vectors and cosine PUT /memory/brand-voice POST /memory/facts POST /memory/executives BrandVoiceProfile OrgFact ExecutiveProfile tone and vocabulary what the org is who speaks get_brand_voice_summary the only function that turns a profile into a system prompt Human edits a draft diffed and categorized _apply_learning_to_profile every third signal BrandVoiceProfile JSON columns tone_attributes vocabulary sentence_style tone, style vocabulary length, structure 5 edit categories land in 3 columns read into the prompt GET /memory/learning counted and shown to a human here never a prompt input POST /memory/examples POST /memory/scan-website embed_memory_row(row) writes three columns or none caller commits all four OrgMemoryEmbedding text_content embedding embedding_provider embedding_dim the memory itself 384 floats which model made it 384 search_similar _brand_examples with query no query filters on provider + dim agent tool search_memory Agent system prompt org facts brand voice executive profiles learned tone + style absent from the prompt vocabulary length, structure relevant past content top 5 by cosine appended after the last cache breakpoint reaches the model written and rendered, never read into the prompt
04 / 14Aug 29, 2026

Three Tiers of Memory, One Write Path

The text, the vector, and the name of the model that made the vector have to move together, or the crew ends up confidently retrieving nothing.

200 already sent one request RATE LIMIT per-IP auth · 10 logins / min per-user AI spend · 20 / min per-org AI spend · 100 / min limit_ai checks both bounds AUTHORIZATION permissions_for_role checked on 19 routes LEGACY_ROLE_ALIASES approver maps to manager unknown role NO_PERMISSIONS five verbs gated STREAM GUARD concurrency slot 3 per org, one per stream idle clock, 60 s catches a stalled provider total clock, 600 s catches an endless loop released in finally, even on disconnect MODEL the crew runs billed per token the bill starts here 429 Retry-After rejected attempts not counted refused never defaults to admin event: error a frame, not a status code
05 / 14Aug 29, 2026

The API Layer Is Where the Model Bill Gets Bounded

Limits keyed by what they cost, two clocks on every stream, and authorization that refuses what it does not recognize

SERVER TRANSPORT DESIGN CHOICE FAILURE MODE unknown event compile error passes silently never renders the server emits event: error already-200 stream from three places one SSE parser lib/sse.ts fetch + getReader { event, data } generic over payload sseEvents<T> event stays string nothing discriminates generic over the message sseEvents<CrewChatEvent> union keyed on event data narrows with it silent continue the frame is dropped typing indicator stops no reply, no error rendered as a message msg.event === "error" hooks/useCrewChat.ts the crew did not answer the load fails GET returns 500 or returns nothing the promise rejects api.get(path, token) inside a keyed effect the error is swallowed .catch(() => {}) data stays initial 21 places still do this error is a return value hooks/useResource.ts loading and error derived stale response dropped empty list looks fine, is wrong "All content is scheduled!" the error is shown in place of the data no false empty state
06 / 14Aug 29, 2026

The frontend layer: turn every silence into a failure

A thin client over a streaming backend, where an unhandled server event is a compile error and a failed load can never render as an empty list.

ONE TRANSACTION python -m app_new.migrate 1 Create the vector extension IF NOT EXISTS 2 Create missing tables only create_all 3 Add missing nullable columns _sync_missing_columns 4 Convert legacy JSON column to vector(N) USING NULL::vector 5 Create the HNSW index USING hnsw a second run changes nothing THE GATE scripts/box-swap.sh step 2 exit code migrate exit code exit 0 nonzero New containers start on alternate ports Health checks pass, Caddy reloads New version takes traffic No new container is started The Caddyfile is untouched Old version keeps serving TWO DIALECTS, ONE SUITE conftest.py reads TEST_DATABASE_URL THE FIVE STEPS ABOVE SQLite sqlite+aiosqlite 1 2 3 4 5 1, 4 and 5 never run JSON float array scored in Python Postgres pgvector/pgvector:pg16 1 2 3 4 5 the only tests that run 1, 4 and 5 vector(N) with HNSW similarity in SQL
07 / 14Aug 29, 2026

Expand Only

The data layer is allowed to add to the schema and to do nothing else, and every other decision in it follows from that one rule.

Dashed: may be absent, and nothing upstream fails. BOUND ONCE AT THE EDGE RequestContextMiddleware binds request_id raw ASGI, one task, no copy auth dependency user_id, org_id bound in deps.py, not in routes structlog contextvars merged into every log line read by the telemetry writer request.completed carries request, user, org No route handler participates. ONE CAPTURE POINT BaseCrewAgent every model call in api and crew a test forbids raw SDK calls _record_run() schedule_agent_run_record returns before any write lands estimate_cost_usd() longest model prefix wins cache read 0.1x, write 1.25x priced cost_usd set unpriced cost_usd NULL NULL, never zero. Shown as unpriced_runs. THREE SINKS Prometheus counters agent_runs_total, agent_tokens_total agent_cost_usd_totalincrements only when priced gen_ai.chat span per-request flow, GenAI conventions inert unless an exporter is set agent_runs row durable, per-org accounting cost_usdnullable, and no foreign keys agent_run.persist_failed logged; the request is unaffected Only the row is durable. Counters reset on deploy. request_id own session, never blocks INHERITED CONTEXT Bound once: request_id by the middleware, user_id and org_id by the auth dependency. Every log line and every agent_runs row inherits them. No route handler passes anything along, and no label set carries a UUID.
08 / 14Aug 29, 2026

A Cost Column That Is Allowed To Be NULL

The observability layer: one capture point, three sinks, and a number that refuses to be confidently wrong.

NOTHING ON THE BOX IS THE ONLY COPY TWO TRIGGERS ONE PATH WHEN IT FAILS, IT FAILS BACKWARD OUTCOME health gate fails → restore api_tag_prev, web_tag_prev if rollback fails, terminate; the ASG relaunches SSM Parameter Store api_tag · web_tag db_url · secrets Neon Postgres off-instance serverless, free tier S3 cert store restored on boot synced every ten minutes ship a release push to main, deploy.sh lose the box spot reclaim, user-data read SSM api_tag web_tag pull images docker pull api · web migrate app_new.migrate additive only start new port 8010 · 3010 traffic on old health gate /health/ready 30 tries · 2s flip Caddy sed + reload about one second retire old docker rm SWAP_WENT_LIVE GATE GATE ABORTABLE COMMITTED GATE: the deploy stops here, the old version keeps serving the tag SSM names is serving deploy: one caddy reload reclaim: three to five minutes
09 / 14Aug 29, 2026

The Deploy Path Is the Recovery Path

One spot instance at six to eight dollars a month, where shipping a release and losing the server run the same code.

KIND OF WRONG RUNS AGAINST THE MECHANISM TURNS RED WHEN IN CI? UNIT code is wrong Real code 436 backend tests TEST_DATABASE_URL SQLite local, Postgres in CI an assertion fails --cov-fail-under=62 runs in ci.yml CONTRACT a boundary you do not own changed Mocked provider _client, no network ProbeAgent + _FakeStream on the real BaseCrewAgent the row is wrong error_class, api_kind runs in ci.yml EVAL behavior regressed Deterministic mock crew one DB per case run_evals.py 10 cases, 3 use decision_policy verify() returns False cap breached, no refusals runs in ci.yml CONTROL the eval itself is broken Known-bad input EMBEDDING_PROVIDER run_retrieval_evals.py control_ceiling() scales it the score is HIGH ceiling 0.368, got 0.143 Keyword baseline _legacy_keyword_search KEYWORD_BASELINE_FLOOR checked before anything else the baseline collapses recall@5 under the floor not in ci.yml
10 / 14Aug 29, 2026

Four Kinds of Test, Because There Are Four Kinds of Wrong

The testing layer: unit tests check code, contract tests check a boundary you do not own, evals check behavior, and a control checks the eval.

CONCERN LAYERS Campaign Agent Memory API Frontend Data Observ. Deploy Testing Context Output contracts Control flow Grounding Evaluation Failure semantics what an agent cannot see forced tools, never prose who may act, and when measured, not asserted a gate that can turn red an outage that reads as one 3 4 3 5 2 4 VOCABULARY how often each post says the words 1 48 4 3 15 1 19 3 32
11 / 14Aug 30, 2026

Agent Engineering Is Not a Layer

Six concerns with no box in the architecture diagram, which is exactly why it is easy to believe you are doing them.

BOUNDARY LOCK 1 · RUNTIME CHECK LOCK 2 · STRUCTURAL CONSTRAINT REFUSED AS unreached unrepresentable REQUEST refuses an unknown role require_permission(verb) an unknown role gets the empty verb set logs authz.unknown_role for the operator 5 gated verbs, 19 of 115 handlers AND CHECK ck_users_role 3 assignable roles, 4 legacy kept valid role='okta:group:engineering' same SQL on team_invites.role ForbiddenError IntegrityError TENANT refuses a moved account's old rows One helper, both predicates user_id == user.id organization_id == user.organization_id 26 modules under api/v1 filter on it AND Tenancy read from the row get_current_user loads the User row the JWT org claim is never authorized on 17 of 26 models carry organization_id 404 no stale org org: Previous Employer conversations and notifications org: the inviting org the only rows a query returns accept_invite changes organization_id. The user_id does not change. OUTBOUND refuses a private address on any hop validate_outbound_url() scheme allowlist, then the address check every resolved address, not the hostname is_link_local: 169.254.169.254 AND The client cannot chase a hop follow_redirects=False the scanner walks the hops itself 2 MB body cap, content-type allowlist scan failed hop refused guard_redirect(url) up to 5 hops, each one re-resolved every hop, back to the check DEPLOY refuses a write to a secret /health/ready runs SELECT 1 against the database 503 if SECRET_KEY is change-me-in-production on-box check and smoke test both use it AND Four ARNs, not the prefix api_tag, web_tag, api_tag_prev, web_tag_prev jwt_secret is not in the list 503 at the gate denied by IAM
12 / 14Aug 30, 2026

Security Is Four Boundaries, and the Bug Lives in the Seam

Authorization that fails closed, queries scoped to a tenant, egress checked on resolved IPs, and a deploy role that cannot overwrite what it deploys past

TWO SUITES backend/evals/ they fail independently Orchestrator evals run_evals.py 10 cases, deterministic mock provider, gates CI Retrieval evals run_retrieval_evals.py 34 documents, 17 queries, top_k 5 graded by three gates on one axis recall@5 one axis, three gates 1.00 0.75 0.50 0.25 0.00 SEMANTIC search_similar, top_k 5 must clear a floor floor 0.75 · pass is above 0.871 measured Measured 0.871 can fall to 0.75 and stay green: two full regressions before it fires. KEYWORD BASELINE _legacy_keyword_search must clear a floor floor 0.15 · pass is above 0.281 measured A dead baseline does not fail, it flatters: 0.00 to 0.87 reads as a triumph. Checked first. CONTROL EMBEDDING_PROVIDER must stay under a ceiling If it scores WELL here, the suite FAILS. ceiling 0.368 = 2.5x chance not a constant · pass is below 0.143 measured Hash embeddings carry no semantics, so this must land at chance: 5 / 34 = 0.147. CALIBRATED AT valid only at the inputs it was measured over CALIBRATED_AT top_k 5 documents 34 queries 17 NO VERDICT TOP_K is not 5, or the ceiling rises above the floor. The gate declines. Why a multiple, not a constant 0.40 at 34 documents = 2.7x chance. 0.40 at 200 documents = 16x chance, blessing a competent retriever as random.
13 / 14Aug 30, 2026

The Eval Has to Be Able to Embarrass You

Two suites, one deterministic mock, thresholds calibrated from a measurement, and the positive control almost nobody builds.

LAYER CHOSEN REJECTED WHAT IT COST THE CREW how the agents are shaped 1 Anthropic SDK only anthropic>=1.1.0 CrewAI a framework that owns the loop You write the loop. Retries, step budget, dispatcher, telemetry row, streaming guard, mock provider. And no tutorial fits it. 2 Four agents Strategist, Writer, Editor, Analyst Ten agents one writer per output format Agent count fell, context count did not. Four agents still assemble a block list, cache breakpoints and three memory layers. THE DATA storage and retrieval 3 SQLite dev, Postgres prod clone to running, two commands Postgres everywhere a container in CI and on laptops Two implementations of search_similar, one contract. The tax is a permanent second CI job. It caught two divergences. 4 Additive migrator only migrate.py, 179 lines, 26 tables Alembic declared, imported by nothing Renames and drops are not available, not merely risky. A column type change needs another tool and a two-deploy dance. 5 Local ONNX embeddings bge-small-en-v1.5, no key voyage-3 better, one env var away 310 MB resident per process, so the box runs 2 gunicorn workers, not 4. And the provider width is welded into the column. NOT PAID YET Zero load tests. The worker count is arithmetic on memory, and says nothing about concurrency. THE BOX one instance, one pipeline 6 One t4g.small on spot min = max = desired = 1 Two behind an ALB costs more than everything else A spot reclaim is 3 to 5 minutes of unattended downtime with no failover. The rate limiter is correct only at one. 7 State lives off the box Neon, SSM, Caddy certs to S3 Postgres on the box secrets in a file Every piece of state is a network hop. A boot that cannot reach SSM has no secrets. Uploads die with the instance. 8 Push to main deploys deploy.yml, no approval gate A required reviewer the author of the commit The automation carries the whole safety burden. A static 200 promoted a broken container, so /health/ready runs SELECT 1. WHAT SHIPS the demo boundary 9 Simulated last mile demo publish, demo billing One real integration weeks to prove nothing The last mile is unproven, and simulated metrics feed the Analyst, so every surface rendering one must carry a DemoDataBadge. the rejected alternative a cost stated but not measured A blank cell in this column is a decision not yet made.
14 / 14Aug 30, 2026

Every Decision in This System Has a Price Tag

Nine choices, the alternative each one rejected, and the bill each one is still paying.

Prefer one narrative pass instead? Read the five-part origin story first.