Every Decision in This System Has a Price Tag

The original plan for CommsCrew is still in the git history. Its tech stack section names PostgreSQL, Redis, ChromaDB, Celery, CrewAI and OpenAI.

Of those six, one shipped.

Postgres is real. The other five are gone, and most were gone long before anyone deleted the line declaring them. Commit 62a0ecc removed eight Python dependencies with zero references anywhere in the application: celery, redis, chromadb, slowapi, pyotp, qrcode, authlib, python-statemachine. Commit bcc2e70 found 2,382 lines of a superseded architecture still sitting in the tree, including a Lambda handler's requirements file that pulled crewai, openai and langchain into a repository whose entire argument is that it uses none of them.

That gap between the plan and the running system is what this post is about. Nine months of it. Each place the two diverge is a choice somebody made, and a choice is only legible afterward if you can say what it cost.

So: nine decisions, and the price of each.

The framework came out before the first agent went in

The plan said CrewAI. The code uses the Anthropic SDK and nothing else. This is the entire AI section of backend/requirements.txt:

# AI
anthropic>=1.1.0

The reason is narrow and I would not generalize it. An agent framework is a bet that its abstractions match the shape of your problem. The shape here is a controller that must refuse illegal moves, specialists pinned to exactly one tool call each, a typed state object passed between them, and per-call context that must never be assigned to a shared instance. Every one of those is a constraint on the orchestration loop. A framework that owns the loop owns the constraint.

The cost is that you write the loop. All of it. Retries, the step budget, the dispatcher that refuses a revision past the cap, the telemetry row per model call, the streaming guard, the mock provider that makes any of it testable without a key. None of that is interesting work, and none of it would exist if you had accepted somebody else's opinions about delegation.

The second cost is subtler. Nobody else has your abstractions, so nobody else's tutorial applies to your code.

Six of the ten planned agents were a lookup table

The plan listed ten agents. Read the list closely and six of them are one writer per output format: a LinkedIn post agent, a Twitter thread agent, a press release writer, a blog writer, an email newsletter agent, an executive quote agent.

Those six are a switch statement with a system prompt in each branch, which is not an architecture. Nothing about writing a press release requires a different controller, a different memory view, or a different output contract than writing a LinkedIn post. It requires different guidance about length and structure.

So the ten became four: Strategist, Writer, Editor, Analyst. The six writers became seven bullet points under a heading called Platform Guidelines in backend/app_new/crew/writer.py, one line each for LinkedIn, blog, press release, newsletter, tweet, executive quote and internal email. That is what they always were.

The half that did not collapse is the interesting one. The number of distinct contexts an agent call assembles stayed where it was. build_system_prompt in backend/app_new/crew/base_agent.py returns a block list: the agent's base prompt, then organizational context with a cache breakpoint on it. The Writer appends a per-executive ghostwriting block with its own breakpoint, then per-query retrieval last, deliberately after every breakpoint so a fresh retrieval cannot invalidate the cached prefix above it. Underneath sit three memory layers: explicit guidelines, implicit signals learned from edits, semantic retrieval over embeddings.

Agents merged because the boundary between them was a formatting difference. Contexts did not, because those boundaries are in the problem. An executive's voice really is separate from an organization's brand voice, and both are separate from what happens to be relevant to this one query. Collapsing agents was free. Collapsing those would have cost the cache, the ghostwriting path, and the ability to say where a sentence came from.

One instance, and the reclaim is in the budget

Production is a single t4g.small on spot, in an autoscaling group whose three capacity numbers are all 1. From infra/lib/commscrew-cheap-stack.ts:

    const asg = new autoscaling.AutoScalingGroup(this, 'ApiAsg', {
      vpc,
      vpcSubnets: { subnetType: ec2.SubnetType.PUBLIC },
      launchTemplate,
      minCapacity: 1,
      maxCapacity: 1,
      desiredCapacity: 1,
      healthChecks: autoscaling.HealthChecks.ec2({
        gracePeriod: Duration.minutes(5),  // Boot + Caddy cert issue ~3-5 min
      }),
    });

The rejected alternative is two instances behind a load balancer. An ALB costs more per month than everything else in this stack put together, which at six to eight dollars of compute is the whole budget rather than a rounding error.

The price is stated in the stack header rather than discovered later: a spot reclaim is roughly three to five minutes of downtime, unattended, and there is no failover. Nothing takes over. The autoscaling group notices, launches a replacement, and the replacement rebuilds itself from scratch.

That single number propagates further than it looks. Because there is exactly one process, the rate limiter's in-memory counters are correct. Because there is exactly one writer, syncing Caddy's certificate store to S3 is safe without a distributed lock. Both guarantees are properties of the deployment rather than of the code, and both stop being true at two instances.

Nothing on the box is the only copy of anything

Postgres is Neon, off-instance. Secrets are SSM Parameter Store SecureStrings, fetched fresh at every boot by an IAM role scoped to one path prefix. Caddy's certificate store syncs to S3 on a timer and restores on boot. The root volume is thirty gigabytes of encrypted gp3 that is expected to die.

The rejected alternative is the obvious one: run Postgres on the box, keep secrets in a file, let Caddy keep its certificates where Caddy keeps them. On a single instance that is simpler in every dimension except the one that matters, which is that the instance is disposable by design.

The cost is that every piece of state is now a network hop and an external dependency. A boot that cannot reach SSM has no secrets. A request that cannot reach Neon has no database. And there is one exception the stack writes down rather than solving: uploads go to local EBS and are lost if the instance dies. Recorded as acceptable, not as handled.

The certificate half earned its complexity the expensive way. Let's Encrypt caps issuance at five certificates per exact set of identifiers per 168 hours, and a replacement with an empty store orders new ones. On 2026-08-21 two reclaims inside 21 hours exhausted that quota and the site served no certificate for 17 hours. The S3 sync exists so a replacement reuses its certificate instead of ordering another.

The failure mode a stateless box guards against is the one where recovery means a person restoring something by hand. The failure mode it introduces is a boot that half-succeeds because one of four external services was slow.

Two databases means two retrievals, and only one of them runs in production

Development and CI run SQLite. Production runs Postgres. The rejected alternative is Postgres everywhere, including a container in CI and on every contributor's laptop.

The reason is that a reference implementation whose quick start requires Docker is a reference implementation most readers never run. SQLite makes git clone to running product a two-command path with no services to start.

The price is exact and it is paid every day. search_similar in backend/app_new/crew/memory_service.py has two implementations behind one contract. Postgres orders by cosine distance in SQL against an HNSW index and returns only the requested rows. SQLite pulls the organization's embedded rows and scores them in Python. Same inputs, same ordering, same score scale, two entirely different pieces of code.

There is no version of that which is safe on inspection alone, so the tax is a permanent second CI job. backend-postgres runs against pgvector/pgvector:pg16, executes the migration against a fresh database, and then runs the model, auth, semantic retrieval and legacy migration suites there.

It has already caught two real divergences, both recorded in the module. The first: the Postgres branch was missing the embedding_dim predicate that the Python branch applied per row, so the two paths disagreed about which rows were eligible. The second: an empty result with rows present but written by a different embedding provider emitted a diagnostic warning on the Python path and silence on the Postgres one. Every local run warned. Production returned an empty list and said nothing.

Both bugs lived in the branch that only production exercises. A job that runs that branch on every commit is the only thing that pays this bill.

The migration tool can only add

There is no Alembic. alembic>=1.19.0 sits in backend/requirements.txt and is imported by nothing: no alembic.ini, no versions directory, no call site anywhere in the backend. The models are the schema, and backend/app_new/migrate.py is the only code that changes production's structure. 179 lines, 26 tables.

It creates missing tables, then adds missing columns, and the guard is one condition:

            # Only safe, additive changes: an existing row must be able to take
            # the new column without a value.
            if not column.nullable and column.server_default is None:

A NOT NULL column with no server default is skipped and printed, loudly, rather than attempted.

The reason is the deploy shape. Old and new containers coexist on alternate ports for the whole health-check window, so the old code has to keep working against the new schema for about a minute. That is the expand half of expand-then-contract, and expand is all this tool can do.

The cost is that renames and drops are simply not available. Not discouraged, not risky: unavailable. Changing a column type, tightening a constraint, backfilling and then enforcing, any of it needs a different tool and a two-deploy dance. The file's one exception is a hand-written conversion of a legacy JSON embedding column to vector(N), written because the first pgvector deploy died on exactly that column. Every environment where the migration had been verified built the table fresh and got the vector type for free. The environment with history was the only one that mattered.

The embedding model is a memory decision wearing a quality decision's clothes

Semantic retrieval runs a local ONNX model, BAAI/bge-small-en-v1.5 via fastembed. The hosted alternative, voyage-3, is better and is implemented and is one environment variable away.

Local won because a retrieval path that requires an API key is a retrieval path CI, the eval suite and the public demo all exercise through a mock instead of for real. Keyless is what makes the measurement meaningful.

Two costs, both structural.

The first is resident memory. The model is roughly 150 MB on disk and about 310 MB resident per process. backend/Dockerfile runs gunicorn with --workers 2 and the comment above the line says why: four workers meant about 1.2 GB of model alone on a box with 2 GB shared with Next.js and Caddy. Worker count here reads like a concurrency setting and is actually a memory budget.

The second is that the choice is welded into the schema:

# The real consequence is worth stating plainly: because this width is baked
# into the column, changing EMBEDDING_PROVIDER is a MIGRATION, not a config
# change. That is more honest than a JSON column would be, which would accept
# vectors of any width and let two providers coexist in one table undetected.
PROVIDER_DIMENSIONS: dict[str, int] = {
    "deterministic": 64,
    "local": 384,
    "voyage": 1024,
}

backend/app_new/services/embeddings.py refuses to make that look free. There is no silent fallback between providers, every stored row records the provider and dimension that produced it, and retrieval filters to the active one. Switching to Voyage makes the old corpus invisible until a backfill re-embeds it, which is louder and more recoverable than quietly poisoning every score computed against it.

A decision you can only describe by what it bought is a decision you have not finished making.

There is no human between a push and production

.github/workflows/deploy.yml says it in its own header:

# Push-to-main → fully automatic production deploy. No manual approval gate:
# the smoke-test + auto-rollback inside scripts/deploy.sh ARE the safety net.

The rejected alternative is a protected environment with a required reviewer. On a one-person project that reviewer is the person who wrote the commit, which is ceremony rather than control.

The cost is that the automation carries the entire safety burden, and that reshaped a component two layers away. The original /health returned a static dictionary. It answered 200 without consulting anything, which meant a container booted with a stale database URL or a missing key would pass the swap gate, pass the smoke test, get promoted, retire the old container, and then fail every real request. Auto-rollback would never fire, because from the pipeline's view nothing failed.

So /health/ready now opens a session, runs SELECT 1, and outside development refuses to call a default secret key or a missing Anthropic key acceptable. Both the on-box health check and the public smoke test point at it. The failure mode a shallow health check hides is the one where every gate reports success and the product is down.

Removing the human moved the work into the check rather than deleting it.

The last mile is a simulation, and every surface has to say so

Publishing and billing do not work. The demo provider imitates the OAuth redirect, the token exchange, the publish call and the resulting engagement numbers. Billing writes a subscription tier to the database and returns "demo_mode": True with hardcoded invoices. No post reaches LinkedIn. No card is charged.

The rejected alternative was one real provider, dropped because the interesting engineering here is upstream of publishing and a half-wired LinkedIn integration costs weeks to demonstrate nothing the reader came for.

The cost is that the last mile is unproven, and it compounds. Simulated engagement metrics feed the analytics surface, which feeds the Analyst agent, so the "AI recommendations from real engagement data" loop stays open. And because a simulated number that looks measured is the most damaging thing a reference implementation can ship, every surface rendering one has to carry provenance. Rows persist with a demo source marker. apps/web/src/components/ui/DemoDataBadge.tsx labels them in the UI, in neutral tokens rather than a warning colour, because this is provenance rather than an error. The Analyst's own prompt forbids presenting demo numbers as measured results.

That discipline was not free either. Commit bcc2e70 caught the marketing page promising "From brief to published" while every dashboard surface correctly said otherwise. The headline is now "From brief to scheduled."

What is not paid for yet

Here is the honest count, and it undercuts one of the nine above.

Two workers. Zero load tests. There is no locust file, no k6 script, nothing in CI that sends concurrent traffic at anything. The worker count is justified entirely by arithmetic on resident memory, 310 MB per process against a 2 GB box, and that arithmetic says nothing about whether two async processes serve real concurrent SSE streams acceptably. The per-organization stream concurrency slot defaults to 3. Whether 3 is right, or whether two workers saturate first, is unmeasured.

So the embedding decision is half-earned. The memory half is measured and the number is defensible. The concurrency half is an assumption written down as though it were a finding.

The rest of the ledger is more comfortable. One instance really does mean three to five minutes of unattended downtime per reclaim. Expand-only migrations really do mean a rename is a project. Two dialects really do mean two retrieval implementations, and the second CI job is not optional maintenance, it is the price of the first decision. Each of those was cheaper than its alternative at this scale, and each has a scale at which it stops being.

The one-liner: an architecture decision you cannot price is one you have not actually made yet.

Elsewhere in the series: the deploy layer, where shipping a release and losing the server turn out to run the same code.

The views expressed here are my own and are not related to or reflective of my work or any organization I am affiliated with.