The Deploy Path Is the Recovery Path

AWS can take this server back whenever it wants it.

That is the deal for a t4g.small on spot, which is what CommsCrew runs on. One ARM instance, six to eight dollars a month of compute, Neon Postgres on the free tier next to it. Total bill around seven to ten dollars, and the deploy pipeline adds nothing to that.

Everything else in this layer follows from taking the reclaim threat literally.

If the box can vanish mid-request, nothing that matters is allowed to live on it. And if a replacement has to rebuild itself from scratch anyway, the code that rebuilds it should be the code that ships a release. Two problems, one mechanism.

Nothing on the box is the only copy of anything

Postgres is Neon, off-instance, serverless. Secrets are SSM Parameter Store SecureStrings, fetched fresh at every boot by an IAM role scoped to a single path prefix. Caddy's certificate store syncs to S3. The root volume is 30 GB of encrypted gp3 that is expected to die.

What runs on the instance is commscrew-api on host port 8000, commscrew-web on 3000, and Caddy terminating TLS in front of both. All three are reconstructible from somewhere that is not the instance.

There is one exception and the stack names it out loud. From the cost-cutting trade-offs in the header of infra/lib/commscrew-cheap-stack.ts: uploads go to local EBS rather than S3, and are lost if the instance dies. Written down as acceptable, not as solved.

SSM says what to run. The launch template says how to run it.

This is the whole idea, and it fits in two lines of user-data. The launch template has no image tag baked into it. It asks SSM what the current tag is, at boot, every boot. From infra/lib/commscrew-cheap-stack.ts:

      `API_TAG=$(aws ssm get-parameter --name ${ssmPrefix}/api_tag --query Parameter.Value --output text 2>/dev/null || echo "${imageTag}")`,
      `WEB_TAG=$(aws ssm get-parameter --name ${ssmPrefix}/web_tag --query Parameter.Value --output text 2>/dev/null || echo "${imageTag}")`,

A CDK-provided tag is the fallback, not the answer. The answer is a parameter that deploys rewrite.

The deploy path and the spot-recovery path are the same path. SSM is what should be running; user-data is how to run it. A deploy writes SSM and swaps containers. A reclaim re-reads SSM and boots. Same mechanism, two triggers.

Which means a deploy starts by writing a pointer, not by touching the server. scripts/deploy.sh saves the current tags as *_prev and then repoints production:

get_tag(){ aws ssm get-parameter --name "${SSM_PREFIX}/$1" --query Parameter.Value --output text 2>/dev/null || echo ""; }
PREV_API=$(get_tag api_tag); PREV_WEB=$(get_tag web_tag)
echo "▶ Previous tags: api=${PREV_API:-none} web=${PREV_WEB:-none}"
[ -n "$PREV_API" ] && aws ssm put-parameter --name "${SSM_PREFIX}/api_tag_prev" --type String --value "$PREV_API" --overwrite >/dev/null
[ -n "$PREV_WEB" ] && aws ssm put-parameter --name "${SSM_PREFIX}/web_tag_prev" --type String --value "$PREV_WEB" --overwrite >/dev/null
aws ssm put-parameter --name "${SSM_PREFIX}/api_tag" --type String --value "$TAG" --overwrite >/dev/null
aws ssm put-parameter --name "${SSM_PREFIX}/web_tag" --type String --value "$TAG" --overwrite >/dev/null

The ordering is deliberate. Pointers move before the containers do.

The failure mode is a reclaim landing in the middle of a deploy. If SSM still pointed at the old tag, the replacement box would boot the version you were in the act of replacing, and the deploy would silently undo itself. Writing forward first means an interrupted deploy heals forward. Rollback, under this model, is a parameter write.

Blue-green on one box is just two ports

There is no load balancer. An ALB would cost more per month than the rest of the stack combined, so the swap does the same trick on the loopback interface.

scripts/box-swap.sh runs on the instance itself, delivered by ssm send-command from CI. It looks at where the current container is bound and picks the other port:

start_new(){
  local name=$1 image=$2 cport=$3 a=$4 b=$5 hp=$6 cur new
  cur=$(cur_port "$name" "$cport"); if [ "$cur" = "$a" ]; then new=$b; else new=$a; fi

The API alternates between 8000 and 8010, the web app between 3000 and 3010. The new container comes up on the port the old one is not using and gets health-checked there while live traffic keeps flowing to the old one. Thirty attempts, two seconds apart, roughly a minute of patience before it gives up.

Only when both new containers pass does Caddy learn they exist:

[ -n "$API_OLD" ] && sed -i "s/localhost:${API_OLD}\b/localhost:${API_NEW}/g" "$CADDY"
[ -n "$WEB_OLD" ] && sed -i "s/localhost:${WEB_OLD}\b/localhost:${WEB_NEW}/g" "$CADDY"
systemctl reload caddy 2>/dev/null || /usr/local/bin/caddy reload --config "$CADDY"

Two lines of sed and a reload that takes about a second. That is the entire downtime of a production deploy, against the three to five minutes of recycling the instance, which is what this replaced.

Everything before the reload is abortable. If the pull fails, if the migration fails, if either health check fails, the script exits, the old containers are still serving, and the Caddyfile was never edited.

Everything after the reload is not abortable, and the script says so:

echo "SWAP_WENT_LIVE"
set +e

That marker exists because of a real ambiguity. Post-flip cleanup is a sequence of docker rm and docker rename calls, and an unguarded rename failure used to make the script exit non-zero after traffic had already moved. The orchestrator read that as "the swap aborted before going live" and ran the wrong rollback, restoring SSM to the previous tag while the box kept serving the new one. Nothing looked broken, and the next reclaim would have quietly downgraded production. Now cleanup is best-effort and idempotent, and the script forces exit 0 once traffic has moved.

A health check that returns 200 for a broken container is worse than none

The original /health was a static dictionary. It returned {"status": "healthy", "version": "2.0.0"} without consulting anything.

The failure mode is precise. A container boots with a stale DATABASE_URL or a missing secret. It answers /health with 200, because answering that endpoint requires nothing. It passes the swap gate, passes the smoke test, gets promoted, retires the old container, and then returns 500 on every real request. Auto-rollback never fires, because from the pipeline's point of view nothing failed.

So there are now two endpoints with two jobs. /health stays as liveness, cheap and dependency-free, for uptime pings. /health/ready gates the deploy, and it does work. From backend/app_new/main.py:

    @app.get("/health/ready")
    async def readiness():
        checks: dict[str, str] = {}
        ok = True

        try:
            factory = get_session_factory()
            async with factory() as session:
                await session.execute(text("SELECT 1"))
            checks["database"] = "ok"

And, outside development, it refuses to call insecure defaults acceptable:

        if settings.ENVIRONMENT.lower() not in ("development", "dev", "test", "local"):
            if settings.SECRET_KEY in ("", "change-me-in-production"):
                ok = False
                checks["secret_key"] = "missing/default"
            else:
                checks["secret_key"] = "ok"
            if not settings.ANTHROPIC_API_KEY:
                ok = False
                checks["anthropic_api_key"] = "missing"
            else:
                checks["anthropic_api_key"] = "ok"

Both the on-box health check and the public smoke test point at /health/ready, not /health. That is the part that matters. A deep check helps nobody if the gate keeps reading the shallow one.

The same review added a guard one layer down. box-swap.sh gives the new container its secrets by copying them out of the old one with docker inspect. If the old container is missing or uninspectable, that copy yields nothing, and the API starts with zero environment, falls back to a localhost database and a placeholder key, and cheerfully answers /health. The script now refuses to start the API with no inherited environment at all. Readiness would catch it too: one guard for the config, one for the runtime.

Migrations run first, and are allowed to stop everything

Before any container starts, the new image runs the migration against Neon:

DB_URL=$(aws ssm get-parameter --name "${SSM_PREFIX}/db_url" --with-decryption --query Parameter.Value --output text)
log "schema migrate (prod-safe, additive)…"
docker run --rm -e DATABASE_URL="$DB_URL" "${ECR_REGISTRY}/commscrew/api:${API_TAG}" python -m app_new.migrate

set -euo pipefail is at the top of that script, so a non-zero exit here ends the deploy before a single new container exists. A bad migration leaves production on the old version.

That gate has held in production, not in theory. The first pgvector deploy failed at exactly this step, because org_memory_embeddings already existed with embedding as JSON. Every environment where the migration had been verified built that table from scratch and got the vector type for free. The one environment with history was production. The swap refused and the old version kept serving.

Because the old and new containers coexist for the whole health-check window, migrations have to be backward compatible. Add now, remove in a later deploy.

The certificate has to outlive the instance

Caddy stores issued certificates and its ACME account key under /var/lib/caddy, on the root volume. The root volume is the thing that dies.

The failure mode is arithmetic. A replacement comes up with an empty store, decides it has no certificate, and orders new ones. Let's Encrypt caps that at 5 certificates per exact set of identifiers per 168 hours. A run of reclaims exhausts the quota, and the next replacement gets nothing at all.

On 2026-08-21 that is what happened. Two reclaims inside 21 hours, Let's Encrypt returning 429 for both hostnames, ZeroSSL's ACME backend simultaneously returning 502 on newNonce, and the site served no certificate for 17 hours. The API stayed up throughout, having obtained its certificate in the narrow window before the fallback broke.

The fix is a sync in both directions. On boot, before Caddy starts, the store is restored from S3, and a failure there is non-fatal because a first-ever boot has nothing to restore. Then a systemd timer pushes it back, first at three minutes after boot and every ten minutes after that:

      `cat > /usr/local/bin/caddy-state-sync.sh <<'SYNC'
#!/bin/sh
exec aws s3 sync /var/lib/caddy/ "s3://${caddyStateBucket.bucketName}/caddy-data/" \\
  --region ${this.region} --delete --only-show-errors
SYNC`,

A replaced instance now reuses its existing certificate instead of ordering another, which also removes the ACME wait from every recovery.

This is safe only because the ASG is fixed at exactly one instance, so there is exactly one writer. Two or more boxes would need Caddy's real S3 or DynamoDB storage module. A sync is not a distributed lock and should not be mistaken for one.

A second timer, every twenty minutes, checks that a certificate file exists for each host and restarts Caddy if one is missing. During the outage Caddy had left a stalled issuance goroutine holding its lock and never retried, logging nothing but renewal polls for the other domain. A restart clears the lock, turning an indefinite silent outage into a self-healing one.

The model ships inside the image and proves itself before the box takes traffic

CommsCrew retrieves organizational memory with a local embedding model. It used to load lazily, on the first request that needed a vector.

On a spot instance, lazily means after every reclaim. Fresh box, health check green, ASG satisfied, and then the first user request tries to pull about 150 MB from HuggingFace. If that fetch fails, retrieval returns an empty list, the agents run with no organizational memory, and the output is fast, polished, and generic. Nothing downstream can distinguish that from an organization that genuinely has no memories yet. Healthy meant the process was up, not that the product worked.

Two changes closed the gap. First, the model is baked into the image, in a layer that sits before the application code so ordinary code changes reuse it. From backend/Dockerfile:

ENV FASTEMBED_CACHE_PATH=/opt/models/fastembed
RUN mkdir -p /opt/models/fastembed && chown -R app:app /opt/models

# Switch to non-root user (before the bake, so file ownership matches runtime)
USER app

RUN python -c "from fastembed import TextEmbedding; TextEmbedding('BAAI/bge-small-en-v1.5')" && \
    test -n "$(ls -A /opt/models/fastembed)" || (echo 'model bake produced an empty cache' && exit 1)

The cache path is pinned with the same environment variable at build and at runtime, so the two cannot disagree about where the weights are, and the build fails if the cache comes out empty. A network fetch on a user's request became part of the ECR pull the box already does.

Second, the application proves it can embed before it accepts anything. From the lifespan handler in backend/app_new/main.py:

    if startup_settings.EMBEDDING_PROVIDER == "local":
        from app_new.services.embeddings import get_embedding_provider

        [probe] = await get_embedding_provider().embed(["warmup probe"])
        if not probe or not any(probe):
            raise RuntimeError("embedding warmup produced an empty vector")
        logger.info("embeddings.warmup_ok", dimension=len(probe))

A worker that cannot embed now dies at boot, where the swap's health gate catches it and keeps production on the old version. The measured cost of that guarantee is small and it varies by boot. Two production boots, read out of the container log as the delta between loading_local_model and warmup_ok, came in at 456 ms and 211 ms, straight off the image layer with no network. It runs twice, once per worker, because each worker proves itself independently.

Two workers, not four, and that number is a memory budget wearing a concurrency costume. The box has 2 vCPUs and 2 GB, shared with Next.js and Caddy, and each worker loads its own copy of the model at roughly 310 MB resident. Four meant about 1.2 GB of model alone, which does not fit. Two matches the cores. With both models loaded, free -m on the box reports about 780 MB available.

When it fails, it fails backward

The pipeline has no human approval gate. Push to main, and .github/workflows/deploy.yml assumes a GitHub OIDC role and runs the deploy. There are no long-lived AWS keys anywhere. That puts the entire safety burden on the automation, so the automation has three tiers of retreat.

If the swap fails before the Caddy flip, deploy.sh restores the previous SSM pointers and stops. The old containers never stopped serving.

If the swap succeeded but the public smoke test fails, it restores the pointers and re-swaps to the previous tags, whose images are still in ECR. The smoke test polls both URLs eighteen times, ten seconds apart, and requires a 200 from /health/ready.

If that rollback swap also fails, SSM is already at the previous version, so the script terminates the instance. The ASG relaunches, user-data reads SSM, and the box heals. It exits 2 rather than 1, so an operator can tell "rolled back" from "rollback failed, box recycling."

The role is narrow in the same spirit. It can write exactly four SSM parameters, the tag pointers, not the prefix holding the database URL and API keys. Its ssm:SendCommand permission is conditioned on the Name tag this stack sets on its own ASG, because one statement covering both the shell-script document and instance/* hands root on every instance in the account to anyone who can push to main.

What this layer does not do

Here is the honest count, and it undercuts the headline.

A deploy swaps exactly two things: the API container and the web container. The same user-data a reclaim runs also writes a Caddyfile, five systemd units, two shell helper scripts, a Route 53 change batch for two A records, and the boot sequence that installs Docker and Caddy in the first place. A deploy touches none of them. infra/** sits in the workflow's paths-ignore list on purpose, because this pipeline never runs cdk deploy and a green check on an infrastructure change would be a lie.

So "the deploy path is the recovery path" is true in one direction only. The recovery path is a superset. Infrastructure changes are applied out of band and sit dormant until the next boot.

That has already cost something. When auto-deploy was first activated, the live launch template had drifted and only started the API container, with no web container at all, which is why the site returned nothing while the API was fine. The pipeline could not have fixed it, because the pipeline does not run that code.

The other limits are simpler. A reclaim is still three to five minutes of downtime, unattended. Deploy downtime went from minutes to about a second; availability did not move. Uploads on local EBS still die with the box. One instance means one writer, and every trick here depends on that staying true.

The one-liner: if losing the box is routine, make shipping to it run the same code that losing it does.

Next in the series: the testing layer, where unit tests, contract tests, evals, and a control on the eval each catch a different kind of wrong.

The views expressed here are my own and are not related to or reflective of my work or any organization I am affiliated with.