The API Layer Is Where the Model Bill Gets Bounded

A route that streams a model's output has already sent its 200 by the time anything can go wrong.

That single fact shapes most of this layer. The status line goes out before the crew starts working, so a failure fifty seconds in cannot be an HTTP error. The connection stays open for the whole run, so it holds a database session and an upstream connection the entire time. And the run is billed per token. A client stuck in a retry loop produces an invoice.

CommsCrew's API is 115 route decorators under backend/app_new/api. Sixteen of them carry a rate limit. Four of them return a StreamingResponse. Nineteen check a permission before doing anything. This post is about those three small numbers, and why each guard is shaped the way it is rather than some other way.

All of it runs on one t4g.small ARM spot instance behind Caddy. That constraint is not incidental. It is why several of these guards are correct today and would be wrong tomorrow, which I will come back to at the end.

The scope of a limit is part of the limit

Two different things get throttled here, and treating them as one thing is the usual mistake.

Auth endpoints are cheap and unauthenticated. The threat is credential stuffing and signup spam, the actor is an address, and an extra request costs a database round trip. Model-backed endpoints are expensive and authenticated. The threat is a runaway loop or a bored user, and an extra request costs money.

So backend/app_new/core/rate_limit.py makes the key an explicit part of the policy:

class Scope(str, Enum):
    """What the limit counts against."""

    IP = "ip"
    USER = "user"
    ORGANIZATION = "organization"

Keying AI spend by IP would be actively wrong in both directions. One authenticated user behind a corporate NAT would throttle their whole office. The same user on a phone hotspot could re-roll drafts all day from a fresh address. So limit_ai takes two bounds and checks both, because neither implies the other: the per-user bound stops one runaway client, and the per-org bound caps what an entire workspace can spend in a window. Twenty users each politely inside their own limit still add up to a bill nobody approved.

In backend/app_new/api/v1/crew.py that reads as a factory per endpoint:

_chat_limit = limit_ai(
    "crew.chat",
    per_user=_settings.RATE_LIMIT_AI_PER_USER_PER_MINUTE,
    per_org=_settings.RATE_LIMIT_AI_PER_ORG_PER_MINUTE,
)

The defaults in backend/app_new/config.py are 20 per user per minute and 100 per org per minute. Website scanning is stricter at 20 per user per hour, because that endpoint hits third parties on our behalf and a loop there is somebody else's outage rather than just our bill. Auth is stricter still: 10 logins per minute, 5 registrations per hour, both per address. A human logs in once. Anything above single digits in a window is a script.

limit_ai returns the User it authenticated, so a route swaps Depends(get_current_user) for Depends(_chat_limit) and nothing else in the signature changes. That detail matters more than it looks: a guard that forces you to restructure a handler is a guard people route around.

Rejected attempts are deliberately not counted

The algorithm is a sliding window log. Timestamps per key, pruned on read. A fixed window has a boundary burst, where a 60-per-minute limit lets 120 requests through across a two-second seam. A token bucket is harder to explain inside a 429, which matters because the client has to act on the message.

The interesting part of InMemoryRateLimitStore.hit is what it does not do:

        if len(window) >= limit:
            # Do NOT record this attempt. Recording rejected attempts turns the
            # limiter into a lockout: a client that keeps retrying would push
            # the window forward forever and never recover.
            retry_after = max(1, int(window[0] + window_seconds - now) + 1)
            return Decision(allowed=False, remaining=0, retry_after=retry_after)

        window.append(now)
        return Decision(allowed=True, remaining=limit - len(window), retry_after=0)

The failure mode is a client that behaves badly for one second and is then locked out indefinitely by its own retries. retry_after is computed from the oldest surviving hit rather than being a constant, because it is a promise: it is the earliest moment a retry can actually succeed.

Getting that promise to the client took one more decision. RateLimitExceeded is deliberately not an HTTPException subclass. FastAPI serializes those directly and would drop Retry-After unless every raise site remembered to attach it. One dedicated exception plus one handler in main.py makes the header impossible to forget.

The budget headers had the same problem in reverse. X-RateLimit-Remaining is stashed on the ASGI scope and emitted by RequestContextMiddleware in backend/app_new/core/request_context.py, not written onto a response object inside the dependency. A raised HTTPException builds a fresh response and discards whatever a dependency set on the old one. The client getting 401s is precisely the client a login limit is aimed at, and that client would never have seen how much budget it had left.

The address you charge has to be the real one

client_ip is twelve lines of code and every one of them is about not trusting a header:

def client_ip(request: Request) -> str:
    trusted = get_settings().TRUSTED_PROXY_COUNT
    if trusted > 0:
        forwarded = request.headers.get("x-forwarded-for", "")
        hops = [h.strip() for h in forwarded.split(",") if h.strip()]
        if len(hops) >= trusted:
            return hops[-trusted]
        logger.warning(
            "ratelimit.forwarded_for_mismatch",
            expected_hops=trusted,
            actual_hops=len(hops),
        )

    return request.client.host if request.client else "unknown"

(Docstring trimmed.) X-Forwarded-For is attacker-controlled. Anyone can send one. Honoring it unconditionally hands out a free bypass: rotate a fake value per request and the per-IP limit never fires. So it is read only when TRUSTED_PROXY_COUNT says a proxy we control is actually in front, and then only the hop that proxy appended, counted from the right. Everything to the left is whatever the client typed.

The default is 0, which means the header is ignored entirely, and that default is exactly what bit production. Caddy reverse-proxies to localhost, so at 0 the API sees 127.0.0.1 as the client address for every request on earth. Every per-IP limit collapses into one shared bucket. The login limit becomes 10 per minute for the entire internet, and the site locks itself out under any real traffic.

Commit 47542a1 set TRUSTED_PROXY_COUNT=1 on the deploy path. Commit 2857214 fixed the same thing again, because two code paths start that container: scripts/box-swap.sh on a deploy, and the CDK user-data on a fresh boot. A spot reclaim runs the second one. The instance would have come back with every per-IP limit silently collapsed, and stayed that way until the next deploy happened to correct it.

The setting is now pinned by a test that reads scripts/box-swap.sh and asserts the string is in it. That is a strange-looking test until you notice the symptom is invisible in code review and only appears under real traffic.

Two clocks, because a stall and a loop look nothing alike

A rate limit caps how many runs an org can start in a window. It says nothing about how many are in flight at once, or about a run that starts fine and never ends.

backend/app_new/core/stream_guard.py handles both. Every guarded stream takes a per-org concurrency slot from a semaphore registry, defaulting to 3 concurrent runs per organization. Each in-flight stream holds a database session and an upstream connection, so that number is really a connection-pool guard wearing a product-shaped name. Twenty simultaneous campaign plans from one workspace will exhaust the pool, and the symptom appears on completely unrelated endpoints.

Then there are two timeouts, and the reason there are two is worth sitting with. An idle timeout bounds the gap between events, which is what a stalled provider actually looks like: the upstream accepted the request and went quiet, async for waits forever, nothing errors and nothing is logged because nothing failed. But an agent loop with a bad exit condition emits events steadily and legitimately forever. It is never idle. By definition an idle timeout cannot see it.

So the loop in guarded_stream runs both clocks at once. Here and in the next block I have dropped a couple of explanatory comment lines for length; every line of code is verbatim.

        while True:
            budget = deadline - loop.time()
            if budget <= 0:
                STREAM_REJECTIONS.labels(operation=operation, reason="total_timeout").inc()
                logger.warning("stream.aborted", operation=operation, reason="total_timeout")
                yield _sse(
                    "error",
                    {"message": "This run took too long and was stopped. Try a narrower brief."},
                )
                break

            step = min(settings.AI_STREAM_IDLE_TIMEOUT_SECONDS, budget)
            try:
                frame = await asyncio.wait_for(iterator.__anext__(), timeout=step)
            except StopAsyncIteration:
                break
            except asyncio.TimeoutError:
                # Which clock actually ran out? Near the end of a run the step
                # is capped by the REMAINING total, so a timeout there is a
                # total-budget failure wearing an idle timeout's clothes.
                exhausted_total = loop.time() >= deadline
                reason = "total_timeout" if exhausted_total else "idle_timeout"

Idle is 60 seconds, total is 600. That exhausted_total check is the payoff of a bug the tests found rather than reading did. Because each step's budget is capped by whatever remains of the total, a timeout in the last moments of a long run fires from the idle clock while being a run-length problem. The first version reported it as an idle timeout, which is the log line that sends an operator to go look at the provider. The reason is now derived from the deadline instead of assumed from which call raised.

Failures come back as event: error frames rather than status codes, for the reason at the top of this post. The alternative, dropping the connection, is indistinguishable from a network fault on the client side.

The slot has to come back on the path where nobody is watching

    finally:
        STREAMS_IN_FLIGHT.labels(operation=operation).dec()
        await _registry.release(org_id)
        aclose = getattr(iterator, "aclose", None)
        if aclose is not None:
            await aclose()

The failure mode here is a leak on the most common path, not the rare one. FastAPI throws GeneratorExit into the response generator when a client disconnects, and a user closing a tab during a slow generation is normal behavior, not an edge case. Without the finally, every abandoned run holds its slot forever, and after three of them that org can never start another stream.

The aclose() is the other half. Abandon the source generator without closing it and the agent loop behind it keeps calling the model. The run you just told the user you stopped goes on spending money headlessly. There is a test for exactly that, which sets an asyncio.Event inside the source's own finally and asserts it fired.

An unknown role gets nothing

permissions_for_role in backend/app_new/api/deps.py used to return admin permissions for anything it did not recognize. The justification, written into the code, was reasonable on its face: do not lock out legacy rows.

    if not role:
        logger.warning("authz.missing_role", granted="none")
        return NO_PERMISSIONS

    canonical = LEGACY_ROLE_ALIASES.get(role, role)
    permissions = ROLE_PERMISSIONS.get(canonical)
    if permissions is None:
        logger.warning("authz.unknown_role", role=role, granted="none")
        return NO_PERMISSIONS
    return permissions

User.role was a String(20) with no enum and no constraint. The failure mode is a typo, a bad CSV import, or a future SSO path writing an unmapped string, and any of the three silently granting org-admin. It was latent, because no route lets a user set their own role, but it is precisely the landmine that detonates the first time roles arrive from an external identity provider.

The legacy concern is real, so it is handled explicitly instead of permissively. LEGACY_ROLE_ALIASES maps approver to manager by hand. Known historical values keep exactly the access they had. Only genuinely unknown values lose privileges, and they get logged so an operator sees the drift rather than discovering it as an escalation.

A guard that quietly stops guarding is worse than no guard. You do not go looking for the thing you already believe is handled.

Failing closed is cheap here by design. Exactly five verbs are gated: publish, approve, accept_memory_rule, manage_integrations, manage_team_billing. Drafting, editing, commenting, scheduling and reading stay open to every member. A mis-tagged user loses publish rights, never access to the product.

Two existing tests had asserted the opposite behavior, in so many words: "missing role maps to admin", "unmapped legacy role behaves as admin". The vulnerability had been encoded as intended behavior and was passing CI.

Then the vocabulary itself turned out to be the deeper problem. Four places disagreed about which roles exist: ROLE_PERMISSIONS knew six, team.py offered three, a comment in the user model listed four, and the invite model simply defaulted to editor without reference to any of them. approver appeared only in that comment. So owner, manager and contributor carried permission sets that no route could assign to anyone.

backend/app_new/core/roles.py is now the only place that answers the question, and it renders its own SQL:

# Roles that can be ASSIGNED through the API today.
ASSIGNABLE_ROLES: tuple[str, ...] = ("admin", "editor", "viewer")

# Historical values that may exist in older rows. Kept valid at the database
# level so a CHECK constraint cannot reject data the app already wrote;
# api/deps.LEGACY_ROLE_ALIASES maps them to a modern role for permissions.
LEGACY_ROLES: tuple[str, ...] = ("owner", "manager", "contributor", "approver")

# Everything the `role` column may contain.
ALL_ROLES: tuple[str, ...] = ASSIGNABLE_ROLES + LEGACY_ROLES

# Rendered once so the model definitions and any migration use identical SQL.
ROLE_CHECK_SQL = "role IN (" + ", ".join(f"'{r}'" for r in ALL_ROLES) + ")"

That string becomes a CheckConstraint on users.role and on team_invites.role. Fail-closed authorization refuses an unknown role at request time. The constraint stops one being stored at all. Those are different guarantees and the SSO path needs both.

The test walks the route table, not the source

Every guard above shares a weakness: it lives in a module that keeps looking correct after someone deletes the one line that mounts it.

So backend/tests/test_rate_limit.py does not read source and grep for limit_ai. It resolves the real dependency tree of the real application:

def _dependency_names(route) -> set[str]:
    """Every dependency callable reachable from a route, by qualified name."""
    names: set[str] = set()
    stack = list(getattr(route.dependant, "dependencies", []))
    while stack:
        dep = stack.pop()
        call = getattr(dep, "call", None)
        if call is not None:
            names.add(getattr(call, "__qualname__", "") or "")
        stack.extend(getattr(dep, "dependencies", []))
    return names

Both factories close over a nested _dependency, so a mounted limit shows up as limit_by_ip.<locals>._dependency in that set. The parametrized test runs across all sixteen limited routes, and deleting a single Depends(...) while leaving the limit defined above it fails it.

There is a test guarding the guard, too. test_route_table_is_traversable asserts the walk finds more than 50 routes, because FastAPI changed how include_router works between 0.136 and 0.141: the older version copied routes eagerly, the newer leaves a placeholder and flattens later. Under the newer one a naive walk finds nothing, and the next(...) the first version used over that empty result raised StopIteration instead of reporting a missing rate limit. The repo pins fastapi>=0.141.0 while a local venv had 0.136.3, so the wiring tests passed locally and would have failed in CI on every run, sixteen times over, for a reason with nothing to do with rate limiting. The guard test makes that say itself once.

The stream guard's equivalent test is the weaker kind. It reads each module's source and checks that every return StreamingResponse( has guarded_stream within the next two hundred characters. It still found a real hole. The script that wrapped the generators had matched on event_stream(), and the onboarding module's generator is named stream(), so the edit did nothing while reporting success. One of four streaming endpoints came out of that pass unguarded, in a diff where nobody would have noticed an absence.

Where this breaks

Now the honest part, and it is a counting exercise.

Sixteen rate-limited routes and four guarded streams, and by default every count behind them lives in one Python process's memory. RATE_LIMIT_SHARED_STORE defaults to False. Run three replicas and a 10-per-minute login limit becomes 30 per minute, silently, because each process only ever counts what it saw.

A DatabaseRateLimitStore exists to fix that half, and it is opt-in and currently off. The stream guard has no equivalent at all. AI_STREAM_MAX_CONCURRENT_PER_ORG is 3 per org per process, and there is no flag that makes it 3 per org. Scale horizontally and it quietly becomes 3N, with nothing in the config surface to notice.

Which means the correctness of these guards today is a property of the deployment, not of this code. One box, one container, one process. The limits hold because there is nowhere else for the traffic to go.

The one-liner: a guard you cannot see from the route table is a guard you do not have.

Next in the series: the persistence layer, and the connection pool every one of these guards is really protecting.

The views expressed here are my own and are not related to or reflective of my work or any organization I am affiliated with.