Streaming responses arrived all at once after adding FastAPI middleware: BaseHTTPMiddleware limits

Our job service, built with FastAPI, exposes a server-sent events (SSE) endpoint that streams progress updates for long-running imports. The frontend shows a live progress bar. After a release that added a request logging middleware, users reported the progress bar jumping: nothing for a while, then several steps at once. Some imports also showed as still "running" long after users had closed the tab.

The middleware

from starlette.middleware.base import BaseHTTPMiddleware
import time, logging

log = logging.getLogger("access")

class TimingMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request, call_next):
        start = time.perf_counter()
        response = await call_next(request)
        duration = time.perf_counter() - start
        log.info("request", extra={"path": request.url.path, "status": response.status_code,
                                   "duration_ms": round(duration * 1000)})
        response.headers["Server-Timing"] = f"app;dur={duration * 1000:.1f}"
        return response

app.add_middleware(TimingMiddleware)

And the streaming endpoint:

from fastapi.responses import StreamingResponse

@app.get("/imports/{import_id}/events")
async def import_events(import_id: str, request: Request):
    async def gen():
        async for event in progress_stream(import_id):
            if await request.is_disconnected():
                break
            yield f"data: {event.json()}\n\n"
    return StreamingResponse(gen(), media_type="text/event-stream")

Three symptoms

1. Timing was wrong. The access log recorded about 3 ms for SSE requests that stayed open for minutes. call_next returns as soon as the response starts, not when the body finishes, so the measured duration covered only the handler's setup.
2. Events arrived in bursts. With curl, events that the server generated one per second sometimes arrived in groups:

curl -N https://jobs.staging.example.com/imports/imp_8812/events

1. Disconnects were not detected promptly. request.is_disconnected() inside the generator kept returning False after the client left, so the stream kept running until the import finished.

Removing the middleware made all three go away.

What we ruled out

• Proxy buffering. We tested directly against the pod, bypassing nginx, which already had proxy_buffering off for this path.
• The progress source. Logging inside the generator showed events produced at the right times.
• Uvicorn settings. The same uvicorn configuration streamed correctly without the middleware.

Root cause

BaseHTTPMiddleware runs the downstream application in a separate task and passes the response body back through an internal stream. That indirection changes how the body and the client connection reach the endpoint: timing based on call_next stops at the start of the response, the extra task and memory stream can change when chunks are flushed, and the request object the endpoint sees no longer reflects the real client connection state reliably. Starlette's documentation lists these limitations and recommends pure ASGI middleware for anything that must not interfere with streaming or the request lifecycle.

How BaseHTTPMiddleware works

BaseHTTPMiddleware gives a convenient dispatch(request, call_next) API. To provide it, Starlette:

• wraps the incoming ASGI messages into a Request;
• runs the rest of the application in a task group;
• collects the downstream http.response.start and body messages into a response object with a body iterator;
• sends that response back through the middleware.

This is fine for small, complete responses. It is a poor fit for streaming, server-sent events, long polling and anything that depends on knowing when the client disconnects. Historically it also had issues with contextvars propagation and with background tasks, some of which newer Starlette versions have improved, but the design remains an extra layer around the response.

Pure ASGI middleware

A pure ASGI middleware is a callable that receives scope, receive and send and wraps them. It sees every message as it happens, adds no task, and does not buffer the body.

Who is affected

FastAPI and Starlette applications that combine BaseHTTPMiddleware (including middleware from libraries built on it, and functions registered with @app.middleware("http"), which use it under the hood) with streaming responses, server-sent events, long polling or large file downloads. Plain JSON endpoints are usually fine, which is why the problem appears only when a streaming endpoint is added later.

Environment

Rewrite the middleware as pure ASGI

import time, logging

log = logging.getLogger("access")

class TimingMiddleware:
    def __init__(self, app):
        self.app = app

    async def __call__(self, scope, receive, send):
        if scope["type"] != "http":
            return await self.app(scope, receive, send)

        start = time.perf_counter()
        status = {"code": None}

        async def send_wrapper(message):
            if message["type"] == "http.response.start":
                status["code"] = message["status"]
                headers = list(message.get("headers", []))
                elapsed = (time.perf_counter() - start) * 1000
                headers.append((b"server-timing", f"app;dur={elapsed:.1f}".encode()))
                message["headers"] = headers
            await send(message)

        try:
            await self.app(scope, receive, send_wrapper)
        finally:
            log.info("request", extra={
                "path": scope["path"],
                "status": status["code"],
                "duration_ms": round((time.perf_counter() - start) * 1000),
            })

app.add_middleware(TimingMiddleware)

What this changes:

• The middleware passes receive through untouched, so disconnect messages reach the endpoint directly.
• Body messages go straight to send, so each SSE event is flushed as soon as the generator yields it.
• The log line is written in finally, after the whole response, including the streamed body, has been sent, so the duration is correct for streams. Server-Timing still reflects time to first byte, which is what that header is for.

Make the SSE endpoint robust to disconnects

With the middleware fixed, disconnect detection worked again. We also stopped relying only on polling is_disconnected(): when the client goes away, the ASGI server cancels the response, so cleanup goes in a finally block of the generator:

async def gen():
    try:
        async for event in progress_stream(import_id):
            yield f"data: {event.json()}\n\n"
    finally:
        await unsubscribe(import_id)        # runs on disconnect or completion

Alternatives and trade-offs

• Exclude streaming paths from BaseHTTPMiddleware with a path check. Works, but every new streaming endpoint must remember the exception.
• Use an observability library's ASGI middleware, such as OpenTelemetry's ASGI instrumentation, which is implemented as pure ASGI.

How to test the fix

A test uses a real Uvicorn server and an HTTP client that reads events with timestamps:

async def test_sse_streams_incrementally(live_server):
    async with httpx.AsyncClient(timeout=10) as client:
        async with client.stream("GET", f"{live_server}/imports/test/events") as r:
            times = []
            async for line in r.aiter_lines():
                if line.startswith("data:"):
                    times.append(time.monotonic())
                if len(times) == 3:
                    break
    gaps = [b - a for a, b in zip(times, times[1:])]
    assert all(0.7 < g < 1.3 for g in gaps)

A second test closes the connection after one event and asserts that unsubscribe was called within a second.

Confirming in production

The progress bar updated smoothly again, once per second. Access logs showed realistic durations for SSE requests (minutes, matching the import), and subscriptions were cleaned up within a second of users closing the tab, which reduced open subscriptions in Redis by about 60%.

Regression guard

• A code review rule: new middleware must be pure ASGI; BaseHTTPMiddleware is allowed only for small, non-streaming apps.
• The streaming test above runs in CI.

Security implications

Streams that keep running after clients disconnect consume server resources and can be abused: opening many connections and dropping them would leave work running. Correct disconnect handling and cleanup in finally limit that. The access log does not record query strings, which can contain tokens for SSE endpoints.

Prevention checklist

• Prefer pure ASGI middleware over BaseHTTPMiddleware, especially with streaming endpoints.
• Measure request duration after the response completes, not when call_next returns.
• Pass receive through untouched so disconnects reach endpoints.
• Clean up streaming resources in finally.
• Test streaming with a real server and timed reads.

References

• Starlette, Middleware (including pure ASGI middleware): https://www.starlette.io/middleware/
• FastAPI, Middleware: https://fastapi.tiangolo.com/tutorial/middleware/
• ASGI specification, HTTP connection scope: https://asgi.readthedocs.io/en/latest/specs/www.html
• MDN, Server-sent events: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events