FastAPI Streaming in 2026: SSE, WebSockets, and NDJSON for LLM and ML APIs

Updated: August 6, 2026

FastAPI streaming responses let a server push data to a client in chunks over a single HTTP or WebSocket connection, using StreamingResponse for raw chunked HTTP, EventSourceResponse for Server-Sent Events (SSE), or a WebSocket route for bidirectional traffic. SSE is the default choice for one-way LLM token streaming in 2026. I've shipped a dozen of these endpoints in the last year for FastAPI backends fronting OpenAI, Anthropic, and self-hosted vLLM services, and the patterns that survive production are boringly consistent. So, let's walk the primitives, the async pitfalls, and the proxy gotchas.

• FastAPI 0.135.1 (March 2026) shipped native SSE via from fastapi.sse import EventSourceResponse. You don't need sse-starlette for new projects unless you want its ping/timeout helpers.
• Use SSE for LLM token streaming, NDJSON for tool-to-tool pipelines, StreamingResponse for file downloads, and WebSockets only when the client must send data mid-stream (chat with interruption, live cursors, etc.).
• Every streaming endpoint needs three things: a request.is_disconnected() check, a heartbeat every 15 seconds, and the X-Accel-Buffering: no header. Miss any one and you leak connections or get delayed events.
• Nginx buffers 16 KB by default and Cloudflare buffers ~100 KB; both silently break token-by-token streaming until you disable buffering per-route.
• GZipMiddleware is incompatible with SSE, because the middleware buffers the entire response before compressing.
• Use async LLM clients (openai.AsyncOpenAI, anthropic.AsyncAnthropic) so a single Uvicorn worker can hold hundreds of concurrent streams without pinning the event loop.

When to stream and when not to

Streaming is not a free upgrade. A streaming endpoint holds a TCP connection open for the entire generation, pins a worker's event loop task, and adds framing overhead per chunk. If your response finishes in under 300 ms, a regular JSON body is faster end-to-end and easier to debug. Streaming pays off in four situations: LLM completions where the first token matters more than the last; long-running ML inference (image generation, batch scoring) where users need progress; large file downloads that would overflow memory if buffered; and log or event feeds where the request is inherently open-ended.

The rule I use on my team: stream when the total generation time exceeds 1.5 seconds and the client can render partial output. A CSV export of 500 MB should stream, because the alternative is out-of-memory. A structured JSON response from a synchronous scikit-learn model shouldn't, because the client can't do anything with half a prediction. If in doubt, measure with ab or wrk before adding streaming complexity to your codebase.

One trap worth naming early: streaming does not reduce total latency, it only shifts the perceived latency. The 30th token arrives at the same wall-clock time whether you stream or buffer. You just show the user the 1st token earlier. That trade is almost always right for LLMs and almost always wrong for classification APIs.

FastAPI StreamingResponse for chunked HTTP

StreamingResponse is the lowest-level primitive FastAPI gives you. It wraps any sync or async iterator and flushes each yielded chunk over HTTP chunked transfer encoding. There's no event framing, no reconnect logic, no keep-alive comments, just bytes going out as fast as your generator produces them. That makes it perfect for file downloads and NDJSON pipelines, and workable (but rough) for LLM tokens.

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import asyncio, json

app = FastAPI()

async def ndjson_rows():
    # Simulate paging through a large query, one row per chunk.
    for i in range(10_000):
        row = {"id": i, "value": i * 2}
        yield json.dumps(row).encode() + b"\n"
        if i % 500 == 0:
            await asyncio.sleep(0)  # yield to the event loop

@app.get("/export/rows")
async def export_rows():
    return StreamingResponse(
        ndjson_rows(),
        media_type="application/x-ndjson",
        headers={"X-Accel-Buffering": "no"},
    )

The await asyncio.sleep(0) inside the loop is a small but important detail. Without it, a fully CPU-bound generator can starve other requests on the same worker. On the async ETL work I described in async ETL in Python with httpx and asyncio, we hit exactly this: a generator pulling from a synchronous pandas call blocked every other in-flight request. Force a yield point every few hundred iterations.

NDJSON is my default format for machine-to-machine streams. Each line is a self-contained JSON object, so downstream tools can parse without buffering. It plays cleanly with jq, DuckDB (read_json_auto('file.ndjson')), and any HTTP client that supports line-delimited reading. For anything user-facing in a browser though, jump straight to SSE. You get automatic reconnect and a native browser API for free.

Native EventSourceResponse in FastAPI 0.135+

FastAPI 0.135.1, released in March 2026, added a first-class EventSourceResponse. Before that, everyone reached for sse-starlette; now you get SSE built into the framework, with headers (Cache-Control: no-cache, X-Accel-Buffering: no, Content-Type: text/event-stream) set automatically and Pydantic serialization pushed down to the Rust core. If you're on 0.135 or newer, prefer this over the third-party library. One fewer dependency, tighter integration, better throughput.

from typing import AsyncIterable
from fastapi import FastAPI, Request
from fastapi.sse import EventSourceResponse, ServerSentEvent
from pydantic import BaseModel
import asyncio

app = FastAPI()

class Token(BaseModel):
    text: str
    index: int

async def token_stream(prompt: str) -> AsyncIterable[ServerSentEvent]:
    # Replace with your real LLM client below.
    for i, word in enumerate(prompt.split()):
        await asyncio.sleep(0.05)
        yield ServerSentEvent(
            event="token",
            data=Token(text=word, index=i).model_dump_json(),
        )
    yield ServerSentEvent(event="done", data="")

@app.get("/chat", response_class=EventSourceResponse)
async def chat(prompt: str, request: Request):
    async def guarded():
        async for event in token_stream(prompt):
            if await request.is_disconnected():
                break
            yield event
    return EventSourceResponse(guarded())

A few things are worth pointing out. The event="token" field lets the browser dispatch on named events (source.addEventListener("token", …)) instead of the default message. That's useful when you want to distinguish token deltas from tool calls or final metadata in the same stream. The guarded() wrapper is my standard pattern for disconnect handling; I'll expand on it in the disconnects section. And using a Pydantic model inside data= means your SSE payloads are validated the same way as any other FastAPI response body, which stops a whole class of "the frontend crashed because content was None" incidents.

On the browser side, consumption is a single API:

const source = new EventSource("/chat?prompt=Explain%20SSE");
source.addEventListener("token", (e) => {
  const t = JSON.parse(e.data);
  document.getElementById("out").textContent += t.text + " ";
});
source.addEventListener("done", () => source.close());

EventSource auto-reconnects with exponential backoff if the connection drops, using the Last-Event-ID header to resume. That's a feature you'd have to hand-roll on WebSockets. That alone is why I default to SSE for anything user-facing.

sse-starlette for older FastAPI versions

If you're pinned below 0.135, or you need features like configurable pings and connection semaphores, sse-starlette is still the reference implementation. It exports its own EventSourceResponse and adds a built-in ping mechanism, an optional per-connection timeout, and a hook to run cleanup code when the client disconnects.

from sse_starlette.sse import EventSourceResponse
from fastapi import FastAPI, Request

app = FastAPI()

async def event_gen(request: Request):
    for i in range(1_000):
        if await request.is_disconnected():
            return
        yield {"event": "tick", "data": str(i)}

@app.get("/ticks")
async def ticks(request: Request):
    return EventSourceResponse(
        event_gen(request),
        ping=15,  # send a comment every 15 seconds
        headers={"X-Accel-Buffering": "no"},
    )

The ping=15 argument makes sse-starlette inject a :ping comment into the stream every 15 seconds. Browsers ignore comments, but proxies see traffic and reset their idle timers. Without it, Nginx will kill an idle connection after 60 seconds by default, and your users will see mysterious reconnection loops during long LLM generations.

One caveat that has bitten me twice: sse-starlette is incompatible with FastAPI's GZipMiddleware. The middleware buffers the entire response before compressing, which defeats the whole point of streaming. Either drop GZip entirely or exclude the SSE routes with a middleware guard. Compression on token streams is almost never worth it anyway. The payloads are already small and the overhead per chunk is significant.

WebSockets: when bidirectional is worth the cost

WebSockets give you full-duplex communication over a single TCP connection. Both sides can send at any time, framing is binary-capable, and you're not tied to HTTP request semantics. That's genuinely useful for collaborative editors, live cursors, multiplayer games, and voice/video signaling. For LLM streaming, where the pattern is "client sends prompt, server streams tokens, done", WebSockets are almost always overkill.

from fastapi import FastAPI, WebSocket, WebSocketDisconnect
import asyncio

app = FastAPI()

@app.websocket("/ws/chat")
async def ws_chat(ws: WebSocket):
    await ws.accept()
    try:
        while True:
            prompt = await ws.receive_text()
            for word in prompt.split():
                await ws.send_json({"token": word})
                await asyncio.sleep(0.05)
            await ws.send_json({"done": True})
    except WebSocketDisconnect:
        return

The reason to reach for a WebSocket here would be something like "the user can interrupt mid-generation" or "the same connection carries typing indicators and tokens simultaneously." Both are real requirements. Neither justifies WebSockets for a first version. Start with SSE, ship it, and upgrade only when you have a specific bidirectional need that HTTP round-trips can't cover.

Operationally, WebSockets are harder. They need sticky sessions when you scale horizontally, they don't play well with HTTP/2 multiplexing, and every load balancer, WAF, and CDN needs explicit WebSocket configuration. Cloudflare, AWS ALB, and GCP HTTPS Load Balancer all support WebSockets, but each has separate timeout knobs you'll discover the hard way. For 90% of AI chat use cases, SSE eliminates every one of these problems.

Streaming OpenAI and Anthropic through your API

The most common production pattern I've built is proxying a hosted LLM provider (OpenAI, Anthropic, Bedrock, or a self-hosted vLLM) through a FastAPI endpoint that adds auth, rate limiting, cost tracking, and observability. The trick is bridging the provider's async iterator to the SSE response without buffering the whole completion in memory.

from fastapi import FastAPI, Request
from fastapi.sse import EventSourceResponse, ServerSentEvent
from anthropic import AsyncAnthropic
import json, time

app = FastAPI()
client = AsyncAnthropic()  # ANTHROPIC_API_KEY from env

@app.post("/v1/chat/stream")
async def stream_chat(payload: dict, request: Request):
    async def event_source():
        start = time.perf_counter()
        input_tokens = output_tokens = 0
        try:
            async with client.messages.stream(
                model="claude-opus-4-7",
                max_tokens=1024,
                messages=payload["messages"],
            ) as stream:
                async for text in stream.text_stream:
                    if await request.is_disconnected():
                        break
                    yield ServerSentEvent(event="token", data=text)
                msg = await stream.get_final_message()
                input_tokens = msg.usage.input_tokens
                output_tokens = msg.usage.output_tokens
        except Exception as exc:
            yield ServerSentEvent(
                event="error",
                data=json.dumps({"message": str(exc)}),
            )
            return
        yield ServerSentEvent(
            event="done",
            data=json.dumps({
                "latency_ms": int((time.perf_counter() - start) * 1000),
                "input_tokens": input_tokens,
                "output_tokens": output_tokens,
            }),
        )
    return EventSourceResponse(event_source())

Four things I always do in this endpoint. First, use the async client (AsyncAnthropic or openai.AsyncOpenAI). A sync client will block the event loop and pin your worker to a single request. Second, catch every exception inside the generator and yield an error event before returning; if you let the exception propagate, the client sees a truncated stream with no explanation. Third, capture usage metadata at the end and emit it in the done event; that's where your billing and analytics hook goes. Fourth, check request.is_disconnected() on every token, because the LLM keeps generating (and billing) even after the user closes the tab if you don't.

If you're managing multiple providers behind one gateway, which is a common pattern for cost optimization and failover, combine this with the router patterns I covered in the LiteLLM Python gateway guide. LiteLLM's async streaming interface is drop-in compatible with the shape above; you just swap the provider client for litellm.acompletion(..., stream=True).

Handling client disconnects and heartbeats

The single most common production bug in streaming endpoints: the server keeps generating after the client has disconnected. In the best case you waste CPU. With an LLM, you keep paying the vendor for tokens no one will see. With a database-backed stream, you leak connections. FastAPI does not tell your generator that the client is gone. You have to ask, on every iteration, with await request.is_disconnected().

from fastapi import Request, FastAPI
from fastapi.sse import EventSourceResponse, ServerSentEvent
import asyncio

app = FastAPI()

async def guarded_stream(gen, request: Request, ping_every: float = 15.0):
    last_ping = asyncio.get_event_loop().time()
    async for event in gen:
        if await request.is_disconnected():
            return
        now = asyncio.get_event_loop().time()
        if now - last_ping > ping_every:
            yield ServerSentEvent(comment="keep-alive")
            last_ping = now
        yield event

Wrap every SSE generator in something like guarded_stream so disconnect and heartbeat behaviour is consistent across your API. A common variant uses asyncio.wait_for with a timeout, so the generator can't hang forever if the upstream provider stops responding. Production LLM APIs occasionally do this, and without a timeout you accumulate zombie tasks that hold TCP connections until the OS finally closes them.

Heartbeats matter for a second reason beyond proxy timeouts: they surface half-open connections. TCP will happily hold a socket open for minutes after the client's device has gone to sleep or its Wi-Fi has dropped. Sending a keep-alive comment every 15 seconds forces a write on the socket. A broken connection produces an error immediately, your generator sees it, and cleanup runs. Without heartbeats you leak these connections until Linux's default TCP keep-alive kicks in, two hours later.

Nginx, Cloudflare, and reverse-proxy buffering

Every streaming outage I've been paged for in the last two years was caused by an intermediate proxy buffering the response. Nginx's default proxy_buffering on holds up to 16 KB before flushing. That's roughly 30 tokens of a typical LLM response. Users see nothing for the first few seconds, then everything in a burst. The fix is per-route:

location /v1/chat/stream {
    proxy_pass http://fastapi_upstream;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_buffering off;
    proxy_cache off;
    proxy_read_timeout 300s;
    chunked_transfer_encoding on;
}

You can also (and I recommend both) send the X-Accel-Buffering: no response header from FastAPI. Nginx honours it and disables buffering for that response even if the location block doesn't. The native EventSourceResponse in FastAPI 0.135+ sets this automatically; for StreamingResponse you pass it via headers=. Honestly, I set both belt-and-suspenders and never think about it again.

Cloudflare has its own buffer, roughly 100 KB, on the Free and Pro plans. There's no header that disables it. You need to be on Enterprise with the "Enable Streaming" feature turned on, or bypass Cloudflare entirely for streaming routes (a subdomain with DNS only mode). Learn this before you launch, not during the launch. AWS CloudFront behaves similarly and needs a specific origin request policy (see the CloudFront custom origin docs) to pass streaming through.

Comparison table: SSE vs WebSocket vs NDJSON vs StreamingResponse

The primitives overlap, and the "right" pick depends on your client, your infrastructure, and your traffic pattern. This is the cheat sheet I hand new hires:

Dimension	StreamingResponse (NDJSON)	SSE (EventSourceResponse)	WebSocket
Direction	Server → Client	Server → Client	Bidirectional
Protocol	HTTP/1.1 or HTTP/2 chunked	HTTP text/event-stream	WS upgrade (ws:// or wss://)
Browser API	fetch() + reader loop	new EventSource()	new WebSocket()
Auto-reconnect	Manual	Built into EventSource	Manual
Event framing	Line-delimited JSON	Named events + data	Text or binary frames
Works through Nginx (default)	Needs proxy_buffering off	Needs proxy_buffering off	Needs upgrade headers + long timeouts
Cloudflare Free plan	~100 KB buffer	~100 KB buffer	Supported
Ideal for	Bulk data export, tool pipelines	LLM tokens, notifications, live UI	Chat with interruption, collaborative UIs
Operational complexity	Low	Low	High

My default: SSE for anything a browser will consume, NDJSON for anything another server will consume, WebSocket only when you can name the bidirectional requirement in one sentence. If you're serving both model inference and streaming endpoints from the same FastAPI service, the ML serving trade-offs in the model serving comparison of BentoML, Ray Serve, FastAPI, and Triton are worth cross-referencing. The streaming layer sits on top of those choices, not next to them.

Testing streaming endpoints

FastAPI's TestClient works for streaming, but you have to consume the response as an iterator rather than reading .text. For SSE endpoints I lean on httpx's streaming client, which mirrors real client behaviour more closely and catches bugs the sync TestClient hides.

import httpx, pytest

@pytest.mark.asyncio
async def test_chat_streams_tokens():
    async with httpx.AsyncClient(app=app, base_url="http://test") as client:
        async with client.stream("GET", "/chat?prompt=hello world") as r:
            assert r.status_code == 200
            assert r.headers["content-type"].startswith("text/event-stream")
            tokens = []
            async for line in r.aiter_lines():
                if line.startswith("data:"):
                    tokens.append(line[5:].strip())
            assert tokens  # got at least one token

Two things this test catches that a naive TestClient test would miss: the response actually streams (rather than buffering server-side and returning everything at once), and the Content-Type is set correctly. Both regressions have shipped to production on my team more than once.

For load testing, k6 has first-class SSE support as of v0.55 (k6 SSE documentation). Simulate hundreds of concurrent streams before you learn the hard way that your event loop can't keep up. The number that matters is time to first byte, not total request time. If TTFB creeps above 500 ms under load, you have a concurrency ceiling to raise before launch.

Frequently Asked Questions

Do I need sse-starlette in FastAPI 0.135 or later?

No. FastAPI 0.135.1 (March 2026) added a native EventSourceResponse at from fastapi.sse import EventSourceResponse. It sets the correct headers automatically and serializes Pydantic models on the Rust side. Keep sse-starlette only if you need its configurable ping interval or connection-limit semaphore.

Why does my FastAPI SSE endpoint work locally but buffer in production?

Almost always a reverse proxy. Nginx buffers 16 KB by default and Cloudflare buffers ~100 KB before flushing. Set proxy_buffering off in your Nginx location block and send the X-Accel-Buffering: no header from FastAPI. For Cloudflare, upgrade to a plan with streaming enabled or bypass CF for streaming routes.

Should I use SSE or WebSockets for streaming LLM responses?

SSE. It's simpler, works over plain HTTP, has native browser support with auto-reconnect, and matches the one-way "server pushes tokens" pattern that LLMs need. Choose WebSockets only when the client must send data mid-stream (interruptible chat, live cursors, voice) and you can articulate the bidirectional requirement in one sentence.

How do I detect when a client disconnects from a FastAPI stream?

Call await request.is_disconnected() inside your generator loop and break when it returns True. FastAPI does not notify your generator automatically. If you skip this check, your endpoint keeps generating tokens (and paying the LLM vendor) after the user has closed the tab.

Can I use GZipMiddleware with FastAPI streaming responses?

No. GZipMiddleware buffers the entire response before compressing, which defeats streaming. Either remove GZip entirely or exclude streaming routes with a middleware guard. Token payloads are already small enough that gzip overhead usually outweighs the bandwidth savings anyway.

What's the difference between NDJSON and SSE for streaming from FastAPI?

NDJSON is newline-delimited JSON over raw chunked HTTP, with no framing, no event types, and no reconnect. SSE adds an event framing layer (event:, data:, id:) and gives browsers EventSource with automatic reconnect. Use NDJSON for tool-to-tool pipelines (jq, DuckDB, curl consumers) and SSE for anything a browser will read.