LLM response streaming: UX and SSE architecture in 2026

A company launched an AI assistant for customer queries. The model generated responses in 4–6 seconds. Users clicked "Send" twice, thinking nothing was happening. After enabling streaming, the same model generated the first words in 400 ms. The total time didn’t change. The number of double clicks dropped by 80%.

This is the essence of streaming: it doesn’t speed up inference, it changes the perception of waiting.

What is streaming and why it changes UX#

A standard LLM call works like this: the client sends a query, the model generates all tokens in memory, the server sends the full response. The user sees a blank page for the entire 4–6 seconds, then suddenly the full text appears.

Token-by-token streaming reverses the transmission order. The model generates a token, the server sends it, the browser renders it. The user sees the text growing letter by letter, just like in ChatGPT. Psychologically, this is a fundamental difference: the system appears responsive, even when it’s computationally just as slow.

Two metrics matter in the context of streaming:

Without streaming, the user waits for the total time. With streaming, they wait only for TTFT, and the rest arrives progressively. For a 300-token response at 40 tok/s throughput, the total time is ~7.5 s. A TTFT of 400 ms means the user starts reading after 0.4 s, not 7.5 s.

SSE architecture: how to build it correctly#

Server-Sent Events (SSE) is an HTTP protocol where the server keeps the connection open and sends text events in the format data: {...}\n\n. The client (browser or API client) receives each event separately and renders it progressively.

Minimal server-side flow (Python/FastAPI):

from fastapi.responses import StreamingResponse

async def generate_stream(prompt: str):
    async for token in llm.stream(prompt):
        yield f"data: {json.dumps({'token': token})}\n\n"
    yield "data: [DONE]\n\n"

@app.post("/v1/chat/stream")
async def chat_stream(req: ChatRequest):
    return StreamingResponse(
        generate_stream(req.prompt),
        media_type="text/event-stream",
        headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
    )

Four things that most often break streaming in production:

Buffering proxies. Nginx buffers responses by default. The header X-Accel-Buffering: no disables buffering for a specific endpoint. Without it, the client receives the entire stream at once after completion, just like without streaming.

BFF layer with gzip. If Next.js or another BFF rewrites requests via rewrite, gzip compression buffers chunks to a threshold size. Solution: File Route Handler returning upstream.body directly, with the header Cache-Control: no-transform. At Cashcrown, we encountered this issue when building the Playground, where the standard BFF rewrite buffered SSE for browsers with gzip until we switched to a direct handler.

No heartbeat. SSE connections idle for 30–60 s are closed by load balancers and CDNs. Send a comment ": ping\n\n" every 15–20 s during long generations.

Backpressure. If the client reads slower than the server writes (e.g., slow mobile connection), the queue buffer grows. The implementation should limit the queue size per connection and close it when the threshold is exceeded, instead of holding data indefinitely. At Cashcrown, we use an asyncio.Queue(100) per connection with automatic closure when the limit is exceeded.

Guardrails on the stream: what works, what doesn’t#

Streaming complicates validation. With a full response, the guardrail can check the entire text before sending. On a stream, you only have a fragment.

Three approaches used in practice:

Input guardrail (pre-stream). Operates on the prompt before sending it to the model. Checks for injection, forbidden topics, PII. If the prompt fails, the stream never starts. This is the simplest and most effective pattern. Every query at Cashcrown goes through screen() before SSE initialization.

Output guardrail on a sliding window. Maintain a buffer of the last N tokens and check each new window for forbidden patterns. You can stop the stream midway by sending an error event. Downside: the user sees half the response, then an error message. Use only for serious violations (sensitive data, content policy).

Post-stream review for irreversible actions. If the response leads to an irreversible action (payment confirmation, sending an email, database write), never execute it based solely on the stream. Buffer the full response, run it through a full guardrail, and wait for human confirmation. Streaming is a UX layer, not a correctness layer. Irreversible decisions require full validation and human approval.

Stream cancellation and resource cleanup#

The user may click "Stop" mid-generation. On the browser side, AbortController closes the SSE connection. On the server side, listen for request.is_disconnected() (FastAPI/Starlette) or equivalent and break the generation loop.

async def generate_stream(prompt: str, request: Request):
    async for token in llm.stream(prompt):
        if await request.is_disconnected():
            break  # stop generation, release semaphores
        yield f"data: {json.dumps({'token': token})}\n\n"

Lack of cancellation handling means the model continues generating in the background, occupying GPU and concurrency semaphores, even when the client is no longer listening. With many concurrent users, this leads to pool exhaustion and increased latency for everyone.

The article LLM token cost: how to measure and optimize it describes semaphore patterns and call budgets that work with streaming pipelines.

Observability: what to measure with streaming#

Traditional HTTP metrics (total time) aren’t enough to assess streaming quality. You need four additional measurement points.

TTFT per query. Timestamp at the first token minus the request send timestamp. The target depends on the use case: for a chat assistant, below 600 ms is comfortable; for document generation, below 2 s is acceptable.

Throughput per stream. Tokens sent divided by the time from the first to the last token. A drop in throughput signals GPU overload or network backpressure.

Cancellation rate. Percentage of streams interrupted by the client before the last token. A high rate (above 15–20%) means users are losing patience. Causes: too long TTFT, poor response quality, UX issues in the interface.

Partial output errors. Streams ended with an error after sending at least one token. This is a harder category to handle than a total error, because the user sees truncated text.

The pattern for collecting these metrics in observability: for each SSE event, log telemetry (timestamp, token count, connection id) to a ring buffer. On connection close, flush to the metrics system. More about the observability layer for AI agents in monitoring AI agent quality.

When streaming doesn’t make sense#

Streaming improves UX for text responses intended for humans. It doesn’t improve—and may complicate—several scenarios:

For structured output (JSON, XML), partial output is invalid until the structure is closed. Streaming JSON fields leads to situations where the client receives {"result": "poz and must wait. Better to buffer and send the full JSON after schema validation.

For batch processing (document processing without a user interface), TTFT doesn’t matter to any human. The overhead of streaming (maintaining connections) is a cost without benefit.

For short responses (below 20–30 tokens), the difference between TTFT and total time is too small to be perceptually significant. A single-sentence classifier doesn’t need streaming.

The article local LLM: what hardware and GPU you really need discusses throughput as a criterion for hardware selection in production pipelines.

FAQ#

Does streaming speed up token generation by the model?#

No. Streaming doesn’t change the speed of token generation. The LLM generates one token at a time in an autoregressive loop regardless of the transmission mode. Streaming only changes the delivery timing to the client: each token is sent immediately after generation, instead of waiting for the entire response to finish. The effect is perceptual but significant: a TTFT of 400 ms instead of 6 s is the difference between a system that feels "responsive" and one that feels "frozen" to the user.

How to handle an error mid-stream?#

Standard HTTP has no mechanism for "error after 200 OK" in the body, because the status is already sent. SSE pattern: send a special error event (event: error\ndata: {...}\n\n) before closing the connection. The client listens for the error event separately from the message event. On the UI side, display the fragment that arrived, annotate the interruption, and offer the option to resume. Never hide the fact that the response is incomplete, because the user may make decisions based on truncated text.

Can you stream through Next.js App Router without buffering?#

Yes, but it requires a File Route Handler (app/api/.../route.ts) returning new Response(upstream.body, {...}) with headers Cache-Control: no-cache, no-transform and X-Accel-Buffering: no. The standard rewrite in next.config.js rewrites the request through the Node.js layer, which gzip-buffers the response for browsers with Accept-Encoding: gzip. The File Route Handler bypasses this layer and forwards the upstream body directly. The article LLM prompt caching describes complementary server-side optimization techniques for LLM pipelines.

How to measure TTFT in production without client access?#

Measure on the server side: timestamp before sending the first data: event minus the request receipt timestamp. This is a proxy for real TTFT (doesn’t account for network time), but sufficient for tracking trends and detecting regressions. Measure real end-to-end TTFT via client instrumentation: performance.mark('request-sent') on send, performance.mark('first-token') on the first SSE event. Export both values to the same metrics system and correlate.

What are the limitations of guardrails on a stream?#

A guardrail on full text has all the information. A guardrail on a stream only sees a prefix. Patterns requiring full-sentence context (double negatives, disguised requests) are hard to detect on a window of a few tokens. Cashcrown’s approach: full input guardrail eliminates most risks before the stream starts; output guardrail detects lexical patterns (forbidden words, card number fragments, PII). For high-risk content, the entire stream is buffered and verified before display, which negates the UX benefit of streaming. This is a conscious trade-off.