AI
11/16/2026
10 min read

How to Stream LLM Responses in Node.js With Server-Sent Events

How to Stream LLM Responses in Node.js With Server-Sent Events

A user types a question into your AI feature and stares at a spinner for 9 seconds. Then the whole answer appears at once. Nothing was broken, but it felt broken. On a slow mobile connection, many users will have refreshed the page or closed the tab before the answer arrived.

Streaming fixes that. The model sends its answer a few words at a time, and your backend passes each piece to the user as it arrives. This guide shows how to stream LLM responses in Node.js from end to end: read the stream from the provider, relay it to the browser with server-sent events, handle errors and cancelled requests, and practice for free with a local model.

Why Stream LLM Responses at All

A language model writes one token at a time. If tokens are new to you, our guide to LLM tokens explains the unit. Without streaming, your server waits for the last token before it sends anything. With streaming, the first words reach the user while the rest is still being written.

Three things change for the better:

  • The wait feels shorter. The total time is about the same, but the user starts reading almost immediately.
  • Users can stop early. If the answer is going the wrong way, the user can cancel, and your backend can stop the model.
  • Long answers stop timing out. Anthropic's documentation says its SDK requires streaming for requests with large output limits, to avoid HTTP timeouts.

Streaming does not make the model cheaper or smarter. It changes when the user sees the work. The cost control comes from what you do when a user leaves, which we cover below.

How Streaming Works Under the Hood

Most LLM providers stream with server-sent events (SSE). The server answers with the text/event-stream content type and keeps the HTTP response open, writing small blocks of text. MDN's guide to server-sent events describes the format: each message uses fields such as event: and data:, and messages are separated by a pair of newline characters. A line that starts with a colon is a comment, which servers send to keep a quiet connection alive.

On the Claude Messages API, you set "stream": true on the request. Anthropic's streaming documentation, read on November 16, 2026, lists the order of events:

  1. message_start carries the message with empty content.
  2. For each content block, a content_block_start, any number of content_block_delta events, and a content_block_stop.
  3. One or more message_delta events with the stop reason and token usage.
  4. A final message_stop.

The text arrives inside the deltas, as {"type": "text_delta", "text": "..."}. The documentation also warns that ping events can appear anywhere, that new event types may be added, and that your code should handle unknown types gracefully. Other providers use different event names for the same idea. OpenAI's streaming guide, for example, sends text in response.output_text.delta events.

Your backend sits in the middle. It reads one stream from the provider and writes a second, simpler stream to the browser. That middle layer is where you keep your API key secret, check the user, cap the cost, and decide what the browser should see.

What You Need

  • Node.js 18 or later, which has fetch and readable streams built in. We tested with Node.js 22.
  • Express for the HTTP server. We used Express 5.3.0.
  • The Anthropic TypeScript SDK, installed with npm install @anthropic-ai/sdk express. We used version 1.8.0.
  • An Anthropic API key in the ANTHROPIC_API_KEY environment variable, or Ollama if you want to follow along without a paid key.

Hosted APIs bill in dollars and usually need an international card. If you do not have one yet, skip to the Ollama section and build the whole flow locally first. Nothing in the browser code changes when you switch providers later.

Add "type": "module" to your package.json so the examples can use import.

Step 1: Build the Streaming Endpoint

This server accepts a question, opens a stream to Claude, and relays every piece of text to the client as its own server-sent event. Create the file server.js:

// server.js
import express from "express";
import Anthropic from "@anthropic-ai/sdk";

const app = express();
app.use(express.json());

const anthropic = new Anthropic(); // reads ANTHROPIC_API_KEY from the environment

function sendEvent(res, event, data) {
  res.write(`event: ${event}\ndata: ${JSON.stringify(data)}\n\n`);
}

app.post("/api/chat", (req, res) => {
  res.writeHead(200, {
    "Content-Type": "text/event-stream",
    "Cache-Control": "no-cache",
    "Connection": "keep-alive",
    "X-Accel-Buffering": "no",
  });

  const stream = anthropic.messages.stream({
    model: process.env.CLAUDE_MODEL ?? "claude-haiku-5-5",
    max_tokens: 1024,
    messages: [{ role: "user", content: req.body.message }],
  });

  const heartbeat = setInterval(() => res.write(": ping\n\n"), 15000);

  function finish(event, data) {
    clearInterval(heartbeat);
    if (res.writableEnded || res.destroyed) return;
    sendEvent(res, event, data);
    res.end();
  }

  stream.on("text", (delta) => sendEvent(res, "token", { text: delta }));
  stream.on("end", () => finish("done", {}));
  stream.on("abort", () => console.log("client left, generation stopped"));
  stream.on("error", (err) => {
    console.error("stream failed:", err.message);
    finish("error", { message: "The model stopped before finishing. Please try again." });
  });

  res.on("close", () => {
    clearInterval(heartbeat);
    if (!res.writableEnded) stream.abort();
  });
});

app.listen(3000, () => console.log("listening on http://localhost:3000"));

Here is what each part does:

  • The headers turn the response into an event stream. X-Accel-Buffering: no tells Nginx not to buffer this response, which the Nginx proxy module documents as a way to switch buffering off per response.
  • messages.stream() is the SDK helper that sets stream: true for you and parses the raw events. Its text event hands you only the new text, so you never parse content_block_delta yourself.
  • max_tokens is a hard cap on how much the model may write. Our guide to LLM temperature and max tokens explains why every call needs one.
  • The model name comes from an environment variable. claude-haiku-5-5 was listed on Anthropic's models overview on November 16, 2026. Model names change, so check that page and keep the name out of your code.
  • The heartbeat writes a comment line every 15 seconds. Browsers ignore it, but proxies and load balancers see traffic and keep the connection open.

The rest of the code handles the two things that go wrong in production. We cover them in Step 3.

Step 2: Read the Stream in the Client

The browser's built-in EventSource object reads server-sent events, but its constructor only takes a URL and a withCredentials option, so there is nowhere to put a request body. Your chat endpoint needs the user's message in a POST body, so read the stream with fetch instead. Create the file read-stream.js:

// read-stream.js
export async function readChatStream(message, onToken) {
  const response = await fetch("http://localhost:3000/api/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ message }),
  });

  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  let buffer = "";

  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    buffer += decoder.decode(value, { stream: true });

    const frames = buffer.split("\n\n");
    buffer = frames.pop();

    for (const frame of frames) {
      if (frame.startsWith(":")) continue; // heartbeat comment
      const event = frame.match(/^event: (.*)$/m)?.[1];
      const data = frame.match(/^data: (.*)$/m)?.[1];
      if (event === "token") onToken(JSON.parse(data).text);
      if (event === "error") throw new Error(JSON.parse(data).message);
      if (event === "done") return;
    }
  }
}

The important line is buffer = frames.pop(). Network chunks do not respect message boundaries. One chunk can hold two events, or half of one. The function splits on the blank line that ends each event, handles the complete ones, and keeps the unfinished tail for the next chunk.

The same function runs in the browser and in Node.js. In a page, pass an onToken callback that appends text to the chat bubble. To try it from the terminal, start the server with node server.js, then send a request with curl:

curl -N -X POST http://localhost:3000/api/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "In one sentence, what does Paystack send when a payment succeeds?"}'

The -N flag turns off curl's own buffering, so you see each event as it arrives:

event: token
data: {"text":"Paystack "}

event: token
data: {"text":"sends "}

The exact words depend on the model. The shape does not.

Step 3: Handle Errors and Cancelled Requests

Two failures are specific to streaming, and both are easy to miss in testing.

The Model Fails Halfway Through

Once you have sent the 200 status and the first tokens, you cannot change the status code. Anthropic's documentation shows that an error can arrive inside the stream as an event, for example an overloaded_error that would normally be an HTTP 529. The SDK turns that into an error event on the stream.

In server.js, the error handler sends a final error event with a message that is safe to show, then closes the response. The client throws, so your interface can keep the partial answer, show a retry button, and avoid pretending the answer was complete. Log the real error on the server. Never send raw provider errors to the browser.

The User Leaves

Users close tabs, lose signal, and switch apps. If your server keeps the upstream request running, the model keeps writing an answer nobody will read, and you pay for every output token.

The res.on("close") handler fixes that. When the connection closes before your server has ended the response, it calls stream.abort(), which cancels the request to the provider. The SDK then emits an abort event. Keep a listener on it: in our test with SDK 1.8.0, aborting a stream that had no abort listener crashed the Node.js process with an unhandled APIUserAbortError.

The finish() helper checks res.writableEnded and res.destroyed before it writes, because the SDK can still emit end after an abort. Without the check, your server tries to write to a socket that no longer exists.

Step 4: Stream From a Local Model With Ollama

Ollama runs open-weight models on your own machine, with no per-token bill. Its /api/chat endpoint streams by default. The Ollama API documentation lists stream as a boolean that defaults to true, with the response sent as newline-delimited JSON (application/x-ndjson). Each object carries a slice of text in message.content, and the last one has "done": true.

That is a different format from server-sent events, so it needs its own reader. Create the file ollama-stream.js:

// ollama-stream.js
export async function* streamOllama(prompt, model = "llama3.2") {
  const response = await fetch("http://localhost:11434/api/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ model, messages: [{ role: "user", content: prompt }] }),
  });

  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  let buffer = "";

  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    buffer += decoder.decode(value, { stream: true });

    const lines = buffer.split("\n");
    buffer = lines.pop();

    for (const line of lines) {
      if (!line.trim()) continue;
      const chunk = JSON.parse(line);
      if (chunk.done) return;
      yield chunk.message.content;
    }
  }
}

It uses the same buffering trick as the client, splitting on single newlines instead of blank lines. Because it is an async generator, you consume it with for await (const text of streamOllama("Hello")) { ... }. To plug it into your endpoint, replace the Anthropic call in server.js with a loop over streamOllama() that calls sendEvent() for each piece. Pull a model first with ollama pull llama3.2, or change the model name to one you have.

For cancellation, pass an AbortController signal to fetch and call abort() in the same res.on("close") handler. The browser side does not change at all, which is the point of keeping your own simple event format between the server and the client.

Production Checklist

Streaming code that works on your laptop can still fail behind real infrastructure. Check these before you ship:

ProblemWhat you seeFix
A proxy buffers the responseNothing for seconds, then the whole answer at onceSend X-Accel-Buffering: no for Nginx, and check your host's documentation for its own buffering setting.
Compression middleware holds outputThe same delayed burstExclude the streaming route from compression, or flush after each write if your middleware supports it.
Idle connections are closedLong answers cut off partwayKeep the heartbeat comment and raise idle timeouts on your load balancer.
Too many open streamsExtra browser tabs hangMDN notes that when SSE is not used over HTTP/2, the limit is 6 open connections per browser and domain. Serve over HTTP/2, where the limit is negotiated and defaults to 100 streams.
Serverless time limitsStreams end at a fixed timeCheck your platform's maximum response duration before you choose it for long answers.
No authenticationAnyone can spend your API budgetCheck the user's session before you open the upstream stream, and rate limit per user.

Store the full answer once the stream ends, not each token. The SDK's finalMessage() method returns the complete message, including token usage, which you need for cost tracking. When answers look wrong after streaming, our guide on how to debug AI backend systems shows how to trace a request end to end.

How We Tested This Code

We ran every block on November 16, 2026, with Node.js 22, Express 5.3.0, and @anthropic-ai/sdk 1.8.0. We did not call the paid API. Instead, we pointed the SDK at a local server that replays the event sequence from Anthropic's streaming documentation, including a mid-stream overloaded_error. We checked three cases: a full answer, a client that disconnects early, and a failure halfway through. The Ollama reader was tested against a local server that sends Ollama's documented line format in deliberately uneven slices, to prove the buffering works when a line arrives in pieces.

Once streaming works, the next steps are retries when the provider is busy, conversation history, and the rest of a production AI backend. The AI Engineering course covers these in order, from AI foundations and prompt engineering to RAG pipelines, agents, and production concerns. If you want to see a complete chat service built around the same ideas in Python, read our guide to designing a production-grade AI chat service with FastAPI.

Sources

Summary

Streaming lets users see an LLM's answer while it is being written. Your Node.js backend reads the provider's stream, relays each piece of text to the browser as a server-sent event, and keeps the API key, the cost cap, and the error handling on the server.

Set the event-stream headers, send a heartbeat, and turn off proxy buffering. Read the stream in the client with fetch and a buffer that handles split chunks. Send a safe error event when the model fails mid-answer, and abort the upstream request when the user leaves so you stop paying for tokens. Practice the whole flow for free with Ollama, then switch providers without touching the browser code.

Frequently Asked Questions

What Does It Mean to Stream an LLM Response?

Streaming means the model's answer is sent to you in small pieces while it is still being generated, instead of as one block at the end. Anthropic's documentation describes setting stream to true so the response arrives incrementally through server-sent events. Your backend can pass each piece to the user as soon as it lands.

Does Streaming Make an LLM Faster?

No. The model takes about the same time to write the full answer. Streaming changes when the user sees the first words, so the wait feels shorter and the user can stop reading or cancel early. It does not change how many tokens are generated or what they cost.

What Is an SSE Stream?

Server-sent events (SSE) is a simple format for pushing text from a server to a client over one HTTP response. The server replies with the text/event-stream content type and writes small blocks of lines such as event: and data:, each block ending with a blank line. MDN documents the format in its guide to server-sent events.

Should I Use Server-Sent Events or WebSockets for LLM Streaming?

For a chat answer that flows one way, from your server to the user, server-sent events are usually enough. They run over plain HTTP, work through most proxies, and need no extra protocol. WebSockets make sense when both sides send messages continuously, such as live collaboration or voice.

Why Is My SSE Stream Hanging During a Long API Call?

The most common cause is a proxy or middleware holding the response in a buffer, so nothing reaches the client until the end. Send the X-Accel-Buffering: no header for Nginx, exclude the route from compression, and send a comment line such as : ping every few seconds so idle connections are not closed.

How Do I Stop Paying for Tokens After the User Leaves?

Listen for the response's close event in your server and abort the upstream request when it fires before you finished. In the Anthropic SDK that is a call to stream.abort(). Without it, the model can keep generating an answer nobody will read.

Can I Stream From a Local Model for Free?

Yes. Ollama runs open-weight models on your own machine, and its chat endpoint streams by default, sending one JSON object per line. There is no per-token bill, so it is a cheap way to practice the full streaming flow before you pay for a hosted API.

Tags

Enjoyed this article?

Subscribe to our newsletter for more backend engineering insights and tutorials.