Streaming

Stream OpenAI Chat Completions or Anthropic Messages from Prism.

Streaming returns model output as it is generated. Set stream: true, process each event in order, and close the response when the terminal event arrives.

OpenAI Chat Completions

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.PRISM_API_KEY,
  baseURL: "https://api.prisminference.com/v1",
});

const stream = await client.chat.completions.create({
  model: "prism-glm53",
  messages: [{ role: "user", content: "Review this TypeScript diff: ..." }],
  stream: true,
  stream_options: { include_usage: true },
});

for await (const chunk of stream) {
  const delta = chunk.choices[0]?.delta?.content;
  if (delta != null) {
    process.stdout.write(delta);
  }

  if (chunk.usage != null) {
    console.error(`\n${chunk.usage.total_tokens} tokens`);
  }
}

Direct HTTP responses use Server-Sent Events:

curl -N "https://api.prisminference.com/v1/chat/completions" \
  -H "Authorization: Bearer $PRISM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "prism-glm53",
    "messages": [{"role": "user", "content": "Explain this diff: ..."}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

Each event starts with data:. The final event is data: [DONE]. A final usage chunk may have an empty choices array.

Anthropic Messages

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({
  apiKey: process.env.PRISM_API_KEY,
  baseURL: "https://api.prisminference.com",
});

const stream = client.messages.stream({
  model: "prism-glm53",
  max_tokens: 1024,
  messages: [{ role: "user", content: "Explain this diff: ..." }],
});

stream.on("text", (text) => {
  process.stdout.write(text);
});

const message = await stream.finalMessage();
console.error(`\n${message.usage.output_tokens} output tokens`);

Anthropic streams typed events including message_start, content_block_delta, message_delta, and message_stop. Use the official SDK to assemble content blocks and tool inputs.

Disconnects and retries

A disconnected stream may already have produced billable tokens. Do not append a blind retry to the partial assistant message.

1. Record whether the terminal event was received.
2. If no user-visible output was consumed, retry with exponential backoff and jitter.
3. If partial output was consumed, either show it as interrupted or restart the turn as a new request.
4. For 429, honor Retry-After before retrying.
5. Do not retry 400 or 401 until the request or credential is corrected.

Use a stable prompt_cache_key for repeated turns in one OpenAI conversation. This improves cache affinity but does not make a request idempotent.