Chat completions (POST /v1/chat/completions) · Local API Guide

Local API Guide contents

Overview

The main endpoint, compatible with OpenAI's Chat Completions API: send a conversation, get the next assistant message. It supports streaming, thinking (reasoning), images, structured JSON output, and two 5090Ai extensions: web_search answers with a live web search and knowledge answers from the documents on your Knowledge page.

Request body

Parameter	Description
`messages`arrayrequired	The conversation, oldest first. Each message has a `role` (`system`, `developer`, `user` or `assistant`) and `content`: a string, or an array of parts (see below).
`model`string	Optional. `qwen3.8-27b` or `qwen3.6-35b-a3b`. The loaded model answers; naming the other bundled model returns `400 model_not_loaded`, any other value is ignored.
`stream`boolean	Send the answer as server-sent events while it is written. Default `false`.
`stream_options.include_usage`boolean	When streaming: send `usage` in an extra last chunk with an empty `choices` array (as OpenAI does).
`max_tokens`integer	The longest answer, in tokens (`max_completion_tokens` works too). Default: until the model finishes or the context is full.
`stop`string \| string[]	Up to 16 strings that end the answer when generated.
`response_format`object	Structured output, as in OpenAI's API: `{"type": "json_object"}` for any JSON object, or `{"type": "json_schema", "json_schema": {"name": "…", "schema": {…}}}` for JSON matching your schema. The answer is always valid JSON (unless `max_tokens` cuts it short). `{"type": "text"}` is the default. See Structured output.
`enable_thinking`boolean	Let the model think before it answers. The thinking comes back in `reasoning_content`. Default `false`. Also accepted as `chat_template_kwargs.enable_thinking`.
`reasoning_effort`string	`low`, `medium` or `high` (default): how long to think. Qwen3.8-27B only.
`web_search`boolean \| string	Extension: `true` (the app's depth setting), `"quick"`, `"balanced"` or `"deep"`. Answers the last user message with a live web search. Pages linked in the message are opened directly: their text, their design (colors, fonts, layout) and a screenshot go to the model, so a link works as a design or content reference. See Extensions.
`knowledge`boolean \| string	Extension: `"all"` (or `true`), a collection name, or a collection id like `"c:3"`. Answers the last user message from your documents.
`temperature, top_p, top_k, min_p, seed, …`number	Sampling. Without them (or with `temperature` 0) answers are greedy: the same request always gets the same answer. `temperature` above 0 samples, with `top_p`, `top_k`, `min_p`, `presence_penalty`, `frequency_penalty`, `repetition_penalty`, `logit_bias` and `seed` (the same seed gives the same answer). Sampled requests run at the engine's full speed. The app's Always greedy API setting ignores these options.
Authorization header	`Authorization: Bearer <key>`, only when Require API key is on in the app.

Content parts

Instead of a string, content can be an array of parts, to send images with the text:

Part	Description
`{"type": "text", "text": "…"}`	Text.
`{"type": "image_url", "image_url": {"url": "data:image/png;base64,…"}}`	An image as a base64 data URL: PNG, JPEG, WebP, GIF, BMP or TIFF. Up to 16 images per request. Web addresses are not fetched.

Response

Without streaming, one chat.completion object:

Field	Description
`id`string	`chatcmpl-…`
`object`string	`chat.completion`
`created`integer	Unix time.
`model`string	The model that answered.
`choices[0].message`object	`role` (`assistant`), `content`, and `reasoning_content` when thinking was on.
`choices[0].finish_reason`string	`stop` (finished or hit a stop string), `length` (reached `max_tokens` or the context) or `cancelled`.
`usage`object	`prompt_tokens`, `completion_tokens`, `total_tokens` and `prompt_tokens_details.cached_tokens` (prompt tokens reused from the previous request).
`timings`object	`prompt_ms`, `prompt_per_second`, `predicted_n`, `predicted_ms`, `predicted_per_second` (generation speed), `cached_tokens`, `max_ctx`, `dropped_messages` (oldest messages left out to fit the context), `images`, `vision_ms`.
`sources`array	Only with `web_search` / `knowledge`: the sources the answer can cite. See Extensions.
`web_search`object	Only with `web_search`: `searched`, `depth`, `queries`, `pages_read`, `seconds`, `note`.

Response

{
  "id": "chatcmpl-5f0c2a9e7d1b4c3a8e6f9012",
  "object": "chat.completion",
  "created": 1792390000,
  "model": "qwen3.6-35b-a3b",
  "choices": [{
    "index": 0,
    "message": { "role": "assistant", "content": "Tokyo." },
    "finish_reason": "stop"
  }],
  "usage": { "prompt_tokens": 21, "completion_tokens": 3, "total_tokens": 24,
             "prompt_tokens_details": { "cached_tokens": 0 } },
  "timings": { "prompt_ms": 22.4, "prompt_per_second": 938.8, "predicted_n": 3,
               "predicted_ms": 4.1, "predicted_per_second": 790.2, "max_ctx": 262144,
               "dropped_messages": 0, "images": 0, "vision_ms": 0.0 }
}

Streaming

With "stream": true the answer arrives as server-sent events, each line data: {…} holding a chat.completion.chunk, the same format as OpenAI's:

The first chunk has delta.role (and, with the extensions, sources and web_search).
Then delta.content pieces, and delta.reasoning_content pieces while the model thinks.
Chunks also carry progress: {"phase": "prefill", "done", "total"} while a long prompt is read, then {"phase": "decode", "tokens", "decode_ms"}. Most clients ignore it; use it for a progress bar.
The last chunk has finish_reason, usage and timings, followed by data: [DONE].

Stream

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1792390000,"model":"qwen3.6-35b-a3b","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1792390000,"model":"qwen3.6-35b-a3b","choices":[{"index":0,"delta":{"content":"Tokyo"},"finish_reason":null}],"progress":{"phase":"decode","tokens":1,"decode_ms":1.2}}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1792390000,"model":"qwen3.6-35b-a3b","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"timings":{…},"usage":{"prompt_tokens":21,"completion_tokens":3,"total_tokens":24}}

data: [DONE]

Structured output (JSON)

Set response_format and the answer is JSON, guaranteed: the model can only write text that fits, so the reply always parses, with every required key and the right types. Use it to extract fields from documents, classify, or feed another program without retries or repair code.

json_object: any JSON object. Say in your prompt which keys you want.
json_schema: the answer matches json_schema.schema (types, required, enum, additionalProperties, nested objects and arrays, minItems/maxItems, string pattern and more). strict is accepted; the schema is always enforced. A schema that can't be used is refused with 400.
With enable_thinking the model thinks freely first (reasoning_content), then writes the JSON as content.
Works with streaming, with stop and max_tokens, and side by side with other requests. If finish_reason is length, the JSON was cut off: raise max_tokens.

Extensions: web search and your documents

web_search runs the same web search as the app's globe button: the model decides what to search for, several search engines are queried, the best pages are read, and the most relevant passages are added to the last user message. If the question doesn't need the web ("hello", maths), no search is made. knowledge does the same with the documents on your Knowledge page. Use both together and the answer draws on the web and your documents.

The last user message must contain text. It can also contain images.
Expect a few seconds before the answer starts: about 5 s for quick, longer for balanced and deep.
The answer cites its sources as [1], [2]…, matching index in sources:

Field	Description
`index`integer	The number the answer cites, as in [1] or [2][3].
`type`string	`web` or `knowledge`.
`title, url, site`string	Web sources: the page title, its address, and the site (with the publication date when known).
`document, document_id, location`string, integer, string	Knowledge sources: the file name, its id and where in it (page, slide, sheet or section).
`similarity`number	Knowledge sources: how close the passage is to the question (0–1).
`snippet`string	The start of the passage the model was given.

The same lists also come in the X-Web-Sources and X-Knowledge-Sources response headers (URL-encoded JSON), for clients that only read headers. To search without an answer, use Web search or Knowledge search.

Examples

A simple question

curl http://localhost:1234/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.6-35b-a3b",
    "messages": [
      {"role": "system", "content": "Answer in one short sentence."},
      {"role": "user", "content": "What is the capital of Japan?"}
    ]
  }'

from openai import OpenAI

client = OpenAI(base_url="http://localhost:1234/v1", api_key="not-needed")
reply = client.chat.completions.create(
    model="qwen3.6-35b-a3b",
    messages=[
        {"role": "system", "content": "Answer in one short sentence."},
        {"role": "user", "content": "What is the capital of Japan?"},
    ],
)
print(reply.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({ baseURL: "http://localhost:1234/v1", apiKey: "not-needed" });
const reply = await client.chat.completions.create({
  model: "qwen3.6-35b-a3b",
  messages: [
    { role: "system", content: "Answer in one short sentence." },
    { role: "user", content: "What is the capital of Japan?" },
  ],
});
console.log(reply.choices[0].message.content);

Streaming

curl -N http://localhost:1234/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3.6-35b-a3b", "stream": true,
       "messages": [{"role": "user", "content": "Write a haiku about GPUs."}]}'

stream = client.chat.completions.create(
    model="qwen3.6-35b-a3b",
    messages=[{"role": "user", "content": "Write a haiku about GPUs."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="", flush=True)

const stream = await client.chat.completions.create({
  model: "qwen3.6-35b-a3b",
  messages: [{ role: "user", content: "Write a haiku about GPUs." }],
  stream: true,
});
for await (const chunk of stream) process.stdout.write(chunk.choices[0]?.delta?.content ?? "");

Thinking

reply = client.chat.completions.create(
    model="qwen3.8-27b",
    messages=[{"role": "user", "content": "Is 2^61 - 1 prime? Explain briefly."}],
    extra_body={"enable_thinking": True, "reasoning_effort": "high"},
)
msg = reply.choices[0].message
print("Thinking:", msg.model_extra.get("reasoning_content"))
print("Answer:", msg.content)

const reply = await client.chat.completions.create({
  model: "qwen3.8-27b",
  messages: [{ role: "user", content: "Is 2^61 - 1 prime? Explain briefly." }],
  enable_thinking: true,
  reasoning_effort: "high",
});
console.log(reply.choices[0].message.reasoning_content, reply.choices[0].message.content);

Structured output (JSON schema)

import json

schema = {
    "type": "object",
    "properties": {
        "company": {"type": "string"},
        "total": {"type": "number"},
        "currency": {"type": "string", "enum": ["USD", "EUR", "GBP"]},
        "paid": {"type": "boolean"},
    },
    "required": ["company", "total", "currency", "paid"],
    "additionalProperties": False,
}
reply = client.chat.completions.create(
    model="qwen3.6-35b-a3b",
    messages=[{"role": "user", "content": "Invoice from Acme Ltd, total 1,240.50 EUR, paid on 3 May."}],
    response_format={"type": "json_schema", "json_schema": {"name": "invoice", "schema": schema}},
)
invoice = json.loads(reply.choices[0].message.content)   # always parses
print(invoice["company"], invoice["total"])

const reply = await client.chat.completions.create({
  model: "qwen3.6-35b-a3b",
  messages: [{ role: "user", content: "Invoice from Acme Ltd, total 1,240.50 EUR, paid on 3 May." }],
  response_format: {
    type: "json_schema",
    json_schema: {
      name: "invoice",
      schema: {
        type: "object",
        properties: {
          company: { type: "string" },
          total: { type: "number" },
          currency: { type: "string", enum: ["USD", "EUR", "GBP"] },
          paid: { type: "boolean" },
        },
        required: ["company", "total", "currency", "paid"],
        additionalProperties: false,
      },
    },
  },
});
const invoice = JSON.parse(reply.choices[0].message.content); // always parses
console.log(invoice.company, invoice.total);

An image

import base64

with open("screenshot.png", "rb") as f:
    url = "data:image/png;base64," + base64.b64encode(f.read()).decode()
reply = client.chat.completions.create(
    model="qwen3.6-35b-a3b",
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "What does this error message mean?"},
        {"type": "image_url", "image_url": {"url": url}},
    ]}],
)
print(reply.choices[0].message.content)

import fs from "node:fs";

const url = "data:image/png;base64," + fs.readFileSync("screenshot.png").toString("base64");
const reply = await client.chat.completions.create({
  model: "qwen3.6-35b-a3b",
  messages: [{ role: "user", content: [
    { type: "text", text: "What does this error message mean?" },
    { type: "image_url", image_url: { url } },
  ] }],
});
console.log(reply.choices[0].message.content);

Web search

curl http://localhost:1234/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3.6-35b-a3b", "web_search": "balanced",
       "messages": [{"role": "user", "content": "What is new in the latest Python release?"}]}'

reply = client.chat.completions.create(
    model="qwen3.6-35b-a3b",
    messages=[{"role": "user", "content": "What is new in the latest Python release?"}],
    extra_body={"web_search": "balanced"},
)
print(reply.choices[0].message.content)
print("Searched for:", reply.model_extra["web_search"]["queries"])
for s in reply.model_extra.get("sources", []):
    print(f"[{s['index']}] {s['title']} - {s['url']}")

const reply = await client.chat.completions.create({
  model: "qwen3.6-35b-a3b",
  messages: [{ role: "user", content: "What is new in the latest Python release?" }],
  web_search: "balanced",
});
console.log(reply.choices[0].message.content);
for (const s of reply.sources ?? []) console.log(`[${s.index}] ${s.title} - ${s.url}`);

Your documents (knowledge)

curl http://localhost:1234/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3.6-35b-a3b", "knowledge": "all",
       "messages": [{"role": "user", "content": "What is our refund policy for digital goods?"}]}'

reply = client.chat.completions.create(
    model="qwen3.6-35b-a3b",
    messages=[{"role": "user", "content": "What is our refund policy for digital goods?"}],
    extra_body={"knowledge": "Policies"},      # "all", a collection name, or "c:<id>"
)
print(reply.choices[0].message.content)
for s in reply.model_extra.get("sources", []):
    print(f"[{s['index']}] {s['document']}, {s['location']} ({s['similarity']:.2f})")

const reply = await client.chat.completions.create({
  model: "qwen3.6-35b-a3b",
  messages: [{ role: "user", content: "What is our refund policy for digital goods?" }],
  knowledge: "Policies",   // "all", a collection name, or "c:<id>"
});
for (const s of reply.sources ?? []) console.log(`[${s.index}] ${s.document}, ${s.location}`);

Errors

Status	Code	When
401	`invalid_api_key`	Require API key is on and the key is missing or wrong.
400	`invalid_json`	The body is not a JSON object.
400	`model_not_loaded`	The request names the other bundled model. 5090Ai runs one model at a time: switch in the app, or leave `model` out.
400	`(none)`	A parameter is invalid, for example `max_tokens` below 1 or a prompt longer than the context.
402	`trial_exhausted`	Today's Local API allowance is used up (type `insufficient_quota`). See Daily limits and plans.
503	`model_not_loaded`	No model is loaded, or it is still loading (retry in a few seconds).
502	`engine_unavailable`	The engine did not respond (for example while the app restarts it).
400	`(none)`	`response_format` is not an object, has an unknown `type`, or its schema is invalid or unsupported (the message says why).
400	`invalid_web_search`	`web_search` is not `true`, `false`, `quick`, `balanced` or `deep`.
400	`invalid_knowledge`	`knowledge` is not a string or boolean.
400	`invalid_request`	`web_search` or `knowledge` is set but the last user message has no text.
404	`collection_not_found`	No knowledge collection has that name or id.
503	`knowledge_unavailable`	The knowledge model (Qwen3-Embedding-4B) is not installed, or no model is loaded.

Requests refused with 400, 404 or 503 don't count toward your daily allowance.