Local API Guide

Quick start: step by step

5090Ai includes a local server that speaks the OpenAI API, so your own scripts, editors, agents and tools can use the same models, at the same speed, on your own PC. Follow these steps once; after that it's just a base URL.

1. Install 5090Ai and load a model

   Install the app and download a model (see the install and first start guides). The Local API uses whichever model is loaded: Qwen3.8-27B (smartest) or Qwen3.6-35B-A3B (fastest).

2. Turn on the Local API server

   In the app, click API in the menu on the left and turn the switch to Running. The address is shown underneath: http://localhost:1234/v1. Port 1234 is the default; change it on the same page if another program uses it.

3. Choose who can connect

   By default only programs on this PC can connect, and no key is needed. On the Local API page you can also turn on:

   • Require API key: clients must send Authorization: Bearer <key>. Copy the key from the Local API page.
   • Serve on local network: other devices on your network connect to http://<this PC's IP>:1234/v1.
   • Allow browser requests (CORS): for web pages that call the Local API from a browser.

4. Check the connection

   List the models. If you get JSON back, you're connected; loaded: true marks the model that will answer.

   curlPythonJavaScript

   Copy

   curl http://localhost:1234/v1/models

5. Send your first chat request

   Any OpenAI client works: point it at the base URL. The model is optional; the loaded model answers.

   curlPythonJavaScript

   Copy

   curl http://localhost:1234/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model": "qwen3.6-35b-a3b", "messages": [{"role": "user", "content": "Hello! Who are you?"}]}'

6. Stream the answer

   Add stream: true to show the answer while it's written. At several hundred tokens per second, it appears almost at once.

   PythonJavaScript

   Copy

   stream = client.chat.completions.create(
       model="qwen3.6-35b-a3b",
       messages=[{"role": "user", "content": "Explain recursion with an example."}],
       stream=True,
   )
   for chunk in stream:
       if chunk.choices:
           print(chunk.choices[0].delta.content or "", end="", flush=True)

7. Add web search or your own documents

   Two extra fields turn on the app's best features: web_search for current information from the web, and knowledge to answer from the documents on your Knowledge page. The response lists the sources the answer cites.

   PythonJavaScript

   Copy

   reply = client.chat.completions.create(
       model="qwen3.6-35b-a3b",
       messages=[{"role": "user", "content": "Summarise this week's AI news."}],
       extra_body={"web_search": "balanced"},       # or {"knowledge": "all"}
   )
   print(reply.choices[0].message.content)
   for s in reply.model_extra.get("sources", []):
       print(s["index"], s.get("title") or s.get("document"), s.get("url", ""))

8. Keep an eye on your daily allowance

   The Local API includes 100 free prompts a day. The counter at the top right of the app's Local API page shows how many you've used today. Need more? See Daily limits and plans.

Authentication and network

The Local API server listens on localhost (this PC only) unless Serve on local network is on. Every endpoint under /v1 needs the API key when Require API key is on:

Authorization: Bearer 5090-sk-…

• With the OpenAI SDKs, pass the key as api_key / apiKey. Without a required key, any value works (the SDKs need one).
• New key on the Local API page replaces the key: update your clients.
• With Allow browser requests (CORS) on, browsers may call the Local API from any page, and can read the X-Web-Sources and X-Knowledge-Sources headers.
• The server speaks plain HTTP. To reach it from outside your network, put it behind your own HTTPS reverse proxy or VPN.

Models, context and speed

• 5090Ai runs one model at a time on the GPU. Switching (in the app) takes about 45 seconds; the Local API never switches models by itself. Leave out model, or use the loaded model's id: qwen3.8-27b or qwen3.6-35b-a3b.
• Context: up to 128K tokens on Qwen3.8-27B and 256K on Qwen3.6-35B-A3B (set on the app's Models page). If a conversation is longer, the oldest messages are left out and timings.dropped_messages says how many.
• Up to four requests are answered at the same time, each with exactly the answer it would get on its own; more wait in a queue and start in order. Answering several at once needs free graphics memory: Qwen3.8-27B does it with its context set to 64K or less, Qwen3.6-35B-A3B at 128K or less (with longer contexts, including the 27B's 128K and the 35B-A3B's 256K defaults, requests are answered one at a time). See Local API speed.
• Repeating the start of a conversation is fast: the prompt is reused (usage.prompt_tokens_details.cached_tokens).
• Without sampling options answers are deterministic: the same request gets the same answer. Set temperature above 0 to sample (with top_p, top_k, min_p, penalties and seed), at the engine's full speed.
• Each response's timings.predicted_per_second is the generation speed in tokens per second.

Streaming

"stream": true returns server-sent events in OpenAI's format, ending with data: [DONE]. The last chunk carries usage and timings; with stream_options.include_usage, usage comes in a separate final chunk with empty choices, as with OpenAI. Chunks also carry a progress object you can use for a progress bar on long prompts. Details: chat completions, streaming.

Thinking

With "enable_thinking": true, the model reasons step by step before it answers: better for maths, logic and hard code, but slower. The reasoning comes back in message.reasoning_content (delta.reasoning_content when streaming) and the answer in content. On Qwen3.8-27B, reasoning_effort (low, medium, high) sets how long it thinks.

Structured output (JSON)

Add "response_format": {"type": "json_object"}, or {"type": "json_schema", "json_schema": {"name": "…", "schema": {…}}} with your JSON Schema, and the answer is guaranteed to be valid JSON (matching the schema), the same as OpenAI's structured outputs. It works on both models, with streaming and with thinking (the model thinks first, then writes the JSON). Ideal for extraction, classification and pipelines: no retries, no repair code. Example: structured output.

Images

Both models see images. Send them as image_url content parts with a base64 data URL (PNG, JPEG, WebP, GIF, BMP or TIFF), up to 16 per request. Web addresses are not fetched: download the image and send its bytes. A 1080p screenshot costs about 2,000 prompt tokens. Example: an image.

Web search and your documents

Add "web_search" and/or "knowledge" to a chat request and 5090Ai does the research before the model answers, exactly like the app. The response has sources (numbered like the citations in the answer) and, for web search, web_search with the queries it ran.

Daily limits and plans

Chat in the 5090Ai app is free and unlimited. The Local API includes 100 free prompts per day (your local calendar day; the count resets at midnight). Anything over 100 Local API prompts per day is considered commercial use, which needs a one-time license:

Licenses are bought in the app (click the counter on the Local API page, or Settings → License → API plans) and work on up to 2 PCs. What counts:

When the allowance is used up, generation requests get 402 with code trial_exhausted (type insufficient_quota) until midnight.

Errors

Errors use OpenAI's format, so SDKs raise their usual exceptions:

{ "error": { "message": "Model 'qwen3.8-27b' is not loaded. 5090Ai runs one model at a time; …",
             "type": "invalid_request_error", "code": "model_not_loaded" } }

Compatibility

• Works with the official OpenAI SDKs and most tools that let you set an OpenAI-compatible base URL: use http://localhost:1234/v1 and any API key (or yours, if required).
• Extra fields (web_search, knowledge, enable_thinking) go in extra_body with the Python SDK, and straight in the request object with the JavaScript SDK. Extra response fields are in model_extra (Python) or on the object (JavaScript).
• Not supported: tool / function calling, several choices per request (n), log probabilities, audio, and the Responses and Assistants APIs.

Endpoint reference

Every endpoint has its own page with the parameters, the response, examples in curl, Python and JavaScript, and the errors.