Fast alone.
Unreal four at once.
Chat on 5090Ai already writes up to 7.2× faster than standard llama.cpp. Its OpenAI-compatible Local API goes further: up to four calls at the same time on your one RTX 5090, at up to 1,145 tokens per second in total. That is up to 8.3× the work of standard llama.cpp answering one call at a time (Qwen3.8-27B: 595 against 72 tokens per second), and still around 2.7–2.8× when llama.cpp runs four at once too.* For four at once, set the context on the app's Models page to 128K or less on Qwen3.6-35B-A3B (its default is 256K) or 64K or less on Qwen3.8-27B; at longer contexts calls are answered one at a time.
At the measured total speed with four calls at once; llama.cpp with four at once is estimated.*
The engine makes it fast. Parallel makes it unreal.
The Local API's speed doesn't replace chat speed, it sits on top of it. The custom engine first makes every call several times faster than standard llama.cpp; then four calls at once multiply that again, on the same card.
the work of standard llama.cpp, from the same card. Every answer is exactly the one it gets on its own.
Even against llama.cpp running four at once
Give standard llama.cpp's server four parallel slots as well and 5090Ai still gets through the same work in a fraction of the time.
595 tok/s against about 220* on standard llama.cpp with four slots.
1,145 tok/s against about 410* on standard llama.cpp with four slots.
25,000 agent steps a day would cost about $168,812 a year on Claude Sonnet 5.5. See the cost
Qwen3.8-27B
Qwen3.6-35B-A3B
Tens of thousands of calls a day, no per‑token bill
Cloud APIs charge for every token in and every token out, every day. 5090Ai runs the same work on the RTX 5090 you already own and never bills a token. Pick a task and how many calls a day to see what the cloud would charge.
Cloud list prices per million tokens, input / output, October 2026: Claude Sonnet 5.5 $2 / $10, Gemini Flash 3.8 $0.75 / $3.75, and on OpenRouter Qwen3.8-27B $0.10 / $2.10 and Qwen3.6-35B-A3B $0.10 / $1.00. Cost = tokens × price, before any caching or batch discounts; tokens per task are typical examples. Card time is estimated from the measured prompt reading speed and the total speed with four calls at once. 5090Ai never bills per token: the Local API is free for personal use up to 100 calls a day; above that, or for business use, a one-time commercial license unlocks unlimited use. Electricity is not included.
One card. About 99 million tokens a day.
Running around the clock with four calls at once, Qwen3.6-35B-A3B writes about 99 million tokens a day on a single RTX 5090. Standard llama.cpp manages about 16 million one call at a time, and about 35 million with four slots.*
Qwen3.6-35B-A3B
Qwen3.8-27B
Four in flight, all day long
Anything that sends more than one request at a time gets through it faster: batch jobs, pipelines, agents and the tools your team already uses.
Summarize, classify and translate a folder of files four at a time.
Parallel tool calls and sub-agents run side by side instead of waiting in line.
Draft replies to four customers at once while the queue keeps moving.
Turn invoices, forms and emails into structured data, many records at a time.
Product descriptions, listings and reports for a whole catalog in one run.
Run whole prompt suites and regression tests in a fraction of the time.
Every answer matches the same call sent on its own, streaming or not.
Extra calls wait for a free place and start the moment one finishes.
Any OpenAI client works: change the base URL, nothing else.
No rate limits, no per-token bills, nothing leaves your machine.
Just send them together
No special setup: send several requests at the same time and 5090Ai writes them in parallel. Here, four summaries at once.
import asyncio from openai import AsyncOpenAI client = AsyncOpenAI(base_url="http://localhost:1234/v1", api_key="not-needed") async def ask(prompt): reply = await client.chat.completions.create( model="qwen3.6-35b-a3b", messages=[{"role": "user", "content": prompt}]) return reply.choices[0].message.content async def main(): reports = ["Summarize report A", "Summarize report B", "Summarize report C", "Summarize report D"] for answer in await asyncio.gather(*(ask(r) for r in reports)): # all four at once print(answer) asyncio.run(main())
import OpenAI from "openai"; const client = new OpenAI({ baseURL: "http://localhost:1234/v1", apiKey: "not-needed" }); const ask = async prompt => (await client.chat.completions.create({ model: "qwen3.6-35b-a3b", messages: [{ role: "user", content: prompt }], })).choices[0].message.content; const reports = ["Summarize report A", "Summarize report B", "Summarize report C", "Summarize report D"]; const answers = await Promise.all(reports.map(ask)); // all four at once answers.forEach(a => console.log(a));
for r in A B C D; do curl -s http://localhost:1234/v1/chat/completions \ -H "Content-Type: application/json" \ -d "{\"messages\": [{\"role\": \"user\", \"content\": \"Summarize report $r\"}]}" & done wait # all four run at once
How these were measured. 5090Ai: measured by XTO Pty Ltd on an RTX 5090 in October 2026, total tokens per second
while every call is writing (the parallel prompts: explanation, story, list, math; greedy decoding, thinking off, short prompts).
Standard llama.cpp one call at a time: measured on the same card (llama-server b11457, default settings, all layers on the GPU, flash
attention). * Estimated: standard llama.cpp with two to four calls at once (-np 2 to -np 4, continuous
batching) is XTO Pty Ltd's estimate for the same card and models, not yet a measurement. Tokens a day assume the speed is held around
the clock; the real-time race uses each engine's total speed with four calls at once. Qwen3.8-27B answers calls in parallel with its
context set to 64K or less (at 128K it answers one at a time); Qwen3.6-35B-A3B at up to 128K (at 256K, its default, one at a time). Results vary with the prompts, the answer
lengths and the context. The Local API includes 100 free prompts a day; for business or heavier use, unlock unlimited use inside the app.
Full methodology.