Benchmarks · 5090Ai

Chat · one answer at a time

Generation speed

Task by task, side by side

Tokens per second while writing the answer, higher is better. Five everyday tasks on each model, with standard llama.cpp running the same model on the same card. llama.cpp already runs Qwen3.6-35B-A3B much faster than Qwen3.8-27B, so its speed-ups start from a higher base: it is the faster model even where its speed-up is smaller.

Qwen3.8-27B

Smartest · tokens per second by task

Standard llama.cpp5090Ai

Qwen3.6-35B-A3B

Fastest · tokens per second by task

Standard llama.cpp5090Ai

Task	Qwen3.8-27B			Qwen3.6-35B-A3B
Task	llama.cpp	5090Ai	Speed-up	llama.cpp	5090Ai	Speed-up
ChatExplain how a CPU cache works	73	247	3.4×	191	675	3.5×
CodeWrite an asyncio web crawler	72	371	5.2×	181	886	4.9×
MathProve there are infinitely many primes	71	306	4.3×	193	807	4.2×
WritingA short science fiction story	74	139	1.9×	178	561	3.2×
Code editAdd docstrings to a Python file	73	526	7.2×	201	1,117	5.6×
Typical	~72	~304	4.2×	~186	~782	4.2×

Tokens per second. Typical speed is stated by XTO Pty Ltd from its testing; llama.cpp's typical speed is the average of the chat, code, math and writing tasks. Results vary with the prompt.

What it feels like

A full answer in seconds

Time to write a 512-token answer (about 380 words) at typical speed. Shorter is better.

Standard llama.cpp5090Ai

Prompt reading

Reading long documents

Prefill measure - how fast each model reads a ~20,000-token document before it starts answering. Higher is better.

Tokens per second

~20,000-token document

Standard llama.cpp5090Ai

At a glance

Time to read the whole ~20,000-token document

Model	llama.cpp	5090Ai
Qwen3.8-27B	~6.0 s	~6.0 s
Qwen3.6-35B-A3B	~2.7 s	~1.4 s

Qwen3.8-27B reads prompts as fast as standard llama.cpp. Qwen3.6-35B-A3B reads at about 15,000 tokens per second, so it suits very long documents best.

Beyond chat

Knowledge, images and the web

The same RTX 5090 indexes your documents, looks at your images and reads the web, all on your PC.

Knowledge (your documents)

Indexing

~20,000 tok/s

About 13,500 tokens per second when GPU memory is short.

40–250 ms

Per question, by meaning and by keyword together.

Accuracy

≥ 0.9997

Cosine similarity to the reference llama.cpp embeddings.

Model

4.3 GB

Qwen3-Embedding-4B, next to either chat model.

Images

1080p screenshot

~2,000 tokens

Of prompt, read like text.

Encoding

~0.4 s

Per 1080p image, on the GPU.

Answer speed

Unchanged

Answers about images are written at full speed.

Per request

Up to 16

Images in one message or Local API call.

Web search (Balanced, warm)

~5 sPlan and searchChoose queries, search several engines, refine
3–5 sRead pagesAbout 16 pages, in parallel
< 1 sRankPick the best passages
1–2 sRead the resultsQwen3.6-35B-A3B ~1 s (~18K tokens) · Qwen3.8-27B ~2 s (~8K tokens)

From sending a question to the answer starting: about 9–13 seconds. The first search after starting the app also loads the search model (about 10 seconds) unless web search was already switched on.

Local API · up to four calls at once

Parallel Local API · head to head

Four calls at once, against llama.cpp doing the same

Total tokens per second while one to four Local API calls write at the same time on one RTX 5090: 5090Ai against standard llama.cpp's server with the same number of parallel slots. Higher is better.

Qwen3.8-27B

Total tokens per second by calls at once

Standard llama.cpp5090Ai

Qwen3.6-35B-A3B

Total tokens per second by calls at once

Standard llama.cpp5090Ai

Calls at once	Qwen3.8-27B			Qwen3.6-35B-A3B
Calls at once	llama.cpp	5090Ai	Faster	llama.cpp	5090Ai	Faster
1One call at a time	72	216	3.0×	186	657	3.5×
22 calls at once	132*	333	2.5×	290*	892	3.1×
33 calls at once	180*	449	2.5×	360*	1,025	2.8×
44 calls at once	220*	595	2.7×	410*	1,145	2.8×

Total tokens per second while every call is writing (the parallel prompts: explanation, story, list, math). 5090Ai and llama.cpp one call at a time: measured by XTO Pty Ltd on an RTX 5090 in October 2026. * Estimated: llama.cpp with two to four calls at once (llama-server b11457 with -np 2 to -np 4 and continuous batching) is XTO Pty Ltd's estimate for the same card and models, not yet a measurement.

On top of chat speed

Two multipliers, one card

The engine makes every call faster; four calls at once multiply that again. Measured, from standard llama.cpp answering one call at a time to 5090Ai answering four.

Speed that stacksTotal tokens per second · one RTX 5090

Standard llama.cppOne call at a time

72tok/s

3.0×The 5090Ai engine

5090AiOne call at a time · chat

216tok/s

2.75×Four calls at once, on top

5090Ai Local APIFour calls at once

595tok/s

8.3×

the work of standard llama.cpp, from the same card. Every answer is exactly the one it gets on its own.

Cost vs the cloud

Tens of thousands of calls a day, no per‑token bill

Cloud APIs charge for every token in and every token out, every day. 5090Ai runs the same work on the RTX 5090 you already own and never bills a token. Pick a task and how many calls a day to see what the cloud would charge.

What the cloud would chargeThe same calls, the same tokens · list prices

Task

Calls a day

10,000

A year of work

Workload, every day	Claude Sonnet 5.5	Gemini Flash 3.8	Qwen3.8-27BOpenRouter	Qwen3.6-35B-A3BOpenRouter	5090Ai
1,000 chat replies1,200 tokens in · 400 out each	$2,336	$876	$350	$190	$0†
5,000 code tasks3,000 tokens in · 1,200 out each	$32,850	$12,319	$5,146	$2,738	$0†
10,000 document summaries10,000 tokens in · 500 out each	$91,250	$34,219	$7,482	$5,475	$0†
25,000 agent steps8,000 tokens in · 250 out each	$168,812	$63,305	$12,091	$9,581	$0†
50,000 classifications500 tokens in · 60 out each	$29,200	$10,950	$3,212	$2,008	$0†

Cloud list prices per million tokens, input / output, October 2026: Claude Sonnet 5.5 $2 / $10, Gemini Flash 3.8 $0.75 / $3.75, and on OpenRouter Qwen3.8-27B $0.10 / $2.10 and Qwen3.6-35B-A3B $0.10 / $1.00. Cost = tokens × price, before any caching or batch discounts; a month is 365 ÷ 12 days. Tokens per task are typical examples; yours will vary. Every workload in the table fits on one RTX 5090 with either model; card time is estimated from the measured prompt reading speed and the total speed with four calls at once. 5090Ai never bills per token: the Local API is free for personal use up to 100 calls a day. † Above that, or for business use, a one-time commercial license unlocks unlimited use. Electricity is not included.

Scaling

5090Ai alone: up to 2.75× the work

Total tokens per second when several Local API requests run at the same time on one RTX 5090, next to the same requests sent one by one. Every answer is exactly the one it gets on its own.

Qwen3.8-27B

Total tokens per second by requests at once

One at a timeAt once

Qwen3.6-35B-A3B

Total tokens per second by requests at once

One at a timeAt once

Requests at once	Qwen3.8-27B		Qwen3.6-35B-A3B
Requests at once	Total tok/s	Gain	Total tok/s	Gain
1One at a time	216	1.0×	657	1.0×
2	333	1.54×	892	1.36×
3	449	2.07×	1,025	1.56×
4	595	2.75×	1,145	1.74×

Through the Local API server, end to end

Qwen3.8-27B

2.0× sooner

Four calls sent together finish in about half the time of sending them one by one.

Qwen3.6-35B-A3B

1.6× sooner

Four calls sent together, at 128K context.

Same answers

Identical

Every answer matched the same request sent alone, streaming or not.

More than four

Queued

Extra calls wait for a free place and start as soon as one finishes.

See the Local API speed Start with the Local API

Measured by XTO Pty Ltd on an RTX 5090 in October 2026 with a separate set of prompts (explanation, story, list, math): greedy decoding, thinking off, short prompts. The table is total speed while every request is writing; end to end includes waiting for the slowest request. Qwen3.8-27B runs requests in parallel with its context set to 64K or less (at 128K it answers one at a time); Qwen3.6-35B-A3B at up to 128K (at 256K, its default, one at a time). Gains vary with the prompts and answer lengths.

The models

Two models, the details

Both run fully on one RTX 5090. One is loaded at a time.

	Qwen3.8-27B	Qwen3.6-35B-A3B
Best for	Hard reasoning, math, large coding tasks	Everyday chat, long documents, fast drafting
Design	Dense	Mixture of experts
Weights	Retrained, Q4 (4-bit) quantized	Retrained, Q4 (4-bit) quantized
Parameters used per token	27 billion (all)	~3 billion of 35 billion
Typical speed	~304 tok/s	~782 tok/s
Speed range by task	139–526 tok/s	561–1,117 tok/s
Prompt reading (~20K tokens)	~3,300 tok/s	~14,800 tok/s
Maximum context	131,072 tokens (128K)	262,144 tokens (256K)
GPU memory in use at maximum context	30.8 GB	27.0 GB
Load time	~49 s	~63 s
Perplexity (lower is better)	3.879	4.451
Understands images	Yes	Yes
Download size	20.3 GB	23.8 GB

Perplexity measures how well a model predicts real text: both models scored on the same text and tokens with a 512-token context. Load time runs from starting the model until it is ready to answer.

Methodology

How we measured

Both engines get the same card, the same models, the same prompts and the same settings.

	5090Ai	Standard llama.cpp
Graphics card	NVIDIA GeForce RTX 5090 (32 GB), the only GPU used
Software	5090Ai engine	llama.cpp b11457 (llama-server), default settings, all layers on the GPU, flash attention
Decoding	Greedy (temperature 0), thinking off
Answer length	512 new tokens per prompt
Runs	Each prompt twice; the faster run counts
Prompts	Chat, code, math, writing and code edit (listed in the table above), identical for both engines
Prompt reading	A ~20,000-token document, read from scratch
Parallel Local API	One to four calls at once (explanation, story, list, math): total tokens per second while every call is writing	One call at a time: measured. Two to four at once (`-np 2` to `-np 4`, continuous batching): estimated by XTO Pty Ltd
Typical speed	Stated by XTO Pty Ltd from its testing	Average of chat, code, math and writing

Speeds measured by XTO Pty Ltd on an RTX 5090 in October 2026. Your results will vary with the prompt, the answer length, the context size, the driver and other programs using the GPU.

Download free for Windows Back to the overview