ai.2it.onl

TL;DR

Concurrency scales on consumer hardware — pooled agents multiplied throughput 8.7x at best. The top all-in result was MiniCPM5 1B at 983 tok/s (433% of its solo speed). The worst was a flatline: Qwen3.5 0.8B with MTP enabled never scaled at all (~180 tok/s at any concurrency) because multi-token prediction serializes requests — with MTP off the same model hit 787 tok/s. Thinking models hide a huge share of their budget in reasoning_content (up to 100%), and time-to-first-token degrades from milliseconds to tens of seconds under load. Errors across all 456 runs: zero.

Test rig

One desktop, fully loaded while every run executed. This is a dirty daily-driver environment — not a clean lab bench — and that was the point: the results are what real consumer hardware actually does.

LM Studio configuration

Identical server settings for every model — differences (MTP, KV cache quant, context caps) are called out per model on its page.

Methodology

The harness is a zero-dependency Node agent loop (benchmark/benchmark.js) plus a sweep driver (benchmark/sweep.js) streaming against LM Studio’s local OpenAI-compatible endpoint. Each model got 24 sequential runs — one per concurrency level 1–24 — of 60 seconds wall time. Each agent holds a 12,000-token budget, runs a multi-turn loop until budget or timeout, and samples temperature 0.9.

Token-level timing captures TTFT (time to first token), per-stream token gaps, content vs reasoning tokens, and per-agent/combined throughput. Reasoning is counted differently per model family: reasoning_content streams (MiniCPM5, Qwen-DeepSeek, Chronos, Qwen3.5), inline <think>...</think> tags stripped from content (Gemma Heretic), or not at all (the clean non-thinking models). Per-model pages note which.

One honesty flag: some per-agent numbers carry * — agents that sat queued for ~11–16s and then generated in a short burst report inflated per-agent tok/s. Those are timing artifacts; combined throughput and TTFT are the trusted columns.

Leaderboard — side by side

Podium rows mark the top 3 by peak combined throughput. Solo = concurrency 1 · Peak = highest combined all-token tok/s across 1–24 · Scaling = peak as % of solo · Content/Reasoning @24 = token mix at 24 agents · Every model page has the full table and raw data.

Footnotes
† Qwen2.5 0.5B ×2 (parallel): the same model loaded as two LM Studio instances running in parallel (two API slots sharing the GPU), with N agents on each instance at every level — total concurrency 2–48. For comparability it is listed here at its ≤24-agent window only (peak inside that window: 1020 tok/s at 22 agents); the full 2–48 sweep is on its own page. Its max achievement: 1815 tok/s at 38 total agents — roughly 2x the single-instance ceiling of ~944 tok/s.
‡ MiniCPM5 1B ×2 (parallel): the same thinking model as two parallel instances — listed here at its ≤24-agent window (peak 843 tok/s at 22 agents). Its max achievement: 1698 tok/s at 48 total agents — vs the single-instance peak of ~983 tok/s, and the highest all-token number recorded in any run so far, parallel or not.

Side-by-side charts

All charts below are generated from the sweep JSON in the site’s dark theme; original matplotlib exports are downloadable per model. The parallel runs (†‡) appear here at their ≤24-agent window for comparability — the full 2–48 sweeps are on their own pages.

Models — individually

Every model gets its own page: the complete 24-row table, key findings, all 8–9 charts, and raw CSV/JSON downloads.

Key takeaways

1. Concurrency pays, up to a wall. Every model except MTP-on Qwen3.5 scaled 2.7–8.7x — the one outlier at the low end being Qwen2.5 0.5B, which barely needs concurrency (527 tok/s at just 2 agents). Diminishing returns beyond ~13–16 agents for most; a few (MiniCPM5, Falcon3, Qwen2.5 0.5B) kept climbing to 20+.
2. MTP serializes — it is not a throughput win for concurrency. The cleanest result of the whole sweep: same model, same box, MTP off = 3.3x scaling; MTP on = flatline. The decode path cannot batch.
3. Thinking tokens are overhead, not content. At 24 agents, reasoning share ran 45–100%. If your agents need visible answers, count content-only throughput.
4. TTFT is the agent-latency wall. Prefill-heavy thinking models push first-token latency to 20–60s under load — Qwen3 4B waits up to the full 60s window — unusable for interactive orchestration, fine for batch pipelines.
5. Reliability is not the bottleneck. 288 runs, zero hard errors. Timeouts (budget exhaustion) are the only non-OK outcome.
6. A 9950X3D + 32GB + 8GB VRAM can host the whole swarm. 24 concurrent 1B–3B agents ran with KV on GPU and no OOMs — this is genuinely consumer-hardware territory.

What’s next — bigger models & real-world use

This sweep is a living dataset — it’s not done yet. I plan to keep testing and adding models to the list as they land, so keep a lookout for new entries in the sidebar. Next up on the bench:

• Bigger models — climbing to roughly 9B parameters, about the max I can fit in 8GB of VRAM, to see how the concurrency curve changes as models outgrow the card.
• MoE models with offloaded experts — sparse models that push expert weights out to system RAM; the 9950X3D’s CCD0 affinity and 32GB of 1:1 DDR5 were built for exactly this. The question: does expert offloading serialize under load, or does MoE’s sparsity actually hold up better under concurrency than dense models?

And throughput only answers how fast a model can go — not how well it works. The strongest performers from this sweep also head onto real-world use datasets under the same harness and the same dirty rig:

• Agentic task mixes — multi-turn tool calling, structured JSON/action output, and code-editing loops, measuring parse success and instruction-following while 8–24 agents are in flight.
• Long-context drift — whether models stay coherent when agents hold full 32k contexts under compaction pressure, not just 60-second sprints.
• Quality vs token budget — the reasoning-share question made concrete: does the hidden thinking actually buy better task outcomes, or just slower answers?
• Latency budgets — which models stay inside interactive TTFT budgets at realistic concurrency, and which are batch-only.

Data & downloads

Every number above is reproducible: each model page links its sweep-summary.csv, sweep-summary.json, the full report markdown, and the original white-background matplotlib chart exports. The generation pipeline lives in build/ (dark-theme chart generator + model page builder) and is re-runnable as the dataset grows.

• data/ directory (per-model sweep summaries)