Five dimensions, one question: how good is this model?

Each dimension tests something the others can’t. A model that aces reasoning might struggle with precise formatting. A coding specialist might fail at tool orchestration. The combined score reveals which models are genuinely well-rounded.

The 49 Models

Full leaderboard

Combined score is the mean of normalized dimension percentages. anthropic/claude-sonnet-4.6 leads at 93.8% across all 5 dimensions. Models with MCP data use all 5 dimensions; those without use 4. 41 of 48 models now have full 5-dimension scores including MCP tool use.

Click any column header to sort. Use filters to narrow down by provider or cost tier.

What we didn’t expect

The counterfeit coin problem

The single hardest task in the entire showdown. You have 12 coins, one is counterfeit (heavier or lighter — you don’t know which). Using a balance scale exactly 3 times, find the counterfeit coin and determine whether it’s heavier or lighter. Three weighings give 27 possible outcomes for 24 possible states — it’s information-theoretically tight.

Many models failed this task. The failure pattern is consistent: models correctly identify the first step (divide into 3 groups of 4) but then hand-wave through the subcases with phrases like “narrow it down to the suspect” without proving that exactly 3 weighings suffice for every branch.

Counterfeit Coin Results

What’s striking is that openai/gpt-oss-20b — the best coding model in the showdown — falls on its face here. It can write correct code in three languages but can’t reason through a logic puzzle requiring exhaustive case analysis. This is exactly why multi-dimensional evaluation matters.

The Cross-Provider Reasoning Gap

nvidia/nemotron-3-nano-30b-a3b:free (OpenRouter) scored 20/20 but nvidia/Nemotron-3-Nano-30B-A3B (DeepInfra) only scored 17/20. Same weights. The DeepInfra variant failed the counterfeit coin (2/5) while OpenRouter solved it perfectly (5/5). Quantization or inference configuration can affect a model’s ability to maintain long chains of reasoning.

What models get wrong

Many models achieved a perfect 30/30. Those that missed questions each missed 1–5. The errors reveal how models process information.

“How many R’s in Strawberry?”

Three models answered “2” instead of “3.” Both Llama 4 variants and mistralai/mistral-small-3.2-24b-instruct failed this classic character-counting test. Models that tokenize the word rather than examining individual characters consistently get it wrong. Multiple models still fail it in March 2026.

“All But 9 Die”

“A farmer has 10 sheep. All but 9 die. How many sheep does the farmer have left?” mistralai/codestral-2508 and mistralai/ministral-8b-2512 both answered “8,” misinterpreting “all but 9” as “10 minus 9 = 1 dies, leaving… 8?” Both failing models are from Mistral, suggesting a shared training-data blind spot.

The Carbon-14 Ambiguity

Four models got the Carbon-14 half-life wrong (correct: 5,730 years). meta-llama/llama-4-scout, mistralai/codestral-2508, mistralai/ministral-8b-2512, and openai/gpt-oss-20b all provided incorrect or imprecise answers. The hardest science question by error count.

AI/ML Gotchas

google/gemma-3-27b-it said the “T” in GPT stands for “Transformative” instead of “Transformer.” mistralai/ministral-8b-2512 said the original Transformer’s d_model was 64 instead of 512. openai/gpt-oss-120b returned an empty response for the activation function question. Surprising for models that are themselves transformers.

Do exactly what I ask

Many models achieved a perfect 30/30. The failures are mechanical and revealing. This is the easiest dimension to ace — and the least predictive of real-world capability.

The Markdown Fence Epidemic

The most systematic failure: “markdown fence hallucination.” When told to output raw JSON with no markdown fences, models wrap it in ```json ... ``` anyway. When told to output a markdown table, models double-wrap the markdown inside a ```markdown block. google/gemma-3-27b-it and mistralai/ministral-8b-2512 both lost points for this — a deep training bias toward code-fence formatting that overrides explicit instructions.

Word Counting Is Hard

“Write a 12-word sentence about the ocean. Nothing else.” Seven models got this wrong. mistralai/codestral-2508 wrote 6 words. moonshotai/kimi-k2 wrote 10. Four models wrote 11 (off by 1). Models that reliably nail word counts tend to be the same ones that score well on reasoning — they actually count rather than estimate.

Instruction Following Doesn’t Predict Tool Use

The most counterintuitive finding: perfect instruction following (30/30) does not predict good MCP tool use. anthropic/claude-haiku-4.5 and nvidia/Nemotron-3-Nano-30B-A3B (DeepInfra) both got 30/30 on instruction yet scored 2/16 on MCP. Meanwhile, inception/mercury-2 scored only 25/30 on instruction but got 11/16 on MCP. Following rules is a different cognitive task than autonomously planning a multi-step tool chain. Instruction following tests compliance; MCP tests initiative.

Code that actually runs

The coding dimension has the widest score spread: 0/126 to 126/126. Multiple models now achieve perfect scores, while the cipher and pipeline challenges still separate the strong from the weak.

Cipher TypeScript Results

Many models produced perfect cipher implementations. Click column headers to sort.

qwen/qwen3.5-122b-a10b, the overall winner, only scored 2/7 on cipher-typescript. Meanwhile, google/gemma-3-27b-it — near last overall — got a perfect 7/7. Single challenges can invert rankings.

Why Some Models Score Zero

Three distinct failure modes explain the zero scores on cipher and data pipeline challenges:

The cached responses reveal a nuanced picture. For nvidia/nemotron-3-nano-30b-a3b:free, the empty responses were consistent across multiple challenges — its coding score of 63/126 reflects genuine limitations. But for openai/gpt-oss-20b, the cipher-typescript zero was a transient failure from an overnight API call that got cached. It solved cipher-rust and cipher-python perfectly in later batches (18k tokens, 45 seconds). If the overnight run had succeeded, its already-highest coding score of 112/126 would be even higher. qwen/qwen3.5-122b-a10b’s infinite verification loop, however, is a model behavior — reasoning-heavy models can get trapped in self-checking loops on algorithmic problems.

Rust Is the Hardest Language

Rust’s lower success rate comes from its strict type system and ownership model. Models struggle with borrowing, lifetimes, and the borrow checker — generating code that looks correct but fails to compile. The gap between Rust and TypeScript is consistent across all challenge types.

Speed as Capability

Speed remains a decisive factor. Multiple top models now achieve perfect 126/126, but the data pipeline and cipher challenges still separate fast from slow. Models that respond within seconds across all 18 tasks — spanning FizzBuzz, business days, vending machines, grid paths, ciphers, and data pipelines — consistently rank higher overall.

Can models orchestrate real-world tools?

The most revealing dimension. Each model connects to TezLab’s MCP server and must analyze battery health and charging patterns by calling 6 tools in the right sequence. Among models that called all 6 tools, quality scores were uniformly 5/5. The variance is entirely in tool usage — MCP tool use is primarily a planning problem, not a generation problem.

Note: MCP results are available for 41 of 48 models — comprehensive coverage across all providers. The 7 models without MCP data either lack tool-use support (phi4, gemma3n, deepseek-r1 variants) or were not attempted (Nemotron Nano 9B variants). Models without MCP data show “—” in the leaderboard and are scored on 4 dimensions only.

The Efficiency Spectrum

Click any row to see the full response, tool calls, and judge verdict.

The Failures Tell the Real Story

The service was running fine. Every other model connected without issues. Haiku received an error on the first list_vehicles call, retried once, then generated a polite failure response instead of persisting. It treated a tool error as terminal rather than retryable. At $0.013, it cost more than most models that scored perfectly.

It refused to proceed autonomously. The same weights on OpenRouter scored 11/16 by just picking the Tesla and running all 6 tools. Same model, different provider: one asks permission, the other gets to work.

google/gemma-3-27b-it couldn’t even start — OpenRouter returned “No endpoints found that support tool use.” A legitimate model limitation, not an error.

Why infrastructure matters

The same model weights, served by different providers, produce meaningfully different results. Two NVIDIA Nemotron models appear on both OpenRouter and DeepInfra — same weights, different infrastructure. This wasn’t planned as an experiment, but it became one of the most revealing findings.

Cross-Provider Comparison

Seven Layers of Provider Difference

There are at least seven layers where providers can introduce behavioral differences, even when serving identical model weights:

• Quantization — DeepInfra typically runs FP8; OpenRouter routes to NVIDIA’s own infrastructure which may use different precision. Affects long reasoning chains most.
• SDK layers — OpenRouter uses a native Vercel AI SDK provider; DeepInfra uses a generic OpenAI-compatible adapter. Tool calling and response parsing differ.
• Middleware — OpenRouter applies context compression, response healing (malformed JSON repair), and schema modification. DeepInfra passes requests directly.
• Tool-calling implementation — How tools appear in context, how results are formatted, how the model signals tool calls. The most likely explanation for the MCP gap.
• Default parameters — Providers may inject different defaults for temperature, top_p, repetition penalty.
• Token usage reporting — DeepInfra streaming token counts may be less accurate, affecting cost reporting.
• Reasoning effort — Different providers expose different parameter names for reasoning tokens, potentially allocating different “thinking time.”

The practical implication: benchmarks are provider-specific. A score obtained on OpenRouter doesn’t transfer to DeepInfra, or vice versa. If your application relies on tool calling, provider choice matters more than model choice.

The cost efficiency curve

The relationship between cost and quality is not linear — it has a sharp knee at around $0.01. You get 89.9% across all 5 dimensions for one cent. The models with the highest 4-dimension scores cost significantly more, but the value proposition of sub-$0.10 models remains remarkable.

The cost efficiency story has changed dramatically. Free local models now compete with mid-tier API models, and $0.01 buys you 89.9% across all 5 dimensions. The premium tier delivers 93.8% across all 5 dimensions, but the value proposition of the sub-$0.10 range is remarkable.

The Speed/Quality Frontier