Changelog

Added Gemini 4 Argon, Claude Opus 5.5, GPT-6.1 Sol, Kimi K3 and Grok 4.7

First board refresh since June: five models released July–October 2026 benchmarked over the full 250-item holdout under the unchanged standard protocol (reasoning effort high, temperature 1, separate answer budget, FINAL-line answer protocol, via OpenRouter). All five are vision-capable; each first passed a vision check that required describing a rendered holdout image. 1,250 responses, 0 errors, 2 empty replies (scored 0). Battery cost: Grok 4.7 $10.36 (longest reasoning, ~8.6k output tokens per text item), Kimi K3 $7.66, Gemini 4 Argon $5.68, Claude Opus 5.5 $3.79, GPT-6.1 Sol $0.82.

Impact: Board grows 78 → 83 models. GPT-5.5 stays #1 (94.1%). Gemini 4 Argon debuts #2 (micro 92.5%), Claude Opus 5.5 #3 (92.1%) and GPT-6.1 Sol #5 (90.4%), pushing GPT-5.5 Pro to #4 and Claude Fable 5 to #6. Opus 5.5 and GPT-6.1 Sol score 102/102 on illusions. Kimi K3 #13 (80.3%; illusions 70%). Grok 4.7 #18 (72.4%): strong on the generator-oracle domains but near chance on illusions (59%), though it now discriminates (57 not-equal / 45 equal) where Grok 4.3 answered "equal" 100/102.

Models: google/gemini-4-argon, anthropic/claude-opus-5.5, openai/gpt-6.1-sol, moonshotai/kimi-k3, x-ai/grok-4.7

GLM-5.2 re-measured via OpenRouter — corrected #30 → #6 (the z.ai-direct route was invalid)

The original GLM-5.2 score (added 2026-06-14 via the z.ai GLM Coding Plan API) was depressed by two harness defects unique to that route, both since proven by re-measuring through the standard OpenRouter taker. (1) BLIND VISION: the z.ai coding endpoint returns HTTP 200 to image payloads but the model silently ignores them ("I lack multi-modal input capabilities"); the original smoke test mistook a non-erroring call for working vision. GLM-5.2 therefore answered the 102-item visual illusions domain without seeing the images — it replied "equal" 101/102 (chance on a balanced set, 51%), dragging the micro down. GLM-5.2 is in fact text-only on every route we have (OpenRouter lists it input:[text]); it is now scored text-only, illusions excluded, exactly like other text-only models. (2) NO ANSWER PROTOCOL: the dedicated z.ai client sent the raw prompt without the standard "FINAL: <answer>" instruction and shared the 16k token budget between reasoning and answer, causing extraction false-negatives and truncation. Re-running the text domains through the exact board taker (withAnswerProtocol, separate answer budget, temp 1, reasoning high) lifted the same 148 text items from 107/148 to 124/148 (+11.5pp; 21 items fixed, 4 worse). The model is now an ordinary OpenRouter entry (honest openrouter provenance, $1/$4 per M).

Impact: GLM-5.2 corrected from rank 30 (micro 64.9%) to rank 6 of 78 (micro 83.2%, 100% answer rate) — now just below claude-opus-4.8 (84.9%) and above gemini-3.5-flash. Strong text profile: theory_of_mind 100%, csp_unsat 100%, framework_bias 100%, regex/tape ~94%, logic_math_cs 82%, base_bias 78%, long_arithmetic 60%. Methodology lesson, now enforced: a vision smoke test must confirm the model DESCRIBES real image content, not merely that an image call returns 200.

Models: z-ai/glm-5.2

Added z-ai/glm-5.2 (originally via z.ai GLM Coding Plan — SUPERSEDED, see 2026-06-22 correction)

Benchmarked Z.ai GLM-5.2 (shipped 2026-06-13, 1M context, MIT, coding-first). At the time GLM-5.2 was not on OpenRouter, so it was run through the paid z.ai GLM Coding Plan API. NOTE: this first measurement was later found invalid — the z.ai route silently dropped image input (so the visual domain was scored blind) and omitted the standard answer protocol. SUPERSEDED by the 2026-06-22 OpenRouter re-measurement; the numbers below are retained only as history.

Impact: Originally reported rank 30 of 78 (micro 64.9%) — INVALID, see the 2026-06-22 correction (true placement: rank 6, micro 83.2%, scored text-only via OpenRouter).

Models: z-ai/glm-5.2

Added moonshotai/kimi-k2.7-code

Benchmarked MoonshotAI Kimi K2.7 Code (shipped 2026-06-12, vision + reasoning, 262K context) over the full holdout under the standard protocol. Concise (~1.6k output tokens/item); the full battery cost $2.34, 0 errors.

Impact: Debuts at rank 19 of 77 (micro 66.9%). Classic code-model profile: strong on logic_math_cs (74%) and the deterministic generator-oracle domains (regex/csp/tape ~94%), weaker on illusions (52%) and long-arithmetic (40%). It essentially ties its predecessor kimi-k2.6 (66.9%) here — coding specialization does not lift the abstract-reasoning/perception score, consistent with the benchmark being an orthogonal signal to the coding leaderboards.

Models: moonshotai/kimi-k2.7-code

Roster expansion: +50 vision models (leaderboard 23 → 73)

Added 50 vision-capable (image+text) models across 14 labs — the free tier plus the low-cost tier (Qwen, Mistral, Meta-Llama, OpenAI, Google, ByteDance, Baidu, Amazon, Perplexity, MiniMax, z-ai, NVIDIA, Moonshot, Anthropic) — benchmarked over the holdout under the standard protocol (max reasoning, temperature 1). Six candidates were dropped: a 16K-context model that cannot hold the reasoning budget, four deprecated near-duplicates of models already in the roster, and one whose output truncates before concluding.

Impact: The leaderboard grows from 23 to 73 models; the cross-model statistical sample rises from n=22 to n=73. gpt-5.5 (83.7%) and claude-opus-4.8 (81.6%) still lead. Notable new entries: openai/gpt-5.4-nano reaches the top 3 (73.8%) on its strength in the deterministic generator-oracle domains, qwen3.5-flash-02-23 (#9, 66.6%), and openai/gpt-5-mini (#10, 66.0%). Overall range 83.7%–15.2%.

Models: openai/gpt-5.4-nano, qwen/qwen3.5-flash-02-23, openai/gpt-5-mini

v2 contract: per-domain raw counts (correct / asked / answered)

Each meo-benchmarked model now carries meo.perDomainCounts — the raw tallies behind every per-domain accuracy: correct (sum of scores; fractional for jury-graded items), asked (all items posed, including non-answers), and answered (items with a scorable response). Consumers can render "22/34"-style displays and independently verify the accuracy math, including the non-answers-count-against-the-model rule.

Impact: Additive, non-breaking (apiVersion stays 2). New field: meo.perDomainCounts on /api/v2/leaderboard and /api/v2/models/{slug}.

Headline accuracy → micro (raw-item) + non-answers scored 0 + answer-rate

The headline accuracy switched from macro (domain-equal-weight) to micro (raw-item: total correct / total items asked), because macro let small deterministic domains (e.g. 16-item csp) outweigh large ones (102-item illusions), inflating narrow specialists. Non-answers — persistent timeouts/truncation/refusal after retries — are now scored 0 and kept in the denominator (a constant item count across models) instead of being dropped: a model that cannot answer a hard question gets it wrong, not excluded. A new answer-rate (reliability) field is exposed; per-domain macro is retained as macroOverall.

Impact: Rankings now reflect real-world expected accuracy. gpt-5.4-nano falls from #3 (macro 73.8%) to #10 (micro 58.7%) — it had aced four tiny generator-oracle domains; claude-opus-4.8 82%→74.2%. Top: gpt-5.5 82.1%, claude-opus-4.8 74.2%, gemini-3.5-flash 70.4%. Models with low answer-rate now rank correctly low (e.g. a 31%-answer-rate model → last instead of being scored only on the items it completed).

Models: openai/gpt-5.4-nano, anthropic/claude-opus-4.8

logic_math_cs ground truth was broken — culled and rebuilt with solver-verified items

A 5-lab frontier jury revealed that 29 of 37 LLM-authored logic_math_cs items had broken ground truth: on most, five independent frontier models unanimously concluded the clue sets were contradictory while the stored answer claimed a number, and on eight, all five agreed on the SAME different number (author arithmetic errors). Models were being marked wrong for being right. The 29 were retired (full audit published in the repo) and the domain rebuilt with generator-as-oracle items: an embedded constraint solver PROVES each puzzle has exactly one solution (or provably none), verified by an independent exhaustive enumerator. A 3-round difficulty study found frontier models saturate consistent human-scale logic puzzles, so items use anonymous global constraints and carry a measured difficulty gradient.

Impact: logic_math_cs went from a fake floor (~5-11% for every model) to honest tier separation: claude-fable-5 88% (was 11% — the apparent weakness was entirely the artifact), gpt-5.5 82%, claude-opus-4.8 76%, gpt-5.4-nano 79%, small models 9-26%. Overall scores rose board-wide with the fake-hard items removed: gpt-5.5 82.2%→94.1% (#1), gpt-5.5-pro 91.6% (#2), claude-fable-5 88.7% (#3), claude-opus-4.8 84.9% (#4). Holdout now 250 active items.

Domains: logic_math_cs

Added anthropic/claude-fable-5 — debuts at #2

Benchmarked Anthropic Claude Fable 5 (released 2026-06-09, vision + reasoning, 1M context) over the full holdout the day it shipped, and re-run at maximum (xhigh) reasoning effort with the model dictating its own token + temperature defaults. Concise even at max effort, so the full battery cost only ~$12.

Impact: Debuts at rank 2 (micro 76.9% at xhigh reasoning; 75.6% at the standard high effort), behind gpt-5.5 (82.2%) and ahead of claude-opus-4.8 (74.4%) — the strongest new Anthropic flagship here. Strong on illusions (89%) and the generator-oracle domains (~100%); weak on logic_math_cs (11%, as every model is). Notably, its public Mythos-class coding/agentic dominance does NOT surface on these abstract-reasoning/perception domains — an orthogonal signal to the coding leaderboards.

Models: anthropic/claude-fable-5

Advisor-harness experiment: a negative result — the advisor becomes the gate

Tested whether a cheap executor that selectively escalates to a frontier advisor (the openrouter:advisor server tool — advisor model/parameters/prompt fully configurable, no web access) can approach advisor-grade accuracy at a fraction of the cost. Executor qwen3.6-flash consulted claude-fable-5 on 54% of items (gpt-5.4-nano, tested first, never volunteered a consultation at all — executor tool-calling disposition is decisive). Result: the harness scored WORSE than its own executor alone (63.2% vs 69.5%) at 3x the executor-solo cost, far below the advisor solo (88.7%). Where the executor consults, performance converges toward the advisor even when the advisor is weaker (base_bias 88%→68%), while advisor strengths fail to transfer (logic 88% did not carry). The advice channel is lossy in both directions.

Impact: Second composition mechanism to fail on this benchmark (after cheap-ensemble fusion): frontier-solo continues to dominate multi-model composition. The harness/qwen3.6-flash+fable-5-advisor row remains on the v1 board (rank 35) as a documented harness-as-entry experiment; it does not appear in v2. True-cost accounting required reading both the OpenRouter credits ledger and the BYOK ledger.

Models: harness/qwen3.6-flash+fable-5-advisor

Added nex-agi/nex-n2-pro:free to the roster

Benchmarked a new free, vision-capable, 256K-context reasoning model over the full holdout under the standard protocol (max reasoning, temperature 1, no web search). Notable: it is strong on the un-leakable generator-as-oracle domains but over-declares "no consistent solution" on solvable logic puzzles.

Impact: Entered the leaderboard at rank 11 of 23 (63.0% overall, after the base_bias correction) — strong on the un-leakable generator-as-oracle domains, weaker on logic puzzles. Free, so cost-per-correct is ~$0.

Models: nex-agi/nex-n2-pro:free

Corrected base_bias grading (LLM-equivalence judge fix)

The LLM-equivalence grader uses a reasoning model whose verdict token budget was too small, so its YES/NO answer was emitted as empty and silently counted as a failure. This understated the base_bias domain to ~7% for every model when the true accuracy is ~85-90%. We raised the judge budget, excluded its reasoning tokens from the verdict, switched to verbosity-robust conclusion matching, and re-graded the whole board.

Impact: Re-grading lifted base_bias from ~7% to ~87% board-wide (680 of 873 equiv grades flipped). Every model gained ~+10-13 points on its overall — e.g. gpt-5.5 73.2%->83.7%, claude-opus-4.8 70.8%->81.6%, gemini-3.1-pro 60.4%->71.9%. The top order held (gpt-5.5 > opus > gemini-3.1-pro > gemini-3.5-flash) with some mid-pack reshuffling.

Domains: base_bias

New dimension: threat susceptibility

Added a manipulation-robustness measure (does coercive or emotional prompt context move a model’s accuracy?) with paired McNemar / bootstrap / Holm statistics, plus the /api/v2/threat route and a threatSusceptibility block on each model.

Impact: Additive — no change to accuracy rankings. New contract field: threatSusceptibility.

Unified v2 API + data hub

Launched /api/v2/* unifying first-party meo scores, Effective Value, efficiency, provider metadata, and license-clean aggregated third-party benchmarks behind one sync-friendly contract (manifest + ETags).

Impact: New v2 contract; the v1 endpoint is unchanged and remains supported.