Methodology & Effective Value (𝕍)

How meoadvisors scores AI models: an un-leaked private holdout, objective-first grading with a bias-controlled multi-lab jury, generator-as-oracle domains, and the Effective Value (𝕍) metric whose ranking inverts with task depth. Full transparency on N/ω/δ, domains, and novelty vs public leaderboards.

as of 2026-06-08 · methodology v4

The Effective Value (𝕍) metric

Accuracy answers “how often is the model right?” Deployment hinges on a harder question: how much value does a model deliver per unit of money and time, given that a long autonomous task fails entirely if any step fails? Effective Value is our answer:

• Numerator — velocity × chain success. (1−E)^N is the probability of completing an N-step chain without human rescue. At E=0.1 a single step succeeds 90% of the time, but a 10-step chain only ≈35%, and a 40-step chain ≈1.5%. Velocity v (tokens/sec) rewards throughput — but only throughput that reaches the finish line.
• Denominator — money plus friction-amplified time. Cost C_f plus time, where time is amplified by a compounding friction term δ^(E·N) (δ>1): deeper tasks with higher error rates lose more time to debugging and hallucination loops. A flawed model is penalized twice.
• The ω ≫ C_f thesis. For autonomous workflows the cost of time far exceeds the cost of money — an hour of a stalled agent dwarfs a few cents of tokens. With per-item costs in cents and times in tens of seconds, the time term dominates C_f even at ω=1 (our default).
• Defaults. N=10 (a moderate agentic chain), ω=1, δ=1.5 (≈50% compounding friction per error-step). These are scenario knobs, not constants — the robust finding is how the ranking moves as N sweeps.

Sensitivity: the rank inversion with depth

The decisive consequence of the formula is a rank inversion with task depth, computed live from the current board. At shallow depth raw speed wins; as the chain deepens the error-cascade makes accuracy decisive:

Fast models top the shallow ranking; accurate flagships dominate deep chains. A cheap-but-accurate model rises sharply with depth while a fast-but-flawed one collapses. Explore it interactively with the N/ω/δ sliders on the dashboard.

What the cross-model statistics show

• More “thinking” predicts less accuracy, not more (ρ=−0.54, p=0.004). Token volume is a struggle signal, not a capability signal — the strongest models are the most concise.
• Price buys accuracy weakly, with steep diminishing returns (ρ=0.58): a 12× cheaper model lands within a few points of mid-priced flagships. Cost-per-correct, not token price, is the decision-relevant quantity.
• 𝕍 is accuracy-anchored but efficiency-adjusted (ρ=0.91 with accuracy, r=0.61) — neither a relabeling of accuracy nor independent of it.
• At moderate depth, pricier models score higher 𝕍 — the chain-success term dominates, so “expensive implies low value” is false for deep agentic tasks and true only for shallow ones.

How we keep the benchmark un-leaked

• Private holdout. The holdout is never served; a tiny public sample is published only for illustration and labelled “assume contaminated.” No grading is ever done against the public sample.
• Server-side answers. Ground truth and rubrics live in a physically separate store, never serialized into any export or page.
• Canary tagging + leak tripwire. Every item carries a namespaced canary; a log-probability tripwire flags contamination (best-effort, since hosted models rarely expose logprobs).
• Rotation. Each cycle retires the oldest/easiest items. For parametric and generator-as-oracle domains, a new seed yields a fresh un-leakable item with the same exact ground-truth formula.

Objective-first grading + a bias-controlled jury

Wherever a ground truth exists we grade objectively (an atomic exact/structured check plus an LLM-equivalence fallback to avoid prose false-negatives). Only for genuinely open-ended prose do we use a multi-lab jury, with four load-bearing rules: one model per lab (disjoint families); independent multi-call scoring aggregated by median/majority with randomized criterion order; never judge your own lab; and per-modality filtering (vision items judged only by vision-capable jurors). We deliberately do not use an answer-fusion router as the verdict mechanism — bias-controlled scoring is not the same as synthesizing one answer.

The eleven domains

Three modalities; seven authored-with-verification, four generator-as-oracle (an embedded solver computes the ground truth — no answer key to leak, infinite parametric items).

Every model runs under an identical protocol: maximum reasoning effort, temperature 1, no web search, one isolated context per item, on a roster pinned by exact provider slug.

Novelty vs existing leaderboards

Crowd arenas (LMArena) are contamination-resistant but reward style and are gameable. Independent aggregators (Artificial Analysis, Epoch) run carefully but inherit the contamination of public benchmarks. Contamination-resistant suites (LiveBench, LiveCodeBench) use rotation + objective scoring; we push further with generator-as-oracle items that are un-leakable by construction and a fully private holdout. To our knowledge the combination of a private multi-modal holdout, generator-as-oracle infinite items, a bias-controlled multi-lab jury, and a real-world-efficacy metric with a demonstrated depth-driven rank inversion is novel.

Resources

• Paper (preprint): Zenodo 10.5281/zenodo.20586608
• Dataset (CC BY 4.0): Zenodo 10.5281/zenodo.20586610 · Hugging Face · Kaggle
• Research deep-dives: Effective Value · Sycophancy · Multi-LLM-as-judge · Cheap-ensemble fusion

FAQ