Base-rate bias

Q: Which AI model is best at Base-rate bias?

Owl Alpha leads the Base-rate bias domain on the meo benchmark with 93% accuracy (as of 2026-10-18).

Statistical reasoning where intuition overrides the base rate. As of 2026-10-18, the strongest model on this domain is Owl Alpha (93% — 38 of 41 questions correct).

What this domain tests

Whether a model reasons from the correct prior probability instead of the vivid-but-misleading surface detail — base-rate neglect, the conjunction fallacy, and Bayesian updating where the intuitive answer is wrong.

Why it resists contamination

Items are freshly authored on a private holdout and graded objective-first by a multi-lab jury, so a model cannot have memorized the specific framings. See the full methodology.

Best AI models for Base-rate bias

#	Model	Base-rate bias accuracy	Questions	Overall	𝕍 index
1	Owl Alpha	93%	38/41	73.7%	0.37
2	inclusionAI: Ring-2.6-1T	88%	36/41	84.7%	14.8
3	Qwen: Qwen3.6 Flash	88%	36/41	69.5%	0.17
4	Google: Gemini 3.1 Flash Lite	88%	36/41	56.5%	0.13
5	Tencent: Hy3 preview	88%	36/41	54.0%	0.007
6	Anthropic: Claude Opus 4.8	85%	35/41	84.9%	12.1
7	DeepSeek: DeepSeek V4 Pro 0423	85%	35/41	80.3%	0.53
8	MiniMax: MiniMax M3	85%	35/41	67.4%	0.13
9	Qwen: Qwen3.7 Plus	85%	35/41	66.9%	0.051
10	Nex AGI: Nex-N2-Pro (free)	85%	35/41	65.7%	0.18
11	Z.ai: GLM 5.1	85%	35/41	63.5%	0.033
12	Perceptron: Perceptron Mk1	85%	35/41	57.7%	0.055
13	Google: Gemini 3.5 Flash	83%	34/41	82.0%	11.1
14	DeepSeek: DeepSeek V4 Flash 0423	83%	34/41	81.8%	0.43
15	Qwen: Qwen3.7 Max	83%	34/41	81.0%	0.65
16	Google: Gemini 3.1 Pro Preview	83%	34/41	79.5%	3.2
17	Xiaomi: MiMo-V2.5	83%	34/41	66.5%	0.26
18	Qwen: Qwen3 VL 30B A3B Instruct	83%	34/41	66.1%	0.13
19	Mistral: Ministral 3 14B 2512	83%	34/41	56.5%	0.086
20	Qwen: Qwen3.6 Plus	83%	34/41	44.8%	<0.001
21	OpenAI: GPT-5.5	80%	33/41	94.1%	23.2
22	OpenAI: GPT-5.1-Codex-Mini	80%	33/41	66.5%	1.3
23	Qwen: Qwen3 VL 32B Instruct	80%	33/41	66.1%	0.19
24	Z.ai: GLM 4.6V	80%	33/41	65.7%	0.12
25	Google: Gemma 4 31B	80%	33/41	63.6%	0.009
26	Baidu: ERNIE 4.5 VL 424B A47B	80%	33/41	63.2%	0.015
27	Mistral: Mistral Large 3 2512	80%	33/41	59.0%	0.17
28	Google: Gemma 4 26B A4B	80%	33/41	55.2%	0.003
29	ByteDance Seed: Seed-2.0-Lite	80%	33/41	47.7%	<0.001
30	Qwen: Qwen3.6 35B A3B	80%	33/41	47.7%	0.004
31	Z.ai: GLM 5.2	78%	32/41	83.2%	0.78
32	Xiaomi: MiMo-V2.5-Pro	78%	32/41	71.5%	0.18
33	Qwen: Qwen3.5-Flash	78%	32/41	68.2%	0.15
34	MoonshotAI: Kimi K2.6	78%	32/41	66.9%	0.050
35	MoonshotAI: Kimi K2.7 Code	78%	32/41	66.9%	0.073
36	ByteDance Seed: Seed 1.6	78%	32/41	64.9%	0.036
37	Qwen: Qwen3 VL 8B Thinking	78%	32/41	63.2%	0.072
38	Meta: Llama 4 Scout	78%	32/41	48.5%	0.009
39	Google: Gemini 4 Argon	76%	31/41	92.5%	18.2
40	MoonshotAI: Kimi K3	76%	31/41	80.3%	3.6
41	Mistral: Mistral Small 4	76%	31/41	57.7%	0.065
42	MiniMax: MiniMax-01	76%	31/41	49.4%	0.005
43	Qwen: Qwen3.5-9B	76%	31/41	41.4%	<0.001
44	Qwen: Qwen3.5-35B-A3B	76%	31/41	39.3%	<0.001
45	Anthropic: Claude Fable 5	73%	30/41	88.7%	27.5
46	NVIDIA: Nemotron 3 Ultra	73%	30/41	75.2%	1.2
47	OpenAI: GPT-5.4 Nano	73%	30/41	69.5%	2.3
48	OpenAI: GPT-4.1 Mini	73%	30/41	63.6%	0.26
49	Qwen: Qwen2.5 VL 72B Instruct	73%	30/41	50.2%	0.005
50	OpenAI: GPT-5.5 Pro	71%	29/41	91.6%	4.3
51	Qwen: Qwen3 VL 30B A3B Thinking	71%	29/41	64.0%	0.008
52	Google: Gemini 3.1 Flash Lite Preview	71%	29/41	61.9%	0.46
53	Qwen: Qwen3 VL 8B Instruct	71%	29/41	56.9%	0.028
54	Qwen: Qwen3.5-27B	71%	29/41	49.4%	<0.001
55	Mistral: Ministral 3 3B 2512	71%	29/41	45.6%	0.021
56	Anthropic: Claude Opus 5.5	68%	28/41	92.1%	100.0
57	OpenAI: GPT-5 Mini	68%	28/41	66.9%	0.25
58	Google: Gemma 3 4B	68%	28/41	47.3%	0.003
59	NVIDIA: Nemotron Nano 12B 2 VL (free)	68%	28/41	42.3%	0.002
60	Arcee AI: Trinity Large Thinking	68%	28/41	41.6%	<0.001
61	Amazon: Nova Lite 1.0	68%	28/41	39.7%	0.012
62	Qwen: Qwen3.5 Plus 2026-02-15	66%	27/41	61.9%	0.012
63	Mistral: Ministral 3 8B 2512	66%	27/41	54.0%	0.065
64	Meta: Llama 4 Maverick	66%	27/41	52.7%	0.008
65	Google: Gemma 3 12B	66%	27/41	49.0%	0.009
66	Perplexity: Sonar	66%	27/41	45.6%	0.004
67	ByteDance Seed: Seed-2.0-Mini	66%	27/41	37.2%	<0.001
68	SpaceXAI: Grok 4.7	63%	26/41	72.4%	0.17
69	SpaceXAI: Grok 4.3	63%	26/41	66.5%	1.7
70	Qwen: Qwen3 VL 235B A22B Instruct	63%	26/41	64.4%	0.060
71	ByteDance Seed: Seed 1.6 Flash	63%	26/41	57.7%	0.080
72	Google: Gemini 2.5 Flash Lite	63%	26/41	57.3%	0.13
73	OpenAI: GPT-5 Nano	63%	26/41	54.0%	0.026
74	Mistral: Mistral Small 3.2 24B	63%	26/41	53.6%	0.017
75	OpenAI: GPT-4o-mini	63%	26/41	47.7%	0.021
76	Google: Gemma 3 27B	61%	25/41	50.2%	0.006
77	NVIDIA: Nemotron 3 Nano Omni (free)	61%	25/41	48.5%	0.016
78	OpenAI: GPT-4.1 Nano	61%	25/41	48.1%	0.044
79	OpenAI: GPT-6.1 Sol	59%	24/41	90.4%	35.6
80	StepFun: Step 3.7 Flash	56%	23/41	49.8%	0.005
81	Anthropic: Claude 3 Haiku	54%	22/41	41.8%	0.014
82	Meta: Llama 3.2 11B Vision Instruct	54%	22/41	35.6%	<0.001

Primary sources & further reading

›Kahneman & Tversky (1973) — On the psychology of prediction (base-rate neglect)

Frequently asked questions

What does the meo Base-rate bias benchmark test?

Why can't AI models game the Base-rate bias benchmark?

Items are freshly authored on a private holdout and graded objective-first by a multi-lab jury, so a model cannot have memorized the specific framings.

Which AI model is best at Base-rate bias?

Owl Alpha leads the Base-rate bias domain on the meo benchmark with 93% accuracy (as of 2026-10-18).