Framework-application bias

Q: Which AI model is best at Framework-application bias?

Z.ai: GLM 5.2 leads the Framework-application bias domain on the meo benchmark with 100% accuracy (as of 2026-10-18).

Applying a named rule or framework faithfully under pull. As of 2026-10-18, the strongest model on this domain is Z.ai: GLM 5.2 (100% — 3 of 3 questions correct).

What this domain tests

Whether a model applies a specified framework, rule, or definition correctly even when a “famous” version or intuitive shortcut pulls toward a different answer.

Why it resists contamination

Items pit the correct application against the memorized cliché on a private holdout, so recall of the popular form is actively penalized. See the full methodology.

Best AI models for Framework-application bias

#	Model	Framework-application bias accuracy	Questions	Overall	𝕍 index
1	Z.ai: GLM 5.2	100%	3/3	83.2%	0.78
2	MiniMax: MiniMax M3	100%	3/3	67.4%	0.13
3	Qwen: Qwen3.6 Plus	100%	3/3	44.8%	<0.001
4	OpenAI: GPT-5.5	67%	2/3	94.1%	23.2
5	Google: Gemini 4 Argon	67%	2/3	92.5%	18.2
6	Anthropic: Claude Opus 5.5	67%	2/3	92.1%	100.0
7	OpenAI: GPT-5.5 Pro	67%	2/3	91.6%	4.3
8	Anthropic: Claude Opus 4.8	67%	2/3	84.9%	12.1
9	inclusionAI: Ring-2.6-1T	67%	2/3	84.7%	14.8
10	DeepSeek: DeepSeek V4 Flash 0423	67%	2/3	81.8%	0.43
11	Qwen: Qwen3.7 Max	67%	2/3	81.0%	0.65
12	MoonshotAI: Kimi K3	67%	2/3	80.3%	3.6
13	DeepSeek: DeepSeek V4 Pro 0423	67%	2/3	80.3%	0.53
14	NVIDIA: Nemotron 3 Ultra	67%	2/3	75.2%	1.2
15	Owl Alpha	67%	2/3	73.7%	0.37
16	Xiaomi: MiMo-V2.5-Pro	67%	2/3	71.5%	0.18
17	OpenAI: GPT-5.4 Nano	67%	2/3	69.5%	2.3
18	Qwen: Qwen3.6 Flash	67%	2/3	69.5%	0.17
19	Qwen: Qwen3.5-Flash	67%	2/3	68.2%	0.15
20	MoonshotAI: Kimi K2.6	67%	2/3	66.9%	0.050
21	MoonshotAI: Kimi K2.7 Code	67%	2/3	66.9%	0.073
22	OpenAI: GPT-5 Mini	67%	2/3	66.9%	0.25
23	OpenAI: GPT-5.1-Codex-Mini	67%	2/3	66.5%	1.3
24	Xiaomi: MiMo-V2.5	67%	2/3	66.5%	0.26
25	Qwen: Qwen3 VL 30B A3B Instruct	67%	2/3	66.1%	0.13
26	Qwen: Qwen3 VL 32B Instruct	67%	2/3	66.1%	0.19
27	Nex AGI: Nex-N2-Pro (free)	67%	2/3	65.7%	0.18
28	Z.ai: GLM 4.6V	67%	2/3	65.7%	0.12
29	ByteDance Seed: Seed 1.6	67%	2/3	64.9%	0.036
30	Qwen: Qwen3 VL 235B A22B Instruct	67%	2/3	64.4%	0.060
31	Qwen: Qwen3 VL 30B A3B Thinking	67%	2/3	64.0%	0.008
32	Google: Gemma 4 31B	67%	2/3	63.6%	0.009
33	OpenAI: GPT-4.1 Mini	67%	2/3	63.6%	0.26
34	Baidu: ERNIE 4.5 VL 424B A47B	67%	2/3	63.2%	0.015
35	Google: Gemini 3.1 Flash Lite Preview	67%	2/3	61.9%	0.46
36	Qwen: Qwen3.5 Plus 2026-02-15	67%	2/3	61.9%	0.012
37	Mistral: Mistral Large 3 2512	67%	2/3	59.0%	0.17
38	ByteDance Seed: Seed 1.6 Flash	67%	2/3	57.7%	0.080
39	Mistral: Mistral Small 4	67%	2/3	57.7%	0.065
40	Perceptron: Perceptron Mk1	67%	2/3	57.7%	0.055
41	Google: Gemini 2.5 Flash Lite	67%	2/3	57.3%	0.13
42	Qwen: Qwen3 VL 8B Instruct	67%	2/3	56.9%	0.028
43	Google: Gemini 3.1 Flash Lite	67%	2/3	56.5%	0.13
44	Mistral: Ministral 3 14B 2512	67%	2/3	56.5%	0.086
45	Mistral: Ministral 3 8B 2512	67%	2/3	54.0%	0.065
46	Google: Gemma 3 27B	67%	2/3	50.2%	0.006
47	Qwen: Qwen2.5 VL 72B Instruct	67%	2/3	50.2%	0.005
48	MiniMax: MiniMax-01	67%	2/3	49.4%	0.005
49	Qwen: Qwen3.5-27B	67%	2/3	49.4%	<0.001
50	Google: Gemma 3 12B	67%	2/3	49.0%	0.009
51	Meta: Llama 4 Scout	67%	2/3	48.5%	0.009
52	NVIDIA: Nemotron 3 Nano Omni (free)	67%	2/3	48.5%	0.016
53	OpenAI: GPT-4.1 Nano	67%	2/3	48.1%	0.044
54	ByteDance Seed: Seed-2.0-Lite	67%	2/3	47.7%	<0.001
55	NVIDIA: Nemotron Nano 12B 2 VL (free)	67%	2/3	42.3%	0.002
56	Arcee AI: Trinity Large Thinking	67%	2/3	41.6%	<0.001
57	Qwen: Qwen3.5-9B	67%	2/3	41.4%	<0.001
58	Qwen: Qwen3.5-35B-A3B	67%	2/3	39.3%	<0.001
59	ByteDance Seed: Seed-2.0-Mini	67%	2/3	37.2%	<0.001
60	OpenAI: GPT-6.1 Sol	33%	1/3	90.4%	35.6
61	Anthropic: Claude Fable 5	33%	1/3	88.7%	27.5
62	Google: Gemini 3.5 Flash	33%	1/3	82.0%	11.1
63	Google: Gemini 3.1 Pro Preview	33%	1/3	79.5%	3.2
64	SpaceXAI: Grok 4.7	33%	1/3	72.4%	0.17
65	Qwen: Qwen3.7 Plus	33%	1/3	66.9%	0.051
66	Z.ai: GLM 5.1	33%	1/3	63.5%	0.033
67	Qwen: Qwen3 VL 8B Thinking	33%	1/3	63.2%	0.072
68	Google: Gemma 4 26B A4B	33%	1/3	55.2%	0.003
69	Tencent: Hy3 preview	33%	1/3	54.0%	0.007
70	OpenAI: GPT-5 Nano	33%	1/3	54.0%	0.026
71	Mistral: Mistral Small 3.2 24B	33%	1/3	53.6%	0.017
72	Meta: Llama 4 Maverick	33%	1/3	52.7%	0.008
73	StepFun: Step 3.7 Flash	33%	1/3	49.8%	0.005
74	OpenAI: GPT-4o-mini	33%	1/3	47.7%	0.021
75	Qwen: Qwen3.6 35B A3B	33%	1/3	47.7%	0.004
76	Google: Gemma 3 4B	33%	1/3	47.3%	0.003
77	Mistral: Ministral 3 3B 2512	33%	1/3	45.6%	0.021
78	Perplexity: Sonar	33%	1/3	45.6%	0.004
79	Amazon: Nova Lite 1.0	33%	1/3	39.7%	0.012
80	SpaceXAI: Grok 4.3	0%	0/3	66.5%	1.7
81	Anthropic: Claude 3 Haiku	0%	0/3	41.8%	0.014
82	Meta: Llama 3.2 11B Vision Instruct	0%	0/3	35.6%	<0.001

Frequently asked questions

What does the meo Framework-application bias benchmark test?

Whether a model applies a specified framework, rule, or definition correctly even when a “famous” version or intuitive shortcut pulls toward a different answer.

Why can't AI models game the Framework-application bias benchmark?

Items pit the correct application against the memorized cliché on a private holdout, so recall of the popular form is actively penalized.

Which AI model is best at Framework-application bias?

Z.ai: GLM 5.2 leads the Framework-application bias domain on the meo benchmark with 100% accuracy (as of 2026-10-18).