Skip to main content
Meo

Multi-step state tracking

Maintaining state across many sequential updates. As of 2026-10-18, the strongest model on this domain is OpenAI: GPT-5.5 (100% — 1 of 1 questions correct).

What this domain tests

Tracking entities and their state across a long sequence of updates (positions, values, ownership) and answering about the final state — long-horizon working memory and consistency.

Why it resists contamination

Update sequences and the resulting state are generated programmatically, so each trace is unique and the answer key is computed, not authored (generator-as-oracle). Because instances are generated with a programmatic ground truth, this is a “generator-as-oracle” domain — effectively unlimited fresh, self-verifying items. See the full methodology.

Best AI models for Multi-step state tracking

#ModelMulti-step state tracking accuracyQuestionsOverall𝕍 index
1OpenAI: GPT-5.5100%1/194.1%23.2
2Google: Gemini 4 Argon100%1/192.5%18.2
3Anthropic: Claude Opus 5.5100%1/192.1%100.0
4OpenAI: GPT-5.5 Pro100%1/191.6%4.3
5OpenAI: GPT-6.1 Sol100%1/190.4%35.6
6Anthropic: Claude Fable 5100%1/188.7%27.5
7Z.ai: GLM 5.2100%1/183.2%0.78
8MoonshotAI: Kimi K3100%1/180.3%3.6
9Google: Gemini 3.1 Pro Preview100%1/179.5%3.2
10Owl Alpha100%1/173.7%0.37
11SpaceXAI: Grok 4.7100%1/172.4%0.17
12Qwen: Qwen3.5-Flash100%1/168.2%0.15
13MoonshotAI: Kimi K2.7 Code100%1/166.9%0.073
14Qwen: Qwen3.7 Plus100%1/166.9%0.051
15SpaceXAI: Grok 4.3100%1/166.5%1.7
16Qwen: Qwen3 VL 30B A3B Instruct100%1/166.1%0.13
17Nex AGI: Nex-N2-Pro (free)100%1/165.7%0.18
18Z.ai: GLM 4.6V100%1/165.7%0.12
19ByteDance Seed: Seed 1.6100%1/164.9%0.036
20Qwen: Qwen3 VL 235B A22B Instruct100%1/164.4%0.060
21MiniMax: MiniMax-01100%1/149.4%0.005
22NVIDIA: Nemotron Nano 12B 2 VL (free)100%1/142.3%0.002
23Anthropic: Claude Opus 4.80%0/184.9%12.1
24inclusionAI: Ring-2.6-1T0%0/184.7%14.8
25Google: Gemini 3.5 Flash0%0/182.0%11.1
26DeepSeek: DeepSeek V4 Flash 04230%0/181.8%0.43
27Qwen: Qwen3.7 Max0%0/181.0%0.65
28DeepSeek: DeepSeek V4 Pro 04230%0/180.3%0.53
29NVIDIA: Nemotron 3 Ultra0%0/175.2%1.2
30Xiaomi: MiMo-V2.5-Pro0%0/171.5%0.18
31OpenAI: GPT-5.4 Nano0%0/169.5%2.3
32Qwen: Qwen3.6 Flash0%0/169.5%0.17
33MiniMax: MiniMax M30%0/167.4%0.13
34MoonshotAI: Kimi K2.60%0/166.9%0.050
35OpenAI: GPT-5 Mini0%0/166.9%0.25
36OpenAI: GPT-5.1-Codex-Mini0%0/166.5%1.3
37Xiaomi: MiMo-V2.50%0/166.5%0.26
38Qwen: Qwen3 VL 32B Instruct0%0/166.1%0.19
39Qwen: Qwen3 VL 30B A3B Thinking0%0/164.0%0.008
40Google: Gemma 4 31B0%0/163.6%0.009
41OpenAI: GPT-4.1 Mini0%0/163.6%0.26
42Z.ai: GLM 5.10%0/163.5%0.033
43Baidu: ERNIE 4.5 VL 424B A47B 0%0/163.2%0.015
44Qwen: Qwen3 VL 8B Thinking0%0/163.2%0.072
45Google: Gemini 3.1 Flash Lite Preview0%0/161.9%0.46
46Qwen: Qwen3.5 Plus 2026-02-150%0/161.9%0.012
47Mistral: Mistral Large 3 25120%0/159.0%0.17
48ByteDance Seed: Seed 1.6 Flash0%0/157.7%0.080
49Mistral: Mistral Small 40%0/157.7%0.065
50Perceptron: Perceptron Mk10%0/157.7%0.055
51Google: Gemini 2.5 Flash Lite0%0/157.3%0.13
52Qwen: Qwen3 VL 8B Instruct0%0/156.9%0.028
53Google: Gemini 3.1 Flash Lite0%0/156.5%0.13
54Mistral: Ministral 3 14B 25120%0/156.5%0.086
55Google: Gemma 4 26B A4B 0%0/155.2%0.003
56Tencent: Hy3 preview0%0/154.0%0.007
57Mistral: Ministral 3 8B 25120%0/154.0%0.065
58OpenAI: GPT-5 Nano0%0/154.0%0.026
59Mistral: Mistral Small 3.2 24B0%0/153.6%0.017
60Meta: Llama 4 Maverick0%0/152.7%0.008
61Google: Gemma 3 27B0%0/150.2%0.006
62Qwen: Qwen2.5 VL 72B Instruct0%0/150.2%0.005
63StepFun: Step 3.7 Flash0%0/149.8%0.005
64Qwen: Qwen3.5-27B0%0/149.4%<0.001
65Google: Gemma 3 12B0%0/149.0%0.009
66Meta: Llama 4 Scout0%0/148.5%0.009
67NVIDIA: Nemotron 3 Nano Omni (free)0%0/148.5%0.016
68OpenAI: GPT-4.1 Nano0%0/148.1%0.044
69ByteDance Seed: Seed-2.0-Lite0%0/147.7%<0.001
70OpenAI: GPT-4o-mini0%0/147.7%0.021
71Qwen: Qwen3.6 35B A3B0%0/147.7%0.004
72Google: Gemma 3 4B0%0/147.3%0.003
73Mistral: Ministral 3 3B 25120%0/145.6%0.021
74Perplexity: Sonar0%0/145.6%0.004
75Qwen: Qwen3.6 Plus0%0/144.8%<0.001
76Anthropic: Claude 3 Haiku0%0/141.8%0.014
77Arcee AI: Trinity Large Thinking0%0/141.6%<0.001
78Qwen: Qwen3.5-9B0%0/141.4%<0.001
79Amazon: Nova Lite 1.00%0/139.7%0.012
80Qwen: Qwen3.5-35B-A3B0%0/139.3%<0.001
81ByteDance Seed: Seed-2.0-Mini0%0/137.2%<0.001
82Meta: Llama 3.2 11B Vision Instruct0%0/135.6%<0.001

Frequently asked questions

What does the meo Multi-step state tracking benchmark test?

Tracking entities and their state across a long sequence of updates (positions, values, ownership) and answering about the final state — long-horizon working memory and consistency.

Why can't AI models game the Multi-step state tracking benchmark?

Update sequences and the resulting state are generated programmatically, so each trace is unique and the answer key is computed, not authored (generator-as-oracle).

Which AI model is best at Multi-step state tracking?

OpenAI: GPT-5.5 leads the Multi-step state tracking domain on the meo benchmark with 100% accuracy (as of 2026-10-18).

Related: all domains · Multi-step state tracking ranking · methodology · full leaderboard.