Skip to main content
Meo

Regex & automata

Regular expressions and finite-automaton behavior. As of 2026-10-18, the strongest model on this domain is OpenAI: GPT-5.5 (100% — 16 of 16 questions correct).

What this domain tests

Whether strings match a given regular expression / are accepted by a finite automaton, and related language-recognition reasoning — exact formal-language execution.

Why it resists contamination

The pattern, the test strings, and the ground-truth accept/reject labels are all generated together, so each instance is fresh and programmatically verifiable. Because instances are generated with a programmatic ground truth, this is a “generator-as-oracle” domain — effectively unlimited fresh, self-verifying items. See the full methodology.

Best AI models for Regex & automata

#ModelRegex & automata accuracyQuestionsOverall𝕍 index
1OpenAI: GPT-5.5100%16/1694.1%23.2
2Google: Gemini 4 Argon100%16/1692.5%18.2
3Anthropic: Claude Opus 5.5100%16/1692.1%100.0
4OpenAI: GPT-5.5 Pro100%16/1691.6%4.3
5OpenAI: GPT-6.1 Sol100%16/1690.4%35.6
6Anthropic: Claude Fable 5100%16/1688.7%27.5
7Anthropic: Claude Opus 4.8100%16/1684.9%12.1
8inclusionAI: Ring-2.6-1T100%16/1684.7%14.8
9Google: Gemini 3.5 Flash100%16/1682.0%11.1
10DeepSeek: DeepSeek V4 Flash 0423100%16/1681.8%0.43
11Qwen: Qwen3.7 Max100%16/1681.0%0.65
12MoonshotAI: Kimi K3100%16/1680.3%3.6
13DeepSeek: DeepSeek V4 Pro 0423100%16/1680.3%0.53
14Google: Gemini 3.1 Pro Preview100%16/1679.5%3.2
15SpaceXAI: Grok 4.7100%16/1672.4%0.17
16Qwen: Qwen3.5-Flash100%16/1668.2%0.15
17OpenAI: GPT-5 Mini100%16/1666.9%0.25
18SpaceXAI: Grok 4.3100%16/1666.5%1.7
19Xiaomi: MiMo-V2.5100%16/1666.5%0.26
20Z.ai: GLM 5.1100%16/1663.5%0.033
21Z.ai: GLM 5.294%15/1683.2%0.78
22NVIDIA: Nemotron 3 Ultra94%15/1675.2%1.2
23OpenAI: GPT-5.4 Nano94%15/1669.5%2.3
24Qwen: Qwen3.6 Flash94%15/1669.5%0.17
25MoonshotAI: Kimi K2.7 Code94%15/1666.9%0.073
26Qwen: Qwen3.7 Plus94%15/1666.9%0.051
27ByteDance Seed: Seed 1.694%15/1664.9%0.036
28Google: Gemma 4 31B94%15/1663.6%0.009
29MiniMax: MiniMax M388%14/1667.4%0.13
30MoonshotAI: Kimi K2.688%14/1666.9%0.050
31OpenAI: GPT-5.1-Codex-Mini88%14/1666.5%1.3
32Qwen: Qwen3 VL 30B A3B Instruct88%14/1666.1%0.13
33Nex AGI: Nex-N2-Pro (free)88%14/1665.7%0.18
34Qwen: Qwen3 VL 235B A22B Instruct88%14/1664.4%0.060
35Qwen: Qwen3 VL 30B A3B Thinking88%14/1664.0%0.008
36Baidu: ERNIE 4.5 VL 424B A47B 88%14/1663.2%0.015
37Qwen: Qwen3 VL 32B Instruct81%13/1666.1%0.19
38OpenAI: GPT-4.1 Mini81%13/1663.6%0.26
39Qwen: Qwen3 VL 8B Thinking81%13/1663.2%0.072
40Google: Gemini 3.1 Flash Lite Preview81%13/1661.9%0.46
41ByteDance Seed: Seed 1.6 Flash81%13/1657.7%0.080
42Google: Gemini 2.5 Flash Lite81%13/1657.3%0.13
43Google: Gemma 4 26B A4B 81%13/1655.2%0.003
44NVIDIA: Nemotron 3 Nano Omni (free)81%13/1648.5%0.016
45Owl Alpha75%12/1673.7%0.37
46Xiaomi: MiMo-V2.5-Pro75%12/1671.5%0.18
47Z.ai: GLM 4.6V75%12/1665.7%0.12
48Mistral: Mistral Small 475%12/1657.7%0.065
49Qwen: Qwen3.5 Plus 2026-02-1569%11/1661.9%0.012
50Perceptron: Perceptron Mk169%11/1657.7%0.055
51Google: Gemini 3.1 Flash Lite69%11/1656.5%0.13
52Mistral: Ministral 3 8B 251269%11/1654.0%0.065
53OpenAI: GPT-5 Nano69%11/1654.0%0.026
54Google: Gemma 3 27B69%11/1650.2%0.006
55Qwen: Qwen2.5 VL 72B Instruct69%11/1650.2%0.005
56Qwen: Qwen3 VL 8B Instruct63%10/1656.9%0.028
57StepFun: Step 3.7 Flash63%10/1649.8%0.005
58Arcee AI: Trinity Large Thinking63%10/1641.6%<0.001
59Mistral: Mistral Large 3 251256%9/1659.0%0.17
60MiniMax: MiniMax-0156%9/1649.4%0.005
61OpenAI: GPT-4.1 Nano56%9/1648.1%0.044
62Tencent: Hy3 preview50%8/1654.0%0.007
63Mistral: Mistral Small 3.2 24B50%8/1653.6%0.017
64OpenAI: GPT-4o-mini50%8/1647.7%0.021
65Mistral: Ministral 3 14B 251244%7/1656.5%0.086
66Meta: Llama 4 Maverick44%7/1652.7%0.008
67Meta: Llama 4 Scout44%7/1648.5%0.009
68ByteDance Seed: Seed-2.0-Lite44%7/1647.7%<0.001
69Perplexity: Sonar38%6/1645.6%0.004
70Qwen: Qwen3.6 Plus38%6/1644.8%<0.001
71ByteDance Seed: Seed-2.0-Mini38%6/1637.2%<0.001
72Qwen: Qwen3.5-27B31%5/1649.4%<0.001
73Mistral: Ministral 3 3B 251231%5/1645.6%0.021
74Anthropic: Claude 3 Haiku31%5/1641.8%0.014
75Google: Gemma 3 12B25%4/1649.0%0.009
76Qwen: Qwen3.6 35B A3B25%4/1647.7%0.004
77Google: Gemma 3 4B25%4/1647.3%0.003
78NVIDIA: Nemotron Nano 12B 2 VL (free)25%4/1642.3%0.002
79Amazon: Nova Lite 1.019%3/1639.7%0.012
80Qwen: Qwen3.5-35B-A3B19%3/1639.3%<0.001
81Meta: Llama 3.2 11B Vision Instruct19%3/1635.6%<0.001
82Qwen: Qwen3.5-9B13%2/1641.4%<0.001

Primary sources & further reading

Frequently asked questions

What does the meo Regex & automata benchmark test?

Whether strings match a given regular expression / are accepted by a finite automaton, and related language-recognition reasoning — exact formal-language execution.

Why can't AI models game the Regex & automata benchmark?

The pattern, the test strings, and the ground-truth accept/reject labels are all generated together, so each instance is fresh and programmatically verifiable.

Which AI model is best at Regex & automata?

OpenAI: GPT-5.5 leads the Regex & automata domain on the meo benchmark with 100% accuracy (as of 2026-10-18).

Related: all domains · Regex & automata ranking · methodology · full leaderboard.