What Is the HLE Benchmark? Definition, How It Works & Examples (2026)

The HLE benchmark (Humanity's Last Exam) is a highly challenging, multi-disciplinary artificial intelligence evaluation dataset specifically designed to measure the absolute upper limits of frontier large language model (LLM) reasoning and expert-level knowledge. It was created as a direct response to the rapid saturation of traditional benchmarks, where models like GPT-4 and Claude began achieving near-human or superhuman scores on tests that were once considered hard, such as MMLU and MATH. The HLE benchmark functions as a collated set of over 3,000 expert-crafted, closed-ended questions that span dozens of academic and professional domains, constructed with the explicit goal of being impossible for current non-specialist humans to answer without extensive research, thereby providing a durable test bed for measuring genuine artificial superintelligence.

What Is the HLE Benchmark?

The HLE benchmark is a collaborative project born from a recognition that existing evaluation methodologies in AI were failing to differentiate between increasingly competent models. Co-introduced by the Center for AI Safety (CAIS) and Scale AI in late 2024, the dataset is colloquially termed "Humanity's Last Exam" because it may represent one of the final written examinations where humans can confidently assert a performance advantage over machines. Unlike prior benchmarks that draw from undergraduate textbooks or standardized test banks, questions for the HLE benchmark were sourced via an open global competition explicitly seeking multi-modal, abstract reasoning problems that require deep synthesis of niche information rather than mere retrieval. The result is a corpus covering advanced mathematics (including unsolved problem derivations), quantum physics, virology, historical linguistics, professional law, and esoteric trivia [1].

As of 2026, the benchmark is widely regarded as the primary "impossible test" for next-generation models, with top-performing systems still struggling to surpass a 25% accuracy rate on the hardest private-held subset, even as they exceed 95% on older standards like GSM8K.

How Does the HLE Benchmark Evaluation Work?

The operational mechanism of the HLE benchmark relies on a strict zero-shot, closed-book evaluation protocol to prevent contamination and shortcuts. The underlying architecture of the test is a curated data pipeline that transforms expert submissions into model-readable prompts.

The process works as follows:

1. Question Submission and Validation: Domain experts submit questions along with a verified correct answer and detailed justifications. Each entry undergoes a multi-stage peer review by other experts to ensure unambiguity and extreme difficulty.
2. Canonical Formatting: Accepted questions are templated into a multiple-choice or exact-match short-answer format. The benchmark explicitly bans open-ended essay formats to enable strict, automatic scoring without subjective judge models (LLM-as-a-Judge), which can introduce bias.
3. Execution Protocol: The model is presented with the question text and, optionally, an image or chart. The model must process the input in a single forward-pass context window or through a controlled chain-of-thought log without access to external tools, search engines, or Python interpreters. This tests the raw latent knowledge and reasoning of the base model.
4. Scoring Metric: Performance is measured by strict exact-match accuracy. Partial credit is rarely awarded unless specified by the question author. The separation of the dataset into a public "dev" set and a larger, private "test" set (the HLE-Private split) is critical to preventing benchmark leakage and overfitting.

The evaluation’s primary technical distinction is its emphasis on "grokking"—models must combine fragmented, inter-disciplinary concepts to reason about a specific query, rather than just pattern-matching on training data.

What Are the Key Variants or Splits of the HLE Benchmark?

There are two principal components that define how the HLE benchmark is used in practice:

HLE-Public A smaller subset of roughly 500 questions released openly with answers, facilitating academic research and model development. This variant is often used for few-shot prompting experiments and validation, though the authors strongly caution that high scores here can be indicative of overfitting rather than genuine reasoning since the set is accessible for training.

HLE-Private The main test set comprising over 2,500 questions, the answers to which are sequestered and held by Scale AI. Frontier labs like OpenAI, Google DeepMind, and Anthropic must submit their models to a semi-automated evaluation harness without ever seeing the ground truth. This private split is the true measure of capability, maintained to ensure the exam remains a lasting challenge.

HLE-MultiModal A sub-component explicitly testing vision-language models (VLMs). This set cannot be solved via text input alone; it requires interpreting complex diagrams, protein folding visualizations, historical manuscripts, or geographic maps embedded within the prompt. As of 2026, the gap between text-only and multi-modal reasoning performance on this set remains a notable research vector [2].

How Does the HLE Benchmark Differ from Other AI Benchmarks?

To understand the HLE benchmark's place in the ecosystem, it is useful to contrast it with the previous generation of evaluation tools.

The HLE benchmark is distinguished not just by difficulty but by a comprehensive abstraction of conceptual integration. GPQA tests deep knowledge but often within a single field; HLE demands cross-disciplinary reasoning, making it a stricter test of fluid intelligence rather than crystallized knowledge.

What Are Real-World Examples of HLE Benchmark Questions?

To characterize the benchmark’s extreme depth, consideration of specific real-world examples is essential. The dataset spans domains that typically require a PhD to parse, let alone answer.

• Mathematical Physics: A question might present a simplified, but novel, derivation of a Hawking radiation flux effect in a modified non-standard quantum field theory and ask the model to identify the specific symmetry break that invalidates the derivation.
• Historical Linguistics: The model is provided with a sentence in the extinct Tocharian B language using unique diacritical marks, followed by a question about the subjunctive mood conjugation inferred solely from the provided isolated text example, requiring on-the-fly morphological reconstruction.
• Organic Chemistry: A multi-step synthesis problem shows a target chiral molecule and two potential reactant paths, asking the model to select the path that preserves steric hindrance integrity given an anomalous solvent effect, utilizing orbital symmetry rules.
• Classical Musicology: An image of an obscure Renaissance-era mensural notation fragment is displayed. The model must identify the implied hexachordum durum shift without audible input, relying purely on visual analysis of the notational anomalies.

What Are the Practical Use Cases and Benefits?

The HLE benchmark serves functions beyond a mere leaderboard score. Its primary practical use cases include:

• Frontier Safety Assessment: The primary user case is not capability framing but risk assessment. By testing the absolute ceiling of a model’s deductive insight, the HLE benchmark allows safety teams at labs like Anthropic to deduce if a model is capable of devising novel, dangerous strategies that might not surface in normal safety dialogs [1].
• Architectural Diagnostics: Researchers use the performance delta between the HLE-Public and HLE-Private sets to diagnose catastrophic overfitting. If a model scores 40% on the public set but nearly 0% on the private set, it indicates the model has likely memorized web data associated with the public answers rather than learned the underlying reasoning patterns.
• Scaling Law Calibration: Developers like Google DeepMind use the HLE as a target for predicting scaling law plateaus. As of 2026, the near-flat scaling curve on HLE compared to sharp improvements on MMLU highlights that simple increases in compute clusters and data volume yield diminishing returns on expert synthesis [3].
• Hybrid Human-AI Performance Baselines: The examine also facilitates a rare human-AI evaluation model. Calibrated human experts given 2 hours per question often score 40-50% (as the questions are designed to be hard even for them), but when paired with a sub-par AI as a research assistant, combined scores jump dramatically, providing a quantifiable metric for AI augmentation at the highest professional levels.

What Are the Limitations and Criticisms of the HLE Benchmark?

Despite its utility, the HLE benchmark carries specific limitations and methodological trade-offs that affect how it should be interpreted.

• Knowledge-Gated, Not Intelligence-General: A dominant criticism is that the benchmark tests for embedded esoteric knowledge rather than pure logical deduction. A model might fail a question about Tocharian B syntax not because it cannot reason, but because the Curated Pre-Training corpus lacked sufficient data on extinct Indo-European tongues. It validates crystallized knowledge, not complete fluid intelligence.
• Human Baseline Instability: The name "Humanity’s Last Exam" is somewhat aspirational. The difficulty bias implies that the average human answer rate is effectively 0%, making it impossible to compare the "average human" against the model. The only valid matched baseline is PhD-level domain experts, and recruiting these individuals for a verified, statistically significant baseline is economically and logistically prohibitive.
• Scoring Brittleness: The strict exact-match format clashes with the stochastic nature of LLMs. A model may provide a correct reasoning chain in its latent space but output a slightly off-by-one value, a wrong LaTeX formatting tag, or the correct answer with a redundant symbol, resulting in a score of zero. This undercounts the actual reasoning ability and inflates the perceived "failure" of models.
• Static Nature and Leakage Risk: Although the private set attempts to mitigate contamination, the web leak of 100+ sample questions in early 2025 demonstrated the difficulty of maintaining integrity in a connected world. This risk promotes a cat-and-mouse dynamic where the benchmark must be continuously expanded with new expert questions at a rate that likely outpaces the organizing budget over the long term.

Frequently Asked Questions

Q: Who created the HLE benchmark?
A: The benchmark was developed and launched by the Center for AI Safety (CAIS) in collaboration with Scale AI. The actual questions, however, were contributed by a global network of nearly a thousand anonymous academic and industry domain experts through a structured competition format.

Q: Why do models score so low on the HLE benchmark?
A: Models perform poorly because the questions intentionally exploit the long-tail distribution of human knowledge. Unlike standardized tests, HLE questions rarely appear in any published internet text exactly; doing well requires genuine abstract synthesis of concepts from disparate, obscure fields, which current transformer architectures struggle to compute in a single feedforward pass.

Q: Is the HLE benchmark truly the “last exam” for AI?
A: No, this is a symbolic framing. The benchmark tests linguistic and symbolic reasoning on static text and images, not embodied agentic tasks, real-world mechanical repair, or long-term planning. Surpassing HLE would indicate massive advances in retrieval and reasoning, but it would not prove AI superiority in physical or dynamic strategic domains.

Q: Can I take the HLE benchmark as a human to test myself?
A: A small selection of the HLE-Public sample questions is available online for the curious, but taking the full exam is not feasible due to the immense required compute-level knowledge. An average specialist would be expected to fail on topics outside their specific area of deep expertise.

Q: How does the HLE benchmark handle multi-modal inputs?
A: For multi-modal questions, the raw image or audio frequency chart is presented as a base64-encoded asset or a direct pixel value matrix alongside the text prompt. The model must identify the semantic connection between the visual signal and the textual query without tool use.

Q: What is a “passing” grade on the HLE benchmark?
A: As of 2026, there is no standard passing threshold, and model performance is reported as a raw accuracy percentage. The community does not anticipate a “passing” score for frontier AIs for at least several more years, as the average university professor is estimated to score below 10% on random subsets outside their field.