The Endless Exam

A benchmark for mathematical constructions with verifiable, uncapped scores.

Leaderboard

No toolsCode + web100 = reference · uncapped scores

Overall results with and without tools on 69 distinct instances
Rank	Model	Effort	Tools	Score	Output tokens per instance	Valid
Loading results…

Evaluation settings

Without tools: one response per instance, with a 128k output-token budget including reasoning.

Code + web: four CPU threads, 16 GiB and two hours per instance, with no cumulative output-token cap.

Score vs. output tokens

No toolsCode + webHover or tap a point for details.

The same data are available in the leaderboard above.

Output tokens include reasoning and all model turns when tools are used. ^* Some usage was not reported; actual token use may be higher. The line at 100 marks reference-level quality. Scores are uncapped.

The construction families

These animations use small, verified examples to make the rules easy to see. The benchmark uses much larger instances—with higher dimensions, longer codes and larger grids—where finding high-quality constructions is substantially harder. These illustrations are separate from the model results above.

Browse all 14 families

Points and patterns

These problems forbid a particular pattern inside a set or colour class. A larger construction must include more points or integers without creating that pattern.

Cap sets

— illustrationGold: selected points

A cap set is a collection of points with no three distinct selected points summing to zero. Here the coordinates are 0, 1 and 2, and addition wraps around modulo three. The objective is to select as many points as possible while respecting that rule.

The grey lattice contains all 27 points in three dimensions. Nine gold points emerge from the rule z = x² + y², with arithmetic modulo three. The lattice rotates so you can see the depth of the construction; the faint lines are coordinate guides, not extra constraints.

The evaluated instances use 9–11 dimensions: 19,683–177,147 candidate points instead of the 27 shown here.

Progression-free sets

— illustrationGold: selected points

An arithmetic progression consists of a starting point, a midpoint and an endpoint, with x + z = 2y. Over a finite field the same equation applies coordinate by coordinate, using modular arithmetic. A good construction contains many points and no such triple.

This example works modulo five. Six selected points satisfy x² + 2y² = 1. Late in the animation, two selected points determine where a third progression term would have to lie; that missing point is marked with a cross. The formal instances use F₅ in five or six dimensions and F₇ in four dimensions, rather than this two-dimensional picture.

Corner-free sets

— illustrationOutline: an excluded point

A corner consists of three grid points: a right-angle vertex and two points the same horizontal and vertical distance away. The distance may be positive or negative. Select as many points as possible without completing such a corner.

This 7 × 7 grid contains twenty selected points. Two of them form the highlighted right angle, but the third point is missing. The condition must hold for every possible corner. The evaluated grids have side lengths 138–185, with up to 34,225 candidate points.

Linear-equation-free sets

— illustrationGold: retained integers

Choose a subset of an integer interval that contains no nontrivial solution to a linear equation. In this example the equation is x + 2y = 3z. The solution x = y = z is harmless; every other solution using selected numbers is forbidden.

Ten integers are retained from 1 to 27. As the layout settles onto a number line, two selected values identify a weighted average. The marked average is outside the set, so that potential equation is not completed. Formal instances vary both the coefficients and the interval, with intervals containing 6,248–19,044 integers.

Schur colourings

— illustrationRows: sum-free colour classes

Colour every integer from 1 to N so that no colour class contains x, y and x + y, including x = y. With the number of colours held constant, a longer fully coloured interval is a better construction.

Here, thirteen integers separate into three sum-free classes. The example 1 + 1 = 2 crosses between classes; every sum of two members of a class must fall outside that class. The benchmark uses six, seven or eight colours and much longer intervals.

Geometry and distance

Geometric quality can be determined by a single limiting triangle or closest pair. In a graph, distance is measured by the number of edges along a path.

Heilbronn triangles

— illustrationGold: the minimum-area triangle

Place a fixed number of points inside a square. Every choice of three points forms a triangle, and the smallest triangle determines the objective. Improving most triangles is not enough if one nearly collinear triple remains.

Seven points create 35 triangles. As the points move, the minimum-area triangle is highlighted and its area is recomputed. The motion shows how the limiting triangle can change; it is not a claimed optimisation trajectory.

Formal instances use 26–50 points in square or triangular domains. Fifty points generate 19,600 triangles to consider.

Spherical codes

— illustrationArc: a closest pair

A spherical code places directions on the surface of a sphere while keeping every pair separated by a required angle. Here that threshold is 60 degrees. The objective is to fit as many directions as possible without violating the separation constraint.

Twelve directions appear around the sphere. The highlighted arc joins a closest pair, separated by approximately 63.4 degrees. Rotating the view helps distinguish points on the near and far sides.

The benchmark uses 12–15 dimensions and includes both a 60-degree threshold and variable-angle variants.

Degree–diameter graphs

— illustrationGold: reached within the step limit

How many vertices can a graph have if each vertex has only a few neighbours and every pair must remain a few steps apart? The degree limits the number of edges at each vertex; the diameter limits the longest shortest path.

The ten-vertex Petersen graph has degree three and diameter two. From the marked vertex, three neighbours are reached in one step and all remaining vertices in a second. The same distance limit holds from every starting vertex. The evaluated instances use different combinations of degree and diameter limits, each ranging from 3 to 7.

Designs and algorithms

Covering designs and Latin squares organise combinations. Matrix multiplication asks for a collection of arithmetic identities that works for every input.

Covering designs

— illustrationGold: covered pairs

A covering design uses larger subsets, called blocks, to cover every smaller subset of a specified size. The aim is to use as few blocks as possible while leaving no required subset uncovered.

On these seven elements, each triple covers three pairs. Seven triples cover all 21 pairs, each exactly once. The benchmark extends this to 18–30 elements, covering triples or four-element subsets with blocks of six or seven elements.

Orthogonal Latin squares

— illustrationPaired symbols: one from each square

A Latin square contains every symbol exactly once in each row and column. Two squares are orthogonal when overlaying them produces no repeated ordered pair. The task asks for as many pairwise orthogonal squares of the given order as possible.

The two order-three squares use the rules r + c and r + 2c, modulo three. Their cells align to form nine different pairs. The two components of each pair retain their visual identity as the squares come together. Formal instances use orders 10–20. Their larger arrays require many more row, column and pairwise-orthogonality checks.

Matrix multiplication

— illustrationGold: an active product or contribution

Construct a scheme that multiplies matrices using few scalar multiplications. Each multiplication may combine several input entries, and its result may contribute to several output entries. The resulting identities must hold for every input matrix, not just one numerical example.

This illustration uses Strassen’s seven-product scheme for 2 × 2 matrices. Watch input combinations feed the shared products, then recombine into the four output entries. The displayed numbers provide a concrete example of the identities.

The evaluated shapes are 4×4 by 4×5, 4×5 by 5×5, and 5×5 by 5×5, with integer coefficient constraints.

Sequences and codes

The constraint changes from controlling correlations within one sequence to separating pairs or triples of words.

Low-autocorrelation sequences

— illustrationBars: nonzero-shift correlations

Choose a sequence of +1 and −1 signs. At each nonzero shift, compare the overlapping signs and add their products. These correlations are squared and summed; a smaller total gives a larger merit factor, the quantity being maximised.

A length-thirteen Barker sequence is compared with its shifted copies. The bars show all twelve nonzero-shift correlations. Their squared sum is six, giving this short sequence a merit factor of 169/12, approximately 14.08.

The benchmark sequences have lengths 240–375, with many more shifts whose correlations must be controlled together.

Shannon codes

— illustrationGold: a distinguishing coordinate

Imagine a channel where equal or neighbouring symbols on a cycle can be confused. Two codewords are distinguishable only if at least one position uses symbols far enough apart on that cycle. The task is to construct as many mutually distinguishable words as possible.

Each circle is a five-symbol channel position. Five two-symbol words are selected, pairing i with 2i modulo five. Two words may be confusable in the first position and still be separated in the second.

Formal instances use seven or nine symbols and lengths 24–64. Verified products of smaller codes can describe very large constructions.

Trifference codes

— illustrationGold: three distinct symbols in one position

A trifference code uses the symbols 0, 1 and 2. For every three distinct codewords, some position must contain all three symbols. It is not enough to distinguish words pair by pair: the same position must separate the entire triple.

The display contains nine words of length four. Three rows are highlighted at a time, and a gold column shows their three different symbols. This small code already has 84 triples, all of which satisfy the condition.

The formal code lengths are 64, 96, 144 and 192. Compact algebraic certificates allow much larger code families to be submitted and checked.

How scoring works

Relative quality

For a valid maximisation answer, relative quality is the objective divided by its reference. Minimisation uses the inverse ratio. A value of 1 matches the reference, and improvements above 1 retain their magnitude.

This evaluation uses bench-v1.0, with unchanged instances and references. Thirty instances use published frontiers to compare models directly with existing mathematical results. The other 39 use verified construction baselines, which need not be the strongest published constructions. Scores are not clipped at reference parity; answer formats and verification budgets constrain which constructions can be evaluated. The overall score is 100 times mean relative quality over all 69 instances. Invalid constructions score zero. Tool-free budget exhaustion also scores zero; tool-assisted runs are scored from constructions saved before the deadline.

RELATIVE QUALITYFIG. 01

1.24relative
quality

Objective / reference Drag to explore

24% above the published reference (relative quality = 1.24).

SCORING ILLUSTRATIONNo cap at 1.

Run the benchmark

Generators, verifiers, reference constructions and model responses are available in the repository. The paper describes the evaluation and results.

VERIFY YOUR FIRST CONSTRUCTION

# A four-point cap set, checked locally.
python bench/exam.py verify \
  --family capset \
  --params '{"d":2}' \
  --answer examples/capset.json

valid: true / objective: 4

Run from the repository after installing its requirements.