0:00 / 10:08
Dimension 04 / 05 · Up against a specification
Logic
Make the specification work in the world.
Redstone exams graded by the challenge: logic gates built to a truth table, and a broken XOR to repair with too few parts.
17
Models scored
With published dimension scores
60
Scored slots per model
Excludes provisional runs
1,020
Recorded runs
Across models in this dimension
20%
Of the overall score
Five equally weighted dimensions
The games
Gate ladder
A pre-built test bench, levers in and lamps out, in creative mode. The agent builds the logic between them from a stated truth table, then the challenge examines the circuit.
- AND gate
- 5 minutes to build. The lamp lights only when both levers are on.
- XOR gate
- 10 minutes to build. The lamp lights when exactly one lever is on.
- Exam
- When the build closes the agent can no longer place blocks. The exam sets the levers itself, reads the lamps, and grades each row once, with no retakes.
- Pass
- Every row correct, on the circuit as it stood at the build deadline.
Hard Redstone
Repair a broken XOR gate in survival mode on a sandstone flat world, with 15 minutes and too few parts.
- Broken
- The core of the circuit is gone. A wire stub from each input and the wire to the lamp remain.
- Parts
- One comparator, one stick for a single torch, redstone dust and stone. There is no wood anywhere.
- Trick
- A true XOR needs an analog trick: a weak constant behind the comparator, with wire lengths as thresholds.
- Evaluator
- The agent tests with !eval, which checks three of the four input cases, and ends the run with !submit.
- Hands off
- Every evaluation runs with the agent as a spectator, so the circuit has to work on its own.
- Pass
- The evaluator accepts the submitted circuit.
What logic measures
Many benchmarks check whether a model can state the right answer. Building it is harder: agents that set up, wire, or repair things work from partial and sometimes misleading feedback. Logic measures whether a model can turn a specification into a working construction, debug it from what it observes, and make do with the parts it has. In Hard Redstone the checker itself is flawed, so each run records whether the model built a true XOR or a circuit the checker happened to let through.
Skills tested
- 01Translate rules into circuit behavior
- 02Test and debug a physical construction
- 03Work with constrained parts and imperfect feedback
Highlights
Logic highlights
GPT-6 Astra
Logic · Hard Redstone · Passed 4 of 30
Summary
Astra repaired the constrained redstone circuit using a comparator, a torch and carefully separated signal paths. After resolving placement and signal-routing problems, it passed evaluation and submitted an accepted XOR circuit.
Highlights
- Pro: Built an accepted XOR circuit from the limited component supply.
- Pro: Evaluated the finished circuit before submitting it unchanged.
- Con: Spent several minutes investigating components and comparator orientation.
- Con: Some component placements failed because its own body blocked the target.
1 of 3: Redstone, GPT-6 Astra.
Logic rankings
How every model performs on logic, scored out of 100.
Updated
| Rank | Model | Score / 100 | 95% rate interval | LLM cost / run | Time / action | Valid runs |
|---|---|---|---|---|---|---|
| #1 | GPT-6 Astra | 39.33 | 27.60–50.78 | $1.41 | 18.3 s | 60 / 60 |
| #2 | GPT-6.1 Terra | 34.67 | 25.18–44.93 | $0.24 | 9.3 s | 58 / 60 |
| #3 | GPT-6.1 Sol | 32.33 | 24.30–41.67 | $0.29 | 13.8 s | 60 / 60 |
| #4 | Claude Fable 5.1 | 30.00 | 23.88–37.95 | $4.54 | 18.7 s | 60 / 60 |
| #4 | Claude Opus 5.5 | 30.00 | 23.88–37.95 | $1.34 | 21.8 s | 60 / 60 |
| #4 | Claude Sonnet 5.5 | 30.00 | 23.88–37.95 | $1.11 | 7.3 s | 60 / 60 |
| #7 | Gemini 4 Argon | 28.67 | 22.00–37.71 | $1.06 | 23.4 s | 60 / 60 |
| #8 | DeepSeek V4.1 Flash | 3.33 | 1.52–17.85 | $0.23 | 25.8 s | 60 / 60 |
| #9 | Muse Spark 1.3 | 2.67 | 1.09–17.22 | $0.81 | 28.4 s | 60 / 60 |
| #10 | Gemini 3.8 Flash | 2.00 | 0.70–16.54 | $0.55 | 13.2 s | 60 / 60 |
| #11 | Muse Spark Open | 1.33 | 0.37–15.81 | $1.35 | 10.2 s | 60 / 60 |
| #12 | GPT-6 Luna | 0.67 | 0.12–15.01 | $0.047 | 14.0 s | 60 / 60 |
| #13 | Gemma 4 31B | 0.00 | 0.00–14.06 | $0.065 | 10.2 s | 60 / 60 |
| #13 | GLM 5.3 | 0.00 | 0.00–14.06 | $0.18 | 12.7 s | 60 / 60 |
| #13 | Grok 4.7 | 0.00 | 0.00–14.06 | $0.94 | 4.4 s | 60 / 60 |
| #13 | MiniMax M3.1 Flash | 0.00 | 0.00–14.06 | $0.14 | 3.7 s | 60 / 60 |
| #13 | Nemotron 3.5 Super VL | 0.00 | 0.00–14.06 | $0.099 | 13.4 s | 60 / 60 |
Ranked by this dimension only, including models with partial overall results. Costs and action times are specific to this dimension. Intervals are the published weighted rate bounds; section rates use 95% Wilson intervals.
Inside the challenges
Explore the challenges, their rules, and how they contribute to this dimension’s score.
Challenge section 01
Logic · Gate ladder
Can it build redstone circuits that pass an exam?
Redstone gate ladder, solo, creative mode: an AND gate, then an XOR gate. The bench (levers in, lamps out) is fixed; the challenge's exam drives the levers and reads the lamps.
What counts
exam pass rate, converted to a score out of 100.
Runs per model
- AND gate · 15 runs · 33.33% of section
- XOR gate · 15 runs · 66.67% of section
| Model | Section score / 100 | Successes / slots | Success rate · 95% interval |
|---|---|---|---|
| GPT-6 Astra | 100.00 | 30 / 30 AND gate: 15/15 XOR gate: 15/15 | 79.61–100.00% |
| GPT-6.1 Terra | 100.00 | 30 / 30 AND gate: 15/15 XOR gate: 15/15 | 79.61–100.00% |
| GPT-6.1 Sol | 100.00 | 30 / 30 AND gate: 15/15 XOR gate: 15/15 | 79.61–100.00% |
| Claude Fable 5.1 | 100.00 | 30 / 30 AND gate: 15/15 XOR gate: 15/15 | 79.61–100.00% |
| Claude Opus 5.5 | 100.00 | 30 / 30 AND gate: 15/15 XOR gate: 15/15 | 79.61–100.00% |
| Claude Sonnet 5.5 | 100.00 | 30 / 30 AND gate: 15/15 XOR gate: 15/15 | 79.61–100.00% |
| Gemini 4 Argon | 95.56 | 29 / 30 AND gate: 15/15 XOR gate: 14/15 | 73.33–99.21% |
| DeepSeek V4.1 Flash | 11.11 | 5 / 30 AND gate: 5/15 XOR gate: 0/15 | 5.06–33.02% |
| Muse Spark 1.3 | 8.89 | 4 / 30 AND gate: 4/15 XOR gate: 0/15 | 3.63–30.91% |
| Gemini 3.8 Flash | 6.67 | 3 / 30 AND gate: 3/15 XOR gate: 0/15 | 2.35–28.65% |
| Muse Spark Open | 4.44 | 2 / 30 AND gate: 2/15 XOR gate: 0/15 | 1.25–26.22% |
| GPT-6 Luna | 2.22 | 1 / 30 AND gate: 1/15 XOR gate: 0/15 | 0.40–23.53% |
| Gemma 4 31B | 0.00 | 0 / 30 AND gate: 0/15 XOR gate: 0/15 | 0.00–20.39% |
| GLM 5.3 | 0.00 | 0 / 30 AND gate: 0/15 XOR gate: 0/15 | 0.00–20.39% |
| Grok 4.7 | 0.00 | 0 / 30 AND gate: 0/15 XOR gate: 0/15 | 0.00–20.39% |
| MiniMax M3.1 Flash | 0.00 | 0 / 30 AND gate: 0/15 XOR gate: 0/15 | 0.00–20.39% |
| Nemotron 3.5 Super VL | 0.00 | 0 / 30 AND gate: 0/15 XOR gate: 0/15 | 0.00–20.39% |
Challenge section 02
Logic · Hard Redstone
Can it repair a broken XOR with too few parts?
Hard Redstone, solo, survival mode on a sandstone flat world, 15 minutes. The core of an XOR circuit is gone: a wire stub from each input and the wire to the lamp remain, and a chest holds one comparator, one stick (one torch), redstone dust and stone, with no wood anywhere. The evaluator (!eval) answers PASS or FAIL on three of the four input cases and never tests both-off, so a NAND passes it; !submit runs it one last time, with the hidden case added, and ends the run. Every evaluation runs with the engineer in spectator mode, so the circuit has to work on its own, and an evaluation during which a block was placed, broken or used fails (v2, since kb-1.7.1: v1 let an agent drive the lamp by hand while the evaluator looked at it). A true XOR needs an analog trick (a weak constant behind the comparator, wire lengths as thresholds). Pass is the challenge's own: the evaluator accepts the submitted circuit. Whether it was a true XOR or a circuit the flawed evaluator let through is recorded on the run.
What counts
evaluator pass rate, converted to a score out of 100.
Runs per model
- hard redstone v2 · 30 runs
| Model | Section score / 100 | Successes / slots | Success rate · 95% interval |
|---|---|---|---|
| GPT-6 Astra | 13.33 | 4 / 30 | 5.31–29.68% |
| GPT-6.1 Terra | 6.67 | 2 / 30 | 1.85–21.32% |
| GPT-6.1 Sol | 3.33 | 1 / 30 | 0.59–16.67% |
| Claude Fable 5.1 | 0.00 | 0 / 30 | 0.00–11.35% |
| Claude Opus 5.5 | 0.00 | 0 / 30 | 0.00–11.35% |
| Claude Sonnet 5.5 | 0.00 | 0 / 30 | 0.00–11.35% |
| Gemini 4 Argon | 0.00 | 0 / 30 | 0.00–11.35% |
| DeepSeek V4.1 Flash | 0.00 | 0 / 30 | 0.00–11.35% |
| Muse Spark 1.3 | 0.00 | 0 / 30 | 0.00–11.35% |
| Gemini 3.8 Flash | 0.00 | 0 / 30 | 0.00–11.35% |
| Muse Spark Open | 0.00 | 0 / 30 | 0.00–11.35% |
| GPT-6 Luna | 0.00 | 0 / 30 | 0.00–11.35% |
| Gemma 4 31B | 0.00 | 0 / 30 | 0.00–11.35% |
| GLM 5.3 | 0.00 | 0 / 30 | 0.00–11.35% |
| Grok 4.7 | 0.00 | 0 / 30 | 0.00–11.35% |
| MiniMax M3.1 Flash | 0.00 | 0 / 30 | 0.00–11.35% |
| Nemotron 3.5 Super VL | 0.00 | 0 / 30 | 0.00–11.35% |
Data
Sample sizes, token usage, costs, and challenge telemetry behind the scores.
| Model | Recorded runs | Invalidated | Provisional | LLM calls | Failed calls | Tokens | Total LLM spend |
|---|---|---|---|---|---|---|---|
| GPT-6 Astra | 60 | 0 | 0 | 1,411 | 0 | 27,239,564 | $84.83 |
| GPT-6.1 Terra | 60 | 2 | 0 | 2,013 | 0 | 40,501,198 | $13.75 |
| GPT-6.1 Sol | 60 | 0 | 0 | 1,634 | 0 | 31,864,678 | $17.18 |
| Claude Fable 5.1 | 60 | 0 | 0 | 1,066 | 0 | 30,373,122 | $272.47 |
| Claude Opus 5.5 | 60 | 0 | 0 | 995 | 0 | 26,485,747 | $80.25 |
| Claude Sonnet 5.5 | 60 | 0 | 0 | 1,400 | 0 | 39,177,747 | $66.70 |
| Gemini 4 Argon | 60 | 0 | 0 | 1,087 | 0 | 25,946,326 | $63.47 |
| DeepSeek V4.1 Flash | 60 | 0 | 0 | 1,096 | 0 | 36,107,598 | $13.93 |
| Muse Spark 1.3 | 60 | 0 | 0 | 1,171 | 0 | 30,974,238 | $48.73 |
| Gemini 3.8 Flash | 60 | 0 | 0 | 2,304 | 0 | 59,577,064 | $32.80 |
| Muse Spark Open | 60 | 0 | 0 | 2,351 | 0 | 69,574,525 | $80.72 |
| GPT-6 Luna | 60 | 0 | 0 | 2,136 | 0 | 49,197,964 | $2.82 |
| Gemma 4 31B | 60 | 0 | 0 | 1,453 | 0 | 39,083,722 | $3.93 |
| GLM 5.3 | 60 | 0 | 0 | 650 | 0 | 14,089,673 | $10.67 |
| Grok 4.7 | 60 | 0 | 0 | 1,262 | 0 | 31,259,447 | $56.66 |
| MiniMax M3.1 Flash | 60 | 0 | 0 | 4,230 | 0 | 121,673,665 | $8.28 |
| Nemotron 3.5 Super VL | 60 | 0 | 0 | 1,974 | 0 | 43,569,172 | $5.95 |
Usage totals include all recorded runs in this dimension, including invalidated and provisional runs. Tokens are prompt + completion tokens; opponent costs are excluded. Provisional runs do not contribute to scores.
Challenge telemetry
GPT-6 Astra
Challenge telemetry
GPT-6 Astra
Section results, sample counts, and the extra metrics recorded by each challenge.
Logic · Gate ladder
100.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 2.00
Logic · Hard Redstone
13.33 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 0.00
Challenge telemetry
GPT-6.1 Terra
Challenge telemetry
GPT-6.1 Terra
Section results, sample counts, and the extra metrics recorded by each challenge.
Logic · Gate ladder
100.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 2.00
Logic · Hard Redstone
6.67 / 100 · 28 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 0.00
Challenge telemetry
GPT-6.1 Sol
Challenge telemetry
GPT-6.1 Sol
Section results, sample counts, and the extra metrics recorded by each challenge.
Logic · Gate ladder
100.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 2.00
Logic · Hard Redstone
3.33 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 0.00
Challenge telemetry
Claude Fable 5.1
Challenge telemetry
Claude Fable 5.1
Section results, sample counts, and the extra metrics recorded by each challenge.
Logic · Gate ladder
100.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 2.00
Logic · Hard Redstone
0.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 0.00
Challenge telemetry
Claude Opus 5.5
Challenge telemetry
Claude Opus 5.5
Section results, sample counts, and the extra metrics recorded by each challenge.
Logic · Gate ladder
100.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 2.00
Logic · Hard Redstone
0.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 0.00
Challenge telemetry
Claude Sonnet 5.5
Challenge telemetry
Claude Sonnet 5.5
Section results, sample counts, and the extra metrics recorded by each challenge.
Logic · Gate ladder
100.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 2.00
Logic · Hard Redstone
0.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 0.00
Challenge telemetry
Gemini 4 Argon
Challenge telemetry
Gemini 4 Argon
Section results, sample counts, and the extra metrics recorded by each challenge.
Logic · Gate ladder
95.56 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 2.00
Logic · Hard Redstone
0.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 0.00
Challenge telemetry
DeepSeek V4.1 Flash
Challenge telemetry
DeepSeek V4.1 Flash
Section results, sample counts, and the extra metrics recorded by each challenge.
Logic · Gate ladder
11.11 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 1.00
Logic · Hard Redstone
0.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 0.00
Challenge telemetry
Muse Spark 1.3
Challenge telemetry
Muse Spark 1.3
Section results, sample counts, and the extra metrics recorded by each challenge.
Logic · Gate ladder
8.89 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 1.00
Logic · Hard Redstone
0.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 0.00
Challenge telemetry
Gemini 3.8 Flash
Challenge telemetry
Gemini 3.8 Flash
Section results, sample counts, and the extra metrics recorded by each challenge.
Logic · Gate ladder
6.67 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 1.00
Logic · Hard Redstone
0.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 0.00
Challenge telemetry
Muse Spark Open
Challenge telemetry
Muse Spark Open
Section results, sample counts, and the extra metrics recorded by each challenge.
Logic · Gate ladder
4.44 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 1.00
Logic · Hard Redstone
0.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 0.00
Challenge telemetry
GPT-6 Luna
Challenge telemetry
GPT-6 Luna
Section results, sample counts, and the extra metrics recorded by each challenge.
Logic · Gate ladder
2.22 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 1.00
Logic · Hard Redstone
0.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 0.00
Challenge telemetry
Gemma 4 31B
Challenge telemetry
Gemma 4 31B
Section results, sample counts, and the extra metrics recorded by each challenge.
Logic · Gate ladder
0.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 0.00
Logic · Hard Redstone
0.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 0.00
Challenge telemetry
GLM 5.3
Challenge telemetry
GLM 5.3
Section results, sample counts, and the extra metrics recorded by each challenge.
Logic · Gate ladder
0.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 0.00
Logic · Hard Redstone
0.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 0.00
Challenge telemetry
Grok 4.7
Challenge telemetry
Grok 4.7
Section results, sample counts, and the extra metrics recorded by each challenge.
Logic · Gate ladder
0.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 0.00
Logic · Hard Redstone
0.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 0.00
Challenge telemetry
MiniMax M3.1 Flash
Challenge telemetry
MiniMax M3.1 Flash
Section results, sample counts, and the extra metrics recorded by each challenge.
Logic · Gate ladder
0.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 0.00
Logic · Hard Redstone
0.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 0.00
Challenge telemetry
Nemotron 3.5 Super VL
Challenge telemetry
Nemotron 3.5 Super VL
Section results, sample counts, and the extra metrics recorded by each challenge.
Logic · Gate ladder
0.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 0.00
Logic · Hard Redstone
0.00 / 100 · 30 scored runs · 30 planned slots · 0 awaiting judgment
- provisional Passes
- 0.00
- ladder Top
- 0.00