Lab 01–04
Down to the metal
Machine bytes, source to native, LLVM IR, and SSA rules that are not actually checked.
Invalid LLVM IR is accepted
Appendix B — the labs
Eight labs. Reproducible
with Python and Clang.
The book demonstrates rather than surveys. Each claim in Part II is shown on a worked example small enough to check by hand, and every one of them ships as a lab you can run. Where a number is owed and has not been taken, the chapter says so rather than estimating one.
Lab 05
AIR compiles
AIR parses, verifies and compiles — and refuses four bad programs for four different declared reasons.
Lab 06
Validation cost
Nine faults injected across languages. Six of them are silent in every language tested.
6 of 9 faults silent
Lab 07
Structured diagnostics
The same fault, two error messages. A conventional message repaired it 40% of the time; a diagnostic naming the violated obligation and enumerating the legal repairs, 100%. Format alone did nothing (p = 0.36).
40% → 100% repair
Lab 08
Regeneration amplification
Repair cost is a straight line in the size of the file, not the size of the mistake (R² = 0.9996). Ask for a patch instead and the cost stops scaling — but 15 of 24 patches would not apply.
505× the fix
Labs 1–6 run offline. Labs 7 and 8 call a model API and are excluded from the default runner. Lab 4’s result — invalid LLVM IR accepted — and Lab 6’s uninitialised-accumulator row are toolchain-specific, verified on Apple Clang 15.0.0 on arm64, and the book says so.
Lab 8 — Chapter 8
What a one-line fix costs
Every AI coding tool is priced per token. So the question is not whether a model can fix a bug — it can — but what you are billed to receive the fix. The same fault, needing the same six-token correction, was dropped into modules of four sizes, with the surrounding code doing nothing but making the file bigger.
Tokens emitted to deliver a six-token fix
The same fault, in modules of four sizes. The surrounding code does nothing but make the file bigger.
Return the whole programReturn a minimal diff
Fitting the upper line: emitted = 141 + 2.06 × program tokens, R² = 0.9996. The cost of the repair is a straight line in the size of the file, and essentially all of the variance is explained by it.
505×
the size of the mistake
A six-token fix in a 1,408-token file cost 3,032 emitted tokens to deliver — and the line was still climbing at the right-hand edge of the table.
99.7%
of it was already correct
Of those 3,032 tokens, the model changed 8.2. Everything else was code that was right when it arrived and right when it left.
8.2 / 6
tokens changed, against the floor
The model was not confused. Across all four sizes its edits ran between 0.9× and 1.6× the minimal fix. It found the answer immediately every time.
So the 505× is not confusion. It is the cost of the only delivery mechanism available — writing the file out again — and it is charged in full whether the fix is one token or a hundred.
Return the whole program
The normal workflow. It works, and you pay for the file.
| Program | Emitted | vs fix | Repaired |
|---|---|---|---|
| 50 tok | 222 | 36.9× | 6/6 |
| 341 tok | 876 | 146.0× | 6/6 |
| 729 tok | 1,632 | 272.0× | 6/6 |
| 1,408 tok | 3,032 | 505.4× | 5/6 |
Return a minimal diff
The cheapest thing it could possibly emit. The cost stops scaling — and it stops working.
| Program | Emitted | Repaired | Would not apply |
|---|---|---|---|
| 50 tok | 191 | 4/6 | 2/6 |
| 341 tok | 198 | 2/6 | 4/6 |
| 729 tok | 181 | 2/6 | 4/6 |
| 1,408 tok | 203 | 1/6 | 5/6 |
Then why not just ask for a patch?
The model was offered that, because the comparison is only fair if it is allowed its best case. Asked for a minimal diff it emits about two hundred tokens no matter how big the program is. The cost stops scaling — exactly as the argument predicts it would, if repair could be made local.
And it does not work. Success falls from four in six to one in six as the file grows, and at the largest size five diffs in six could not be applied to the program at all: wrong line numbers, wrong context, hunks that did not match. Fifteen of twenty-four patch attempts were unusable.
The producer can be cheap or it can be reliable, and at present it cannot be both.
What defeats it is that a line number in a text file is not a stable address, so the patch is aimed at coordinates that shift, and it misses more often the more file there is to miss in. This is not an argument that models will not improve at emitting diffs. They will. It is an argument that they are being asked to hit a moving target — and Chapter 20 is about giving them a fixed one.
Limits, as the book states them. One fault class, one language, two models from one vendor, and a largest program of 1,408 tokens — which is a page, not a codebase. The straight line is very straight over that range and the book claims no evidence about what it does beyond it. Lab 8 runs in a few minutes.
Lab 7 — Chapter 21
Forty per cent to a hundred
Six programs, each carrying a distributed fault — one wrong assumption touching several sites, because a fault at a single site cannot tell these conditions apart. Each is rejected three ways. Two models, five trials, 180 attempts. A repair counts only if the program passes a hidden test suite it never saw.
A prose what a test runner emits now: expected 2.5, got 5.0
B structured the same facts in a fixed schema, nothing added
C full plus obligation, cause, legal alternatives, repair_scopeA against B is pure format — identical information, one as prose and one in a schema. The difference is ten points, and on a permutation test that is p = 0.36: indistinguishable from chance.
B against C is fifty points. It adds the obligation, the cause and the enumerated legal repairs. So the value is in what the record says, not the shape it is written in. A schema is a container; what pays is the payload.
The repairs under C were also slightly smaller than the reference fix rather than larger — 0.81× minimal edit size against 0.92×.
Distributed faults repaired
Two models · five trials · 180 attempts. A repair counts only if the program passes a hidden test suite.
AProse
40%24 / 60
what a test runner emits now
BStructured
50%30 / 60
the same facts in a fixed schema
CFull record
100%60 / 60
plus obligation, cause, legal, repair_scope
- A vs B — format only
- +10 pts · p = 0.36
- B vs C — content
- +50 pts · p < 0.00001
The weaker model gained far more
Haiku goes from 23% to 100%, a factor of 4.3. Sonnet from 57% to 100%, a factor of 1.8. The representation substitutes for model capability — a better producer infers more of what the representation failed to state, so the worse the producer, the more the statement is worth.
The clearest single row
Chapter 1’s altitude bug, run as an experiment rather than asserted. Repair rate by fault, under each condition.
| Injected fault | A prose | B struct | C full |
|---|---|---|---|
| unit metres/feet | 0% | 0% | 100% |
| duration ms/s | 0% | 50% | 100% |
| threshold inclusive | 20% | 40% | 100% |
| offset one-based | 50% | 50% | 100% |
| rate per-hundred | 70% | 60% | 100% |
| half-open range (control) | 100% | 100% | 100% |
Zero out of ten under prose. Zero out of ten under structure. Ten out of ten once the obligation is named. A unit error distributed through a function is not recoverable from a symptom, by either model, in fifteen attempts — and it is trivial the moment something writes the unit down.
That is the argument of the book reduced to one row of a table.
The bottom row is the control. A single-site, local fault repairs every time under every condition: when the fault is already local, the diagnostic does not matter. Chapter 1 predicts exactly that, and it is better to see it than to have to argue it.
What this does not show
The limits, stated in the book
The obvious objection to condition C is that it carries strictly more information than A, so of course it wins. That is true. The reply is the argument of the book rather than a result: a current toolchain already holds most of what C carries and throws it away at the point of reporting — and the part it does not hold, the unit, is exactly what Part IV proposes to declare.
Beyond that: six tasks, programs of under a dozen lines, one vendor’s models, and a format designed by the person reporting the result. Chapter 27 says this experiment should be run by somebody else, and that is still true.
Lab 7 is in the repository so that disagreeing with it costs you an afternoon rather than an argument.