Lab 01–04
Down to the metal
Machine bytes, source to native, LLVM IR, and SSA rules that are not actually checked.
Invalid LLVM IR is accepted
Appendix B — the labs
Eight labs. Reproducible
with Python and Clang.
The book demonstrates rather than surveys. Each claim in Part II is shown on a worked example small enough to check by hand, and every one of them ships as a lab you can run. Where a number is owed and has not been taken, the chapter says so rather than estimating one.
Lab 05
AIR compiles
AIR parses, verifies and compiles — and refuses four bad programs for four different declared reasons.
Lab 06
Validation cost
Nine faults injected across languages. Six of them are silent in every language tested.
6 of 9 faults silent
Lab 07
Structured diagnostics
The same fault, two error messages. A conventional message repaired it 40% of the time; a diagnostic naming the violated obligation and enumerating the legal repairs, 100%. Format alone did nothing (p = 0.36).
40% → 100% repair
Lab 08
Regeneration amplification
Repair cost scales linearly with program size when the unit of change is the whole program.
R² = 0.9996
Labs 1–6 run offline. Labs 7 and 8 call a model API and are excluded from the default runner. Lab 4’s result — invalid LLVM IR accepted — and Lab 6’s uninitialised-accumulator row are toolchain-specific, verified on Apple Clang 15.0.0 on arm64, and the book says so.
Lab 7 — Chapter 21
Forty per cent to a hundred
Six programs, each carrying a distributed fault — one wrong assumption touching several sites, because a fault at a single site cannot tell these conditions apart. Each is rejected three ways. Two models, five trials, 180 attempts. A repair counts only if the program passes a hidden test suite it never saw.
A prose what a test runner emits now: expected 2.5, got 5.0
B structured the same facts in a fixed schema, nothing added
C full plus obligation, cause, legal alternatives, repair_scopeA against B is pure format — identical information, one as prose and one in a schema. The difference is ten points, and on a permutation test that is p = 0.36: indistinguishable from chance.
B against C is fifty points. It adds the obligation, the cause and the enumerated legal repairs. So the value is in what the record says, not the shape it is written in. A schema is a container; what pays is the payload.
The repairs under C were also slightly smaller than the reference fix rather than larger — 0.81× minimal edit size against 0.92×.
Distributed faults repaired
Two models · five trials · 180 attempts. A repair counts only if the program passes a hidden test suite.
AProse
40%24 / 60
what a test runner emits now
BStructured
50%30 / 60
the same facts in a fixed schema
CFull record
100%60 / 60
plus obligation, cause, legal, repair_scope
- A vs B — format only
- +10 pts · p = 0.36
- B vs C — content
- +50 pts · p < 0.00001
The weaker model gained far more
Haiku goes from 23% to 100%, a factor of 4.3. Sonnet from 57% to 100%, a factor of 1.8. The representation substitutes for model capability — a better producer infers more of what the representation failed to state, so the worse the producer, the more the statement is worth.
The clearest single row
Chapter 1’s altitude bug, run as an experiment rather than asserted. Repair rate by fault, under each condition.
| Injected fault | A prose | B struct | C full |
|---|---|---|---|
| unit metres/feet | 0% | 0% | 100% |
| duration ms/s | 0% | 50% | 100% |
| threshold inclusive | 20% | 40% | 100% |
| offset one-based | 50% | 50% | 100% |
| rate per-hundred | 70% | 60% | 100% |
| half-open range (control) | 100% | 100% | 100% |
Zero out of ten under prose. Zero out of ten under structure. Ten out of ten once the obligation is named. A unit error distributed through a function is not recoverable from a symptom, by either model, in fifteen attempts — and it is trivial the moment something writes the unit down.
That is the argument of the book reduced to one row of a table.
The bottom row is the control. A single-site, local fault repairs every time under every condition: when the fault is already local, the diagnostic does not matter. Chapter 1 predicts exactly that, and it is better to see it than to have to argue it.
What this does not show
The limits, stated in the book
The obvious objection to condition C is that it carries strictly more information than A, so of course it wins. That is true. The reply is the argument of the book rather than a result: a current toolchain already holds most of what C carries and throws it away at the point of reporting — and the part it does not hold, the unit, is exactly what Part IV proposes to declare.
Beyond that: six tasks, programs of under a dozen lines, one vendor’s models, and a format designed by the person reporting the result. Chapter 27 says this experiment should be run by somebody else, and that is still true.
Lab 7 is in the repository so that disagreeing with it costs you an afternoon rather than an argument.