The claim, stated so you can attack it: "general" intelligence is not the finish line — it is the flaw. A generalized model has to re-derive your context from scratch every time it shows up, and wherever that re-derivation runs thin, it fills the gap with something plausible. General means hallucinating. Specific means fit. And nobody on earth can prove an AI's work is good — that was settled mathematics in 1953 (Rice's theorem), and it is why every wider promise on the market is unprovable exactly where it counts. Here is the move that survives it: you cannot inspect what a system means, but you can measure where its output lands. Grip is geometric — declare the lane before the work starts, count the crossings, sign the result. Where it landed and how far it drifted comes back the same for you as for a stranger who re-runs it on their own machine. We build that.

Open this commit's attestation — verify it yourself →

Why we believe this matters: the difference between what a system says it is doing and what it is doing has weight — that gap is where every AI failure and every uninsurable liability lives. But the same measurement, read the other way, is the most personal thing in the book: it means you are not about to be averaged out by a generalist. Today's passage walks from the anxiety to the physics to what it does for you — read it to the last two paragraphs.

“A paragraph I had spent two evenings on came back from a cleanup pass shorter, flatter, easier, and -- I want to be exact, because the exactness is the whole problem -- better.”

The Edit Has No Control Group

A paragraph I had spent two evenings on came back from a cleanup pass shorter, flatter, easier, and -- I want to be exact, because the exactness is the whole problem -- better. I read it twice. I felt the small physical relief of a sentence that no longer snags. I kept it.

Four days later I went looking for a clause about a boiler inspector who was permitted to refuse the certificate, and it was not in the file, and it was not in the file before that either. Nobody had argued with it. It had been averaged.

The previous section asked where you inject the fake. An edit is the case where there is nowhere left to inject it, because by the time anything exists to be judged, the thing it would have been judged against has already been overwritten.

Start with what a rewrite is when you strip the interface off it. You hand over a passage and ask for a better one. What comes back is the most likely passage given everything the model has read -- a conditional expectation, which is a fancy way of saying an average with the conditions you supplied holding some of it in place. That is not a defect and it is not anybody being careless. It is the arithmetic doing exactly what it says on the tin. But the sentence in your passage that was doing the most work is, by construction, the least likely one in it. Improbability is not a stylistic property. It is the definition of information: Shannon put the whole of it in one line in 1948, and the line says that a symbol carries information in proportion to how unexpected it was. A pass that makes prose more probable removes information, and it removes the most distinguishing information first, and it does this on every pass whether or not anyone wanted it.

You would catch that immediately if you could see what left. You cannot, and this is the part worth carrying out of the chapter.

Consider who is available to complain. The reader of the edited paragraph never met the clause about the inspector; nothing in the smoothed version points at a hole, because smooth is what a hole looks like from the outside. The author could complain, in principle, and does not, because the author has just spent their attention on the passage and the edit arrives as relief. So the only testimony anyone ever collects about an edit is collected from the survivor. Thumbs up, thumbs down, a rating, a shrug -- all of it is an emission from the wrong side of a boundary that has already closed. The data processing inequality is blunt about what that means: no operation on a summary recovers what the summary discarded, uniformly over every procedure anyone might invent. The edit's account of itself cannot contain what the edit displaced. Not expensively. At all.

Which gives the phenomenon its right name. This is not bad taste and it is not bad training. It is a control group that was destroyed before the trial began, and every measurement afterwards is a mirror with a serial number on it, hung in the room where you were counting on being able to see.

Say it as a general form, because it costs nothing to carry and it applies far past writing. Any process that improves an artifact by replacing it, and is evaluated only on the replacement, will move that artifact toward the population average and will report success the entire way down. Code review where the reviewer sees the diff but never the requirement. A restructure judged on the org chart it produced. A translation graded by someone who does not read the source. A summary that becomes the thing the next step acts on. Each one is the same missing counterfactual wearing local clothes, and each one is silent by construction, which is the whole of its danger.

The escape is the same escape as before, and it is embarrassingly cheap. Keep the pre-image. Not the memory of it, not a description of it -- the bytes, addressed, so a later reader can put the two side by side and ask what stopped being checkable. My own version of this is a diff read against the immutable commit rather than the working tree, for the unglamorous reason that the working tree is written by the same hand that did the edit, and I do not get to grade my own eviction. It is not sophisticated. It is a boiler inspector who is allowed to refuse the certificate, which is the clause the pass deleted, which is the joke I would rather have not been in.

Read this in the book, in context →