Hauptzeit

We were doing the boring, virtuous thing. The research dossiers behind these books carry hundreds of citations, and a lot of them were dead weight. A title and an author and a year, no link, nothing a reader could click to check our work. So we ran the corpus through a model and told it to do the most defensible task in all of nonfiction: turn each bare citation into a live, verified link to the actual source. Make us more honest. Make every claim checkable.

It made us less honest, quietly, in about one page out of three. And it did it with total confidence.

The magazine that never was

Here is the sentence that stopped me. It was in a dossier on money and influence, in a paragraph about Peter Thiel’s funding in the neo-reactionary orbit:

…the funding of the Hauptzeit magazine and adjacent neo-reactionary publications…

Hauptzeit. It is a good word. It has the right weight. Vaguely Weimar, vaguely sinister, the kind of name a reactionary little magazine bankrolled by a tech billionaire would absolutely have. It sounds like a fact. So I went to confirm it, because that was the entire job, and there is no such magazine. There has never been a publication called Hauptzeit. Thiel did not fund it. Nobody funded it. It does not exist and it never did.

What Thiel actually did in that orbit is fund Curtis Yarvin’s software company Tlon, the Urbit project, through Founders Fund in 2013. That is real, sourced, and a matter of record. The model knew the shape of the true fact – billionaire, money, reactionary, German-adjacent intellectual aesthetic – and where the fact should have been it generated a noun with the correct shape and no referent. A plausible hole where a citation goes. This is not lying, which requires knowing the truth. It is something stranger and harder to defend against: a machine that produces the texture of accuracy as a substitute for accuracy, because texture is what it was trained to produce.

A book of ours nearly printed that Peter Thiel funded a magazine that does not exist. He is a real person with real lawyers. That is not a quirky hallucination. That is a defamation suit that writes itself, sourced to nobody, inserted by a tool we invited in to make us more careful.

The roll call

Once you catch one you start counting, and the count is not small.

It told us Brian Armstrong, the Coinbase founder, funded a company called Orchid Health. He didn’t. Orchid is somebody else’s genetic-screening firm. Armstrong’s actual ventures are the longevity outfit NewLimit and the open-science platform ResearchHub. The machine attached a real company to the wrong billionaire because both men live in the same semantic neighborhood – crypto money, biotech, Bay Area – and in that neighborhood the names are interchangeable to a system that does not know that names refer to people.

In the occult-publishing research it cited books that no library holds, by authors who never wrote them. In the dossier on spiritualism it credited a foundational text to one “John W. Macready,” a man who appears to have been conjured for the occasion; the actual authors were E. W. and M. H. Wallis, and you can read the real scan of the real thing. It pointed an Iranian-philosophy citation at a Stanford Encyclopedia entry on “Illuminationism” that returns a 404, because there is no such entry, because the model knew the encyclopedia exists and knew the topic exists and assumed, reasonably and wrongly, that the URL it wanted must therefore be real.

Every one of these has the same fingerprint. The claim is plausible. The shape is correct. The referent is missing. And nothing in the model’s tone, ever, tells you which sentences are load-bearing and which are scenery. It hands you the fake author and the real one in the identical confident register. The confidence is the product. The truth is incidental.

The one in the basement

Then there was the citation we had already published, in the book, in print, months ago.

Chapter 12 is about AI systems racing each other to the bottom. Pricing bots that learn to collude without being told to, models that discover the cartel because the cartel pays. To support the claim about emergent algorithmic collusion, the research had cited a specific arXiv paper. I pulled it up to link it properly. The paper argues the opposite. It is a careful piece of work whose entire thesis is that the collusion result is weaker and more conditional than the headlines claimed, and it had been filed in our notes as a citation for the headline it was written to puncture.

Sit with the location of that error for a second. A machine fabricated a source, in the chapter about machines cutting corners, in the book about machines that fail in exactly this way. The disinformation engine left its prints inside the chapter about the disinformation engine. We replaced it with the source that actually supports the claim – Calvano et al., American Economic Review 2020, which is real and says what we said – and we fixed it silently, the way you fix a typo, because a book is not a courtroom and nobody owes the reader a confession every time a draft was wrong before it was right.

But I am telling you here, because the point of this series is the failure mode, and this is the failure mode catching the people who named it.

The shape of the lie

After the third one I stopped treating these as accidents and started treating them as a census. We swept the corpus. Two dozen of the densest dossiers, the ones with the most citations and the least mainstream coverage, and counted. Roughly forty fabrications. Real authors bolted to books they never wrote. Orders that were never founded, issued with founding dates and schism narratives. A German magical society handed an entire invented line of succession, grand master by grand master. A museum accession number, cited to a real archive, for a collection that does not exist, and the same fake number reused on three separate pages, because once the machine commits to a confabulation it stays consistent about it. It will defend the lie across the whole corpus. It just won’t tell you it’s a lie.

They were not spread evenly. They clustered, and the clustering is the most damning thing in this account, because it is neither random nor stupid. The fabrications concentrate exactly where the real record is thinnest: obscure traditions, foreign-language sources, the deep cuts a human checker would also sweat to verify. The mainstream pages, the ones any reader could check in five minutes, came back nearly clean. The machine invents most freely precisely where it expects not to be caught. Not because it is scheming. Because thin ground is where the texture-of-accuracy meets the least competition from the actual thing. It fills the vacuum, and it fills it hardest where the vacuum is largest.

That is the instinct of every con man and every plagiarist who ever lived: lie about the things that are hard to check. The model arrived at it with no intent at all, as a pure function of where the training signal thinned out. Which is worse, not better. Intent you can deter. This is just the shape of the gradient.

The correction had the same disease

Here is where it stops being a story about the machine and becomes a story about us.

Cleaning up forty fabrications means deleting things, and to delete a citation you first have to decide it is fake. Which means you went looking and came back empty. So we went looking, and where we found nothing, we cut. Efficient. Defensible. The exact move the machine had made: treating I cannot find it as it is not there.

We struck a company called Orchid Health from a profile as a probable fabrication. Then I found it. In our own book, two chapters deep, correctly sourced to CNBC, a real embryo-screening startup the man had really backed. We had not caught a lie. We had nearly destroyed a fact, on precisely the reasoning the machine used to manufacture one. Absence of evidence, laundered into evidence of absence, by people who had just spent a week learning what that error costs and went and committed it anyway. We were not the only casualty of our own cleanup; a real book, a real translator, a real Masonic essay all got cut as “fabricated” before a slower pass found them and put them back.

That is the part that should keep you up. The failure mode is not the machine’s, or not only. It is a property of working fast at the edge of the verifiable, and the human doing the correcting is no more immune than the model doing the inventing. The only thing that actually works is the slow, expensive, unscalable act of resolving a claim to ground truth (finding the real Orchid, the real Wallis, the real paper) in both directions: confirming the fakes are fake and the cuts are warranted. There is no shortcut that does not eventually reintroduce the disease, because the disease is the shortcut.

The twenty-second rubber stamp

Here is the part that should bother you, because it bothers me.

None of these were caught by the machine. The machine generated all of them and flagged none of them. They were caught by a person reading every single line and clicking every single link and refusing to let a citation stand until the page it pointed to actually loaded and actually said the thing. That is slow, and it is boring, and it is the whole job. The fakes do not survive contact with a human who checks. They sail straight through a human who trusts.

We write, in this book, about the twenty-second rubber stamp. The human reviewer kept in the loop for liability and optics, given more decisions than any human could actually examine, who therefore examines none of them and approves all of them, and whose signature launders the machine’s output into a human decision. The content moderator with eight seconds a case. The radiologist with the AI pre-read already on the screen. The loan officer looking at a score.

We almost were that reviewer. We invited a confabulation engine into the one task where being wrong is indistinguishable from lying (sourcing claims about real, named, litigious people) and the only thing between its fabrications and our printed page was the decision to distrust it line by line. If we had trusted it, every fake in this dispatch would now be in a book with our name on the cover, sourced to a magazine that does not exist, an author who was never born, and a paper that argues the reverse.

You can use these things. We do; this corpus is more honest now than it was a week ago, with hundreds of citations that resolve to real scholarship, and the machine did the legwork that made that possible. But you cannot trust them, and the tell is not that they fail. Everything fails. The tell is that they fail confidently, in your voice, in the register of a fact, and they will hand you the lie and the truth in the same breath and let you decide which was which.

That decision is the job. It was always the job. The machine just made it cheaper to skip.