How results here get checked, and what a full recheck after an AI model switch found

In July 2026 an AI model I had not chosen did about two and a half days of my work. When all of it was redone from scratch, every computed number came out the same. The errors were in the wording around the numbers and in how some checks had been done (one database search gave a wrong "not found"), and one real finding had gone unnoticed. This page describes the levels of checking every result here goes through, and then that recheck.

I do research with AI agents: Claude agents working under my direction write code, run computations and draft text, and I direct and check the work. Agents can produce work that looks right and is not, so every result on this site goes through up to five levels of checks before it is published. The more a result matters, the more levels it goes through.

1. Rereading. The agent that produced the work rereads it. This catches typos and little else.
2. Consistency. The numbers have to agree with each other and with the source files.
3. A second program. A second implementation, written from scratch without looking at the first, has to produce the same numbers. This level catches a lot, and AI agents make it cheap enough to do every time.
4. An outside source. The numbers are compared with something I do not control that already has the answer: a published table, or the OEIS (the Online Encyclopedia of Integer Sequences, a public database of number sequences).
5. An expert looking for flaws. For example, when I extended the published tables for a set of octal games (two-player games in which players take turns removing counters from heaps), Achim Flammenkamp, who maintains the reference tables for these games, reviewed my solver for one game, 0.007, and reran parts of the computation on his own machines with his own programs: one game over his old table's range, two others over their full new ranges, and later a fourth, 0.114, over its full new range. Then he listed the results on his page.

The recheck after the model switch#

In mid-July, partway through a session, I found that my sessions had been running on a different AI model from the one I had chosen, for about two and a half days, without my knowledge. The session record shows only that the model changed. I still don't know why. A model I had not picked could make different mistakes from the one I had been checking all along, so I asked for a full recheck of everything done under it.

The session transcripts record which model wrote each message, so I could find exactly when the switch started and ended and list every file produced in between. Items about to go out to other people, such as draft emails and text prepared for the OEIS, were checked first. Then fresh Claude agents, back on the model I had chosen and working under my direction, redid the work instead of rereading it: they wrote new code, went back to the original sources, and were told not to trust the documents being checked.

The computed numbers held up: the data files for the octal games were recomputed from scratch and came out identical, byte for byte. That covers the numbers themselves. What was said about them, and how they had been checked, is a separate matter, and that is where the errors were.

The main errors:

• A finding nobody had marked. In these games every heap size has a number called its nim value, which tells you how to play it (a value of 0 means the player whose turn it is will lose against perfect play). The values fall into two classes by a fixed test on their binary digits, and one class turns up only rarely: these are the rare values. People who study these games watch for them, because if a game's rare values stop for good, its values must eventually repeat and the game is solved. The recomputed data for one game, called 0.314, contained a new rare value. It was a real result. It had sat in data that had already passed its checks, and nobody had noticed it, so the write-up undersold the work.
• A wrong draft reply. A draft reply to Flammenkamp, prepared by an agent during the switch, got the status of some games wrong. It said the work of extending one game's table was essentially finished when it was only a little over half done.
• A search done the wrong way. An agent had searched the OEIS to see whether the lists of nim values for ten of my games were already there, and reported that none of them were. My lists began with the value for a heap of size 0, which the OEIS lists leave out, and five of the games are filed there under a related game whose list is shifted by a few places, so a search that matched from the start of my list found nothing. Done again correctly, all ten were already in the database. So the right step was to add my new values to those existing entries rather than propose new ones; the OEIS entries that came out of this work are mostly those existing entries, extended (of the twelve entries for these games, one is new and eleven are older entries I extended). This one was found only on a deeper second pass that I asked for after the first recheck.

Each was caught by comparing the work with the source records, in its own way: the first by recomputing the data, the second by comparing the draft with the actual progress of each game, and the third by noticing that an older note on my project status page already listed one of the games in the OEIS, which contradicted the search.

What I changed#

I now check the checks as well as the results. For every check that has passed, I ask what method it used and whether that method could miss something, such as a different starting position, a different version of the data, or a different encoding. I scan data that has already passed its checks for findings nobody has marked. And every archived agent report now states which AI model produced it.

Scope

This covers one model switch and one person's program, and it does not establish a general rule.

For readers in software:how the software was built, and which parts you could reuse

Treat AI output like a binary from a build server you do not control: rebuild it from source, compare it with known-good outputs, checksum long runs, and make sure the system that produced a result is never the only one that checked it.

@misc{orden2026verificationladder,
  author       = {Tyler Satchel Orden},
  title        = {How results here get checked, and what a full recheck after an AI model switch found},
  year         = {2026},
  howpublished = {Cyborg Mathematics},
  url    = {https://cyborgmathematics.com/writing/verification-ladder}
}