On the side, I run a small lab that tests quantum compilers. A quantum compiler takes a program written for a quantum computer and tries to make it smaller, because on today's hardware every extra operation is another chance for an error.
The first study asked a simple question. Three popular toolkits, Qiskit, pytket and PyZX, already ship plenty of optimisation steps. If you search for a better order to run those existing steps in, can you beat GUOQ, a newer optimiser, on programs nobody had tuned for?
No. Across 145 fresh test programs, my search left about 3% more of the expensive two-qubit operations than GUOQ did. Given a twentieth of GUOQ's time, it did about 5% worse. It lost clearly.
So I published it.
Why publish a loss
Before running anything, I wrote down what would count as a win and saved it where I couldn't quietly change it later.
That matters because a disappointing result invites slicing. Maybe it wins on the bigger programs. Maybe with more time. Each cut sounds reasonable. Stack enough of them and a loss turns into a headline. Fixing the rule first takes that option away. Ideas for doing better go into the next study, with a new rule of their own.
Every output also went through an equivalence checker, a tool that confirms the smaller program still does exactly what the original did. A smaller program is no use if it's wrong.
I ran two other studies alongside it. One checked the three toolkits for wrong outputs. Out of 5,888 outputs, 172 differed, and every one was either a problem the Qiskit team already knew about or within tolerance. Nothing new to report.
The other tested whether AI models could sort those findings. My own rule said I had to grade every model's report without knowing which model wrote it, and I didn't. So that study failed by its own standard, whatever the numbers looked like. That one stings more than the loss, because the procedure I skipped was my own.
The habits travel: decide what "working" means before you test, check that the output is correct as well as faster, and write down the result you got rather than the one you wanted.
The fine print
Nobody outside the lab has reviewed this yet. Some of the planning dates rest on my word, and every study had deviations, all listed in the records. Everything else, including the data and a script that recomputes every number, is on GitHub at heyfavas-cell/ohrf-results. If you find a mistake, open an issue. Finding mistakes is the whole point of the lab.