Debugging a SCSI Hang Under Load on the Origin 200

A real OpenBSD 6.9 installer was running on the emulated Origin 200, extracting its distribution sets onto a SCSI disk through the modelled QLogic ISP1040 adapter. This was megabytes of real DMA write traffic driven by a real vendor driver, not by any test I could have written, and it was the most convincing thing the machine had done. It got partway through and stopped.

No panic, no fault, no log line. The output simply stopped and nothing further happened. It stalled somewhere around 31 or 32 percent of the first set, at a different byte offset each run. When I sampled the processor’s registers over and over and resolved them against the kernel’s symbols, the guest was sitting in its own idle loop. Not spinning on something, asleep, waiting for a completion that never came.

It took me four wrong diagnoses to find out why, and the answer was one register that two different things write, in two different directions, with a model that did not keep them apart. What finally settled it was a reproduction on a machine with no SGI hardware in it at all.

The adapter and its driver trade work through two rings in host memory. The driver writes command entries into a request ring. The adapter writes completion entries into a response ring. Each ring needs two indices, one saying how far the producer has written and one saying how far the consumer has read. On this chip family the response ring’s two indices share a single register address. The adapter publishes its producer index there, and the driver writes its consumer index to the same place as an acknowledgement, and the direction is implied by who is writing. That sharing is real hardware, not a shortcut I took: the guest’s own driver header gives the response-in and response-out names the same offset, so a model with one storage location behind that address is being faithful to the chip. The request ring has the same shape one slot lower.

What my model got wrong was what sat behind that address. There was a single array element. The adapter’s completion path published its producer count into it, and the generic register-write path dropped any guest write to that mailbox into the same element. So a driver acknowledgement did not acknowledge anything. It overwrote the adapter’s published producer count with the driver’s consumer count.

Now run OpenBSD’s actual interrupt handler against that. The handler reads the producer index once, at the top. It processes every completion up to what it read, and then it writes back what it consumed. If a second completion posts while the handler is running, the adapter bumps the producer index to 2, and then the handler’s write-back of consumer 1 lands on the same register and stamps it back to 1. The handler re-reads, sees producer equal to consumer, decides the ring is empty, and goes to sleep waiting for a completion that already happened. The command behind that second completion never finishes, the filesystem layer waits on it forever, and the machine goes quiet.

That is why the defect needed concurrency to show up. With one command outstanding at a time there is no window, because nothing new can arrive between the handler’s read and its write. The window opens the instant a second completion lands while the handler for the first is still running, and the emulator makes that window as wide as it goes: under sustained writes, one command’s completion callback is what kicks off the next, and the model has no bus latency, so the next command’s doorbell is answered synchronously, inside the same call. On real silicon the second completion turns up microseconds later, after the handler has returned. Here it turned up in the middle of it, every time. The measurement that made it unambiguous came from a debugger attached to a parked instance: the adapter’s internal producer count read 116, and the guest-visible register read back 115, one behind, and exactly equal to the last value the driver had acknowledged.

The uncomfortable part is that the device’s own reviewed datasheet had required the right behaviour before the bug existed. It said the response-queue register must answer the adapter’s own producer count and never whatever a guest last wrote to that shared address, and that the adapter must track what the driver has acknowledged and reserve one slot so a full ring and an empty ring stay distinguishable. It went further: it named the ordering, that the handler reads the index once at the top and does not re-read before writing back, and that a completion can run and finish synchronously inside that handler because one command’s completion starts the next. It even named the workload, the shape of a disk-image extraction rather than an edge case. The document had the failure, the workload, and the fix written down before the code was wrong, and the model diverged from it anyway. That is the datasheet-before-device rule paying for itself in the way that is hardest to notice, by nobody reading the thing it produced. The datasheet also constrained the fix, which matters because the obvious repair is wrong. Keeping the two directions apart has to happen behind the one shared address. Adding a second register is the shape you would reach for in a hurry, and it would be a fidelity regression dressed up as a fix.

Reproducing the erasure turned up a second, real defect in the same register: nothing reserved a slot, so a completely full ring is indistinguishable from an empty one. On a four-entry response ring, four completions posted and none acknowledged bring the producer index to rest exactly on the consumer index, and the register reads back the value every driver spells as empty. It is a real defect, but it cannot fire under the ring the real driver programs. The model reports 512 maximum requests, OpenBSD derives a 256-entry response ring from that and allows at most 170 commands outstanding, and one request yields one completion, so unacknowledged completions can never exceed outstanding commands and 170 is always short of 256. Live observation agreed: a producer index at 248 in a 256-entry ring with about twenty commands in flight. So it went on the list as a real bug that is not this bug, which is a distinction worth keeping, because closing the investigation on it would have felt like success and fixed nothing.

The four wrong turns are the part I would actually keep.

The first was a double completion. Instrumentation reported a per-device count of completion callbacks exceeding dispatches by an impossible margin, which points straight at a use-after-free: free a request without clearing the driver-private pointer, and a second completion reads a dangling one. I killed it structurally rather than by argument. QEMU’s own SCSI layer asserts that a request is still uncompleted before it sets status, that assertion was live in the build, and no run ever aborted, so a literal second completion call is excluded by the code and not by a probability story about counters. Which was just as well, because the counters turned out to be untrustworthy here: the dispatch-versus-complete numbers flipped direction between otherwise identical runs.

The second was a lost interrupt edge, and it is the best wrong idea I had. The Bridge that gathers a device’s interrupt sends its packet only on a rising transition, and the adapter’s line is level. So a second completion arriving while the guest is already in its handler produces no transition and no packet. The guest clears the Hub bit, sees nothing, and sleeps. It fit everything I had. I killed it by attaching to a live stall and reading the Bridge’s interrupt status register directly, and it was zero. No pin asserted at all. That read proved more than I had set out to check: nothing clears the adapter’s interrupt state except a guest-issued clear or a reset, so a clear line at the stall is positive proof the guest ran its handler. A hang in which the wakeup arrived cannot be a lost-wakeup hang, and that retires the whole class rather than one member of it. There was a second reason it had to go, too. OpenBSD’s own bridge driver documents this exact loss as silicon errata, and its documented workaround is for the driver to notice the situation and re-inflict the lost interrupt on itself. That recovery runs through the driver re-reading the Bridge status and storing into the Hub register the model already implements. Adding level re-delivery to the Bridge to paper over the hang would have broken a recovery a real guest depends on.

The third was the idea that the Origin was special. The whole thing had been framed as an Origin-specific fault, because the same OpenBSD install on an ordinary PC machine had completed. That was a wrong reading of a correct measurement, and the experiment that showed it is below. The fourth was a byte-swap or DMA-translation defect, discarded for the same reason the fabric was: the reproduction does not involve any of it.

Both defects reproduced as failing register-level tests on the plain PC machine, with no guest running at all. The first is the ring wrapping into looking empty, on a four-entry ring with four commands posted as four separate doorbells and nothing acknowledged. The second is the erasure, run deliberately on an eight-entry ring so a wrap cannot be the explanation. That is the whole trick: the fabric, the endianness handling, the crosstalk routing, and the entire interrupt path are excluded from the reproduction by construction rather than by argument. Nobody had to be talked out of the Origin being involved, because the Origin was not present.

Then a second experiment, on an x86 machine under a real OpenBSD driver, varying one thing:

The discriminator is concurrency, not volume and not architecture. Which reframes the evidence that had made the fault look Origin-specific: the x86 install that succeeded extracts largely sequentially, so the adapter was never sound on that machine either. That workload just never opened the window. The measurement had been honest, the instrument had worked, the number was real. What failed was the step from the measurement to the conclusion, the assumption that the thing I had measured was the thing that mattered. A run that cannot tell two hypotheses apart does not support either one, however carefully it was taken.

The fix keeps the adapter’s published producer count separate from the consumer cursor a guest writes. Capacity is checked before anything goes into the ring, with (producer + 1) modulo length not equal to consumer, which gives a usable capacity one below the ring length and keeps producer-equals-consumer meaning empty. Completions that cannot fit are held on a pending list and drained oldest-first, and an entry leaves that list only just before it posts, through the ordinary posting path, so the producer advances and both interrupt-status bits are re-asserted. The device’s test suite grew three subtests named for the three things that had to hold: four completions do not read as empty, an acknowledgement does not erase a completion, and a held completion re-signals when it drains.

The ring-full-looks-empty sibling is still on the list. It cannot be reached by the driver I run, but it is a real bug, and the machine is going to meet drivers I have not run yet.