Three Origin 200 Timer Bugs, One Number

Forty-seven seconds turned up three times on this project, in three unrelated places, as the signature of three completely different bugs. Each time it came back my first instinct was that the same bug had returned. It never had. The number was the same for a reason that had nothing to do with the cause, and if I’d trusted that instinct I’d have gone back to re-fix something that was already correct.

The reason it’s always the same is the counter. Every periodic tick on this machine is driven by a 32-bit counter compared against a target, and the interrupt fires when the counter reaches the target. If a target is ever missed, set to a value the counter has already passed or skipped for any other reason, the counter doesn’t stop and wait. It keeps counting, wraps back through zero, and reaches the target again on its next lap. At this machine’s rate one lap is 47.7 seconds. So any bug that makes the comparator miss a single match, for whatever reason, produces exactly one dead lap: 47.7 seconds with no tick, then a self-heal when the counter comes back around. The number is the counter’s width. It names the wrap, not the fault.

The first time, the guest never armed the timer at all. Mainline Linux keeps time off a different clock and legitimately leaves this one masked, so the model’s armed timer sat permanently pending and every processor paid for it on every pass. A storm rather than a lost tick, but stamped with the same wrap period, because that’s when the model’s timer first came due.

The second time, the guest armed the timer constantly and correctly and lost a race with the counter. IRIX rearms the tick dozens of times a second by writing the target a little ahead of the counter, and under host load some of those writes landed a little behind it instead. The model took a behind-Count target literally and waited a full wrap for it, and an IRIX disk boot that looked like a dead clock was really waiting out the counter.

The third was ours, one layer below the second, and it survived the second’s fix because it was a different mechanism wearing the same face. Our processor model carries an extension to the standard comparator, and that extension read the counter twice for a single Compare write: once to decide what the write meant relative to the counter now, and once to set the host-side deadline that actually delivers the interrupt. Between those two reads, on a loaded host, the emulator thread could be preempted long enough for the counter to advance past the value the guest had just written. The first read saw the target ahead and armed normally. By the time the second read set the deadline, the counter had crossed the target, the match was gone, and the deadline was clamped to the next lap, a full wrap out. Reading the counter exactly once, and deciding and arming from that single sample, closes the window. The standard equality comparator this project inherited doesn’t have the extension and was never affected.

I checked that one the hard way, because it only shows up under load and I didn’t trust a boot that happened to pass. A reproducer injects a stall into exactly that seam between the two reads, deterministically, and it fails the old code and passes the new every run. And I ran the two builds side by side on one host for hours under real load: the old build lost several ticks, the fixed one lost none, with the processor pinned near full the whole time so the clean result was earned rather than idle.

What separated all three from ordinary host-load jitter was the exactness. Main-loop starvation can gap a tick too, but it produces gaps of arbitrary length. A missed comparator match produces one length only, a single lap, 47.7 seconds, every time. That exact-wrap signature was the fingerprint that named the mechanism each time, and it’s why the same number could mean three different things. Three guests reached it by three unrelated routes: one ignored the timer, one raced it, one was modelled wrong. All three ended in the same place, a comparator that missed its match and waited a full wrap for the next.