The Random Register That Killed Init on the Origin 200

On the four-processor, two-node machine, init died about one boot in five, always on a second-node processor, always of pages it could not translate. The register behind it was the R10000’s TLB Random, and its fault was a strange one. It was too regular to be random.

A random index that was not random enough

Random picks the slot a random TLB refill lands in. My model stepped it down by one on every consultation, a tidy approximation that is strictly deterministic: consecutive refills picked adjacent, descending slots. The guest recycles those TLB slots on a regular cadence of its own, and that cadence phase-locked with the descending placement. The refill for one page kept landing in a slot the guest was about to overwrite next, several times in a row, so the page’s translation was eaten as fast as it was made. The processor that could not keep that page was a second-node processor, and init died on it.

I measured the lock directly: a perfect unit decrement of Random across twenty-five consecutive refills. The MIPS architecture decrements the register once per graduated processor cycle, not once per TLB consultation, so on real silicon its placement scatters across the array and the coincidence that killed init is vanishingly rare. The model was wrong not by being random in the wrong way, but by not being random at all.

Stepping it by the clock instead

The fix steps Random down by the virtual-clock nanoseconds elapsed since its last consultation, at least one, wrapping at the Wired floor. The clock, and not the Count register that is nearer the MIPS architecture manual’s units, because Count is guest-writable and this kernel rewrites it live, which would hand the decrement a basis that runs backward. The virtual clock advances with execution the way the hardware’s graduated cycles advance with it, so placement scatters the way the manual’s basis scatters.

Everything the manual mandates is kept. Reset and a write to Wired reload the upper bound, a consultation returns the value from before the step so the first read after a reload still sees the bound, and every value stays between the two bounds. The one field the fix adds, the timestamp it steps from, is deliberately left out of a migration: a stale anchor costs a single step of arbitrary size, and the whole contract of the register is that its value between consultations is unpredictable, so a one-time arbitrary step is squarely within spec.

With placement scattered, the guest no longer overwrites its own fresh refills, and the four-processor two-node machine stopped losing init this way. It was the last of the reliability faults that boot had been carrying.