Running Firmware Out of an I/O Port on the Origin 200

For months the stock SGI firmware would reach the step where it relocates its own code into memory, print that it was doing so, and then appear to stop. It always finished, several minutes later and always with a completion message, but several minutes of nothing is indistinguishable from a wedge, and more than once it was treated as one. That step now takes about fifteen seconds. What was in the way wasn’t the firmware and wasn’t the processor model. It was which kind of memory region the instructions were being fetched from.

The slowness had a note against it: the emulator was built unoptimised, and an unoptimised build is slow. Nobody had profiled it. It doesn’t survive arithmetic. The boot flash is one megabyte, and a one-megabyte copy from flash into RAM is a sub-second operation even on an unoptimised build. Minutes for one megabyte isn’t a machine running slowly, it’s a machine doing something pathological per access, and the gap between those two readings is about four orders of magnitude. That’s worth pausing on, because “it’s slow because the build is slow” explains any amount of slowness and so constrains nothing. It sat unexamined precisely because it was never surprising.

So I traced every device read and write on one boot. Most of the accesses, better than eight in ten, hit a single region: the Hub-special mirror of the boot flash. Nothing else came close. Then the detail that turned a hotspot into a diagnosis. Almost all of those accesses had their address equal to the program counter. They weren’t data reads. The firmware was executing out of that region, fetching its relocation loop one instruction at a time through a memory-mapped I/O mirror, and individual loop bodies were being fetched tens of thousands of times each.

That’s catastrophic rather than merely slow because of how QEMU works. It translates guest instructions into host code in blocks and caches the result, and that caching is the entire performance model. An I/O region can’t back translated code. A region whose reads are served by a callback might return something different every time, which is what makes it I/O rather than memory, so the translator can’t hold on to anything it read there. Every fetch goes through the read callback, one instruction is translated, the block isn’t cached, and the next time round the loop the whole thing happens again. The relocation loop was being retranslated per instruction, for a megabyte of copying.

The same chip is fast through its other view. Early boot runs from the flash’s reset view, which is a rom_device: reads come straight from the backing store and translate normally. The copy phase switches to a 64-bit addressing window into Hub space and lands on the mirror. One chip, two views, and only one of them can hold a translation.

The uncomfortable part is that the hazard wasn’t unknown. A comment in the machine model, written about a different region by the person who built it, states the rule plainly: an I/O region cannot back translated code. Nobody had connected it to the symptom. The knowledge was in the tree, in the right words, next to the wrong region, and the symptom it explained was filed under a different cause entirely. That’s a worse failure than not knowing, and it’s the reason I wrote this one down.

The fix overlays a romd alias of the flash at the address the firmware actually fetches from, sitting above the mirror. Fetches there come from the backing store and translate and cache like any other memory, while the mirror underneath keeps serving what needs to be I/O, the repeated decode and the byte-wide command port. A second change flushes any cached translation if the guest reprograms the chip, which is what makes the first one safe rather than merely fast: code cached because it lives in memory has to stop being cached the moment that memory changes underneath it. With the change, and with the firmware’s own diagnostics switched off, the fetches through the mirror during the copy went from about seven million to zero, and the copy step went from a few minutes to about fifteen seconds.

The zero is the number that matters. A wall-clock figure on a loaded host is soft, and a fifteen-fold speedup could be a dozen things, but seven million fetches going to none is a statement about mechanism: they aren’t happening because that region isn’t serving them at all any more. I keep the diagnostics-off condition attached to the timing on purpose, because the diagnostics lengthen this step, and a run with them on takes longer and always did. A property test pins it: it reads the machine’s own memory map and asserts the flash’s Hub-space base is a cacheable range rather than an I/O one. It fails without the alias and passes with it, never boots the firmware, and never times anything, which is what makes it a stable check rather than a flaky one.