Inside the Origin 200

Most emulated machines have a bus. A processor drives an address, a chip somewhere decodes it, and a device answers. Once you have modelled the decode, you have modelled the machine’s skeleton and everything else is devices.

The Origin 200 is not that machine, and this post is about the four places where it is not.

The node number is inside the physical address. The processor drives a two-bit signal beside the address that decides which of four spaces the address means. A device interrupt is not a wire but a packet with a destination register. And the crossbar that ought to sit in the middle of the machine is not there at all, in a way the operating system depends on.

The parts, and how they connect

One node board with a processor, its memory and the Hub. One base I/O board with the Bridge and everything behind it. On a bigger Origin there would be more node boards, crossbars, routers between them, and more I/O boards. None of that changes how anything is addressed, which is the property the rest of this post keeps coming back to.

The node number is in the address

Memory is physically attached to nodes. Each node’s memory, each node’s Hub registers, each node’s crosstalk windows and each node’s boot flash appear at a fixed stride in one flat 40-bit physical address space, with the node number in the top eight bits.

So reaching another node’s memory is not a routing decision. It is a different value in bits 39 through 32.

SGI’s own firmware manual describes addressing a remote node as exactly that: write the node’s number into the high address bits, for memory, registers, devices and flash alike. That is why one aliasing scheme covers the whole machine.

Alongside the global view, each processor also sees position-independent aliases that always resolve to its own node, so firmware and per-node kernel code can be written once and run anywhere. The Hub’s small window has two halves for exactly this: one reaches whichever node the reading processor is attached to, the other reaches the node named in the address.

The consequence for an emulator is a pleasant one. Node count becomes machine configuration rather than architecture, because all routing is node-first already. A second node is a second Hub and its memory placed at a non-zero node number, plus a fabric that is currently deferred. That is why the very first commit of this machine carried the striding, on a machine that will only ever have one node for months.

The consequence for testing is unpleasant, and worth stating in the same breath. A single node hides the addressing mode, the node count and the whole node table. Everything working today is untested evidence for the multi-node case. There is a specific trap: a guest that rejects the global data area degrades to master node zero and node count zero, which on a single-node machine is the right answer. So the machine boots, nobody notices, and the first second node exposes it.

The space is not in the address either

Here is the part that bites an emulator author first.

The processor has four uncached windows, one for the Hub’s own special space, one for crosstalk I/O, one for special memory operations, and one for plain uncached memory. All four decode to the same 40-bit physical address. What tells them apart is a two-bit attribute the processor takes from high bits of the virtual address and drives on its system bus as a separate signal beside the address.

So a flat 40-bit space cannot tell the boot-flash window from node memory at the 256 MB mark. The two are the same physical address in different spaces.

This fork’s memory system therefore carries a 42-bit machine bus address: the 40-bit physical address with a two-bit window number above it. That is a QEMU-internal carrier and nothing else, not the virtual address, which stays in the processor, and not a widening of physical addressing, which stays at 40 bits.

One detail in it is worth the paragraph, because it is the kind of decision that pays quietly forever. The window number is not the attribute number. The model rotates the attribute by one, so that memory lands at window zero.

Two things fall out of that. A cached reference drives no attribute at all, and a compatibility-segment reference resolves at window zero, so a plain 40-bit physical address still means memory and nothing that ignores the window has to know windows exist. And the Bridge’s translated DMA output targets memory, which is window zero, so the Bridge emits no window bits on a transfer at all. Numbering the windows with the raw attribute would force every DMA transfer to carry a 3, a silent and expensive class of bug for no gain.

The cost is one thing to remember: read the window number as the attribute and every device lands one space out.

An interrupt is a write

On an ordinary machine, a device raises a line, the line reaches an interrupt controller, and the controller raises a line to the processor. Model the lines and you have modelled interrupts.

On this machine the middle hop is a memory write.

A device asserts its PCI pin. The Bridge sets a bit in its interrupt status register, if the matching enable allows it. Then the Bridge issues a crosstalk write to an address it was programmed with, carrying an interrupt number it was configured with, aimed at a target widget. That write lands on a Hub register, and the register sets one bit of a pending bank. Pending AND that processor’s mask becomes a bit in the processor’s cause register, which becomes an exception.

Three things follow from that shape, and each of them is a bug I either hit or deliberately avoided.

The Bridge has to be told where to write. Its interrupt-destination registers hold the target address and the interrupt number. If the machine description, the synthesized firmware and the Bridge’s own configuration disagree about which widget carries interrupts, the write goes to a widget that does not answer and every interrupt disappears.

The status register must follow the level rather than latching. OpenBSD deliberately enables interrupt-clear packets so that it does not have to clear interrupts itself. A Bridge that emits a write on the rising edge only leaves every interrupt permanently asserted, and the machine wedges at the first one. Servicing an interrupt does not clear the bit. Only the device dropping its line does, and a second source still holding the pin has to keep the bit set, because that is the read a guest performs to decide whether to re-arm.

And you must not “improve” it by adding re-delivery. OpenBSD’s own bridge driver documents real silicon errata in which interrupts are lost when two sources interrupt too close together, and the documented workaround is for the driver to notice and re-inflict the lost interrupt on itself, through the Hub register above. Making the Bridge re-deliver would break the recovery a real guest depends on. That mattered concretely: a plausible hypothesis during a disk-I/O hang was exactly a lost edge, and had it been “fixed” that way the machine would have become less faithful while looking more correct.

The crossbar that is not there

An Origin 2000 has a real eight-port crossbar between the Hub and the I/O widgets, and a guest discovers its topology by reading the crossbar’s registers and walking the firmware’s configuration graph for a probe order.

An Origin 200 has no crossbar at all. A jumper sits where the larger machine takes one, the Hub’s crosstalk link goes straight to the I/O bridge, and nothing routes by widget number. The way software finds that out is indirect, and a lot depends on it.

Widget 0 answers with the Bridge’s identification, because it is the Bridge. Every widget address on this machine reaches the one chip, so the widget numbers are a convention the firmware and the guest agree on rather than ports on anything. OpenBSD keys off exactly that: bridge identity at widget 0 sends it down a restricted-topology path that probes widget 8 alone, where crossbar identity would send it down the full-crossbar path.

And real firmware running on an Origin 200 omits the crossbar component from the configuration graph entirely, observed in a real machine’s own graph dump. So the guest’s fallback assumption about where the node controller sits from the I/O side is the normal path on this machine, not a degraded one.

A model that presents a proper crossbar identity would send a real OpenBSD down a path this machine cannot satisfy, which is the whole difference between modelling this machine and modelling something that merely resembles it.

I found that out the long way. The first attempt at getting the guest through autoconfiguration modelled the crossbar, because that is the device the guest’s panic named, and the panic did not move. The synthesized firmware had never programmed the address-translation entry that makes the probe reach a crossbar at all. Two gaps, one symptom, and the cheap gap was not the visible one.

A memory bank’s megabytes are not contiguous

A machine declares how much memory each bank holds, and you would expect that many bytes starting where the bank starts. On this machine you get a quarter of them, four times, 128 MB apart.

Access to each DIMM is interleaved, so a bank larger than 128 MB maps into four slots on a 128 MB stride, a quarter of its size at each, with three holes inside its own address range. A bank of 128 MB or smaller lands entirely in the first slot, because interleaving something that small buys nothing.

Both kernels that boot this machine compute it identically from the same configuration field, down to the special case: four slots per bank, a quarter of the declared size in each, and everything in slot zero below the threshold. One walks the four slots in a loop while the other indexes the bank by dividing the slot number, the same relationship written from opposite ends. Two unrelated kernels agreeing on arithmetic this specific, including the same threshold and the same exception beneath it, is close to the strongest evidence available without a vendor specification. It is still evidence about what each kernel believes, and what follows from it needs no more than that: my machine has to satisfy both of them or they fault.

Mine backs each bank as one contiguous run and declares the whole fitted total as bank zero. At some sizes those two disagreements cancel and at others they do not.

At 512 MB they cancel exactly, the size that was the machine’s default while this was undiscovered. A declared 512 gives each slot 128 MB on a 128 MB stride, contiguous and entirely backed. At 256 MB it comes apart: each slot gets 64 MB, at 0, 128, 256 and 384 MB, and the machine’s RAM ends at 256, so the guest registers two ranges nothing answers for.

The total registered is 256 MB, which is exactly the fitted size, so every check on the total passes while half the pages are missing. The guest survives until something reaches the top of the highest registered range, which a top-down allocator does last, which is why the machine gets as far as /etc/rc before taking a bus error on a store to the last page below 320 MB.

Nothing caught it because the arrangement that hid it was the default. Any run that did not pass a memory size got the one fitting where the defect cannot appear. It surfaced from the other direction entirely, when the networking work needed the installed system to stay up long enough to move a file, and found that it would do so at 128 MB and not at 256.

The default has since moved to 64 MiB, described as one matched pair of 32 MiB DIMMs. The reason given was not this defect: 512 MiB was much larger than the machine anyone used for bring-up, and it made every PROM and guest run initialize memory nobody needed. The side effect is that the default now sits below the 128 MB interleaving threshold, so the bank lands entirely in the first slot and the arithmetic never runs at all. The defect is still there for anyone who fits more, and it is recorded as a live divergence rather than a closed one.

The Bridge, and its IOMMU

The Bridge is a crosstalk widget that presents a 33 MHz PCI bus, gathers device interrupts, and translates every transfer a device initiates, either through a direct-mapped window or through a lookup in its own address-translation entry table.

That is an IOMMU, in a server that shipped in 1997, and it maps cleanly onto QEMU’s existing mechanisms. One in-tree PCI host bridge performs architecturally the same walk, which is a large part of why I chose QEMU as the base.

It has a practical consequence for every device behind it: use the IOMMU-respecting DMA helpers. The unqualified physical-memory helpers bypass translation entirely, which is wrong for any PCI device on this machine.

It also has a byte-order dimension, and I keep three separate byte-swap questions apart on purpose because collapsing them wasted time:

• The programmed-I/O device-window swap, which a little-endian device on a big-endian machine needs on narrow accesses. Settled as an address-only correction, scoped by PCI device identity, with SGI’s own I/O controller exempt, mirroring what OpenBSD’s driver does with its own per-device hook. This is what stopped the SCSI adapters’ mailbox registers reading wrong after reset.
• The DMA swap bits, deliberately not implemented and deliberately not decided. One reading is that remapping byte positions is a no-op in a byte-addressed model. The other is that the swap is real. Neither settles from the model alone, because both predict the bytes I already have. The experiment is a real disk boot.
• The clock’s register byte order, which turned out not to be a guest disagreement at all but an artifact of reading the wrong one of two reference checkouts.

The middle one is the interesting entry, because it is a place where I declined to write code. A model where both hypotheses predict the same observation is a model that cannot be validated, so the honest move is to leave the register storing its value and wait for an experiment that discriminates.

The chip that is four devices and one silence

The IOC3 is where a machine like this stops being elegant. One PCI device provides two serial ports, a 10/100 ethernet controller with its own buffer RAM, a PS/2 keyboard and mouse controller, general-purpose I/O, a one-wire bus and a byte-wide expansion bus with the real-time clock on it.

Bringing it up was a wall per register block, in the order the guest reads them during attachment: a status register, the ethernet block, the serial port’s DMA control block, the PS/2 block, the ethernet buffer RAM. Each one a commit, each one a qtest, each one moving the boot a few lines further.

The most interesting thing on that chip is a bus where nothing answers, on purpose.

OpenBSD’s driver for the controller matches a one-wire product string against a table of board part-number prefixes, none of which is an Origin 200. The branch that recognises this machine is the one taken when no serial-memory part attaches, and that conclusion drives which subdevices exist, the clock’s base address, and which interrupt pin the chip requests. So making the bus answer with an identity part would not degrade a fallback. It would stop the guest recognising the machine.

The reason it works out is narrower than “nothing answers”, and the story corrects itself twice before it lands. There are two drivers on that bus matching different part families, and only one of them sets the field the recognition branch tests. Counted across the thirty-one real-machine captures I hold: forty I/O controllers, forty one-wire buses, forty ethernet-address parts, and zero serial-memory parts. So a real machine reaches the recognition branch with a part present that is the wrong family to trigger the other branch, and this machine reaches the same branch with nothing present.

Same field, same value, same path, by different routes.

The residue was real, was owed, and has since been paid. For a while nothing answered that bus at all, so OpenBSD filled the interface address with all ones and printed it, which the committed login transcript still shows as address ff:ff:ff:ff:ff:ff, correctly, because that is what that boot did. The ethernet-address part is now modelled and answers the read both guests perform before they will trust a MAC address.

Which leaves the recognition property intact, and that was the part I checked rather than assumed. The bus is no longer silent, and the machine still reaches the branch that recognises it, because the driver’s test is not did anything answer but did a serial-memory part attach, and the part now present is the other family. Exactly the arrangement the thirty-one real-machine captures show.

What the shape of this machine costs an emulator

Three of the four peculiarities above make the machine easier to emulate than it looks, once you have accepted them. Node striding makes multi-node a configuration change. The window carrier makes the four spaces decode without a second channel. The interrupt-as-a-write is more code than a wire but it is code you can check exactly, and a bare-metal test ROM can follow the whole path and assert that the Hub holds the interrupt number the Bridge sent rather than merely some bit.

The fourth is the expensive one, and it is not really about the crossbar. It is that a guest’s behaviour on this machine depends on facts about the machine that no register reports. Whether widget 0 answers as a bridge. Whether a component appears in the configuration graph. Whether a part is present on a one-wire bus. Each of those is a topology fact, each one changes which code path a real operating system takes, and each one is invisible in a register dump.

Which is why my machine description is a checked artifact rather than a set of constants in a C file, why its firmware builder and its machine builder are driven from one source so they cannot disagree, and why the answer to “does the model work” is a register-level test per device rather than a boot that reaches a prompt.