Debugging a Slow Gigabit Adapter on the Origin 200

The emulated gigabit adapter worked, and it moved data slowly. Not broken-slowly, but nowhere near the number on the box, and for a while I took that for the ordinary price of emulation and left it alone. Then I went looking for where the time actually went. Most of it was in a byte swap the model was doing twice.

A swap that cancelled, at full price

A frame’s payload reached guest memory through the Bridge’s byte-lane window, which reverses byte order so that a big-endian processor and a little-endian device agree on a word. The adapter model then reversed the same bytes again on its own side. The two reversals cancelled, so the bytes that landed in the guest’s buffer were correct. That is precisely why nothing looked wrong and nothing complained.

Cancelling is not the same as not doing it. Every four-byte word of every frame went out to memory as its own access, came back byte-reversed, and was reversed a second time to undo the first. A standard frame is nearly four hundred of those words, a jumbo frame several times more, and each word was a separate memory dispatch whose entire effect was to hand back what it had been given. The payload was paying, per word, for a correction that corrected nothing.

Removing the one that did nothing

The fix is to stop swapping the payload at all: move it through the plain DMA path that carries raw bytes, and keep the byte-lane correction for the descriptors and registers that genuinely need it. Because the two swaps cancelled, dropping both halves for the payload leaves the delivered bytes identical, which a paired before-and-after run confirmed to the byte in every arm. What changed was the cost. Receive throughput rose by two to three times, transmit by a smaller factor, and UDP, which was not bottlenecked there, stayed flat. The per-word swap had been the dominant device cost, and it had been buying nothing.

The ceiling underneath was not the adapter

With the swap gone I expected the next bottleneck to be somewhere else in the adapter. It was not in the adapter at all. Driving real nine-thousand-byte jumbo frames through a kernel tap peer, on one emulated processor, from inside IRIX, the throughput settled at 239.7 megabits per second in each direction on the run that carries the verdict stamp, every byte accounted for. The more useful fact sits underneath that number: all three of the adapters this machine models reach the same band once each one’s own artifact is removed. They converge because the thing setting the speed is no longer the device. It is the guest, running its own network stack one instruction at a time through the emulator’s interpreter, and no work on a device model moves a ceiling the device is no longer holding up.

Building the test that showed this turned up two more real bugs in its first ten frames, both on paths that only a genuine jumbo frame exercises. The single-CPU ceiling is where it is because every lever inside the devices has been spent, so reaching an actual gigabit was a question about the guest, not the adapter. Spreading the network stack across more than one emulated processor has since answered it: at a jumbo frame size, on more than one processor, at least one of these adapters now clears a gigabit. The ceiling this note is about is the single-processor one, and it moved exactly where the interpreter said it would.