softgpu

A driver stack, measured on a 6-vCPU ARM VM

From a mailbox and two spinning cores to a char device.

softgpu is a software model of a GPU. The device is threads and memcpy on a 1 GHz clock. The stack above it is the real subject: rings, fences, interrupts, a page table, a runlist, and the user/kernel boundary those things sit on.

softgpu, sketched in isometric A hand-drawn card. A host page on the left stores into a command ring on the die. Two small blocks are the compute and copy engines. A rust-colored dot is the doorbell. host softgpu doorbell
Submit stores into the mapped ring and rings the doorbell. It does not ioctl. The two engines are threads on a 1 GHz clock.

Ten stages, in the order they were built.

Each one started from a number in the previous record. The stage ids follow the design notes, which is why 7 landed before 5.

What idle costs.

One 64-byte fill after a quiet gap. spin keeps both engines hot. sleep gates immediately and pays the wake on every command. Hybrid 50 µs is the default: gaps inside the budget are free, and a quiet device drops to a few hundredths of a core.

    Latency of the command that breaks the gap. Log scale. A 10 µs budget gates on the 30 µs row. 50 µs and 200 µs both wait until 300 µs; the longer budget does not make that command faster. Source: gate-spin.json, gated.json, gate-sleep.json, vm7-idle10k.json, vm7-idle200k.json.

    Both engines together. 2.0 is the bill this project paid from stage 0 until stage 7, without printing it.

    A few rows that decided something.

    All of these are from the VM, canary-gated, best pass per row. The fast cluster itself drifted from about 208 ns to about 375 ns across the project, so a later round trip is not automatically a regression.

    Pinned host-to-device, 1 MiB. The page table is the break.
    StageGB/s
    3a pinned65.6
    4 no global lock52.0
    7 idle gating39.5
    5 virtual memory9.0
    Managed sequential copy, 1 MiB. Prefetch is 16 pages.
    ModeMB/sµs/page
    Prefetch on4698.7
    Prefetch off29140
    Engines spinning8834.6
    Stage 6, fair. A higher priority does not wait out the slice. A 5 µs slice spends the engine on switching.
    RunFirst probePair GB/sPreempts / op
    No timeslice, equal priority210 µs39.20
    50 µs slice, equal78 µs39.20.09
    5 µs slice, equal pair2.7 µs24.90.94
    Priority, no slice28 µs—0.10