[TUHS] Re: Porting the SysIII kernel: boot, config & device drivers

On Sat, Dec 31, 2022 at 3:02 PM Paul Ruizendaal <pnr(a)planet.nl> wrote: Warner answered this better than I could. :-) Virtio is solving a very, very different problem: this is limited paravirtualization of devices for virtual machines. The idea of virtio devices is predicated on a hypervisor somewhere providing a matching implementation that, in turn, plugs into whatever primitives the host provides for device management. Put another way, devices don't expose virtio interfaces directly (generally speaking), and you need something somewhere to translate from the virtio interfaces a guest sees to actual useful services that the guest makes use of via those interfaces. A host _could_ emulate something like an actual physical hardware device (for some category of device) but it turns out that's often not particularly efficient; for instance virtio-net tends to be somewhat faster than an emulated e1000. Still, virtio has some problems; you can program DMA to anywhere in the guest physical address space, which can be expensive to handle without an exit from the guest context: virtio-net isn't as fast as Google's gvnic. This, I think, is really the hard part. The issue is that systems are configured a myriad of different ways depending on the platform, vendor, particular combination of IO devices, etc. That makes figuring out where the host OS image is and getting it loaded into memory a real pain in the booty; not to mention on modern systems where you've got to do memory training as part of initializing DRAM controllers and the like. Not to mention power-sequencing, initializing IO buses and all that business: who assigns PCI BARs? What if the kernel is on a PCI device, like an NVMe SSD or similar? Turning on a modern CPU is really a non-trivial exercise, let alone getting the OS loaded. Once you get it into memory, though, _starting_ the kernel is comparatively easy. But all the gritty details before that are how we ended up with enormous systems like UEFI. But backed by what? It's not clear to me what this would buy you; the MMU tends to be architecturally defined, in a way that e.g. the boot process, IO bus initialization, and even CPU topology discovery are not. And then there are soft-TLBs (which are a bad idea, but that's a separate discussion). I mean, most VM systems since SunOS 4 have a HAL and a portable part already. I believe that AIX on the PC/RT did basically this. Certainly, this was true on the 370. The PALcode stuff on the Alpha was similar in terms of having a BIOS-like abstraction with specific implementations for Windows or VMS/Unix. But there's an argument that this is functionality that _belongs_ in the OS, which was the traditional Unix approach. Could it have been done differently? I guess so, but would that have been a good system design? I don't know. This is challenging with virtio because some devices basically rely on the guest trapping into the host with a particular exit code to kick off IO. I suppose you could do that with a shared semaphore, but then you're burning cycles somewhere polling. You could interrupt yourself, either synchronously or with a self-IPI, but then the interrupt handler has to run the virtio handling code. In any event, it's no longer a simple matter of writing to a device register somewhere to make the "device" actually do something. Sadly, that's pretty baked into the interface. But you've got to have some CPU run that software somewhere; either you've got a satellite processor doing it or you're stealing cycles from the OS. The latter is how a BIOS traditionally works and I suspect that if we were doing this in the 1980s, having a dedicated IO processor would have added non-trivial cost to a system, for little gain: let's remember that Unix was _very_ success on a variety of systems in that era without relying on a BIOS-type abstraction!