On Sat, Dec 31, 2022 at 2:08 PM Clem Cole <clemc(a)ccc.com> wrote: I'm going to push back on this slightly: UEFI+ACPI get a bad rap because, well, they're really pretty bad. Oh sure, some things are reasonable: the ACPI table formats aren't awful. But I've been inside a couple of these now and phew golly, they stink pretty badly. The code is poor quality, and encourages running blobs of really dubious provenance. See below.... Perhaps this is what it is, but I think taking a step back and looking at the problem more generally, it's because they're mutated into solving the wrong problem. Consider AML: isn't this something that ought to be handled in, I don't know, a device driver? Yes, that driver may need access to some platform-specific firmware to abstract the details, but an entire virtual machine running random code to provide abstractions for the firmware writers who are running, basically, a parallel operating systems is a bit on the nose. This goes way beyond OpenBoot's forth modules on option ROMs (which were kind of a nifty idea for device discovery and such things). Mothy Roscoe gave a really interesting keynote at OSDI'21: https://www.youtube.com/watch?v=36myc8wQhLo I love how he describes the interfaces we have now as having "congealed." But in some ways, UEFI+ACPI are the antithesis of what the OS should be doing. - Dan C.

On Sat, Dec 31, 2022 at 4:39 PM Clem Cole <clemc(a)ccc.com> wrote: Agreed. Yeah. We canc'd it from our product (Oxide machines boot directly from the x86 reset vector and go almost immediately into Unix; no BIOS/UEFI+ACPI at all! Of course, there's a bunch of stuff that happens before the x86 cores come out of reset, but that's a different kettle of fish). We can get away with this because we control the hardware and the host software, and the unit of compute our customers interact with is a virtual machine, not the bare metal. When you control the horizontal _and_ the vertical, you can put it all in the host OS, where it arguably belongs. But in a world where software and hardware are completely decoupled? I don't know how to make the holistic boot approach work without a lot of system-specific code. It's obvious that you need some kind of system interface, and for most vendors, UEFI+ACPI is there and supported by the chip and firmware vendors, so it's the default. No one ever got fired for using UEFI/ACPI, I guess. - Dan C.

A theory about organizations, inspired by OpenBoot. As a company grows, which capitalism says it must, jobs change from something someone needs to do from time to time, to something a person is hired to do, to something a whole organization does. There are countless examples, but OpenBoot serves as an illustration. Like some others on this list, I'm sure, I've written a number of boot ROMs over the years, in the service of a project for which a bootable system was a prerequisite. Not hard, a day or two's work followed by some maintenance. So when I first saw a major software project in the form of a boot ROM, which may well have been my first taste of OpenBoot (it was a Sun thing, I'm pretty sure) I was bemused. But over time I've come to understand. As Sun grew (if I have the wrong company, I apologize, but it's endemic in any industry, and if I have the details of OpenBoot wrong, ditto), for various reasons the maintenance of the boot ROM became more time consuming, until eventually it became someone's full job, and then the boot ROM department's job. But allocating people's time to tasks is an inexact process, plus once one's full focus becomes boot ROMs, one's head fills with ideas. If one works on boot ROMs, every problem one thinks about becomes one the boot ROM can solve or at least help. (This particular disease is a major infection for compiler teams.) And so the boot ROM grows and grows and grows, accumulating features that are fun for the team to work on, but in the high-altitude view not really worth it. As I said, though, it's really about allocating people's time. "I have nothing to do at the moment, why don't I put a FORTH interpreter into the boot ROM? And then I'll make it an industry standard. Anyway it's my job to do boot ROMs so what else should I do now?" But you wouldn't have done that if the boot ROM stayed at the level of the stack it should have, doing the minimum necessary to get the operating system up and running to let it do the heavy lifting. I never needed a FORTH interpreter in my boot ROM. Maybe some thought they did but did they really? And was it the right use of resources to put it there? This is what happens in organizations. Employee performance evaluation at Google followed this same arc until the need to keep the organization responsible busy generated a process that placed a staggeringly expensive tax on the rest of the company, to the point that a secondary tax was paid in trying to figure out how to deal with the primary tax. This then is my theory of capitalism: Things grow until they break into pieces that must each be fed individually, triggering more growth that eventually become tumors that sap the strength of the organization. Small companies can do well because they haven't grown big enough yet to face this problem. Not an original observation, but perhaps no one has used OpenBoot as its exemplar before. Happy New Year. -rob But

All true except for the Forth choice. It's as bad, maybe worse, as choosing Tcl for your language. I've written a ton of Tcl but I need the Tk GUI part so I put up with Tcl to get it. I'd never push Tcl as a language that other people had to use. Same thing with Forth. I dunno what I'd pick, Perl in the old days, Python now (not that I care for Python but everyone can program it). Just pick something that is trivial for someone to pick up. On Sun, Jan 01, 2023 at 11:16:16AM +1000, George Michaelson wrote: -- --- Larry McVoy Retired to fishing http://www.mcvoy.com/lm/boat

On Sun, Jan 01, 2023 at 12:02:12PM +1100, Rob Pike wrote: Yeah, that was Mitch's baby and you have the details as right as I can remember. I barfed at Forth, I've coded in Forth and didn't care for it. I didn't like lisp either but I could sort of see why some people get all worked up for lisp, wasn't for me, but I could see the point. I never saw the point of Forth for anything. But, for whatever reason, Mitch did. Yuck. The one time it was useful in my memory, was during bringup and the kernel debugger wasn't working yet. So Mitch, or someone, coded up forth code to walk kernel data structures, a set of a dump_$whatever. Briefly useful. Over all, I agree with everything Rob said.

This is not precisely what I remember, but I remember at USENIX when there was a talk about the Open Boot work, someone sang a song. I looked for the song, but only found this page which doesn't read right to me: https://everything2.com/title/The+Open+Firmware+Song ===== nygeek.net mindthegapdialogs.com/home <https://www.mindthegapdialogs.com/home> On Sat, Dec 31, 2022 at 5:38 PM Theodore Ts'o &lt;tytso(a)mit.edu&gt; wrote:

Wise words, I was indeed jumping to a conclusion without defining the question. Douglas Adams comes to mind. On the positive side, it did generate a lot of insightful posts that I hope to digest in the coming days. What an interesting keynote! At first he makes the case that OS research is dead (not unlike Rob’s similar observation 20 years before). However, he goes on to point out a specific challenge that he feels is in dire need of research and innovation. In short his case is that a modern SoC is nothing like a VAX, but Linux still treats all hardware like a VAX. Having watched his keynote I can better place my generic discomfort with the amount of code required outside of the classical OS on a SoC. ==== Below some clarification on the topics that were on my mind: That was my immediate pain point in doing the D1 SoC port. Unfortunately, the manufacturer only released the DRAM init code as compiler ‘-S’ output and the 1,400 page datasheet does not discuss its registers. Maybe this is a-typical, as I heard in the above keynote that NXP provides 8,000 page datasheets with their SoC’s. Luckily, similar code for ARM chips from this manufacturer was available as C source and I could reverse engineer the assembler file back to about 1,800 lines of C. See https://gitlab.com/pnru/xv6-d1/-/blob/master/boot0/sdram.c It does all the expected things (set voltage, switch on the main clock, set up the PLL, calculate delays in terms of clocks, train for the line delays, probe address multiplexing, etc.) by accessing a multitude of registers that appear directly connected to the controller fabric. Yet, at the same time it has only a single entry point (“init_DRAM”) that takes 24 (poorly documented) words of 32 bits to define the desired configuration (with options to specify “default” or “auto”). Why does the main processor need to run this code? Why is there not a small RV32E CPU inside this controller that runs the 1,800 lines and exposes just the 24 words as registers to the main CPU? Are the square mils really that expensive? Or is this the reason that many SoC’s (but not the D1) have a small side CPU to do this for all devices? Next, to read from the SD Card, its controller needs to be set up. Luckily here the manufacturer gives out actual C code and there is documentation. However it is another 1,400 lines of code. About half the code goes to initialising the controller hardware, the other half to negotiating with the card to find out the data width and maximum speed. Together with setting up the SoC clock distribution and the console UART I have about 3,500 lines of setup code. And this is just the 1st stage boot loader that has to run from a little bit of SRAM space on the SoC. I have not looked yet at setting up the ethernet, wifi, display, usb, etc., but undoubtedly this is between 1,000 and 2,000 lines each. This compares to about 7,000 lines for the SysIII kernel. There is also the question of what goes where. For the SD Card, does a boot level module need to decide or operate the transfer mode? Should this not be in the kernel? Maybe some of this should even be in a daemon? It may seem to be a modern problem, but conceptually these very same questions were at play when doing a compare and contrast between the organisation of networking in 4.2BSD and 8th Edition. The former has nearly everything in the kernel, the latter does a split over device drivers, stream filters and daemons. This same question also comes up for all the other devices, in particular usb. ==== I’m not quite sure I agree with Rob’s observation on boot ROM’s. While I get his point about organisational causes of bloat, it is not just about the boot ROM. On a RISC-V or ARM SoC the boot sequence is lengthy and the boot rom does little more than bring the system to a safe state and load the first stage boot loader. The full boot process is quite lengthy (e.g. https://riscv.org/wp-content/uploads/2019/12/Summit_bootflow.pdf) and several parts are SoC specific. One of the things on my mind was that SoC boards change quite quickly: the D1 was last year’s hit with hobbyists, next year it will be something else. Not nice if one has to redo 3,500-10,000 lines of code for each board. Although I am not a fan of Forth, I can see that it is useful when the controller IP blocks of a SoC (or the PCI cards of discrete systems) expose some form of re-targetable program needed to boot it up. The manufacturer BSP code is just not plug & play. Having watched Mothy Roscoe’s keynote, I realise that I was trying to create a way to make a modern SoC look like a VAX. Maybe valid for my archeology purposes, but not in the general case. On top it did not solve the crappy BSP code issue. Maybe the correct solution for my archeology is to just use the simpler FPGA SoC as a target. ==== Yes, this is where it started, but it seems to get traction as a standardised embedded device interface, especially in automotive (Google for "automotive virtio”). Also in FPGA implementations it seems to get more popular as a device interface e.g. the RVSoc system from the Uni of Tokyo. This one has a very simple & partial HW implementation, but it is enough to support an unmodified Linux. I don’t think I agree with every observation made on direct virtio implementation (I certainly agree with some) -- but that is maybe off topic for this list. I will observe however that many device controllers of the early-mid-80’s contained some sort of embedded processor, some even on the low cost end of the market -- think of the WD-1002 hard disk controller variant used in the PC-AT. For mini’s this was not unusual either, I already mentioned the M68K on the Unibus DEULA board, the T11 on the Qbus RQDX2 board is another example. Having smart controllers was cost competitive, even in the 80’s. ==== Today, yes, but not in the early 80’s. For the M68K and the Z8000 there were both segmented and paged MMU’s and often custom designs were used (think Onyx C8000, SUN-1). However, my thinking here was driven by an entirely different thing. I have referred before to the early history of Xenix, for example this chart: http://seefigure1.com/images/xenix/xenix-timeline.jpg (the whole page http://seefigure1.com/2014/04/15/xenixtime.html makes for an interesting read). For a long time I have wondered why early Xenix did not make the jump to a product that was split between a BIOS and a BDOS part, so that they could sell BDOS-part updates to the total installed base. The BDOS part could even be in some kind of p-code. Considering that they heavily invested in their “revenue bomb” C-compiler at the time, this type of thinking was close to their hart (the Amsterdam Compiler Kit was a similar idea). I am talking ’81-’83 here, thereafter it is clear that their economic interest was to focus on DOS. There are 3 possibilities: 1. It did not make technical sense 2. It did not make economic sense 3. It did make sense, but it simply did not happen So, yes, I was conflating a lot of different thoughts into a single solution, without first thinking clearly about the question. For sure, that is undisputed. But could it have been even more successful? Maybe the main reason for "no BIOS attempt" was purely economic: for the companies having to port to each and every machine created customer lock-in, and for the professionals it created an industry of well paying porting & maintenance jobs. The customers were willing to pay for it. Why kill the golden goose?

Seems like a good time to pull out the "wheel of reincarnation" paper. http://cva.stanford.edu/classes/cs99s/papers/myer-sutherland-design-of-disp… I think about it a lot when I ponder the state of graphics these days. The wheel does seem finally to have rusted in place, although I'm not sure I'd choose this exact angle of rotation. -rob On Mon, Jan 2, 2023 at 8:27 AM G. Branden Robinson < g.branden.robinson(a)gmail.com&gt; wrote:

That is interesting. Are you referring to this: https://www.tuhs.org/cgi-bin/utree.pl?file=4.1cBSD/a/sys/conf https://www.tuhs.org/cgi-bin/utree.pl?file=4.1cBSD/usr/man/man8/config.8 Or is auto configuration something else? When it comes to abstracting devices, it would seem to me that virtio has gotten quite some traction in the last decade. Looking at both the systems of 40 years ago and the Risc-V SoC’s of last year, I could imagine something like: - A simple SBI, like the Berkeley Boot Loader (BBL) without the proxy kernel & host interface - Devices presented to the OS as virtio devices - MMU abstraction somewhat similar in idea to that in SYSV/68 [1] Together this might be a usable Unix BIOS that could have worked in the early 80’s. One could also think of it as a simple hypervisor for only one client. The remaining BBL functionality is not all that different from the content in mch.s on 16-bit Unix (e.g. floating point emulation for CPU’s that don’t have it in hardware). A virtio device is not all that different from the interface presented by e.g. PDP-11 ethernet devices (e.g. DELUA), the MMU abstraction was contemporary. High end systems could have had device hardware that implemented the virtio interface directly, low end systems could use simpler hardware and use the BIOS to present this interface via software. [1] From mmu.h in the SYSV/68 source tree: /* @(#)mmu.h 2.26 This is the header file for the UNIX V/68 generic MMU interface. This provides the information that is used by the various routines that call: mmufork () mmuexec () mmuexit () mmuread () mmuwrite () mmuattach () mmudetach () mmuregs () mmualloc () mmuinit () mmuint () The above routines and secondary routines called by them are contained in io/mmu1.c and io/mmu2.c. */

On Sat, Dec 31, 2022 at 3:02 PM Paul Ruizendaal &lt;pnr(a)planet.nl&gt; wrote: Warner answered this better than I could. :-) Virtio is solving a very, very different problem: this is limited paravirtualization of devices for virtual machines. The idea of virtio devices is predicated on a hypervisor somewhere providing a matching implementation that, in turn, plugs into whatever primitives the host provides for device management. Put another way, devices don't expose virtio interfaces directly (generally speaking), and you need something somewhere to translate from the virtio interfaces a guest sees to actual useful services that the guest makes use of via those interfaces. A host _could_ emulate something like an actual physical hardware device (for some category of device) but it turns out that's often not particularly efficient; for instance virtio-net tends to be somewhat faster than an emulated e1000. Still, virtio has some problems; you can program DMA to anywhere in the guest physical address space, which can be expensive to handle without an exit from the guest context: virtio-net isn't as fast as Google's gvnic. This, I think, is really the hard part. The issue is that systems are configured a myriad of different ways depending on the platform, vendor, particular combination of IO devices, etc. That makes figuring out where the host OS image is and getting it loaded into memory a real pain in the booty; not to mention on modern systems where you've got to do memory training as part of initializing DRAM controllers and the like. Not to mention power-sequencing, initializing IO buses and all that business: who assigns PCI BARs? What if the kernel is on a PCI device, like an NVMe SSD or similar? Turning on a modern CPU is really a non-trivial exercise, let alone getting the OS loaded. Once you get it into memory, though, _starting_ the kernel is comparatively easy. But all the gritty details before that are how we ended up with enormous systems like UEFI. But backed by what? It's not clear to me what this would buy you; the MMU tends to be architecturally defined, in a way that e.g. the boot process, IO bus initialization, and even CPU topology discovery are not. And then there are soft-TLBs (which are a bad idea, but that's a separate discussion). I mean, most VM systems since SunOS 4 have a HAL and a portable part already. I believe that AIX on the PC/RT did basically this. Certainly, this was true on the 370. The PALcode stuff on the Alpha was similar in terms of having a BIOS-like abstraction with specific implementations for Windows or VMS/Unix. But there's an argument that this is functionality that _belongs_ in the OS, which was the traditional Unix approach. Could it have been done differently? I guess so, but would that have been a good system design? I don't know. This is challenging with virtio because some devices basically rely on the guest trapping into the host with a particular exit code to kick off IO. I suppose you could do that with a shared semaphore, but then you're burning cycles somewhere polling. You could interrupt yourself, either synchronously or with a self-IPI, but then the interrupt handler has to run the virtio handling code. In any event, it's no longer a simple matter of writing to a device register somewhere to make the "device" actually do something. Sadly, that's pretty baked into the interface. But you've got to have some CPU run that software somewhere; either you've got a satellite processor doing it or you're stealing cycles from the OS. The latter is how a BIOS traditionally works and I suspect that if we were doing this in the 1980s, having a dedicated IO processor would have added non-trivial cost to a system, for little gain: let's remember that Unix was _very_ success on a variety of systems in that era without relying on a BIOS-type abstraction!

On Sat, Dec 31, 2022 at 10:09 PM Warner Losh &lt;imp(a)bsdimp.com&gt; wrote: This is sort of the issue I have with ACPI+UEFI et al. If we stopped at what you say (a small nucleus of primordial software that runs once at the beginning of time intended only to provide essentially static information to the OS in some relatively sane format, but then dies and is never consulted again) then either is fine: the ACPI table formats where this sort of information is encoded are well-defined and not horrible; ripping through the MADT to find all of your CPUs is fine. But that's not all that either did, and the amount of functionality being shoved into UEFI/ACPI in particular seems to show no sign of slowing down. I get that having this sort of parallel OS that exposes functionality in a manner transparent to what we normally consider the operating system coupled with CPU "features" like SMM means that vendors can write clever software to hide the fact that you've got a USB keyboard from an OS that doesn't understand USB; this touches on the thing Ted mentioned about needing to support old OSes like Windows 95 or whatever. But we're so far beyond compatibility crutches and into the land of magical black boxes running opaque blobs that do all sorts of stuff well hidden from the OS, and indeed, the OS has no control over, by design. THAT's the problem. Generally speaking, the hypervisor won't expose hardware devices directly to the guest. Even with SR-IOV and the like, the HV necessarily synthesizes, say, PCI config space as seen by the guest and tightly controls what virtual functions the guest sees, as anything else allows the guest to usurp the host. This implies that the HV is providing the guest with virtualized devices anyway, and once you're doing that the question becomes: what devices to parameterize the guest with? The HV could emulate things it is fairly sure the guest already knows about because they're relatively common: say, an e1000 for a NIC, or AHCI for storage, a 16550 for a console UART, etc, but what's the relative cost of doing so? It turns out, most of these devices aren't super great fits for virtualization; they generate too many exits. Enter virtio, designed for the use case. That said, hypervisor bypass for virtual devices is a big deal, and not something that's easily done with virtio. Even offloading virtio handling to dedicated cores is hard, because there's no easy way for the guest to generate a doorbell interrupt in the host (kicking a virtio queue involves a guest exit, which implies some local processing on the processor running the VCPU: that may be as simple as kicking off an IPI to another CPU, but you're still exiting, which is expensive). If that were all they did, I think a lot of the complaints would melt away. The data formats and the general concept of providing that data to the OS in semi-portable manner, yes. The real problem addressed here is the decoupling of hardware and systems software at scale; the problem was simply smaller on the PDP-11 and VAX, but is orders of magnitude larger now. You need _something_ unless you're in the downright luxurious position that, say, we're in at Oxide. UEFI+ACPI serve in that capacity, but it's important to note that that doesn't make them good. Lots of things that are very useful kind of suck. - Dan C.