On Dec 14, 2023, at 3:29 PM, Noel Chiappa <jnc(a)mercury.lcs.mit.edu> wrote: Exactly! If blocking was not required, you can do the work in an interrupt handler. If blocking is required, you can't just use the stack of a random process (while in supervisor mode) unless you are doing some work specifically on its behalf. I suspect IBM operating systems probably did use them. At least TSO must have. Once you start *accounting* (and charging) for cpu time, this idea must fall out naturally. You don't want to charge a process for kernel time used for an unrelated work! *Accounting* is an interesting thing to think about. In a microkernel where most of the work is done by user mode services, how to you keep track of time and resources used by a process (or user)? This can matter for things like latency, which may be part of your service level agreement (SLA). This should also have a bearing on modern network based services. There was a race condition in V7 swapping code. Once a colleague and I spent two weeks of 16 hour debugging days! We were encrypting before swapping out and decrypting before swapping in. This changed timing enough that the bug manifested itself (in the end about 2-3 times a day when the system was running 5 or 6 kernel builds in parallel!). This is when I really understood "You are not expected to understand this." :-)

On Thu, Dec 14, 2023 at 7:07 PM Noel Chiappa <jnc(a)mercury.lcs.mit.edu> wrote: Assuming we're talking about Unix, yes, each process has two stacks: one for userspace, one in the kernel. The way I've always thought about it, every process has two parts: the userspace part, and a matching thread in the kernel. When Unix is running, it is always running in the context of _some_ process (modulo early boot, before any processes have been created, of course). Furthermore, when the process is running in user mode, the kernel stack is empty. When a process traps into the kernel, it's running on the kernel stack for the corresponding kthread. Processes may enter the kernel in one of two ways: directly, by invoking a system call, or indirectly, by taking an interrupt. In the latter case, the kernel simply runs the interrupt handler within the context of whatever process happened to be running when the interrupt occurred. In both cases, one usually says that the process is either "running in userspace" (ie, normal execution of whatever program is running in the process) or "running in the kernel" (that is, the kernel is executing in the context of that process). Note that this affects behavior around blocking operations. Traditionally, Unix device drivers had a notion of an "upper half" and a "lower half." The upper half is the code that is invoked on behalf of a process requesting services from the kernel via some system call; the lower half is the code that runs in response to an interrupt for the corresponding device. Since it's impossible in general to know what process is running when an interrupt fires, it was important not to perform operations that would cause the current process to be unscheduled in an interrupt handler; hence the old adage, "don't sleep in the bottom half of a device driver" (where sleep here means sleep as in "sleep and wakeup", a la a condition variable, not "sleep for some amount of time"): you would block some random process, which may never be woken up again! An interesting aside here is signals. We think of them as an asynchronous mechanism for interrupting a process, but their delivery must be coordinated by the kernel; in particular, if I send a signal to a process that is running in userspace, it (typically) won't be delivered right away; rather, it will be delivered the next time the process is scheduled to run, as the process must enter the kernel before delivery can be effected. Signal delivery is a synthetic event, unlike the delivery of a hardware interrupt, and the upcall happens in userspace. Note that this says, "starting of writes." Presumably, the writes themselves were asynchronous; this just initiates the operations. It certainly adds latency to the page fault handler, but not as much as waiting for the operations to complete! I can kinda sorta see a method in the madness of the Multics approach. If you think that page faults are relatively rare, and initiating IO is relatively cheap but still more expensive than executing "normal" instructions, then it makes some sense that you might want to amortize the cost of that by piggybacking one on the other. Of course, that's just speculation and I don't really have a sense for how well that worked out in Multics (which I have played around with and read about, but still seems largely mysterious to me). In the Unix model, you've got scheduling latency to deal with to run the pageout daemon; of course, that all happened as part of a context switch, and in early Unix there was no demand paging (and so I suppose page faults were considered fatal). That said, using threads as an organizational metaphor for structured concurrency in the kernel is wonderful compared to many of the alternatives (hand-coded state machines, for example). - Dan C.