17 Nov
2018
17 Nov
'18
8:09 p.m.
From: Grant Taylor
> as an operating system person, I was, and remain,
a big fan of Multics
> ... which I still think was a better long-term path for OSes (long
> discussion of why elided to keep this short).
Can I ask for the longer discussion?
Sure. (Note that I'm not on COFF, so if anyone replies, please make sure I'm
CC'd directly.)
Basically, it boils down to 'monolithic OS's are bad' - for all the
reasons
laid out in the usual ukernel places (I first saw them in the BSTJ MERT
paper, IIRC).
{What follows is not the crispest way to explain it; sorry. I didn't feel like
deleting it all and re-writing to get straight to the point, which is 'Multics
had many of the advantages of a ukernel - and a long time before the ukernel
idea arrived - but without the IPC overhead, which seems to be the biggest
argument against ukernels'.}
Now, early UNIX may have done pretty well in a _tiny_ amount of memory (I
think our -11/40 V6 had about 64KB of main memory _total_, or some such
ridiculous number), and to do that, maybe you have to go monolithic, but
monolithic is pretty Procrustean.
This was brought home to me in doing early TCP/IP work on PDP-11 Unix. The
system was just not well suited to networking work - if what you wanted to do
didn't fit into the UNIX model, you were screwed. Some people (e.g. BBN) did
TCP as a process (after adding IPC; no IPC in UNIX, back then), but then
you're talking a couple of process switches to do anything.
I looked at how Dave Clark was doing it on Multics, and I was green with envy.
He added, debugged and improved his code _on the running main campus system_,
sharing the machine with dozens of real users! Try doing that on UNIX
(although nowadays it's getting there, with loadable kernel stuff - but this
was in the 70's)!
To explain how this was even possible, you need to know a tiny bit about
Multics. It was a 'single level store' system; process memory and files were
not disjoint (as in UNIX), there were only segments (most of which were
long-lived, and survived reboots); processes had huge address spaces (up to
2^18 segments), and if you needed to use one, you just mapped it into your
address space, and off you went.
So he wrote his code, and if I in my command process executed the 'TELNET'
command, when it needed to open a TCP connection, it just mapped in his TCP
code, and called TCP$open() (or whatever he named it). It fiddled around in
the networking state (which was in a data segment that Dave had set up in his
'networking' process when it started up), and set things up. So it was kind of
doing a subroutine call to code in another process.
The security wasn't good, because Multics didn't have set-uid (so that only
Dave's code would have had access to that state database) - when they later
productized the code, they used Multics rings to make it secure.
But still, that was pretty amazing. The key thing here is that nobody had to
modify the Multics 'kernel' to support adding networking - the basic
mechanisms (the single-level store, etc) were so powerful and general you
could write entirely new basic things (like networking) and add them 'on the
fly'.
What Multics had was quite different from ukernels, but it amounted to the
same thing; the system wasn't this monolithic blob, it was
layered/modularized, and you could see (and gain access to, and in some cases
replace - either just for yourself, or for everyone) the layers/modules.
The nice thing was that to call up some subsystem to perform some service for
you, you didn't have to do IPC and then a process switch - it was a
_subroutine call_, in the CPU's hardware.
I don't think anyone else ever tried to go down that path as a way to
structure an operating system (in part because you need hardware support), and
it's a pity. (UNIX, which would run on anything, killed just about _everything
else_ off.)
The 386-Pentium actually had support for many segments, but I gather they are
in the process of deleting it in the latest machines because nobody's using
it. Which is a pity, because when done correctly (which it was - Intel hired
Paul Karger to architect it) it's just what you need for a truly secure system
(which Multics also had) - but that's another long message.
Noel
18 Nov
18 Nov
5:33 a.m.
New subject: [TUHS] man-page style
On 11/17/2018 01:09 PM, Noel Chiappa wrote:
Sure. (Note that I'm not on COFF, so if anyone
replies, please make sure
I'm CC'd directly.)
Thank you for the reply Noel. I found this to be an interesting read.
I occasionally bump into some Multicians and am always pleasantly
surprised at how different their world is.
The 386-Pentium actually had support for many
segments, but I gather
they are in the process of deleting it in the latest machines because
nobody's using it. Which is a pity, because when done correctly (which
it was - Intel hired Paul Karger to architect it) it's just what you
need for a truly secure system (which Multics also had) - but that's
another long message.
If you're ever feeling bored and would like to share that story. ():-)
Thank you again for typing the message and sending it to a mailing list
that you're not subscribed to. If we're ever in a pub together, I owe
you a round.
--
Grant. . . .
unix || die
6:28 a.m.
New subject: [TUHS] man-page style
KeyKOS had a lot in common with Multics. It had a ingle level store.
A KeyKOS system could stay on for a long time or can be restarted
from a snapshot. The kernel had no state of its own that needed
saving so snapshotting was easy. Unlike Multics' protection rings
it was a capability system so provided better security. KeyKOS
inter domain call was synchronous: the caller blocked until the
callee process was ready to accept the call. A later system EROS
was an evolution of KeyKOS and open source.
All of this would be easily possible on the Mill arch. if ever
it gets built. Mill has segments and protected function calls.
set-uid has its own issues. Plan9 doesn't have it.
On Nov 17, 2018, at 12:09 PM, Noel Chiappa <jnc(a)mercury.lcs.mit.edu> wrote:
To explain how this was even possible, you need to know a tiny bit about
Multics. It was a 'single level store' system; process memory and files were
not disjoint (as in UNIX), there were only segments (most of which were
long-lived, and survived reboots); processes had huge address spaces (up to
2^18 segments), and if you needed to use one, you just mapped it into your
address space, and off you went.
So he wrote his code, and if I in my command process executed the 'TELNET'
command, when it needed to open a TCP connection, it just mapped in his TCP
code, and called TCP$open() (or whatever he named it). It fiddled around in
the networking state (which was in a data segment that Dave had set up in his
'networking' process when it started up), and set things up. So it was kind of
doing a subroutine call to code in another process.
The security wasn't good, because Multics didn't have set-uid (so that only
Dave's code would have had access to that state database) - when they later
productized the code, they used Multics rings to make it secure.
But still, that was pretty amazing. The key thing here is that nobody had to
modify the Multics 'kernel' to support adding networking - the basic
mechanisms (the single-level store, etc) were so powerful and general you
could write entirely new basic things (like networking) and add them 'on the
fly'.
What Multics had was quite different from ukernels, but it amounted to the
same thing; the system wasn't this monolithic blob, it was
layered/modularized, and you could see (and gain access to, and in some cases
replace - either just for yourself, or for everyone) the layers/modules.
The nice thing was that to call up some subsystem to perform some service for
you, you didn't have to do IPC and then a process switch - it was a
_subroutine call_, in the CPU's hardware.
I don't think anyone else ever tried to go down that path as a way to
structure an operating system (in part because you need hardware support), and
it's a pity. (UNIX, which would run on anything, killed just about _everything
else_ off.)
The 386-Pentium actually had support for many segments, but I gather they are
in the process of deleting it in the latest machines because nobody's using
it. Which is a pity, because when done correctly (which it was - Intel hired
Paul Karger to architect it) it's just what you need for a truly secure system
(which Multics also had) - but that's another long message.
9:24 p.m.
New subject: [TUHS] man-page style
On Sat, Nov 17, 2018 at 03:09:08PM -0500, Noel Chiappa wrote:
I looked at how Dave Clark was doing it on Multics, and I was green with envy.
He added, debugged and improved his code _on the running main campus system_,
sharing the machine with dozens of real users! Try doing that on UNIX
(although nowadays it's getting there, with loadable kernel stuff - but this
was in the 70's)!
One of the things that made Multics amazing is that you could replace
a shared library while other processes was using it --- and without
anything crashing. To achieve this, there is a tag field which
identifies the object type and its length. So if a library defines an
expanded version of the data structure, the new fields are tacked onto
the end of the data structure and the length field is bumped. Older
callers of the library might pass in a version of the data structure
with the original length field; hence, fields can't be accessed unless
without first checking the structure tag.
I stole this idea and used in Kerberos v5 and Linux's userspace
ext2/3/4 utilities, where we use a error table code --- another
Multics concept --- as the structure tag. So in the error_table file,
we might have:
ec EXT2_ET_MAGIC_BADBLOCKS_ITERATE,
"Wrong magic number for badblocks_iterate structure"
And that in each function that uses that structure, there'd be
something like this:
EXT2_CHECK_MAGIC(iter, EXT2_ET_MAGIC_BADBLOCKS_ITERATE);
Where EXT2_CHECK_MAGIC is defined as:
#define EXT2_CHECK_MAGIC(struct, code) \
if ((struct)->magic != (code)) return (code)
(All MIT KerberosV5 and libext2fs structures have a 32-bit unsigned
magic field as the first 4 bytes of the structure.)
This technique is also useful so when I needed to add support for
64-bit block numbers, I could use the structure magic numbers to
disambiguate which version of the object we were using. Hence unlike
some shared libraries where the magic number has been incremented to
indicate an ABI break every few months, e2fsprogs has not had an ABI
break in over ten years.
This also made it a bit easier to find use-after-free bugs in an era
before valgrid/purify, by the simple expedient of zeroing the magic
field when deallocating an object.
The security wasn't good, because Multics
didn't have set-uid (so that only
Dave's code would have had access to that state database) - when they later
productized the code, they used Multics rings to make it secure.
So that's a bit misleading. Setuid isn't really a good analogue for
protection rings. The proper analogue for user mode versus kernel
mode in the Unix world. (Where user mode is roughly speaking, Multics
ring 4, and Kernel mode is Multics ring 0 --- the Honeywell hardware
had support for 8 rings, but processes running below ring 4 have so
little access that using them isn't terribly practical for general
purpose programs. Processes ringing at rings 5 and higher wouldn't
have access to most of what we in the Unix world would call "the
standard POSIX system calls".)
Code running in one ring can transition to higher rings via "gates",
which would be the Unix equivalent of a system call. Hence, a Multics
program running at Ring 4 could create its own gates that would
provide an extremeted limited set of system services to programs
running at Ring 5. Those programs wouldn't have access to the normal
system calls, but only via the specified functions in the Ring 4
gates. This is sort of like Capsicum, but it's more powerful --- and
it was designed decades before FreeBSD's Capsicum.
The nice thing was that to call up some subsystem to
perform some service for
you, you didn't have to do IPC and then a process switch - it was a
_subroutine call_, in the CPU's hardware.
Well, when you call a system call, you don't do a process switch
either. So when Ring 4 code calls a ring 0 service, you chan think of
it as a system call. It might not have been any slower than a normal
function call, but remember, this is a CISC system. So another way of
saying things is that normal function calls weren't any faster than a
privilege transition via a system call!
The 386-Pentium actually had support for many
segments, but I gather they are
in the process of deleting it in the latest machines because nobody's using
it. Which is a pity, because when done correctly (which it was - Intel hired
Paul Karger to architect it) it's just what you need for a truly secure system
(which Multics also had) - but that's another long message.
One unfortunate thing about the 386 VM is that a segment plus offset
gets translated to a 32-bit global virtual address, which is then
translated to a physical address via a single page table. With
Multics, each segment had its own page table which translated the
segment+offset to a physical address.
With only 32-bits of virtual address space on the 386, it's not at all
clear aggressive use of segments ala Multics would have worked
terribly well, due to the internal fragmentation of that 32-bit
address space. So I've talked to some Multicians at MIT who might
quibble with the claim that 386's deisgn was "done correctly".
In any case, since no one really used segments on 32-bit x86, segment
support ended up getting mostly dropped in 64-bit mode. (The FS and
GS segments still kinda work, mostly to keep Windows happy. The CS,
DS, ES, and SS segments are basically no-ops in the 64-bit x86 world.)
Which is too bad. I suspect that with a 64-bit address space,
designing an OS with a Multics-style segmentation architecture might
have been possible.
(But see Rob Pike's "Systems Software Research is Irrelevant" rant for
the argument that even if was *possible* it was very unlikely to have
happened, so for AMD and Intel to have neutered segmentation in the
x86-64 architecture might have been a well justified decision.)
Cheers,
- Ted
1992
days inactive
1993
days old
3 comments
4 participants
participants (4)
-
Bakul Shah -
Grant Taylor -
jnc＠mercury.lcs.mit.edu -
Theodore Y. Ts'o