31 Mar
2020
31 Mar
'20
4:44 a.m.
[...] the whole mbuf's stuff is something [...]
I would prefer not to go into the BSD code history at this point in time, as I’d like to
stay focused on what was going on at Bell Labs between 7th and 8th edition, as far as
networking is concerned. Of course it was all interconnecting, and the topic of
mbuf's is a nice arc to take us ultimately back too streams.
Looking at network buffering in Unix, I think the story begins in 1973 with Spider. In
Spider, the network driver was a character mode device, but used disk block buffering for
storage. This worked well, because the Spider interface sent/received data through DMA, in
cells of 256 bytes and at speeds comparable to contemporary disks:
https://minnie.tuhs.org/cgi-bin/utree.pl?file=V4/nsys/dmr/tdir/tiu.c.
In 1975 the UoI guys doing Arpa Unix had to interface to an IMP, which was much slower
than a disk but much faster than a terminal. They concluded that neither clists not disk
buffers were a good match and Steve Bunch came up with ’netbuf’s, which were 64 byte
blocks (carved out of a pool of disk blocks). Later (1978?) this code was modified to
optionally use separate buffer heads and buffer data, as part of the UoC buffer mod’s (to
alleviate the pressure on a limited address space):
https://minnie.tuhs.org/cgi-bin/utree.pl?file=SRI-NOSC/h/net/netbuf.h
In 1979 the first Datakit software was done (Chesson), where I don’t have much
information. My hypothesis is that this used clists, but I’m still trying to find a copy
of his paper about this implementation. Maybe the driver used a precursor to a streams
like data structure. Maybe this is also where MPX files fit in. Unfortunately, currently I
just don’t know.
In 1980 new TCP software for the VAX is being written, and again it needs to find a data
buffer approach. This is ‘mbuf’s, which are similar to the earlier ’netbuf’s. The main
changes are that the size is increased to 128 bytes and that a start offset is added,
which makes it possible to remove headers without copying the buffer. The ‘memberof’ field
is replaced by a second linkage pointer. The UoC idea to separate the header and data is
dropped again. They (mbufs) may have other origins as well, but as far as Unix networking
is concerned they are an evolution of the Arpa Unix ‘netbuf’s:
https://minnie.tuhs.org/cgi-bin/utree.pl?file=BBN-Vax-TCP/bbnnet/mbuf.h
Then we get to (late) 1981. On the Labs side streams are created, in my current
understanding mainly to support networking and in particular Datakit. The difference with
Spider is that the Datakit interface was not DMA based and that the data arrives in 9 bit
envelopes instead of 8 bit bytes. The 9th bit distinguishes between data bytes and control
bytes. Again neither clists nor disk buffers are a good match for buffering and a new
solution is created. Looking at buffers, these consist of a header and a separate
datablock. The datablock can be of various sizes: 4, 16, 64 and 1024 bytes - a match with
the varying nature of network traffic. Each block has a type (a.o. DATA and CONTROL) which
has various uses, but certainly gives a home to the 9th Datakit bit where needed:
https://github.com/Alhadis/Research-Unix-v8/blob/master/v8/usr/include/sys/…
https://github.com/Alhadis/Research-Unix-v8/blob/master/v8/usr/sys/dev/stre…
Over on the west coast the mbuf structure is extended with a new field, “type” at the same
time, and it transitions from being a pure network data buffer to a more generic buffer.
It remains at a fixed size of 128 bytes with inline data; the option to store data in a
separate block of varying size will only come back late in 4.3BSD. For the 4.2 version:
https://minnie.tuhs.org/cgi-bin/utree.pl?file=4.2BSD/usr/src/sys/h/mbuf.h
When viewed at this code level it would seem to me that by various routes everybody had
arrived at more or less the same the point in network buffering approach by 1981. That has
some interesting implications. At this level, is there really a difference between a pipe
implemented as cross connected streams and a pipe implemented as cross connected sockets?
Is an unbound socket not about the same as an unconnected stream head?
But before getting into such discussions, I’d really like to get the development story
from 7th to 8th edition as clear and precise as possible after 40 years.
On 30 Mar 2020, at 21:31, Clem Cole
<clemc(a)ccc.com> wrote:
On Mon, Mar 30, 2020 at 12:02 PM Paul Ruizendaal <pnr(a)planet.nl> wrote:
Comments in line:
On 30 Mar 2020, at 15:26, Clem Cole
<clemc(a)ccc.com> wrote:
Your date of June '81 for the 4.1BSD release
seems late, but I'll accept it. 3BSD was 1979, and I thought 4BSD was a year later,
with 4.1BSD a few months after 4BSD (few people actually got 4BSD)
I am aware of that. “20 years of Berkely Unix” says November 1980 for 4BSD and June 1981
for 4.1BSD. From the SCCS log I’d say November 9th and July 1st respectively.
Ok, that sounds right 6-7 months between them. That was the time of the
'FASTVAX' work and the fight with Stanford over if BSD or VMS was to be the
official Arpa OS. I remember Joy was incredibly prolific during that time. From a user
standpoint, 4BSD and 4.1BSD are really similar, but he was hacking the kernel. It seemed
like a new change came out all the time. The CAD group (where I was) always wanted the
fastest system, so there was pressure on me to follow him, but I was trying to make
progress on the AP work (my thesis) so I was a little loath to take the churn.
As far as I can tell CSRG integrated the BBN stack with (almost) 4.1 in April 1981 - most
of that is in a separate tree, with a few #ifdef’s in the main tree.
I'm pretty sure that was mostly Eric Cooper (ecc) IIRC. We had a tape directly from
Gurwitz running in the CAD lab in Cory Hall, running on Xerox 3M Ethernet and because we
needed it between our three 780s systems. We had ordered the 3C501's from 3COM and we
were going to be the Beta for the Interlan board - which I wrote the original driver and
gave it to Sam. I'm not sure when IngVAX arrived, Eric Schmidt had Ing70 downstairs
(and that was the ArpaNet connect via a VDH up the hill to the IMP at LBL. We had been
running BerkNET over serial lines. Ernie and Kim were the two 780s over in Evan's
(the big pile of 750s and the C30 IMP were still in the future).
From SCCS it seems that implementation work on sockets started in Oct/Nov 1981 and 4.1a
was released in April or March 1982.
That sounds about right. It was a bunch of wnj marathon hacking sessions.
From the partial coverage in SCCS I’d say there is no BBN code in 4.1a anymore.
Mumble ... I sort of have a hard time with that. I knew and worked with both of them
in my day and have heard both sides of the story. Joy had Gurwitz's code and was
hacking it - that is just a fact. Claiming pure authorship is a tad extreme. For
instance, Andy VanDam told me that the whole mbuf's stuff is something Rob originally
wrote for a project for him at Brown when he was an undergrad. Rob brought it BBN and he
used it for the OS-independent TCP (i.e. the HP3000 version) originally and it landed in
the UNIX version. Joy hacked on it, but that's a data structure directly from Rob.
On the other hand, Joy types open curly brace, close curly brace, and hacks in code the
fastest of anyone I ever knew (someone, maybe Mary Ann Horton, once said he 'wrote
code at 9600'). But if you look at it, its sometimes not pretty. It works and its
usually fast, which is hard to argue with.
The socket API in that release was different from what ended being in 4.1c and 4.2,
Definitely. I thought other hands besides Joy started to mess with it (like Sam)
and seems to have had some wider visibility in 2.9BSD and UniPlus System V - both of
which used the 4.1a API (and probably code base).
Yeah, Keith was taking things was Sam at pretty hectic pace trying to keep the 11's
running. I would guess that took more of a 'good enough' attitude, they got
the TCP stack in the limited address space and did not want to try much more
Yes, Summer 81 - Summer 82 must have been a vibrant year, with much interaction between
various Unix groups.
It was a fun time to be apart of it all.
By the way, dmr was on the 4.2BSD steering group.
Yes, I know. That's how I first got to know him.