29 Jul
2020
29 Jul
'20
5:03 a.m.
OK, I was able to locate 2bsd.tar.gz and spencer_2bsd.tar.gz in the
archive. Neither is an installation tape. It appears that they are just
tarballs of their respective systems (there are very minor differences
between the two).
In the TAPE file in the tarball, it talks about reading the tar program
off of the tape using:
dd if=/dev/mt0 bs=1b skip=1 of=tar
Well, tar is definitely not located at that address, which implies that
the tarball isn't a distro tape. This note in the archive used to read:
...
The remaining gzipped tar files are other 2BSD distributions supplied by
Keith Bostic, except for spencer_2bsd.tar.gz which came from Henry Spencer.
They do not contain installation tape images. The 2.9BSD-Patch directory
contains patches to 2.9BSD dated August 85, and again supplied by Keith Bostic.
...
now it reads:
...
2.11BSD 2.11BSD-pl195.tar is a copy of 2.11BSD at patch level 195, supplied
by Tom Ivar Helbekkmo. spencer_2bsd.tar.gz is a version of 2BSD which came
from Henry Spencer.
...
I recall having to do something with cont.a files, which are not present
on these images. So, my questions is, does anyone know of or have an
actual 2bsd tape/tape image?
Thanks,
Will
Here's where I found the tarballs:
https://www.tuhs.org/Archive/Distributions/UCB/
--
GPG Fingerprint: 68F4 B3BD 1730 555A 4462 7D45 3EAA 5B6D A982 BAAF
29 Jul
29 Jul
6:09 a.m.
On Tue, Jul 28, 2020 at 5:04 PM Will Senn <will.senn(a)gmail.com> wrote:
OK, I was able to locate 2bsd.tar.gz and
spencer_2bsd.tar.gz in the
archive. Neither is an installation tape. It appears that they are just
tarballs of their respective systems (there are very minor differences
between the two).
In the TAPE file in the tarball, it talks about reading the tar program
off of the tape using:
dd if=/dev/mt0 bs=1b skip=1 of=tar
Well, tar is definitely not located at that address, which implies that
the tarball isn't a distro tape. This note in the archive used to read:
...
The remaining gzipped tar files are other 2BSD distributions supplied by
Keith Bostic, except for spencer_2bsd.tar.gz which came from Henry Spencer.
They do not contain installation tape images. The 2.9BSD-Patch directory
contains patches to 2.9BSD dated August 85, and again supplied by Keith Bostic.
...
now it reads:
...
2.11BSD 2.11BSD-pl195.tar is a copy of 2.11BSD at patch level 195, supplied
by Tom Ivar Helbekkmo. spencer_2bsd.tar.gz is a version of 2BSD which came
from Henry Spencer.
...
I recall having to do something with cont.a files, which are not present on these images.
So, my questions is, does anyone know of or have an actual 2bsd tape/tape image?
Both of the 2bsd tapes you found are from the days when Berkeley just sent
patches
to the 7th Edition out. The 2.8BSD tape was the first one to have a
kernel that was bootable from the tape. The 2BSD tapes originally had 2
files on them. The first one was a binary copy of tar that ran on V7. The
second was a tarball of all the rest. As you discovered, they shipped with
a label like:
Second Berkeley Software Tape
May 10, 1979 TAR 800BPI
%dd if=/dev/mt0 bs=1b skip=1 of=tar
%chmod 755 tar
% tar x
10000 blocks
but the 2bsd.tar.gz file has just the second file.
The spensor_2bsd.tar.gz has a tar binary in it:
tar tvf spencer_2bsd.tar.gz | head
-rw-r--r-- 0 0 0 24688 Feb 17 1980 tar
-rw-r--r-- 0 0 10 3687 Feb 17 1980 tar.1
-rw-r--r-- 0 0 10 456 Feb 17 1980 tar.ms
-rw-r--r-- 0 0 10 15216 Feb 17 1980 install.ms
if you are looking for that pre-built. If you are looking to create a tape
with tar on it to extract other tar tapes, you'd need to use a variation on
the maketape.pl with a block size of 1 so the above dd will work on the
target system...
Warner
6:19 a.m.
Note the UID/GID are numeric not ASCII ;-)
I suspect henry's tape is closer to the official tape!!
On Tue, Jul 28, 2020 at 8:11 PM Warner Losh <imp(a)bsdimp.com> wrote:
On Tue, Jul 28, 2020 at 5:04 PM Will Senn <will.senn(a)gmail.com> wrote:
OK, I was able to locate 2bsd.tar.gz and
spencer_2bsd.tar.gz in the
archive. Neither is an installation tape. It appears that they are just
tarballs of their respective systems (there are very minor differences
between the two).
In the TAPE file in the tarball, it talks about reading the tar program
off of the tape using:
dd if=/dev/mt0 bs=1b skip=1 of=tar
Well, tar is definitely not located at that address, which implies that
the tarball isn't a distro tape. This note in the archive used to read:
...
The remaining gzipped tar files are other 2BSD distributions supplied by
Keith Bostic, except for spencer_2bsd.tar.gz which came from Henry Spencer.
They do not contain installation tape images. The 2.9BSD-Patch directory
contains patches to 2.9BSD dated August 85, and again supplied by Keith Bostic.
...
now it reads:
...
2.11BSD 2.11BSD-pl195.tar is a copy of 2.11BSD at patch level 195, supplied
by Tom Ivar Helbekkmo. spencer_2bsd.tar.gz is a version of 2BSD which came
from Henry Spencer.
...
I recall having to do something with cont.a files, which are not present on these images.
So, my questions is, does anyone know of or have an actual 2bsd tape/tape image?
Both of the 2bsd tapes you found are from the days when Berkeley just
sent patches
to the 7th Edition out. The 2.8BSD tape was the first one to
have a kernel that was bootable from the tape. The 2BSD tapes originally
had 2 files on them. The first one was a binary copy of tar that ran on V7.
The second was a tarball of all the rest. As you discovered, they shipped
with a label like:
Second Berkeley Software Tape
May 10, 1979 TAR 800BPI
%dd if=/dev/mt0 bs=1b skip=1 of=tar
%chmod 755 tar
% tar x
10000 blocks
but the 2bsd.tar.gz file has just the second file.
The spensor_2bsd.tar.gz has a tar binary in it:
tar tvf spencer_2bsd.tar.gz | head
-rw-r--r-- 0 0 0 24688 Feb 17 1980 tar
-rw-r--r-- 0 0 10 3687 Feb 17 1980 tar.1
-rw-r--r-- 0 0 10 456 Feb 17 1980 tar.ms
-rw-r--r-- 0 0 10 15216 Feb 17 1980 install.ms
if you are looking for that pre-built. If you are looking to create a tape
with tar on it to extract other tar tapes, you'd need to use a variation on
the maketape.pl with a block size of 1 so the above dd will work on the
target system...
Warner
6:45 a.m.
On 7/28/20 7:09 PM, Warner Losh wrote:
Both of the 2bsd tapes you found are from the days when Berkeley just
sent patches to the 7th Edition out. The 2.8BSD tape was the first one
to have a kernel that was bootable from the tape. The 2BSD tapes
originally had 2 files on them. The first one was a binary copy of tar
that ran on V7. The second was a tarball of all the rest. As you
discovered, they shipped with a label like:
Second Berkeley Software Tape
May 10, 1979 TAR 800BPI
%dd if=/dev/mt0 bs=1b skip=1 of=tar
%chmod 755 tar
% tar x
10000 blocks
but the 2bsd.tar.gz file has just the second file.
The spensor_2bsd.tar.gz has a tar binary in it:
tar tvf spencer_2bsd.tar.gz | head
-rw-r--r-- 0 0 0 24688 Feb 17 1980 tar
-rw-r--r-- 0 0 10 3687 Feb 17 1980 tar.1
-rw-r--r-- 0 0 10 456 Feb 17 1980 tar.ms <http://tar.ms>
-rw-r--r-- 0 0 10 15216 Feb 17 1980 install.ms
<http://install.ms>
if you are looking for that pre-built. If you are looking to create a
tape with tar on it to extract other tar tapes, you'd need to use a
variation on the maketape.pl <http://maketape.pl> with a block size of
1 so the above dd will work on the target system...
Warner
OK. That makes sense. I'll just work with it as is on my v7 system.
Will
--
GPG Fingerprint: 68F4 B3BD 1730 555A 4462 7D45 3EAA 5B6D A982 BAAF
6:46 a.m.
On 7/28/20 7:09 PM, Warner Losh wrote:
Both of the 2bsd tapes you found are from the days
when Berkeley just
sent patches to the 7th Edition out. The 2.8BSD tape was the first one
to have a kernel that was bootable from the tape. The 2BSD tapes
originally had 2 files on them. The first one was a binary copy of tar
that ran on V7. The second was a tarball of all the rest. As you
discovered, they shipped with a label like:
Second Berkeley Software Tape
May 10, 1979 TAR 800BPI
%dd if=/dev/mt0 bs=1b skip=1 of=tar
%chmod 755 tar
% tar x
10000 blocks
but the 2bsd.tar.gz file has just the second file.
The spensor_2bsd.tar.gz has a tar binary in it:
tar tvf spencer_2bsd.tar.gz | head
-rw-r--r-- 0 0 0 24688 Feb 17 1980 tar
-rw-r--r-- 0 0 10 3687 Feb 17 1980 tar.1
-rw-r--r-- 0 0 10 456 Feb 17 1980 tar.ms <http://tar.ms>
-rw-r--r-- 0 0 10 15216 Feb 17 1980 install.ms
<http://install.ms>
if you are looking for that pre-built. If you are looking to create a
tape with tar on it to extract other tar tapes, you'd need to use a
variation on the maketape.pl <http://maketape.pl> with a block size of
1 so the above dd will work on the target system...
This definitely makes sense. Thanks. This
--
GPG Fingerprint: 68F4 B3BD 1730 555A 4462 7D45 3EAA 5B6D A982 BAAF
6:21 a.m.
Cross posting to simh - since much of this has been discussed in the last
few days there also....
in for penny, in for pound ... here is the history ... man ... I lived
this and I'll need a strong drink later tonight after I write it all up.
On Tue, Jul 28, 2020 at 7:04 PM Will Senn <will.senn(a)gmail.com> wrote:
I recall having to do something with cont.a files,
which are not present on these images. So, my questions is, does anyone know of or have an
actual 2bsd tape/tape image?
cont.a is a tp-v6 and earlier ism.
DECtape had a directory at the front of the tape (think superblock/ilist),
but could do cool things and be treated more like a disk.
When tp was created for a very early version of Unix (I'm not sure which,
could be V2), Ken/Dennis et al had DECtape units and so the original scheme
followed the media. This also meant that the program could write files
and go back and update the directory, which is a no-no with many tape
systems. Then research got a 9-track unit. So tp was changed to
calculate how much space was going to be needed, write the directory, then
the datablocks. All good ... except...
9-track could write more files than the directory could take. So for many
years, people would use the ar(1) program to take a number of files in a
directory, create a file called cont.a and then delete the files. Then the
tree would be written with tp, when you read it, you reversed the ar(1)
process. If you look at the USENIX/Harvard tape on the TUHS you'll see
this scheme in use.
BTW: tp was written in assembler and all the data structures were using
PDP-11 binary formats. Eventually, Harvard wrote stp (super-tp) in C
(which is on the USENIX tape Warren has in the archives) that worked like
the original assembler tp but also put a redundant directory at the end of
the tape. Another issue with tp was if the you had a bad block in the
first few blocks you could not decode the rest of the tape. [There were
some other issues with the UNIX tree structure as disks got bigger but I'm
going to ignore all that - other than to say, tp had lived it life].
Enter Mashey and the PWB 1.0 folks (which is based on V6). Someone in USG
created cpio (and volcpy) as part of the PWB 1.0. Like tp it was a PDP-11
binary format, but unlike tp, the tape directory is threaded. *i.e.* block
one describes the first file only and includes the size of the following
file, then file itself, then a new directory block for the next file and
again that file (rinse and repeat). So it improved on tp in the directory
threading, but was still binary and for a reasons I'll leave out had a
different user interface.
As part of V7, Ken wrote a new program, tar [you can ask him why]. But
like cpio he used a threaded tape directory, but unlike cpio it was always
ASCII and not PDP-11 specific. Furthermore, the user interface was made to
parrot tp. So, certainly, it had the advantage that changing tp scripts to
use tar was pretty easy i.e. s/tp/tar/ not so for coil. And it was
muscle memory compliant.
For PWB 2.0, cpio was updated to allow a -c option to write the header in
ascii and -s to byte swap the binary. But the damage had been done ...
Thus began 'tar wars' which was a battle that raged officially over tape
archive formats, but really was an argument about user interfaces. Since
tar was part of Research and the Universities and commercial people used
it, only USG and the folks inside the Bell System were using cpio, as
officially none of us had it since PWB was not released to us (although
thanks to many AT&T employees doing an OYOC year, many schools like UCB,
MIT and CMU all had the sources to cpio anyway -- for instance you'll see
it hidden away on Kirk's CD).
I personally had used both, preferred tar for easy of use and ASCII
directories. But, note I had written car at Masscomp, but never tpio.
This was our trick to use the file scripting list that cpio could do, but
create tar format tapes - which was handy. I never wrote tpio which would
have been cpio format using tp/tar user interface as I did not need it.
Roll forward to the /usr/group UNIX standard that Heinz chaired. We ended
up not being able to agree on a distribution format, but the ISVs were PO
because now they could create UNIX programs that might actually work across
systems, but they had not standard way to distribution.
Roll forward again to IEEE. Heinz's committee was officially disbanded
(story discussed elsewhere) and we were created as IEEE P1003 with Jim
Issack as Chair. This time the ISV's said we had to have a distribution
format. Since *.1 was only an API we were allowed to avoid the user
interface issue but only examine the on tape format.
It turns out while it seems to have been unintended, Ken's original V7
implementation has an interesting coding feature/bug which turns out to be
what clinched the deal. When Ken creates the directory block for each
file, he did bcopy of 0's to the buffer before he wrote that data that
fills it in. Then when he calculated the checksum for the directory header
block, he summed the entire block (which because of the bcopy was zeros).
This means if you write beyond the end of Ken's original header and include
that extra data in the chksum, the original program will ignore the new
information but accept the directory block as valid. i.e. he had built an
extension mechanism into the tar on-tape format.
cpio's ASCII on tape format was not able to do that as the checksum used a
sizeof(header struct) in the checksum routine.
USTAR was born ... Ken had written things like the UID/GID as ASCII
representations of the integer value in the original header. USTAR added
the ASCII representation of the username and the group name since that was
more often portable between systems than the numbers. There were other
additions like more room for the pathname new file types *etc*. But the
key is that a USTAR tape can be read by the original V7 (and follow on)
tape formats, although may not recognize all the filetype or use all of the
new information.
A few years later during *.2 discussions, we finally got into the user
interface stuff and pax(1) was born. Knowing my hack with car, Keith
Bostic, Jim McGuiness and I wrote up a description of a program that could
with both users interfaces scheme. USENIX provided funding for a student
to implement it and put the sources out on comp.unix.sources at some
point. That proposal was originally accepted at the first tape user
interface program in *.2 [a few years later after I stopped being part of
the committee, the USG folks did get an alternate CPIO format accepted and
cpio as an allowed program. USENIX paid to have the program updated to
operate like cpio if it was called that, pure V7 tar if called that and if
pax, user USTAR].
'nuf said ... I hope.
Clem
3:50 p.m.
New subject: [simh] 2bsd tarball
Just a small comment. Whoever it was that thought DECtape was a tape was
making a serious mistake. DECtapes are very different from magtapes.
Johnny
On 2020-07-29 02:21, Clement T Cole wrote:
Cross posting to simh - since much of this has been discussed in the
last few days there also....
in for penny, in for pound ... here is the history ... man ... I lived
this and I'll need a strong drink later tonight after I write it all up.
On Tue, Jul 28, 2020 at 7:04 PM Will Senn <will.senn(a)gmail.com
<mailto:will.senn@gmail.com>> wrote:
I recall having to do something with cont.a files, which are not
present on these images. So, my questions is, does anyone know of or
have an actual 2bsd tape/tape image?
cont.a is a tp-v6 and earlier ism.
DECtape had a directory at the front of the tape (think
superblock/ilist), but could do cool things and be treated more like a disk.
When tp was created for a very early version of Unix (I'm not sure
which, could be V2), Ken/Dennis et al had DECtape units and so the
original scheme followed the media. This also meant that the program
could write files and go back and update the directory, which is a no-no
with many tape systems. Then research got a 9-track unit. So tp was
changed to calculate how much space was going to be needed, write the
directory, then the datablocks. All good ... except...
9-track could write more files than the directory could take. So for
many years, people would use the ar(1) program to take a number of files
in a directory, create a file called cont.a and then delete the files.
Then the tree would be written with tp, when you read it, you reversed
the ar(1) process. If you look at the USENIX/Harvard tape on the TUHS
you'll see this scheme in use.
BTW: tp was written in assembler and all the data structures were using
PDP-11 binary formats. Eventually, Harvard wrote stp (super-tp) in C
(which is on the USENIX tape Warren has in the archives) that worked
like the original assembler tp but also put a redundant directory at the
end of the tape. Another issue with tp was if the you had a bad block
in the first few blocks you could not decode the rest of the tape.
[There were some other issues with the UNIX tree structure as disks got
bigger but I'm going to ignore all that - other than to say, tp had
lived it life].
Enter Mashey and the PWB 1.0 folks (which is based on V6). Someone in
USG created cpio (and volcpy) as part of the PWB 1.0. Like tp it was a
PDP-11 binary format, but unlike tp, the tape directory is threaded.
/i.e./ block one describes the first file only and includes the size of
the following file, then file itself, then a new directory block for the
next file and again that file (rinse and repeat). So it improved on tp
in the directory threading, but was still binary and for a reasons I'll
leave out had a different user interface.
As part of V7, Ken wrote a new program, tar [you can ask him why]. But
like cpio he used a threaded tape directory, but unlike cpio it was
always ASCII and not PDP-11 specific. Furthermore, the user interface
was made to parrot tp. So, certainly, it had the advantage that
changing tp scripts to use tar was pretty easy i.e. s/tp/tar/ not so
for coil. And it was muscle memory compliant.
For PWB 2.0, cpio was updated to allow a -c option to write the header
in ascii and -s to byte swap the binary. But the damage had been done ...
Thus began 'tar wars' which was a battle that raged officially over tape
archive formats, but really was an argument about user interfaces.
Since tar was part of Research and the Universities and commercial
people used it, only USG and the folks inside the Bell System were using
cpio, as officially none of us had it since PWB was not released to us
(although thanks to many AT&T employees doing an OYOC year, many schools
like UCB, MIT and CMU all had the sources to cpio anyway -- for instance
you'll see it hidden away on Kirk's CD).
I personally had used both, preferred tar for easy of use and ASCII
directories. But, note I had written car at Masscomp, but never tpio.
This was our trick to use the file scripting list that cpio could do,
but create tar format tapes - which was handy. I never wrote tpio which
would have been cpio format using tp/tar user interface as I did not
need it.
Roll forward to the /usr/group UNIX standard that Heinz chaired. We
ended up not being able to agree on a distribution format, but the ISVs
were PO because now they could create UNIX programs that might actually
work across systems, but they had not standard way to distribution.
Roll forward again to IEEE. Heinz's committee was officially disbanded
(story discussed elsewhere) and we were created as IEEE P1003 with Jim
Issack as Chair. This time the ISV's said we had to have a distribution
format. Since *.1 was only an API we were allowed to avoid the user
interface issue but only examine the on tape format.
It turns out while it seems to have been unintended, Ken's original V7
implementation has an interesting coding feature/bug which turns out to
be what clinched the deal. When Ken creates the directory block for
each file, he did bcopy of 0's to the buffer before he wrote that data
that fills it in. Then when he calculated the checksum for the
directory header block, he summed the entire block (which because of the
bcopy was zeros). This means if you write beyond the end of Ken's
original header and include that extra data in the chksum, the original
program will ignore the new information but accept the directory block
as valid. i.e. he had built an extension mechanism into the tar on-tape
format.
cpio's ASCII on tape format was not able to do that as the checksum used
a sizeof(header struct) in the checksum routine.
USTAR was born ... Ken had written things like the UID/GID as ASCII
representations of the integer value in the original header. USTAR
added the ASCII representation of the username and the group name since
that was more often portable between systems than the numbers. There
were other additions like more room for the pathname new file types
/etc/. But the key is that a USTAR tape can be read by the original V7
(and follow on) tape formats, although may not recognize all the
filetype or use all of the new information.
A few years later during *.2 discussions, we finally got into the user
interface stuff and pax(1) was born. Knowing my hack with car, Keith
Bostic, Jim McGuiness and I wrote up a description of a program that
could with both users interfaces scheme. USENIX provided funding for a
student to implement it and put the sources out on comp.unix.sources at
some point. That proposal was originally accepted at the first tape
user interface program in *.2 [a few years later after I stopped being
part of the committee, the USG folks did get an alternate CPIO format
accepted and cpio as an allowed program. USENIX paid to have the
program updated to operate like cpio if it was called that, pure V7 tar
if called that and if pax, user USTAR].
'nuf said ... I hope.
Clem
_._,_._,_
------------------------------------------------------------------------
Groups.io Links:
You receive all messages sent to this group.
View/Reply Online (#62) <https://groups.io/g/simh/message/62> | Reply To
Group
<mailto:simh@groups.io?subject=Re:%20Re%3A%20%5Bsimh%5D%20%5BTUHS%5D%202bsd%20tarball>
| Reply To Sender
<mailto:clemc@ccc.com?subject=Private:%20Re:%20Re%3A%20%5Bsimh%5D%20%5BTUHS%5D%202bsd%20tarball>
| Mute This Topic <https://groups.io/mt/75856261/4814011> | New Topic
<https://groups.io/g/simh/post>
Your Subscription <https://groups.io/g/simh/editsub/4814011> | Contact
Group Owner <mailto:simh+owner@groups.io> | Unsubscribe
<https://groups.io/g/simh/leave/8625569/104597204/xyzzy> [bqt(a)softjar.se]
_._,_._,_
--
Johnny Billquist || "I'm on a bus
|| on a psychedelic trip
email: bqt(a)softjar.se || Reading murder books
pdp is alive! || tryin' to stay hip" - B. Idol
7:52 p.m.
New subject: [simh] 2bsd tarball
When I talk about DECtape in my capacity as the local old fart, i describe
it as "a disk with one track and about 1500 small sectors that spins ve-ry
ve-ry slow-ly..
On Wed, Jul 29, 2020 at 5:58 AM Johnny Billquist <bqt(a)softjar.se> wrote:
Just a small comment. Whoever it was that thought
DECtape was a tape was
making a serious mistake. DECtapes are very different from magtapes.
Johnny
On 2020-07-29 02:21, Clement T Cole wrote:
Cross posting to simh - since much of this has been discussed in the
last few days there also....
in for penny, in for pound ... here is the history ... man ... I lived
this and I'll need a strong drink later tonight after I write it all up.
On Tue, Jul 28, 2020 at 7:04 PM Will Senn <will.senn(a)gmail.com
<mailto:will.senn@gmail.com>> wrote:
I recall having to do something with cont.a files, which are not
present on these images. So, my questions is, does anyone know of or
have an actual 2bsd tape/tape image?
cont.a is a tp-v6 and earlier ism.
DECtape had a directory at the front of the tape (think
superblock/ilist), but could do cool things and be treated more like a
disk.
When tp was created for a very early version of
Unix (I'm not sure
which, could be V2), Ken/Dennis et al had DECtape units and so the
original scheme followed the media. This also meant that the program
could write files and go back and update the directory, which is a no-no
with many tape systems. Then research got a 9-track unit. So tp was
changed to calculate how much space was going to be needed, write the
directory, then the datablocks. All good ... except...
9-track could write more files than the directory could take. So for
many years, people would use the ar(1) program to take a number of files
in a directory, create a file called cont.a and then delete the files.
Then the tree would be written with tp, when you read it, you reversed
the ar(1) process. If you look at the USENIX/Harvard tape on the TUHS
you'll see this scheme in use.
BTW: tp was written in assembler and all the data structures were using
PDP-11 binary formats. Eventually, Harvard wrote stp (super-tp) in C
(which is on the USENIX tape Warren has in the archives) that worked
like the original assembler tp but also put a redundant directory at the
end of the tape. Another issue with tp was if the you had a bad block
in the first few blocks you could not decode the rest of the tape.
[There were some other issues with the UNIX tree structure as disks got
bigger but I'm going to ignore all that - other than to say, tp had
lived it life].
Enter Mashey and the PWB 1.0 folks (which is based on V6). Someone in
USG created cpio (and volcpy) as part of the PWB 1.0. Like tp it was a
PDP-11 binary format, but unlike tp, the tape directory is threaded.
/i.e./ block one describes the first file only and includes the size of
the following file, then file itself, then a new directory block for the
next file and again that file (rinse and repeat). So it improved on tp
in the directory threading, but was still binary and for a reasons I'll
leave out had a different user interface.
As part of V7, Ken wrote a new program, tar [you can ask him why]. But
like cpio he used a threaded tape directory, but unlike cpio it was
always ASCII and not PDP-11 specific. Furthermore, the user interface
was made to parrot tp. So, certainly, it had the advantage that
changing tp scripts to use tar was pretty easy i.e. s/tp/tar/ not so
for coil. And it was muscle memory compliant.
For PWB 2.0, cpio was updated to allow a -c option to write the header
in ascii and -s to byte swap the binary. But the damage had been done
...
Thus began 'tar wars' which was a battle that raged officially over tape
archive formats, but really was an argument about user interfaces.
Since tar was part of Research and the Universities and commercial
people used it, only USG and the folks inside the Bell System were using
cpio, as officially none of us had it since PWB was not released to us
(although thanks to many AT&T employees doing an OYOC year, many schools
like UCB, MIT and CMU all had the sources to cpio anyway -- for instance
you'll see it hidden away on Kirk's CD).
I personally had used both, preferred tar for easy of use and ASCII
directories. But, note I had written car at Masscomp, but never tpio.
This was our trick to use the file scripting list that cpio could do,
but create tar format tapes - which was handy. I never wrote tpio which
would have been cpio format using tp/tar user interface as I did not
need it.
Roll forward to the /usr/group UNIX standard that Heinz chaired. We
ended up not being able to agree on a distribution format, but the ISVs
were PO because now they could create UNIX programs that might actually
work across systems, but they had not standard way to distribution.
Roll forward again to IEEE. Heinz's committee was officially disbanded
(story discussed elsewhere) and we were created as IEEE P1003 with Jim
Issack as Chair. This time the ISV's said we had to have a distribution
format. Since *.1 was only an API we were allowed to avoid the user
interface issue but only examine the on tape format.
It turns out while it seems to have been unintended, Ken's original V7
implementation has an interesting coding feature/bug which turns out to
be what clinched the deal. When Ken creates the directory block for
each file, he did bcopy of 0's to the buffer before he wrote that data
that fills it in. Then when he calculated the checksum for the
directory header block, he summed the entire block (which because of the
bcopy was zeros). This means if you write beyond the end of Ken's
original header and include that extra data in the chksum, the original
program will ignore the new information but accept the directory block
as valid. i.e. he had built an extension mechanism into the tar on-tape
format.
cpio's ASCII on tape format was not able to do that as the checksum used
a sizeof(header struct) in the checksum routine.
USTAR was born ... Ken had written things like the UID/GID as ASCII
representations of the integer value in the original header. USTAR
added the ASCII representation of the username and the group name since
that was more often portable between systems than the numbers. There
were other additions like more room for the pathname new file types
/etc/. But the key is that a USTAR tape can be read by the original V7
(and follow on) tape formats, although may not recognize all the
filetype or use all of the new information.
A few years later during *.2 discussions, we finally got into the user
interface stuff and pax(1) was born. Knowing my hack with car, Keith
Bostic, Jim McGuiness and I wrote up a description of a program that
could with both users interfaces scheme. USENIX provided funding for a
student to implement it and put the sources out on comp.unix.sources at
some point. That proposal was originally accepted at the first tape
user interface program in *.2 [a few years later after I stopped being
part of the committee, the USG folks did get an alternate CPIO format
accepted and cpio as an allowed program. USENIX paid to have the
program updated to operate like cpio if it was called that, pure V7 tar
if called that and if pax, user USTAR].
'nuf said ... I hope.
Clem
_._,_._,_
------------------------------------------------------------------------
Groups.io Links:
You receive all messages sent to this group.
View/Reply Online (#62) <https://groups.io/g/simh/message/62> | Reply
To
Group
<mailto:simh@groups.io?subject=Re:%20Re%3A%20%5Bsimh%5D%20%5BTUHS%5D%202bsd%20tarball>
| Reply To Sender
<mailto:clemc@ccc.com?subject=Private:%20Re:%20Re%3A%20%5Bsimh%5D%20%5BTUHS%5D%202bsd%20tarball>
| Mute This Topic
<https://groups.io/mt/75856261/4814011> | New Topic
<https://groups.io/g/simh/post>
Your Subscription <https://groups.io/g/simh/editsub/4814011> | Contact
Group Owner <mailto:simh+owner@groups.io> | Unsubscribe
<https://groups.io/g/simh/leave/8625569/104597204/xyzzy> [bqt(a)softjar.se
]
_._,_._,_
--
Johnny Billquist || "I'm on a bus
|| on a psychedelic trip
email: bqt(a)softjar.se || Reading murder books
pdp is alive! || tryin' to stay hip" - B. Idol
8:30 p.m.
New subject: [simh] 2bsd tarball
Yes. That is a much better way to look at it.
Johnny
On 2020-07-29 15:52, John Cowan wrote:
When I talk about DECtape in my capacity as the local
old fart, i
describe it as "a disk with one track and about 1500 small sectors that
spins ve-ry ve-ry slow-ly..
On Wed, Jul 29, 2020 at 5:58 AM Johnny Billquist <bqt(a)softjar.se
<mailto:bqt@softjar.se>> wrote:
Just a small comment. Whoever it was that thought DECtape was a tape
was
making a serious mistake. DECtapes are very different from magtapes.
Johnny
On 2020-07-29 02:21, Clement T Cole wrote:
--
Johnny Billquist || "I'm on a bus
|| on a psychedelic trip
email: bqt(a)softjar.se || Reading murder books
pdp is alive! || tryin' to stay hip" - B. Idol
Cross posting to simh - since much of this has been discussed in the
last few days there also....
in for penny, in for pound ... here is the history ... man ... I
lived
this and I'll need a strong drink later
tonight after I write it
all up.
On Tue, Jul 28, 2020 at 7:04 PM Will Senn <will.senn(a)gmail.com
<mailto:will.senn@gmail.com>
<mailto:will.senn@gmail.com
<mailto:will.senn@gmail.com>>> wrote:
I recall having to do something with cont.a files, which are not
present on these images. So, my questions is, does anyone
know of or
have an actual 2bsd tape/tape image?
cont.a is a tp-v6 and earlier ism.
DECtape had a directory at the front of the tape (think
superblock/ilist), but could do cool things and be treated more
like a disk.
When tp was created for a very early version of
Unix (I'm not sure
which, could be V2), Ken/Dennis et al had DECtape units and so the
original scheme followed the media. This also meant that the
program
could write files and go back and update the
directory, which is
a no-no
with many tape systems. Then research got a
9-track unit. So
tp was
changed to calculate how much space was going to
be needed, write
the
directory, then the datablocks. All good ...
except...
9-track could write more files than the directory could take.
So for
many years, people would use the ar(1) program to
take a number
of files
in a directory, create a file called cont.a and
then delete the
files.
Then the tree would be written with tp, when you
read it, you
reversed
the ar(1) process. If you look at the
USENIX/Harvard tape on the
TUHS
you'll see this scheme in use.
BTW: tp was written in assembler and all the data structures were
using
PDP-11 binary formats. Eventually, Harvard wrote
stp (super-tp)
in C
(which is on the USENIX tape Warren has in the
archives) that worked
like the original assembler tp but also put a redundant directory
at the
end of the tape. Another issue with tp was if
the you had a bad
block
in the first few blocks you could not decode the
rest of the tape.
[There were some other issues with the UNIX tree structure as
disks got
bigger but I'm going to ignore all that -
other than to say, tp had
lived it life].
Enter Mashey and the PWB 1.0 folks (which is based on V6).
Someone in
USG created cpio (and volcpy) as part of the PWB
1.0. Like tp
it was a
PDP-11 binary format, but unlike tp, the tape
directory is threaded.
/i.e./ block one describes the first file only and includes the
size of
the following file, then file itself, then a new
directory block
for the
next file and again that file (rinse and
repeat). So it improved
on tp
in the directory threading, but was still binary
and for a
reasons I'll
leave out had a different user interface.
As part of V7, Ken wrote a new program, tar [you can ask him
why]. But
like cpio he used a threaded tape directory, but
unlike cpio it was
always ASCII and not PDP-11 specific. Furthermore, the user
interface
was made to parrot tp. So, certainly, it had the
advantage that
changing tp scripts to use tar was pretty easy i.e. s/tp/tar/
not so
for coil. And it was muscle memory compliant.
For PWB 2.0, cpio was updated to allow a -c option to write the
header
in ascii and -s to byte swap the binary. But
the damage had
been done ...
Thus began 'tar wars' which was a battle that raged officially
over
tape
archive formats, but really was an argument about
user interfaces.
Since tar was part of Research and the Universities and commercial
people used it, only USG and the folks inside the Bell System
were using
cpio, as officially none of us had it since PWB
was not released
to us
(although thanks to many AT&T employees doing
an OYOC year, many
schools
like UCB, MIT and CMU all had the sources to cpio
anyway -- for
instance
you'll see it hidden away on Kirk's
CD).
I personally had used both, preferred tar for easy of use and ASCII
directories. But, note I had written car at Masscomp, but never
tpio.
This was our trick to use the file scripting list
that cpio could
do,
but create tar format tapes - which was handy. I
never wrote
tpio which
would have been cpio format using tp/tar user
interface as I did not
need it.
Roll forward to the /usr/group UNIX standard that Heinz chaired. We
ended up not being able to agree on a distribution format, but
the ISVs
were PO because now they could create UNIX
programs that might
actually
work across systems, but they had not standard
way to distribution.
Roll forward again to IEEE. Heinz's committee was officially
disbanded
(story discussed elsewhere) and we were created
as IEEE P1003
with Jim
Issack as Chair. This time the ISV's said we
had to have a
distribution
format. Since *.1 was only an API we were
allowed to avoid the user
interface issue but only examine the on tape format.
It turns out while it seems to have been unintended, Ken's
original V7
implementation has an interesting coding
feature/bug which turns
out to
be what clinched the deal. When Ken creates the
directory block
for
each file, he did bcopy of 0's to the buffer
before he wrote that
data
that fills it in. Then when he calculated the
checksum for the
directory header block, he summed the entire block (which because
of the
bcopy was zeros). This means if you write beyond
the end of Ken's
original header and include that extra data in the chksum, the
original
program will ignore the new information but
accept the directory
block
as valid. i.e. he had built an extension
mechanism into the tar
on-tape
format.
cpio's ASCII on tape format was not able to do that as the
checksum used
a sizeof(header struct) in the checksum routine.
USTAR was born ... Ken had written things like the UID/GID as ASCII
representations of the integer value in the original header. USTAR
added the ASCII representation of the username and the group name
since
that was more often portable between systems than
the numbers.
There
were other additions like more room for the
pathname new file types
/etc/. But the key is that a USTAR tape can be read by the
original V7
(and follow on) tape formats, although may not
recognize all the
filetype or use all of the new information.
A few years later during *.2 discussions, we finally got into the
user
interface stuff and pax(1) was born. Knowing my
hack with car,
Keith
Bostic, Jim McGuiness and I wrote up a
description of a program that
could with both users interfaces scheme. USENIX provided funding
for a
student to implement it and put the sources out
on
comp.unix.sources at
some point. That proposal was originally
accepted at the first tape
user interface program in *.2 [a few years later after I stopped
being
part of the committee, the USG folks did get an
alternate CPIO
format
accepted and cpio as an allowed program. USENIX
paid to have the
program updated to operate like cpio if it was called that, pure
V7 tar
if called that and if pax, user USTAR].
'nuf said ... I hope.
Clem
_._,_._,_
------------------------------------------------------------------------
Groups.io Links:
You receive all messages sent to this group.
View/Reply Online (#62) <https://groups.io/g/simh/message/62> |
Reply
To
Group
<mailto:simh@groups.io
<mailto:simh@groups.io>?subject=Re:%20Re%3A%20%5Bsimh%5D%20%5BTUHS%5D%202bsd%20tarball>
| Reply To Sender
<mailto:clemc@ccc.com
<mailto:clemc@ccc.com>?subject=Private:%20Re:%20Re%3A%20%5Bsimh%5D%20%5BTUHS%5D%202bsd%20tarball>
| Mute This Topic
<https://groups.io/mt/75856261/4814011> | New
Topic
<https://groups.io/g/simh/post>
Your Subscription <https://groups.io/g/simh/editsub/4814011> |
Contact
Group Owner <mailto:simh+owner@groups.io
<mailto:simh%2Bowner@groups.io>> | Unsubscribe
<https://groups.io/g/simh/leave/8625569/104597204/xyzzy>
[bqt(a)softjar.se
<mailto:bqt@softjar.se>]
_._,_._,_
--
Johnny Billquist || "I'm on a bus
|| on a psychedelic trip
email: bqt(a)softjar.se <mailto:bqt@softjar.se> ||
Reading murder books
pdp is alive! || tryin' to stay hip" - B. Idol
12 Aug
12 Aug
5:41 a.m.
New subject: [simh] 2bsd tarball
On Wed, 29 Jul 2020, John Cowan wrote:
When I talk about DECtape in my capacity as the local
old fart, i
describe it as "a disk with one track and about 1500 small sectors that
spins ve-ry ve-ry slow-ly..
According to an ex-boss of mine, a demonstration of the DECtape's
reliability was to blow cigarette smoke over the heads when the tape was
moving (this was back when smoking was socially acceptable, if not
actually encouraged).
-- Dave
29 Jul
29 Jul
7:42 p.m.
New subject: 2bsd tarball -> pdtar, with a side of uuslave
Clem Cole <clemc(a)ccc.com> wrote:
'nuf said ... I hope.
Not quite, Clem!
There was another chapter to the "tar wars" after UNIX and after POSIX.
After I left Sun in about 1985, I worked on a project with GNU and the
BSD folks, to find or write freely available implementations of many
popular UNIX commands. Since we didn't find a free "tar" program, I
wrote one from scratch, based on the SunOS man page and on running the
tar binary from SunOS 3.3.
This was eventually posted to Usenet's mod.sources as "pdtar" on
1986-12-10, as Volume 7 Issue 88. A July 1987 version went on the Sun
User Group tape. A later version was posted to comp.sources.unix on
1987-11-29 as Volume 12, issue 68 in 3 parts. By then, it built and ran
on SunOS, Xenix, Unisoft, Vax 4.2BSD, V7 and USG systems, MSDOS, and
Minix. And "Utzoonix", Henry Spencer's customized system.
In the process I discovered various quirks in the Unix tar program. It
null-terminated some of the fixed-length octal fields in the tar headers
and space-terminated others. I made my code produce bit-for-bit
identical tar files, except for the leftover buffer garbage at the ends
of files and in the last block after the second end-of-file all-zero
header, which I zeroed.
The UNIX tar program was also reading and writing the files in the file
system in 512-byte reads and writes. Eg when extracting a file from a
tape with the default blocking factor of 20, it would do 20 writes to
the file rather than a single 10k write. pdtar did better.
Oh, and by the way, the tar format DID use numerical uid and gid fields,
encoded in ascii numerals in octal. Thus they were both numeric and
ascii! What it didn't do was put in the NAMES of the user and their
group; that idea came in with POSIX drafts, and went into separate
fields that had been undefined & ignored in UNIX tar.
I put the pdtar code into the public domain, so it could be widely used.
This produced a variety of support headaches. In particular, people who
used it on MSDOS kept contacting me over the years about bugs or
documentation or not working with certain hardware. Those ports were
largely distributed as binaries by other people who never bothered to
include nor publish the matching source code. So I couldn't support the
users who were having trouble, which was frustrating for both them and
me. This eventually led me to understand more of the value in using the
GNU General Public License.
When the GNU Project later wanted someone else to enhance tar to satisfy
some of the user requirements from the absence of dump(8), like
incremental backups, they asked if I would mind if they put their
improved version under the GPL. I had no problem with that, and the
pdtar code became the base code for GNU Tar, which (along with the
simple BusyBox tar) seems to have become the main implementation in the
wild today.
John
PS: In that period, I also wrote the first reliable free implementation
of uucp, called gnuucp. This code was based on "uuslave.c 1.7 08/12/85
12:04:20" which had been posted to the ACGNJ BBS system at +1 201 753
9758. Original author unknown. BSD and I went through a legal
interaction with AT&T to verify that the uuslave code had NOT come from
any proprietary UNIX code. The uuslave code barely limped (and had CP/M
ifdef's), but it did point out the basics of the 'g' protocol. Once I
got it working on UNIX, I ran it for a few weeks with my friends at
lll-crg in Livermore, CA, then posted it to net.sources on 1987-03-25,
article-id hoptoad.1925. By this time it had been ported to MSDOS by
Tim Pozar. Soon it became the backbone of Tim's FIDOnet <-> Usenet
gateway (UFGATE) software, which enabled thousands of (largely teenage)
FIDOnet nodes to join the Usenet and swap email with UNIX machines and
thus with the early Internet. Many years later, Ian Taylor wrote a
complete free replacement for uucp, called Taylor UUCP, which seems to
be what people were using last time I saw anyone using uucp. By
coincidence, Ian also later worked with me at Cygnus, the first company
doing commercial support for free software.
A modern web search turns up this about uuslave's BBS:
http://bbslist.textfiles.com/201/oldschool.html
"201-753-9758 ACGNJ BBS #2 (1983-1990) Kevin Tillbrook RBBS
"I used to run an RBBS system for the Amateur Computer Group of NJ
(ACGNJ) for a number of years. It was run on a Zenith PC w/20 meg HD
and later using DesqView for multi-tasking (which was way too slow on
that hardware). I had a BBS running before that, but this is what I am
noted for." - Kevin Tillbrook
Does anybody here (from New Jersey perhaps) know where uuslave came
from? It's always been a mystery to me.
9:40 p.m.
New subject: 2bsd tarball -> pdtar, with a side of uuslave
On Wed, Jul 29, 2020 at 9:42 AM John Gilmore <gnu(a)toad.com> wrote:
There was another chapter to the "tar wars"
after UNIX and after POSIX.
Ah .. indeed - I left out the Gnu Tar story as I was not 100% sure of how
it came about as I had not taken part in it. And pretty much for the
purposes of how we go to where we are today, other than it exists, works,
is a popular implementation and can read/write things when called upon ...
I did not think it would add to the (already) long story. The null vs space
filling is an interesting point which I had left out. Thank you for that
detail - I do remember it.
If I missspoke/was confusing (I hope not) about the UIDs I thought I had
said the way you did. The key was that USTAR added the names in ASCII
which was not there before in Ken's original version.
Again, thanks for the friendly addition/update.
After I left Sun in about 1985, I worked on a project with GNU and the
BSD folks, to find or write freely available
implementations of many
popular UNIX commands.
Yep, I do remember all that...
Since we didn't find a free "tar"
program, I wrote one from scratch,
based on the SunOS man page and on running the
tar binary from SunOS 3.3.
I always found that strange the folks that wrote that that tar
implementation (i.e. you and your mates) had not found the pax code, as the
USENIX version had been previously posted/was in the wild by then. Keith
certainly knew about it (he could have even been part of the finding a
student to write it, but I don't remember), but he also might have been off
at BSDi by that time. I think by then that the USENIX FOSS implementation
even knew how to behave like cpio, tar, or pax depending on its name.
I'm fairly sure, that Apple and HP had picked it up soon after it's
release. DEC had a different set of tar switches, so pax was put in the
Ultrix contributed library, and they left theirs alone. That said, the
USENIX version did have an MIT/UCB/CMU style license, not the gpl, which
our common 'friend' in Cambridge often (??always??) found suspicious. So,
I had always >>suspected<< the licensing style was driver for yet
another version, and
have always been a little curious.
But to me it was like C compilers, as long as they all worked, I didn't
care. As you know, I have never been super religious about the different
license flavors as long as I could use it. Probably a good beer
discussion/story behind it all when I see you next at a future conference
post CV-19.
Clem
30 Jul
30 Jul
1:34 a.m.
New subject: 2bsd tarball -> pdtar, with a side of uuslave
Wasn't the tour header Fields the reason for the strange strncpy semantics?
Which came first?
1:42 a.m.
New subject: 2bsd tarball -> pdtar, with a side of uuslave
On Wed, Jul 29, 2020 at 1:35 PM Richard Salz <rich.salz(a)gmail.com> wrote:
Wasn't the tour header Fields the reason for the
strange strncpy
semantics? Which came first?
strncpy appeared in V7 as far as I can tell. I can't find it in v6 or
earlier. I can't find any of the str functions in fact...
Also, there's a new libarchive-based tar as well that the BSDs are using
since it understands many other formats. It's largely replaced the gnutar
that had previously been in BSD.
Warneer
1376
days inactive
1390
days old
13 comments
8 participants
participants (8)
-
Clem Cole -
Dave Horsfall -
John Cowan -
John Gilmore -
Johnny Billquist -
Richard Salz -
Warner Losh -
Will Senn