> Doug, if you insist on applying your superb editing skills on wiki material, we will never hear from you again!
Thanks, Bill, for the wise advice. If I'm putting out stuff like this
you shouldn't hear from me again.
Apologies for again(!) posting to the wrong mailing list.
Doug
Hello All,
I am attempting to restore 4.3BSD-Tahoe to a usable state on the VAX. It
appears, based on the work that I have done already, that this is
possible. Starting with stock 4.3BSD I got a source tree into /usr/tahoe
and using it I replaced all of /usr/include and /sys, recompiled and
replaced /bin, /lib, and /etc, recompiled a GENERIC kernel, and from there
I was able to successfully reboot using the new kernel. As far as I can
tell (fingers crossed!) the hardest part is over and I'm in the process of
working on /usr.
My question is: how was this sort of thing done in the real world? If I
was a site running stock 4.3BSD, would I have received (or been able to
request) updated tapes at regular intervals? The replacement process that
I have been using is fairly labor intensive and on a real VAX would have
been very time intensive too. Fortunately two to three years' worth of
changes were not so drastic that I ever found myself in a position where
the existing tools were not able to compile pieces of Tahoe that I needed
to proceed, but I could easily imagine finding myself in such a place.
(This was, by the way, what I ran into when attempting to upgrade from
2.9BSD to 2.10BSD, despite a fully documented contemporary upgrade
procedure).
-Henry
I can't speak to the evolution and use of specific
groups; I suspect it was all ad-hoc early on.
Groups appeared surprisingly late (given how familiar
they seem now): they don't show up in the manual
until the Sixth Edition. Before that, chown took
only two arguments (filename and owner), and
permission modes had three bits fewer.
I forget how it came up, but the late Lee McMahon
once told me an amusing story about their origin:
Ken announced that he was adding groups.
Lee asked what they were for.
Ken replied with a shrug and `I dunno.'
Norman Wilson
Toronto ON
Groups appeared surprisingly late (given how familiar
they seem now): they don't show up in the manual
until the Sixth Edition.
Mea culpa; read too hastily. The change actually
came with the Fourth Edition, at the same time as
several other landmark system changes:
-- Time changing from a 32-bit count of 60Hz clock
ticks (which ran out so quickly that its epoch kept
moving) to the modern 32 bits of whole seconds, based
at 1970-01-01T00:00:00 GMT (which takes much longer
to run out, though the horizon is now visible).
-- The modern super-block. In 4/e, the super-block
began at block 0, not 1 (so bootstrapping was rather
more complicated); the free list was a bitmap rather
than the later list of blocks containing lists of
free block numbers.
-- The super-block contained a bitmap of free
i-numbers too. All told, the free block and free
i-node map made up a 1024-byte super-block.
-- I-numbers 1-40 were device files; the root
directory was i-number 41. The only file-type
indication in the mode word was a single bit to
denote directory.
It was already clear that the lifetime of the
bitmaps was running out: the BUGS section says
two blocks isn't enough for the bitmaps for a
20-megabyte RP02.
Norman Wilson
Toronto ON
Hi all,
I was reading a recent thread, over on the FreeBSD forums about groups that quickly devolved into a discussion on the origin of the operator group:
https://forums.freebsd.org/threads/groups-overview.82303/
I thought y’all would be the best place to ask the questions that arose in me during my read of the thread.
Here they are in no special order:
1. Where did operator come from and what was it intended to solve?
2. How has it evolved.
3. What’s a good place to look/ref to read about groups, generally?
I liked one respondent’s answer about using find, heir, and the files themselves to learn about groups being used in a running system, paying attention to the owner, Audi, etc along the way and this is how I do it now, but this approach doesn’t account for the history and evolution.
Thanks!
Willu
Sent from my iPhone
Greg wrote:
> I guess pipe(2) kind of started this mess, [...] Maybe I'm
> going to far with thinking pipe() could/should have just been a library
> call that used open(2) internally, perhaps connecting the descriptors by
> opening some kind of "cloning" device in the filesystem.
At times I’ve been pondering this as well. All of creat/open/pipe could have been rolled into just open(). It is not clear to me why this synthesis did not happen around the time of 7th edition; although it seems the creat/open merger happened in BSD around that time.
As to pipe(), the very first implementation returned just a single fd where writes echoed to reads. It was backed by a single disk buffer, so could only hold ~500 bytes, which was probably not enough in practice. Then it was reimplemented using an anonymous file as backing store and got the modern two fd system call. The latter probably arose as a convenient hack to store the two file pointers needed.
It would have been possible to implement the anonymous file solution still using a single fd, and storing the second file pointer in the inode. Maybe this felt as a worse hack at the time (the conceptual split in vnode / inode was still a decade into the future.)
With a single fd, it would also have been possible to have a cloning device for pipe’s as you suggest (e.g. /dev/pipe, somewhat analogous to the implementation of /dev/stdin in 10th edition). Arguably, in total code/data size this would not have been much different from pipe().
My guess is that from a 1975 perspective, creat/open/pipe was not perceived as something that needed fixing.
Dan wrote:
> 3BSD and I think 4.1BSD had vread() and vwrite(), which looked like
> regular read() and write() but accessed pages only on demand. I was a
> grad student at Berkeley at the time and remember their genesis. Bill
> and I were eating lunch from Top Dog on the Etcheverry Hall plaza, and
> were talking about memory-mapped I/O. I remember suggesting the actual
> names, perhaps as a parallel to vfork(). I had used both TENEX and
> Multics, which both had page mapping. Multics' memory-mapped segments
> were quite fundamental, of course. I think we were looking for something
> vaguely upward compatible from the existing system calls. We did not
> leap to an mmap() right away just because it would have been a more
> radical shift than continuing the stream orientation of UNIX. I did not
> implement any of this: it was just a brainstorming session.
Thank you for reminding me of these.
On a substrate with a unified buffer cache and copy-on-write, vread/vwrite would have been very close to regular read/write and maybe could have been subsumed into them, using flags to open() as the differentiator. The user discernible effect would have been the alignment requirement on the buffer argument.
John Reiser wrote that he "fretted” over adding a 6 argument system call. Perhaps he was considering something like the above as the alternative, I never asked.
I looked at the archives and vread/vwrite were introduced with 3BSD, present in 4BSD but marked deprecated, and absent from 4.1BSD. This short lifetime suggests that using vread and vwrite wasn’t entirely satisfactory in 1980/81 practice. Maybe the issue was that there was no good way to deallocate the buffer after use.
Hello,
I've recently started to implement a set of helper functions and
procedures for parsing Unix-like command-line interfaces (i.e., POSIX +
GNU-style long options, in this case) in Ada.
While doing that, I learned that there is a better way to approach
this problem – beyond using getopt(s) (which never really made sense to
me) and having to write case statements in loops every time: Define a
grammar, let a pre-built parser do the work, and have the parser
provide the results to the program.
Now, defining such a grammar requires a thoroughly systematic approach
to the design of command-line interfaces. One problem with that is
whether that grammar should allow for sub-commands. And that leads to
the question of how task-specific tool sets should be designed. These
seem to be a relatively new phenomenon in Unix-like systems that POSIX
doesn't say anything about, as far as I can see.
So, I've prepared a bit of a write-up, pondering on the pros and cons
of two different ways of having task-specific tool sets
(non-hierarchical command sets vs. sub-commands) that is available at
https://www.msiism.org/files/doc/unix-like_command-line_interfaces.html
I tend to think the sub-command approach is better. But I'm neither a UI
nor a Unix expert and have no formal training in computer things. So, I
thought this would be a good place to ask for comment (and get some
historical perspective).
This is all just my pro-hobbyist attempt to make some people's lives
easier, especially mine. I mean, currently, the "Unix" command line is
quite a zoo, and not in a positive sense. Also, the number of
well-thought-out command-line interfaces doesn't seem to be a growing
one. But I guess that could be changed by providing truly easy ways to
make good interfaces.
--
Michael
> one other thing that SLS breaks, for data files, is the whole Unix 'pipe'
> abstraction, which is at the heart of the whole Unix tools paradigm.
Multics had an IO system with an inherent notion of redirectable data
streams. Pipes could have--and eventually did (circa 1987)--fit into
that framework. I presume a pipe DIM (device interface manager)
was not hard to build once it was proposed and accepted.
Doug
> From: Larry McVoy
> If you read(2) a page and mmap()ed it and then did a write(2) to the
> page, the mapped page is the same physical memory as the write()ed
> page. Zero coherency issues.
Now I'm confused; read() and write() semantically include a copy operation
(so there are then two copies of that data chunk, and possible consistency
issues between them), and the copied item is not necessarily page-sized (so
you can't ensure consistency between the original+copy by mapping it in). So
when one does a read(file, &buffer, 1), one gets a _copy of just that byte_
in the process' address space (and similar for write()).
Yes, there's no coherency issue between the contents of an mmap()'d page, and
the system's idea of what's in that page of the file, but that's a
_different_ coherency issue.
Or am I confused?
PS:
> From: "Greg A. Woods"
> I now struggle with liking the the Unix concept of "everything is a
> file" -- especially with respect to actual data files. Multics also got
> it right to use single-level storage -- that's the right abstraction
Oh, one other thing that SLS breaks, for data files, is the whole Unix 'pipe'
abstraction, which is at the heart of the whole Unix tools paradigm. So no
more 'cmd | wc' et al. And since SLS doesn't have the 'make a copy'
semantics of pipe output, it would be hard to trivially work around it.
Yes, one could build up a similar framework, but each command would have to
specify an input file and an output file (no more 'standard in' and 'out'),
and then the command interpreter would have to i) take command A's output file
and feed it to command B, and ii) delete A's output file when the whole works
was done. Yes, the user could do it manually, but compare:
cmd aaa | wc
and
cmd aaa bbb
wc bbb
rm bbb
If bbb is huge, one might run out of room, but with today's 'light my cigar
with disk blocks' life, not a problem - but it would involve more disk
traffic, as bbb would have to be written out in its entirety, not just have a
mall piece kept in the disk cache as with a pipe.
Noel
> From: "Greg A. Woods"
> the elegance of fork() is incredible!
That's because in PDP-11 Unix, they didn't have the _room_ to create a huge
mess. Try reading the exec() code in V6 or so.
(I'm in a bit of a foul mood today; my laptop sorta locked up when a _single_
Edge browser window process grew to almost _2GB_ in size. Are you effing
kidding me? If I had any idea what today would look like, back when I was 20 -
especially the massive excrement pile that the Internet has turned into - I
never would have gone into computers - cabinetwork, or something, would have
been an infinitely superior career choice.)
> I now struggle with liking the the Unix concept of "everything is a
> file" -- especially with respect to actual data files. Multics also got
> it right to use single-level storage -- that's the right abstraction
Well, files a la Unix, instead of the SLS, are OK for a _lot_ of data storage
- pretty much everything except less-common cases like concurrent access to a
shared database, etc.
Where the SLS really shines is _code_ - being able to just do a subroutine
call to interact with something else has incredible bang/buck ratio - although
I concede doing it all securely is hard (although they did make a lot of
progress there).
Noel
>> > It's part of my academic project to work on provable compiler security.
>> > I tried to do it according to the "Reflections on Trusting Trust" by Ken
>> > Thompson, not only to show a compiler Trojan horse but also to prove that
>> > we can discover it.
>>
>> Of course it can be discovered if you look for it. What was impressive about
>> the folks who got Thompson's compiler at PWB is that they found the horse
>> even though they weren't looking for it.
> I had not heard this story. Can you elaborate, please? My impression from having
> read the paper (a long time ago now) is that Ken did the experiment locally only.
Ken did it locally, but a vigilant person at PWB noticed there was an
experimental
compiler on the research machine and grabbed it. While they weren't looking for
hidden stuff, they probably were trying to find what was new in the
compiler. Ken
may know details about what they had in the way of source and binary.
Doug
> It's part of my academic project to work on provable compiler security.
> I tried to do it according to the "Reflections on Trusting Trust" by Ken
> Thompson, not only to show a compiler Trojan horse but also to prove that
> we can discover it.
Of course it can be discovered if you look for it. What was impressive about
the folks who got Thompson's compiler at PWB is that they found the horse
even though they weren't looking for it.
Then there was the first time Jim Reeds and I turned on integrity control in
IX, our multilevel-security version of Research Unix. When it reported
a security
violation during startup we were sure it was a bug. But no, it had snagged Tom
Duff's virus in the act of replication. It surprised Tom as much as it did us,
because he thought he'd eradicated it.
Doug
This is FYI. No comment on whether it was a good idea or not. :-)
Arnold
> From: Niklas Rosencrantz <niklasro(a)gmail.com>
> Date: Sun, 19 Sep 2021 17:10:24 +0200
> To: tinycc-devel(a)nongnu.org
> Subject: Re: [Tinycc-devel] Can tcc compile itself with Apple M1?
>
>
> Hello!
>
> For demonstration purpose I put my experiment with a compiler backdoor in a
> public repository
> https://github.com/montao/ddc-tinyc/blob/857d927363e9c9aaa713bb20adbe99ded7…
>
> It's part of my academic project to work on provable compiler security.
> I tried to do it according to the "Reflections on Trusting Trust" by Ken
> Thompson, not only to show a compiler Trojan horse but also to prove that
> we can discover it.
> What it does is inject arbitrary code to the next version of the compiler
> and so on.
>
> Regards \n
One of the things I really appreciate about participating in this community
and studying Unix history (and the history of other systems) is that it
gives one firm intellectual ground from which to evaluate where one is
going: without understanding where one is and where one has been, it's
difficult to assert that one isn't going sideways or completely backwards.
Maybe either of those outcomes is appropriate at times (paradigms shift; we
make mistakes; etc) but generally we want to be moving mostly forward.
The danger when immersing ourselves in history, where we must consider and
appreciate the set of problems that created the evolutionary paths leading
to the systems we are studying, is that our thinking can become calcified
in assuming that those systems continue to meet the needs of the problems
of today. It is therefore always important to reevaluate our base
assumptions in light of either disconfirming evidence or (in our specific
case) changing environments.
To that end, I found Timothy Roscoe's (ETH) joint keynote address at
ATC/OSDI'21 particularly compelling. He argues that what we consider the
"operating system" is only controlling a fraction of a modern computer
these days, and that in many ways our models for what we consider "the
computer" are outdated and incomplete, resulting in systems that are
artificially constrained, insecure, and with separate components that do
not consider each other and therefore frequently conflict. Further,
hardware is ossifying around the need to present a system interface that
can be controlled by something like Linux (used as a proxy more generally
for a Unix-like operating system), simultaneously broadening the divide and
making it ever more entrenched.
Another theme in the presentation is that, to the limited extent
the broader systems research community is actually approaching OS topics at
all, it is focusing almost exclusively on Linux in lieu of new, novel
systems; where non-Linux systems are featured (something like 3 accepted
papers between SOSP and OSDI in the last two years out of $n$), the
described systems are largely Linux-like. Here the presentation reminded me
of Rob Pike's "Systems Software Research is Irrelevant" talk (slides of
which are available in various places, though I know of no recording of
that talk).
Roscoe's challenge is that all of this should be seen as both a challenge
and an opportunity for new research into operating systems specifically:
what would it look like to take a holistic approach towards the hardware
when architecting a new system to drive all this hardware? We have new
tools that can make this tractable, so why don't we do it? Part of it is
bias, but part of it is that we've lost sight of the larger picture. My own
question is, have we become entrenched in the world of systems that are
"good enough"?
Things he does NOT mention are system interfaces to userspace software; he
doesn't seem to have any quibbles with, say, the Linux system call
interface, the process model, etc. He's mostly talking about taking into
account the hardware. Also, in fairness, his highlighting a "small" portion
of the system and saying, "that's what the OS drives!" sort of reminds me
of the US voter maps that show vast tracts of largely unpopulated land
colored a certain shade as having voted for a particular candidate, without
normalizing for population (land doesn't vote, people do, though in the US
there is a relationship between how these things impact the overall
election for, say, the presidency).
I'm curious about other peoples' thoughts on the talk and the overall topic?
https://www.youtube.com/watch?v=36myc8wQhLo
- Dan C.
> Maybe there existed RE notations that were simply copied ...
Ed was derived from Ken's earlier qed. Qed's descendant in Multics was
described in a 1969 GE document:
http://www.bitsavers.org/pdf/honeywell/multics/swenson/6906.multics-condens….
Unfortunately it describes regular expressions only sketchily by
example. However, alternation, symbolized by | with grouping by
parentheses, was supported in qed, whereas alternation was omitted
from ed. The GE document does not mention character classes; an
example shows how to use alternation for the same purpose.
Beginning-of-line is specified by a logical-negation symbol. In
apparent contradiction, the v1 manual says the meanings of [ and ^ are
the same in ed and (an unspecified version of) qed. My guess about the
discrepancies is no better than yours.
(I am amused by the title "condensed guide" for a manual in which each
qed request gets a full page of explanation. It exemplifies how Unix
split from Multics in matters of taste.)
Doug
> From: Roland Huisman
> I have a PDP11/20 and I would love to run an early Unix version on
> it. ... But it seems that the earliest versions of Unix do not need the
> extra memory. Does anyone have RK05 disk images for these early Unix
> versions?
Although the _kernel_ source for V1 is available:
https://minnie.tuhs.org//cgi-bin/utree.pl?file=V1
most of the rest is missing; only 'init' and 'sh' are available. So one would
have to write almost _everything_ else. Some commands are available in PDP-11
assembler in later versions, and might be movable without _too_ much work -
but one would have to start with the assembler itself, which is luckily in
assembler.
If I were trying to run 'UNIX' on an -11/20, I think the only reasonable
choice would be MINI-UNIX:
https://gunkies.org/wiki/MINI-UNIX
It's basically V6 UNIX with all use of the PDP-11 memory management
removed. The advantage of going MINI-UNIX is that almost all V6 source
(applications, drivers, etc) will run on it 'as is'.
It does need ~56KB of main memory. If you don't have that much on the -11/20,
LSX (links in the above) would be an option; it's very similar to MINI-UNIX,
but is trimmed down some, to allow its use on systems with less main memory.
I'm not sure if MINI-UNIX has been run on the -11/20, but it _should_ run
there; it runs on the -11/05, and the only differences between the /20 and the
/05 are that the /20 does not have the RTT instruction (and I just checked,
and MINI-UNIX doesn't use RTT), and SWAB doesn't clear the V condition code
bit. (There are other minor differences, such as OP Rn, (Rn)+ are different on
the -11/20, but that shouldn't be an issue.)
Step 1 would be to get MINI-UNIX running on an -11/20 under a simulator; links
in the above to get you there.
Noel
> From: Clem Cole
> The KS11 MMU for the 11/20 was built by CSS ... I think Noel has done
> more investigation than I have.
I got a very rough description of how it worked, but that was about it.
> I'm not sure if the KS11 code is still there. I did not think so.
No, the KS11 was long gone by later Vn. Also,I think not all of the -11/20
UNIX machines had it, just some.
> The V1 work was for a PDP-7
Actually, there is a PDP-11 version prior to V2, canonically called V1.
The PDP-7 version seems to be called 'PDP-7 UNIX' or 'V0'.
> I'm fairly sure that the RK05, used the RK11-D controller.
Normally, yes. I have the impression that one could finagle RK03's to work on
the RK11-D, and vice versa for RK05's on the RK11-C, but I don't recall the
details. The main difference between the RK11-C and -D (other then the
implementation) was that i) the RK11-C used one line per drive for drive
selection (the -D used binary encoding on 3 lines), and ii) it had the
'maintenance' capability and register (al omitted from the -D).
> The difference seems to have been in drive performance.
Yes, but it wasn't major. They both did 1500RPM, so since they used
the same pack format, the rotational delay, transfer rate, etc were
identical. The one peformance difference was in seeks; the
average on the RK01 was quoted as 70 msec, and 50 msec on the
RK05.
> Love to see the {KT11-B prints] and if you know where you found them.
They were sold on eBait along with an -11/20 that allegedly had a KT11-B. (It
didn't; it was an RK11-C.) I managed to get them scanned, and they and the
minimal manual are now in Bitsavers. I started working on a Tech Manual for
it, but gave up with it about half-way done.
> I wonder if [our V1 source] had the KS-11 stuff in it.
No; I had that idea a while back, looked carefully, our V1 listings
pre-date the KS11.
> From: Roland Huisman
> There is a KT11B paging option that makes the PDP11/20 a 18 bit
> machine.
Well, it allows 2^18 bytes of main memory, but the address space of the
CPU is still2^16 bytes.
> It looks a bit like the TC11 DECtape controller.
IITC, it's two backplanes high, the TC11 is one. So more like the RK11-C...
:-)
> I have no idea how it compares to the later MMU units from the
> software perspective.
Totally different; it's real paging (with page tables stored in masin
memory). The KT11-B provides up to 128 pages of 512 bytes each, in both Exec
and User mode. The KT11-C, -D etc are all segmentation, with all the info
stored in registers in the unit.
> I wonder is there is some compatibility with the KT11-B [from the KS11]
I got the impression that the KS11 was more a 'base and bounds' kind
of thing.
Noel
Hello Unix fanatics,
I have a PDP11/20 and I would love to run an early Unix version on it. I've been working on the hardware for a while and I'm getting more and more of the pieces back online again. The configuration will be two RK05 hard disks, TU56H tape, PC11 paper tape reader/puncher and a RX01 floppy drive. Unfortunately I don't have a MMU or paging option. But it seems that the earliest versions of Unix do not need the extra memory.
Does anyone have RK05 disk images for these early Unix versions? That would be a great help. Otherwise it would be great to have some input about how to create a bootable Unix pack for this machine.
A bit about the hardware restoring is on the vcfed forum:https://www.vcfed.org/forum/forum/genres/dec/78961-rk05-disk-drive-ve…
Booting RT11 from RK05https://youtu.be/k0tiUcRBPQATU56H tape drive back onlinehttps://youtu.be/_ZJK3QP9gRA
Thanks in advance!Roland Huisman
Hoi,
I'm interested in the early design decisions for meta characters
in REs, mainly regarding Ken's RE implementation in ed.
Two questions:
1) Circumflex
As far as I see, the circumflex (^) is the only meta character that
has two different special meanings in REs: First being the
beginning of line anchor and second inverting a character class.
Why was it chosen for the second one? Why not the exclamation mark
in that case? (Sure, C didn't exist by then, but the bang probably
was used to negate in other languages of the time, I think.)
2) Symbol for the end of line anchor
What is the reason that the beginning of line and end of line
anchors are different symbols? Is there a reason why not only one
symbol, say the circumflex, was chosen to represent both? I
currently see no disadvantages of such a design. (Circumflexes
aren't likely to end lines of text, neither.)
I would appreciate if you could help me understand these design
decisions better. Maybe there existed RE notations that were simply
copied ...
meillo
You can check the Computer History Museum's holdings on line. If they don't
have the documents already, they would probably like them.
The Living Computer Museum in Seattle had a working blit on display. If
they don't already have the manual, I'm sure they would love to have one.
Alas, their website says they've "suspended all operations for now", a
result of the double whammy of Covid and the death of their principal
angel, Paul Allen.
more garage cleaning this last weekend. i came across some memorabilia
from my time at Bell Labs, including a lovely article titled
The Electrical Properties of Infants
Infants have long been known to grow into adults. Recent experiments
show they are useful in applications such as high power circuit breakers.
Not to mention a lovely article from the “Medical Aspects of Human Sexuality”
(July 1991) titled “Scrotum Self-Repair”.
the two items are
1) “Documents for UNIX Volume 1” by Dolotta, Olson and Petrucelli (jan 1981)”
2) The complete manual for the Blit. this comes in a blue Teletype binder and includes
the full manual (including man pages) and circuit diagrams.
i’d prefer to have them go to some archival place, but send me a private email
if you interested and we’ll see what we can do.
andrew
I’d be interested in a scan of the Blit schematics, and it seems that a few others might be as well:
https://minnie.tuhs.org/pipermail/tuhs/2019-December/thread.html#19652https://github.com/aiju/fpga-blit
(for clarity: I’m not ‘aiju')
Paul
> Message: 1
> Date: Wed, 8 Sep 2021 01:29:13 -0700
> From: Andrew Hume <andrew(a)humeweb.com>
> To: The Eunuchs Hysterical Society <tuhs(a)tuhs.org>
> Subject: [TUHS] desiderata
> Message-ID: <34E984D3-AD92-402D-9A9C-E84B6362DF77(a)humeweb.com>
> Content-Type: text/plain; charset=utf-8
>
> more garage cleaning this last weekend.
[...]
> 2) The complete manual for the Blit. this comes in a blue Teletype binder and includes
> the full manual (including man pages) and circuit diagrams.
>
> i’d prefer to have them go to some archival place, but send me a private email
> if you interested and we’ll see what we can do.
>
> andrew
I recently upgraded my machines to fc34. I just did a stock
uncomplicated installation using the defaults and it failed miserably.
Fc34 uses btrfs as the default filesystem so I thought that I'd give it
a try. I was especially interested in the automatic checksumming because
the majority of my storage is large media files and I worry about bit
rot in seldom used files. I have been keeping a separate database of
file hashes and in theory btrfs would make that automatic and transparent.
I have 32T of disk on my system, so it took a long time to convert
everything over. A few weeks after I did this I went to unload my
camera and couldn't because the filesystem that holds my photos was
mounted read-only. WTF? I didn't do that.
After a bit of poking around I discovered that btrfs SILENTLY remounted the
filesystem because it had errors. Sure, it put something in a log file,
but I don't spend all day surfing logs for things that shouldn't be going
wrong. Maybe my expectation that filesystems just work is antiquated.
This was on a brand new 16T drive, so I didn't think that it was worth
the month that it would take to run the badblocks program which doesn't
really scale to modern disk sizes. Besides, SMART said that it was fine.
Although it's been discredited by some, I'm still a believer in "stop and
fsck" policing of disk drives. Unmounted the filesystem and ran fsck to
discover that btrfs had to do its own thing. No idea why; I guess some
think that incompatibility is a good thing.
Ran "btrfs check" which reported errors in the filesystem but was otherwise
useless BECAUSE IT DIDN'T FIX ANYTHING. What good is knowing that the
filesystem has errors if you can't fix them?
Near the top of the manual page it says:
Warning
Do not use --repair unless you are advised to do so by a developer
or an experienced user, and then only after having accepted that
no fsck successfully repair all types of filesystem corruption. Eg.
some other software or hardware bugs can fatally damage a volume.
Whoa! I'm sure that operators are standing by, call 1-800-FIX-BTRFS.
Really? Is a ploy by the developers to form a support business?
Later on, the manual page says:
DANGEROUS OPTIONS
--repair
enable the repair mode and attempt to fix problems where possible
Note there’s a warning and 10 second delay when this option
is run without --force to give users a chance to think twice
before running repair, the warnings in documentation have
shown to be insufficient
Since when is it dangerous to repair a filesystem? That's a new one to me.
Having no option other than not being able to use the disk, I ran btrfs
check with the --repair option. It crashed. Lesson so far is that
trusting my data to an unreliable unrepairable filesystem is not a good
idea. Since this was one of my media disks I just rebuilt it using ext4.
Last week I was working away and tried to write out a file to discover
that /home and /root had become read-only. Charming. Tried rebooting,
but couldn't since btrfs filesystems aren't checked and repaired. Plugged
in a flash drive with a live version, managed to successfully run --repair,
and rebooted. Lasted about 15 minutes before flipping back to read only
with the same error.
Time to suck it up and revert. Started a clean reinstall. Got stuck
because it crashed during disk setup with anaconda giving me a completely
useless big python stack trace. Eventually figured out that it was
unable to delete the btrfs filesystem that had errors so it just crashed
instead. Wiped it using dd; nice that some reliable tools still survive.
Finished the installation and am back up and running.
Any of the rest of you have any experiences with btrfs? I'm sure that it
works fine at large companies that can afford a team of disk babysitters.
What benefits does btrfs provide that other filesystem formats such as
ext4 and ZFS don't? Is it just a continuation of the "we have to do
everything ourselves and under no circumstances use anything that came
from the BSD world" mentality?
So what's the future for filesystem repair? Does it look like the past?
Is Ken's original need for dsw going to rise from the dead?
In my limited experience btrfs is a BiTteR FileSystem to swallow.
Or, as Saturday Night Live might put it: And now, linux, starring the
not ready for prime time filesystem. Seems like something that's been
under development for around 15 years should be in better shape.
Jon