1 Mar
2023
1 Mar
'23
2:38 a.m.
I've been doing a lot of reading of systems admin books lately including:
Frisch, E. (1991). Essential System Administration (3rd edition is my
fattest book other than Unabridged Shakespeare)
Hunter, B. H., & Hunter, K. B. (1991). UNIX Systems Advanced
Administration and Management Handbook (Opinionated praxis)
Nemeth, E., Synder, G., & Seebass, S. (1989). UNIX System Administration
Handbook (5th edition is another fatty)
Tons of other more recent drivel.
I have been working on my ancient and not so ancient Unix library for a
while now, and it's kind of funny. It seems like once I read a book, be
it new or old, I hardly need it anymore - most of them wind up back at
half-price books. The exceptions are those that I find myself going back
to over and over and over again and wow are those few and far between.
An example of one of the gems is S. R. Bourne's The UNIX System, another
is Kernighan and Pike's The UNIX Programming Environment, and a couple
of newcomers for me are Volumes 3 and 8 of O'Reilly's The Definitive
Guides to the X Window System. I've written in the margins so many times
with these that there are sections where I can't fit any more notes.
That's the kind of sys admin guide I'd like to hear about. So, my
question for y'all is, what did y'all think about sys admin texts as
they were coming out? Were they well received, were they water to a
dying horse, were they paperweights, what? If you are of the camp, "we
don't need no stinking admin guide", or "we did it all by muscle memory
and didn't use books", don't reply. I'm curious about the experience
of
those of y'all who actually used them. Were there any early standouts
and why did they stand out?
Anything from 1970 on is fair game.
Later,
Will
P.S. Can you believe that 2000 is fast becoming 'history' worth
preserving? In 1997, we were rewriting our gas pump and credit card
transaction systems, which were written in C, to deal with upcoming Y2K
bugs. Oh, how the worm has turned :).
1 Mar
1 Mar
3:34 a.m.
On Tue, Feb 28, 2023 at 8:38 PM Will Senn <will.senn(a)gmail.com> wrote:
I'm curious about the experience of those of
y'all who actually used them. Were there any early standouts and why did they stand
out?
This is not going to be popular, but...
Nemeth, E., Synder, G., & Seebass, S. (1989). UNIX
System Administration Handbook (5th edition is another fatty)
This book gave me some terrible advice when I was very young and impressionable.
In there somewhere it says something about not doing something unless
you're prepared to do it right lest one spend more time working around
a work-around than one would have spent just doing it well in the
first place.
The conclusion is, of course, true, but the admonition ignores all
sorts of externalities, like waiting users. And in some cases it could
really lead to paralysis ("omg _everything_ is broken and I can't do X
until I do Y, but to do Y I have to do this other thing and if I
really want to do it right I need to start by traveling to Nepal and
shaving this Yak, but I need to get my passport renewed and damn I
really oughta lose 20 pounds before I get my passport photo taken for
the next ten years, so I guess I oughta join a gym...but I can't even
find one because the network is down and I can't get to Google, but
how can I fix the network if I have not shaved the Nepalese yak?!").
Talk about letting the perfect be the enemy of the good.
Hopefully nowadays we have a better appreciation of the power of
incrementalism; those grand plans for the perfect system rarely come
to fruition. It's better to be flexible and make small, impactful
changes along the way towards a better system, always being mindful of
and tamping down encroaching entropy.
- Dan C.
4:24 a.m.
On Tue, Feb 28, 2023 at 09:34:33PM -0500, Dan Cross wrote:
On Tue, Feb 28, 2023 at 8:38???PM Will Senn
<will.senn(a)gmail.com> wrote:
Yet another reason I'm not a Evi Nemeth fan. I could go on but I won't.
I'm curious about the experience of those of
y'all who actually used them. Were there any early standouts and why did they stand
out?
This is not going to be popular, but...
Nemeth, E., Synder, G., & Seebass, S. (1989).
UNIX System Administration Handbook (5th edition is another fatty)
This book gave me some terrible advice when I was very young and impressionable. 
10:03 a.m.
Dan,
I used the Nemeth book when I didn’t know how to do the odd thing or finding a tool,
but the book was predicated on a very specific Academic environment.
You can see why she’d suggest making tools reliable, robust and ‘complete’ (done
‘properly’)
in their environment with a large, constantly changing workforce, kids all too prone to
‘exploring’ things
or breaking something and then not knowing what to do…
From 1995, I spent around a decade doing contract SysAdmin.
The well run, well staffed and “no drama” sites never needed me :)
I got “parachuted behind enemy lines”, having to fight my way out,
so many times that I discovered I had a process for “Digging myself Out”
<https://stevej-on-it.blogspot.com/2007/06/digging-out-turning-around-challenged.html>
The key is to gain time by automating tasks, starting with what gains the most time for
you,
not business or managerial priorities.
There’s at least three levels of script / tool,
I thought were all covered in that Bill Plauger PDF recently but aren’t:
<https://indico.cern.ch/event/318305/attachments/612388/842557/PJPlauger-ITSeminar-Fifty_years.pdf>
1. Stuff for yourself.
2. A "Program Product" for a small, competent group.
3. A hardened product / Application that the unwashed masses will use and test to
destruction.
As a lone Sys Admin it’s incumbent on you to leverage the Automation under your control,
not drown in entropy.
If you do Admin in seriously large sites, eg AWS or GOOG, practices have to be adapted to
avoid outages - any outage will have massive impact.
At Internet Scale, the only methodology that can work is:
"systems are cattle not pets”- create tools that easily scale to the entire fleet
& are particularly robust & reliable.
Admins can’t be allowed to ever type at a production console - much more controlled than
the old Boulder Uni environment.
HTH
sj
On 1 Mar 2023, at 13:34, Dan Cross
<crossd(a)gmail.com> wrote:
--
Steve Jenkin, IT Systems and Design
0412 786 915 (+61 412 786 915)
PO Box 38, Kippax ACT 2615, AUSTRALIA
mailto:sjenkin@canb.auug.org.au http://members.tip.net.au/~sjenkin
Nemeth, E., Synder, G., & Seebass, S. (1989).
UNIX System Administration Handbook (5th edition is another fatty)
This book gave me some terrible advice when I was very young and impressionable.
In there somewhere it says something about not doing something unless
you're prepared to do it right lest one spend more time working around
a work-around than one would have spent just doing it well in the
first place.
The conclusion is, of course, true, but the admonition ignores all
sorts of externalities, like waiting users. And in some cases it could
really lead to paralysis ("omg _everything_ is broken and I can't do X
until I do Y, but to do Y I have to do this other thing and if I
really want to do it right I need to start by traveling to Nepal and
shaving this Yak, but I need to get my passport renewed and damn I
really oughta lose 20 pounds before I get my passport photo taken for
the next ten years, so I guess I oughta join a gym...but I can't even
find one because the network is down and I can't get to Google, but
how can I fix the network if I have not shaved the Nepalese yak?!").
Talk about letting the perfect be the enemy of the good.
Hopefully nowadays we have a better appreciation of the power of
incrementalism; those grand plans for the perfect system rarely come
to fruition. It's better to be flexible and make small, impactful
changes along the way towards a better system, always being mindful of
and tamping down encroaching entropy.
- Dan C. 
7:34 p.m.
On Tue, Feb 28, 2023 at 09:34:33PM -0500, Dan Cross wrote:
On Tue, Feb 28, 2023 at 8:38 PM Will Senn
<will.senn(a)gmail.com> wrote:
i'll agree here with you on that, but i will say that as a front line sysadmin
at the time it was one of the few resources i had that covered how to do basic
tasks correctly across many unix systems. it helped me alot as a sysadmin for
hire early in my career - esp when i had to fix something at a site with a poorly
maintained unix i may not have had much experience with at the time - "sure i can
fix your print spooler running hpux".
-pete
--
Pete Wright
pete(a)nomadlogic.org
I'm curious about the experience of those of
y'all who actually used them. Were there any early standouts and why did they stand
out?
This is not going to be popular, but...
Nemeth, E., Synder, G., & Seebass, S. (1989).
UNIX System Administration Handbook (5th edition is another fatty)
This book gave me some terrible advice when I was very young and impressionable.
In there somewhere it says something about not doing something unless
you're prepared to do it right lest one spend more time working around
a work-around than one would have spent just doing it well in the
first place.
11:57 p.m.
On Tue, Feb 28, 2023, at 21:34, Dan Cross wrote:
On Tue, Feb 28, 2023 at 8:38 PM Will Senn
<will.senn(a)gmail.com> wrote:
[...]
I'm curious about the experience of those of
y'all who actually used them. Were there any early standouts and why did they stand
out?
This is not going to be popular, but...
Nemeth, E., Synder, G., & Seebass, S. (1989).
UNIX System Administration Handbook (5th edition is another fatty)
This book gave me some terrible advice when I was very young and impressionable.
In there somewhere it says something about not doing something unless
you're prepared to do it right lest one spend more time working around
a work-around than one would have spent just doing it well in the
first place.
The conclusion is, of course, true, but the admonition ignores all
sorts of externalities, like waiting users. And in some cases it could
really lead to paralysis Hopefully nowadays we have a better appreciation of
the power of
incrementalism; those grand plans for the perfect system rarely come
to fruition. It's better to be flexible and make small, impactful
changes along the way towards a better system, always being mindful of
and tamping down encroaching entropy.
- Dan C.
Yeah, good or bad advice at just the right time can have quite an
impact.
In the under-celebrated "Minimal Perl"[0], Tim Maher notes in the
last paragraph of section 5.8:
<quote>
In your own career, I'd advise you to develop an appreciation an
appreciation and an aptitude for both the /quick-and-dirty/ and
/elegant-and-formal/ styles of programming, and to cultivate the
ability to produce either kind on demand, as circumstances
warrant.
</quote>
Seems obvious, in retrospect -- but of course many things do with
the benefit of hindsight. For me, that articulated something that I
sensed was the right way to approach things, but was contrary to
much of the one-dimensional advice I had received up to that
point. It pairs well with one of the "lesser tenets" noted by
Gancarz: "Look for the 90 percent solution"[1].
In my own career, I've found I can often use the quick-and-dirty
approach to buy myself time to afford the "detour to build the
tools"[2] that could not be justified (to others) up-front. And
nothing gets it done faster than a shell script. Five or ten scrappy
N-line shell scripts that get the job done sub-optimally, and
lacking any real thought toward usability or generality buy time to
build better tools (usually more, better-written shell scripts). And
sometimes a scrappy script is "good enough" (for years, even).
-Al
[0] Minimal Perl for Unix People and Linux People
by Tim Maher
Forward by Damian Conway
Manning 2007
p. 175
ISBN-10: 1-932394-50-8
[1] The Unix Philosophy
by Mike Gancarz
Digital Press 1995
p. 117
ISBN-10: 1-55558-123-4
[2] [McIlroy78] The Bell System Technical Journal. Bell Laboratories.
M. D. McIlroy, E. N. Pinson, and B. A. Tague.
"Unix Time-Sharing System Forward". 1978. 57 (6, part 2). p. 1902.
https://archive.org/details/bstj57-6-1899/page/n3/mode/2up
Also quoted in ESR's "The Art of Unix Programming"
Addison-Wesley 2004
p. 12
ISBN-13: 9-780131-429017
https://www.catb.org/~esr/writings/taoup/html/ch01s06.html
--
a l a n d. s a l e w s k i
ads(a)salewski.email
salewski(a)att.net
https://github.com/salewski
2 Mar
2 Mar
1:41 a.m.
I liked the Frisch and Limoncelli/Hogan books. Nemeth less so.
Adam
On Wed, Mar 1, 2023 at 3:59 PM Alan D. Salewski <ads(a)salewski.email> wrote:
On Tue, Feb 28, 2023, at 21:34, Dan Cross wrote:
On Tue, Feb 28, 2023 at 8:38 PM Will Senn
<will.senn(a)gmail.com> wrote:
> I'm curious about the experience of those of y'all who actually used
them. Were there any early standouts and why did they stand out?
This is not going to be popular, but...
> Nemeth, E., Synder, G., & Seebass, S. (1989). UNIX System
Administration
Handbook (5th edition is another fatty)
This book gave me some terrible advice when I was very young and
impressionable.
In there somewhere it says something about not doing something unless
you're prepared to do it right lest one spend more time working around
a work-around than one would have spent just doing it well in the
first place.
The conclusion is, of course, true, but the admonition ignores all
sorts of externalities, like waiting users. And in some cases it could
really lead to paralysis
[...]
Hopefully nowadays we have a better appreciation
of the power of
incrementalism; those grand plans for the perfect system rarely come
to fruition. It's better to be flexible and make small, impactful
changes along the way towards a better system, always being mindful of
and tamping down encroaching entropy.
- Dan C.
Yeah, good or bad advice at just the right time can have quite an
impact.
In the under-celebrated "Minimal Perl"[0], Tim Maher notes in the
last paragraph of section 5.8:
<quote>
In your own career, I'd advise you to develop an appreciation an
appreciation and an aptitude for both the /quick-and-dirty/ and
/elegant-and-formal/ styles of programming, and to cultivate the
ability to produce either kind on demand, as circumstances
warrant.
</quote>
Seems obvious, in retrospect -- but of course many things do with
the benefit of hindsight. For me, that articulated something that I
sensed was the right way to approach things, but was contrary to
much of the one-dimensional advice I had received up to that
point. It pairs well with one of the "lesser tenets" noted by
Gancarz: "Look for the 90 percent solution"[1].
In my own career, I've found I can often use the quick-and-dirty
approach to buy myself time to afford the "detour to build the
tools"[2] that could not be justified (to others) up-front. And
nothing gets it done faster than a shell script. Five or ten scrappy
N-line shell scripts that get the job done sub-optimally, and
lacking any real thought toward usability or generality buy time to
build better tools (usually more, better-written shell scripts). And
sometimes a scrappy script is "good enough" (for years, even).
-Al
[0] Minimal Perl for Unix People and Linux People
by Tim Maher
Forward by Damian Conway
Manning 2007
p. 175
ISBN-10: 1-932394-50-8
[1] The Unix Philosophy
by Mike Gancarz
Digital Press 1995
p. 117
ISBN-10: 1-55558-123-4
[2] [McIlroy78] The Bell System Technical Journal. Bell Laboratories.
M. D. McIlroy, E. N. Pinson, and B. A. Tague.
"Unix Time-Sharing System Forward". 1978. 57 (6, part 2). p. 1902.
https://archive.org/details/bstj57-6-1899/page/n3/mode/2up
Also quoted in ESR's "The Art of Unix Programming"
Addison-Wesley 2004
p. 12
ISBN-13: 9-780131-429017
https://www.catb.org/~esr/writings/taoup/html/ch01s06.html
--
a l a n d. s a l e w s k i
ads(a)salewski.email
salewski(a)att.net
https://github.com/salewski
3:02 a.m.
Adam Thornton <athornton(a)gmail.com> wrote:
I liked the Frisch and Limoncelli/Hogan books. Nemeth
less so.
Having taught a system administration grad level class
for many years now, I think there's an evolution of
the profession whereby books that were previously
practically useful are less so now.
Nemeth et al was useful in a very practical sense
(e.g., Pete's example of "fix printer spooler on
HP-UX"), but Limoncelli's books were always stronger
in principles.
Taking it further up the layer of abstractions, I then
found Mark Burgess's works more valuable, but that
very much leaves the realm of vocation (which I
maintain is hard to capture in books anyway) and
goes towards the more theoretical side.
The profession has evolved (or even splintered)
dramatically, with DevOps, SRE, and cloud computing
covering overlapping but different sets of work in
this area. All that is very different from the early
days of administering Unix systems.
I still struggle with the balance of teaching students
basic Unix skills, conveying (true) Unix multiuser
system concepts, translating administrative principles
into containerized, ephemeral environments, especially
at large scale...
-Jan
1 Mar
1 Mar
9:34 a.m.
I liked the book by Tom Limoncelli & Christine Hogan. Limoncelli went on to write more
books.
I’ve got a copy of Garfield & Spafford, "Practical UNIX and Internet
Security", but I don’t know what the Cool Kids use these days.
Security is something you need to do all the time and stay current with… It’s a constant
battle with no end, something I didn’t want to devote my life to.
Evi Nemeth taught a course in Sys Admin & maybe had a group/dept as well.
The book came out of the course - undergrads had to do prac work, put in ‘hours’.
The Book ’the standard’ for Admins I knew for many years.
Nemeth et al built “sudo” to solve the “give limited privileges to undergrads” problem. [
Implementing a "Capability-based” kernel & tools may have been a better solution
]
It was a good enough design and spread widely.
I never used it’s full feature set, never in an environment that needed it.
However, the University of Colorado Boulder environment was much larger in some ways than
any I worked in and met very different needs.
They had to cope with many students printing jobs, resetting passwords, filling up disk
quotas, email queues and more - with an admin group staffed mainly by undergrads, I
believe.
On 1 Mar 2023, at 12:38, Will Senn
<will.senn(a)gmail.com> wrote:
Frisch, E. (1991). Essential System Administration (3rd edition is my fattest book other
than Unabridged Shakespeare)
Hunter, B. H., & Hunter, K. B. (1991). UNIX Systems Advanced Administration and
Management Handbook (Opinionated praxis)
Nemeth, E., Synder, G., & Seebass, S. (1989). UNIX System Administration Handbook
(5th edition is another fatty)
--
Steve Jenkin, IT Systems and Design
0412 786 915 (+61 412 786 915)
PO Box 38, Kippax ACT 2615, AUSTRALIA
mailto:sjenkin@canb.auug.org.au http://members.tip.net.au/~sjenkin
7:41 p.m.
On Wed, Mar 1, 2023 at 1:34 AM steve jenkin <sjenkin(a)canb.auug.org.au>
wrote:
However, the University of Colorado Boulder
environment was much larger in
some ways than any I worked in and met very different needs.
They had to cope with many students printing jobs, resetting passwords,
filling up disk quotas, email queues and more - with an admin group staffed
mainly by undergrads, I believe.
csops at CU did all the sysadmin for a time of the Unix machines. It was
mostly random undergrads that could be trusted to do these tasks (many
would later go on to be well known in open source and other places like
Todd Miller), with the occasional more senior person overseeing them. This
was not a group that had an excess of resources. The user community was
also not very forgiving.
The preference for doing things right vs hacks came from many painful
instances where the hacks wound up costing extra time. It was also an
ideal, not an absolute. Evi and the csops folks were very pragmatic, and
often would do a hack to get things going or keep them going while doing
things right to keep what we'd call today 'technical debt' sane.
Evi was a hoot! I still miss seeing her from time to time... I'm still half
expecting her to show up after being lost at sea for a decade now :(.
Warner
622
days inactive
623
days old
9 comments
9 participants
participants (9)
-
Adam Thornton -
Alan D. Salewski -
Dan Cross -
Jan Schaumann -
Larry McVoy -
Pete Wright -
steve jenkin -
Warner Losh -
Will Senn