10 Jul
2021
10 Jul
'21
9:18 p.m.
Once again, thanks to everybody who as contributed to making this a
better letter. Many of you have asked to be co-signers. Please
let me know if I've included your name by mistake or if you'd like
your name to be added. And, of course, let me know if any more
edits are required.
BTW, except where I'm quoting the paper I use UNIX instead of Unix
as that's what I believe is the correct form. Please let me know
if that's not correct.
Thanks,
Jon
I read the "Unix Shell Programming: The Next 50 Years" paper
expecting some well thought out wisdom. I was sorely disappointed.
The paper is lacking the generally accepted form of:
o What problem are you trying to solve?
o What have others done?
o What's our approach?
o How does it do?
Some particulars:
o The paper never defines what is meant by the term "Unix shell."
I think that you're using it to mean a command interpreter as
described in the POSIX 1003.2 documents.
o The paper makes liberal use of the term "Unix" such as "... in
every Unix distribution." While systems descended from UNIX
abound few actual instances of UNIX exist today.
o There is no 50-year-old UNIX shell. I started using UNIX in the
early 1970s, and the command interpreter at the time (Ken Thompson's
shell) was nothing like later shells such as the Bourne shell (sh
since research V7 UNIX), Korn shell (ksh), C shell (csh), and the
Bourne again shell (bash). UNIX mainstreamed the notion of a
command interpreter that was not baked into the system. The paper
lacks any discussion of prior art. In practice, shell implementations
either predate the POSIX standard or were built afterwards and
include non-standard extensions.
o The paper repeatedly claims that the shell has been largely ignored by
academia and industry. Yet, it does not include any references to
support that claim. In fact, the large body of published work on
shells and ongoing work on shells such as zsh shows that claim to be
incorrect.
o The large number of pejorative statements detract from the academic
value of the paper. And, in my opinion, these statements are provably
false. It reads as if the authors are projecting their personal
opinions onto the rest of the world.
o The paper applies universal complaints such as "unmaintainable" to the
shell; it doesn't call out any shell-specific problems. It doesn't
explain whether these complaints are against scripts, implementations,
or both. One of the reasons for the longevity of the family of shells
descended from Bourne's sh is that experienced practitioners have been
able to write easily maintainable code. Scripts written in the 1980s
are still around and working fine.
o The paper seems to complain about the fact that the shell is documented.
This is astonishing. Proper documentation has always been a key
component of being a professional, at least in my decades of experience.
As a matter of fact, my boss telling me that "nobody will give a crap
about your work unless you write a good paper" when I was a teenager
at Bell Labs is what led me to UNIX and roff.
o The paper includes non-sequiturs such as discussions about Docker
and systemd that have nothing to to with the shell.
o The paper has many "no-op" statements such as "arguably improved"
that
provide no useful information.
o The example on page 105 don't work as there is no input to "cut".
o The single result in Figure 1 is insufficient evidence that the
approach works on a wide variety of problems.
o The paper gives the appearance that the authors don't actually understand
the Bourne shell semantics. Not just my opinion; Steve Bourne expressed
that in an email to me after he read your paper, and I consider him to be
a subject matter expert.
o The paper confuses the performance of the shell with the performance of
external commands executed by the shell.
o Proofreading should have caught things like "improve performance
performance" on page 107 among others.
I think that the paper is really trying to say:
o Programmable command interpreters such as those found in UNIX based
systems have been around for a long time. For this paper, we're
focusing on the GNU bash implementation of the POSIX P1003.2 shell.
Other command interpreters predate UNIX.
o This implementation is used more often than many other scripting
languages because it is available and often installed as the default
command interpreter on most modern systems (UNIX-based and otherwise).
In particular, it is often the default for Linux systems.
o The shell as defined above is being used in more complex environments
than existed at the time of its creation. This exposes a new set of
performance issues.
o While much work has been done by the bash implementers, it's primarily
been in the area of expanding the functionality, usually in a
backward-compatible manner. Other shells such as the original ksh and
later ash and zsh were implemented with an eye towards the performance
of the internals and user perspectives.
o Performance optimization using modern techniques such as JIT compilation
have been applied to other languages but not to POSIX shell implementations.
This paper looks at doing that. It is unsurprising that techniques that
have worked elsewhere work here too.
It's hard to imagine that the application of this technique is all that's
required for a 50-year life extension. The title of this paper implies
that it's going to be comprehensive but instead concentrates on a couple
of projects. It ignores other active work on shells such as "fish". While
it wouldn't eliminate the issues with the paper, they would not have been
quite so glaring had it had a more modest title such as "Improving POSIX
Shell Performance with JIT Compilation".
Jon Steinhart plus John Cowan, Warner Losh,
John Dow, Steve Bourne, Larry McVoy, and Clem Cole