[TUHS] The Unix shell: a 50-year view - hopefully final review of letter

Once again, thanks to everybody who as contributed to making this a better letter. Many of you have asked to be co-signers. Please let me know if I've included your name by mistake or if you'd like your name to be added. And, of course, let me know if any more edits are required. BTW, except where I'm quoting the paper I use UNIX instead of Unix as that's what I believe is the correct form. Please let me know if that's not correct. Thanks, Jon I read the "Unix Shell Programming: The Next 50 Years" paper expecting some well thought out wisdom. I was sorely disappointed. The paper is lacking the generally accepted form of: o What problem are you trying to solve? o What have others done? o What's our approach? o How does it do? Some particulars: o The paper never defines what is meant by the term "Unix shell." I think that you're using it to mean a command interpreter as described in the POSIX 1003.2 documents. o The paper makes liberal use of the term "Unix" such as "... in every Unix distribution." While systems descended from UNIX abound few actual instances of UNIX exist today. o There is no 50-year-old UNIX shell. I started using UNIX in the early 1970s, and the command interpreter at the time (Ken Thompson's shell) was nothing like later shells such as the Bourne shell (sh since research V7 UNIX), Korn shell (ksh), C shell (csh), and the Bourne again shell (bash). UNIX mainstreamed the notion of a command interpreter that was not baked into the system. The paper lacks any discussion of prior art. In practice, shell implementations either predate the POSIX standard or were built afterwards and include non-standard extensions. o The paper repeatedly claims that the shell has been largely ignored by academia and industry. Yet, it does not include any references to support that claim. In fact, the large body of published work on shells and ongoing work on shells such as zsh shows that claim to be incorrect. o The large number of pejorative statements detract from the academic value of the paper. And, in my opinion, these statements are provably false. It reads as if the authors are projecting their personal opinions onto the rest of the world. o The paper applies universal complaints such as "unmaintainable" to the shell; it doesn't call out any shell-specific problems. It doesn't explain whether these complaints are against scripts, implementations, or both. One of the reasons for the longevity of the family of shells descended from Bourne's sh is that experienced practitioners have been able to write easily maintainable code. Scripts written in the 1980s are still around and working fine. o The paper seems to complain about the fact that the shell is documented. This is astonishing. Proper documentation has always been a key component of being a professional, at least in my decades of experience. As a matter of fact, my boss telling me that "nobody will give a crap about your work unless you write a good paper" when I was a teenager at Bell Labs is what led me to UNIX and roff. o The paper includes non-sequiturs such as discussions about Docker and systemd that have nothing to to with the shell. o The paper has many "no-op" statements such as "arguably improved" that provide no useful information. o The example on page 105 don't work as there is no input to "cut". o The single result in Figure 1 is insufficient evidence that the approach works on a wide variety of problems. o The paper gives the appearance that the authors don't actually understand the Bourne shell semantics. Not just my opinion; Steve Bourne expressed that in an email to me after he read your paper, and I consider him to be a subject matter expert. o The paper confuses the performance of the shell with the performance of external commands executed by the shell. o Proofreading should have caught things like "improve performance performance" on page 107 among others. I think that the paper is really trying to say: o Programmable command interpreters such as those found in UNIX based systems have been around for a long time. For this paper, we're focusing on the GNU bash implementation of the POSIX P1003.2 shell. Other command interpreters predate UNIX. o This implementation is used more often than many other scripting languages because it is available and often installed as the default command interpreter on most modern systems (UNIX-based and otherwise). In particular, it is often the default for Linux systems. o The shell as defined above is being used in more complex environments than existed at the time of its creation. This exposes a new set of performance issues. o While much work has been done by the bash implementers, it's primarily been in the area of expanding the functionality, usually in a backward-compatible manner. Other shells such as the original ksh and later ash and zsh were implemented with an eye towards the performance of the internals and user perspectives. o Performance optimization using modern techniques such as JIT compilation have been applied to other languages but not to POSIX shell implementations. This paper looks at doing that. It is unsurprising that techniques that have worked elsewhere work here too. It's hard to imagine that the application of this technique is all that's required for a 50-year life extension. The title of this paper implies that it's going to be comprehensive but instead concentrates on a couple of projects. It ignores other active work on shells such as "fish". While it wouldn't eliminate the issues with the paper, they would not have been quite so glaring had it had a more modest title such as "Improving POSIX Shell Performance with JIT Compilation". Jon Steinhart plus John Cowan, Warner Losh, John Dow, Steve Bourne, Larry McVoy, and Clem Cole