TUHS

tuhs@tuhs.org

47 participants
6502 discussions

Re: [TUHS] head/sed/tail (was The Unix shell: a 50-year view)

by Paul Ruizendaal

> Message: 7 > Date: Thu, 15 Jul 2021 10:28:04 -0400 > From: "Theodore Y. Ts'o" > Subject: Re: [TUHS] head/sed/tail (was The Unix shell: a 50-year view) > > On Wed, Jul 14, 2021 at 10:38:06PM -0400, Douglas McIlroy wrote: >> Head might not have been written if tail didn't exist. But, unlike head, >> tail strayed from the tao of "do one thing well". Tail -r and tail -f are >> as cringeworthy as cat -v. >> >> -f is a strange feature that effectively turns a regular file into a pipe >> with memory by polling for new data, A clean general alternative >> might be to provide an open(2) mode that makes reads at the current >> file end block if some process has the file open for writing. > > OTOH, this would mean adding more functionality (read: complexity) > into the kernel, and there has always been a general desire to avoid > pushing <stuff> into the kernel when it can be done in userspace. Do > you really think using a blocking read(2) is somehow more superior > than using select(2) to wait for new data to be appended to the file? > > And even if we did this using a new open(2) mode, are you saying we > should have a separate executable in /bin which would then be > identical to cat, except that it uses a different open(2) mode? Yes, it would put more complexity into the kernel, but maybe it is conceptually elegant. Consider a classic pipe or a socket and the behaviour of read(2) for those objects. The behaviour of read(2) that Doug proposes for a file would make it in line with that for a classic pipe or a socket. Hence, maybe it should not be a mode, but the standard behaviour. I often think that around 1981 the Unix community missed an opportunity to really think through how networking should integrate with the foundations of Unix. It seems to me that at that time there was an opportunity to merge files, pipes and sockets into a coherent, simple framework. If the 8th edition file-system-switch had been introduced already in V6 or V7, maybe this would have happened. On the other hand, the installed base was probably already too large in 1981 to still make breaking changes to core concepts. V7 may have been the last chance saloon for that. Paul

3 years, 10 months

Re: [TUHS] Unix Shell Programming: The Next 50 Years - author's response and my response

by Jon Steinhart

Below is a response from two of the authors with my response to it inline. Not very impressed. Hopefully they'll get a clue and up their game. In any case, enough time spent on it. Jon Michael Greenberg writes: > > HotOS isn't exactly a conventional venue---you might notice that many > if not most HotOS papers don't fit the outline you've given. I'm aware of that, and have participated in many unconventional venues myself. I wasn't saying that papers must fit that outline, but I do believe that they should contain that information. There's a big difference between a discussion transcript and a paper; I believe that papers, especially those published under the auspices of a prestigious organization such as the ACM, should adhere to a higher standard. > I'd definitely appreciate detailed feedback on any semantic errors > we've made! Unfortunately I can't help you here; that was feedback from a reader who doesn't want to spend any more time on this. > Your summary covers much of what we imagined! > > As I understand it, the primary goals of the paper were to (a) help > other academics think about the shell as a viable area for research, (b) > highlight some work we're doing on JIT compilation, (c) make the case > for JIT approaches to the shell in general (as its well adapted to its > dynamism), and (d) explore various axes on which the shell could be > improved. It seems like we've done a good job communicating (b) to you, > but perhaps not the rest. Sorry again to disappoint. I certainly hope that you understand the primary goals of your own paper. Point-by-point: (a) While this is a valid point I don't understand why the paper didn't just state it in a straightforward manner. There are several common forms. One is to list issues in the introduction while explaining which one(s) will be addressed in the paper. Another is in the conclusion where authors list work still to be done. (b) At least for me this goal was not accomplished because there were no examples. Figure 1 by itself is insufficient given that the code used to generate the "result" is not shown. It would have been much more illuminating had the paper not only shown that code but also the optimized result. Professionals don't blithely accept magic data. (c) The paper failed to make this case to me for several reasons. As I understand it, the paper is somewhat about applying JIT compilation techniques to interconnected processes. While most shells include syntax to support the construction of such, it's really independent of the shell. For completeness, I have a vague memory of shell implementations for non-multitasking systems that sequentially ran pipelined programs passing intermediate results via temporary files. The single "result" reminds me of something that I saw at a science fair when my child was in middle school; I looked a one team's results and asked "What makes you think that a sample size of one is significant?" The lack of any benchmarks or other study results that justified the work also undermined the case. It reads more like the authors had something that they wanted to play with rather than doing serious research. The paper does not examine the percentage of shell scripts that actually benefit from JIT compilation; for all the reader may know it's such a small number that hand-optimizing just those scripts might be a better solution. I suppose that the paper fits into the apparently modern philosophy of expecting tools to fix up poorly written code so that programmers don't have to understand what they're doing. (d) In my opinion the paper didn't do this at all. There was no analysis of "the shell" showing weaknesses and an explanation of why one particular path was taken. And there was no discussion of what was being done with shells to cause whatever problems you were addressing and possibly ameliorating problems with some up-front sanity. Again, being a geezer I'm reminded of past events that repeat themselves. I joined a start-up company decades ago that was going to speed up circuit simulation 100x by plugging custom-designed floating-point processing boards into a multiprocessor machine. I managed to beat that 100x just by cleverly crafting the database and memory management code. The fact that the company founder never verified his idea led to a big waste of resources. But, he did manage to raise venture capital which is akin to getting DARPA funds. Nikos Vasilakis writes: > > To add to Michael's points, HotOS' "position" papers are often > intended as provocative, call-to-arms statements targeting primarily > the research community (academic and industrial research). Our key > position, which we possibly failed to communicate, is "Hey researchers > everywhere, let's do more research on the shell! (and here's why)". While provocative name-calling and false statements seem to have become the political norm in America I don't think that they're appropriate in a professional context. In my experience a call-to-arms isn't productive unless those called understand the nature of the call. I'm reminded of something that happened many decades ago; my partner asked me to proof a paper that she was writing for her masters degree. I read it over with a confused look and asked her what she was trying to say. She responded, and I told her to write that down and stop trying to sound like someone else. Turned her into a much better writer. So if the paper wanted to say "Hey researchers ..." it should have done so instead of being obtuse. To continue on this point and Michael's (a) above, I don't see a lot of value in proclaiming that research can be done. I think that a more fruitful approach is to cover what has been done, what you're doing, and what you see but aren't doing. > For our more conventional research papers related to the shell, which > might cover your concerns about semantics, correctness, and > performance please see next. These three papers also provide important > context around the HotOS paper you read: > ... Tracking down your other work was key to understanding this paper. It's my opinion that my having to do is illustrative of the problems with the paper. > Thank you for taking the time to read our paper and comment on it. > Could you please share the emails of everyone mentioned at the end of > your email? We are preparing a longer report on a recent shell > roundtable, and would love to get everyone's feedback! While I appreciate the offer to send the longer report, it would only be of interest if it was substantially more professional. There is no interest in reviewing work that is not clearly presented, has not been proofread and edited, includes unsubstantiated, pejorative, or meaningless statements, includes incorrect examples, or statistically irrelevant results. Likewise, there is also no interest if the homework hasn't been done to put the report in context with regards to prior art and other current work. Jon Steinghart <jsacm(a)fourwinds.com> Warner Losh <imp(a)bsdimp.com> John Cowan <cowan(a)ccil.org> Larry McVoy <lm(a)mcvoy.com> John Dow <jmd(a)nelefa.org> Andy Kosela <akosela(a)andykosela.com> Clem Cole <clemc(a)ccc.com> Steve Bourne does not want to give out his address

3 years, 10 months

Re: [TUHS] head/sed/tail (was The Unix shell: a 50-year view)

by Nelson H. F. Beebe

On the subject of tac (concatenate and print files in reverse), I can report that the tool was written by my late friend Jay Lepreau in the Department of Computer Science (now, School of Computing) at the University of Utah. The GNU coreutils distribution for src/tac.c contains a copyright for 1988-2020. I searched my TOPS-20 PDP-10 archives, and found no source code for tac, but I did find an older TOPS-20 executable in Jay's personal directory with a file date of 17-Mar-1987. There isn't much else in that directory, so I suspect that he just copied over a needed tool from his Department of Computer Science TOPS-20 system to ours in the College of Science. ---------------------------------------- P.S. Jay was the first to get Steve Johnson's Portable C Compiler, pcc, to run on the 36-bit PDP-10, and once we had pcc, we began the move from writing utilities in Pascal and PDP-10 assembly language to doing them in C. The oldest C file for pcc in our PDP-10 archives is dated 17-Mar-1981, with other pcc files dated to mid-1983, and final compiler executables dated 12-May-1986. Four system header files are dated as late as 4-Oct-1986, presumably patched after the compiler was built. Later, Kok Chen and Ken Harrenstien's kcc provided another C compiler that added support for byte datatypes, where a byte could be anything from 1 to 36 bits. The oldest distribution of kcc in our archives is labeled "Fifth formal distribution snapshot" and dated 20-Apr-1988. My info-kcc mailing list archives date from the list beginning, with an initial post from Ken dated 27-Jul-1986 announcing the availability of kcc at sri-nic.arpa. By mid-1987, we had a dozen Sun workstations and NFS fileserver; they marked the beginning of our move to a Unix workstation environment, away from large, expensive, and electricity-gulping PDP-10 and VAX mainframes. By the summer of 1991, those mainframes were retired. I recall speaking to a used-equipment vendor about our VAX 8600, which cost about US$450K (discounted academic pricing) in 1986, and was told that its value was depreciating about 20% per month. Although many of us missed TOPS-20 features, I don't think anyone was sad to say goodbye to VMS. We always felt that the VMS developers worked in isolation from the PDP-10 folks, and thus learned nothing from them. ------------------------------------------------------------------------------- - Nelson H. F. Beebe Tel: +1 801 581 5254 - - University of Utah FAX: +1 801 581 4148 - - Department of Mathematics, 110 LCB Internet e-mail: beebe(a)math.utah.edu - - 155 S 1400 E RM 233 beebe(a)acm.org beebe(a)computer.org - - Salt Lake City, UT 84112-0090, USA URL: http://www.math.utah.edu/~beebe/ - -------------------------------------------------------------------------------

3 years, 10 months

Re: [TUHS] head/sed/tail (was The Unix shell: a 50-year view)

by norman＠oclsc.org

Some comments from someone (me) who tends to be pickier than most about cramming programs together and endless sets of options: I, too, had always thought sed was older than head. I stand corrected. I have a long-standing habit of typing sed 10q but don't spend much time fussing about head. When I arrived at Bell Labs in late summer 1984, tail -f was in /usr/bin and in the manual, readslow was only in /usr/bin. readslow was like tail -f, except it either printed the entire file first or (option -e) started at the end of the file. I was told readslow had come first, and had been invented in a hurry because people wanted to watch in real time the moves logged by one of Belle's chess matches. Vague memory says it was written by pjw; the name and the code style seem consistent with that. Personally I feel like tail -r and tail -f both fit reasonably well within what tail does, since both have to do with the bottom of the file, though -r's implementation does make for a special extra code path in tail so maybe a separate program is better. What I think is a bigger deal is that I have frequently missed tail -r on Linux systems, and somehow hadn't spotted tac; thanks to whoever here (was it Ted?) pointed it out first! On the other hand, adding data-processing functions to cat has never made sense to me. It seems to originate from a mistaken notion that cat's focus is printing data on terminals, rather than concatenating data from different places. Here is a test: if cat -v and cat -n and all that make sense, why shouldn't cat also subsume tr and pr and even grep? What makes converting control characters and numbering lines so different from swapping case and adding page headers? I don't see the distinction, and so I think vis(1) (in later Research) makes more sense than cat -v and nl(1) (in Linux for a long time) more sense than cat -n. (I'd also happily argue that given nl, pr shouldn't number lines. That a program was in V6 or V7 doesn't make it perfect.) And all those special options to wc that amounted to doing arithmetic on the output were always just silly. I'm glad they were retracted. On the other other hand, why didn't I know about tac? Because there are so damn many programs in /usr/bin these days. When I started with UNIX ca. 1980, the manual (even the BSD version) was still short enough that one could sit down and read it through, section by section, and keep track of what one had read, and remember what all the different tools did. That hasn't been true for decades. This could be an argument for adding to existing programs (which many people already know about) rather than adding new programs (which many people will never notice). The real problem is that the system is just too damn big. On an Ubuntu 18.04 system I run, ls /usr/bin | wc -l shows 3242 entries. How much of that is redundant? How much is rarely or never used? Nobody knows, and I suspect few even try to find out. And because nobody knows, few are brave enough to throw things away, or even trim out bits of existing things. One day in the late 1980s, I helped out with an Introduction to UNIX talk at a DECUS symposium. One of the attendees noticed the `total' line in the output of ls, and asked why is that there? doesn't that contradict the principles of tools' output you've just been talking about? I thought about it, and said yes, you're right, that's a bit of old history and shouldn't be there any more. When I got home to New Jersey, I took the `total' line out of Research ls. Good luck doing anything like that today. Norman Wilson Toronto ON

3 years, 10 months

First machine to run rogue?

by Dan Cross

What was the first machine to run rogue? I understand that it was written by Glenn Wichman and Michael Toy at UC Santa Cruz ca. 1980, using the `curses` library (Ken Arnold's original, not Mary Ann's rewrite). I've seen at least one place that indicates it first ran on 6th Edition, but that doesn't sound right to me. The first reference I can find in BSD is in 2.79 ("rogue.doc"), which also appears to be the first release to ship curses. Anyone have any info? Thanks! - Dan C.

3 years, 10 months

[tuhs] The Unix shell: a 50-year view

by Nelson H. F. Beebe

In this week's BSDNow.tv podcast, available at https://www.bsdnow.tv/409 there is a story about a new conference paper on the Unix shell. The paper is available at Unix shell programming: the next 50 years HotOS '21: Workshop on Hot Topics in Operating Systems, Ann Arbor, Michigan, 1 June, 2021--3 June, 2021 https://doi.org/10.1145/3458336.3465294 The tone is overall negative, though they do say nice things about Doug McIlroy and Steve Johnson, and they offer ideas about improvements. List readers will have their own views of the paper. My own is that, despite its dark corners, the Bourne shell has served us extraordinarily well, and I have been writing in it daily for decades without being particularly bothered by the many issues raised by the paper's authors. Having dealt with so-called command shells on numerous other operating systems, at least the Unix shells rarely get in my way. ------------------------------------------------------------------------------- - Nelson H. F. Beebe Tel: +1 801 581 5254 - - University of Utah FAX: +1 801 581 4148 - - Department of Mathematics, 110 LCB Internet e-mail: beebe(a)math.utah.edu - - 155 S 1400 E RM 233 beebe(a)acm.org beebe(a)computer.org - - Salt Lake City, UT 84112-0090, USA URL: http://www.math.utah.edu/~beebe/ - -------------------------------------------------------------------------------

3 years, 10 months

Re: [TUHS] The Unix shell: a 50-year view - hopefully final review of letter

by jnc＠mercury.lcs.mit.edu

> From: Jon Steinhart > I use UNIX instead of Unix as that's what I believe is the correct form. Well, Bell documentation uses "UNIX" through V6: https://minnie.tuhs.org//cgi-bin/utree.pl?file=V6/usr/doc/start/start "Unix" starts to appear with V7: https://minnie.tuhs.org//cgi-bin/utree.pl?file=V7/usr/doc/setup As mentioned, the trademark is "UNIX". I don't really have a fixed position on _the_ way to spell it: when I'm wiriting about a specific version (e.g. V6) I use the capitalization as of that version; for general text I'd probably use 'Unix', as that seems to be general now. But I could easily be convinced otherwise. Noel

3 years, 10 months

The Unix shell: a 50-year view - hopefully final review of letter

by Jon Steinhart

Once again, thanks to everybody who as contributed to making this a better letter. Many of you have asked to be co-signers. Please let me know if I've included your name by mistake or if you'd like your name to be added. And, of course, let me know if any more edits are required. BTW, except where I'm quoting the paper I use UNIX instead of Unix as that's what I believe is the correct form. Please let me know if that's not correct. Thanks, Jon I read the "Unix Shell Programming: The Next 50 Years" paper expecting some well thought out wisdom. I was sorely disappointed. The paper is lacking the generally accepted form of: o What problem are you trying to solve? o What have others done? o What's our approach? o How does it do? Some particulars: o The paper never defines what is meant by the term "Unix shell." I think that you're using it to mean a command interpreter as described in the POSIX 1003.2 documents. o The paper makes liberal use of the term "Unix" such as "... in every Unix distribution." While systems descended from UNIX abound few actual instances of UNIX exist today. o There is no 50-year-old UNIX shell. I started using UNIX in the early 1970s, and the command interpreter at the time (Ken Thompson's shell) was nothing like later shells such as the Bourne shell (sh since research V7 UNIX), Korn shell (ksh), C shell (csh), and the Bourne again shell (bash). UNIX mainstreamed the notion of a command interpreter that was not baked into the system. The paper lacks any discussion of prior art. In practice, shell implementations either predate the POSIX standard or were built afterwards and include non-standard extensions. o The paper repeatedly claims that the shell has been largely ignored by academia and industry. Yet, it does not include any references to support that claim. In fact, the large body of published work on shells and ongoing work on shells such as zsh shows that claim to be incorrect. o The large number of pejorative statements detract from the academic value of the paper. And, in my opinion, these statements are provably false. It reads as if the authors are projecting their personal opinions onto the rest of the world. o The paper applies universal complaints such as "unmaintainable" to the shell; it doesn't call out any shell-specific problems. It doesn't explain whether these complaints are against scripts, implementations, or both. One of the reasons for the longevity of the family of shells descended from Bourne's sh is that experienced practitioners have been able to write easily maintainable code. Scripts written in the 1980s are still around and working fine. o The paper seems to complain about the fact that the shell is documented. This is astonishing. Proper documentation has always been a key component of being a professional, at least in my decades of experience. As a matter of fact, my boss telling me that "nobody will give a crap about your work unless you write a good paper" when I was a teenager at Bell Labs is what led me to UNIX and roff. o The paper includes non-sequiturs such as discussions about Docker and systemd that have nothing to to with the shell. o The paper has many "no-op" statements such as "arguably improved" that provide no useful information. o The example on page 105 don't work as there is no input to "cut". o The single result in Figure 1 is insufficient evidence that the approach works on a wide variety of problems. o The paper gives the appearance that the authors don't actually understand the Bourne shell semantics. Not just my opinion; Steve Bourne expressed that in an email to me after he read your paper, and I consider him to be a subject matter expert. o The paper confuses the performance of the shell with the performance of external commands executed by the shell. o Proofreading should have caught things like "improve performance performance" on page 107 among others. I think that the paper is really trying to say: o Programmable command interpreters such as those found in UNIX based systems have been around for a long time. For this paper, we're focusing on the GNU bash implementation of the POSIX P1003.2 shell. Other command interpreters predate UNIX. o This implementation is used more often than many other scripting languages because it is available and often installed as the default command interpreter on most modern systems (UNIX-based and otherwise). In particular, it is often the default for Linux systems. o The shell as defined above is being used in more complex environments than existed at the time of its creation. This exposes a new set of performance issues. o While much work has been done by the bash implementers, it's primarily been in the area of expanding the functionality, usually in a backward-compatible manner. Other shells such as the original ksh and later ash and zsh were implemented with an eye towards the performance of the internals and user perspectives. o Performance optimization using modern techniques such as JIT compilation have been applied to other languages but not to POSIX shell implementations. This paper looks at doing that. It is unsurprising that techniques that have worked elsewhere work here too. It's hard to imagine that the application of this technique is all that's required for a 50-year life extension. The title of this paper implies that it's going to be comprehensive but instead concentrates on a couple of projects. It ignores other active work on shells such as "fish". While it wouldn't eliminate the issues with the paper, they would not have been quite so glaring had it had a more modest title such as "Improving POSIX Shell Performance with JIT Compilation". Jon Steinhart plus John Cowan, Warner Losh, John Dow, Steve Bourne, Larry McVoy, and Clem Cole

3 years, 10 months

The Unix shell: a 50-year view -- feedback wanted

by Jon Steinhart

I not only found this paper offensive, but was more offended that ACM would publish something like this and give it an award to boot. I'm planning to send the authors and ACM what's below. Would appreciate any feedback that you could provide to make it better. Thanks, Jon I read your "Unix Shell Programming: The Next 50 Years" expecting some well thought out wisdom from learned experiences. I was sorely disappointed. o The paper never defines what is meant by the term "Unix shell." I think that you're using to mean a command interpreter as described in the POSIX 1003.2 documents. o There is no 50 year old Unix shell. I started using Unix in the early 1970s, and the command interpreter at the time (Ken Thompson's shell) was nothing like later shells such as the Bourne shell (sh since research V7 Unix), Korn shell (ksh), C shell (csh), and the Bourne again shell (bash). The paper is missing any discussion of prior art. In practice, shell implementations either predate the POSIX standard or were built afterwards and include non-standard extensions. o The paper repeats a claim that the shell has been largely ignored by academia and industry. Yet, it does not include any references that support that claim. My own experience and thus opinion is the opposite making the veracity of your claim questionable. As a reader, such unsubstantiated claims make me treat the entire content as suspect. o The paper applies universal complaints such as "unmaintainable" to the shell; it doesn't call out any shell-specific problems. It doesn't explain whether these complaints are against the scripts, the implementation, or both. One of the reasons for the longevity of the sh/bash shells is that experienced practicioners have been able to write easily maintainable code. Scripts written in the 1980s are still around and working fine. o The paper seems to complain that the fact that the shell is documented is a problem. This is an astonishing statement. In my decades as an acting professional, teacher, and author, proper documentation is a key component of being a professional. o The paper is full of non-sequiturs such as discussions about Docker and systemd that have nothing to to with the shell. o The paper has many "nop" statements such as "arguably improved" that don't actually say anything. o Examples, such as the one on page 105 don't work. o Proofreading should have caught things like "improve performance performance" on page 107 among others. o The references contain many more items than the paper actually references. Did you plagerize the bibliography and forget to edit it? o The single result in Figure 1 is insufficient evidence that the approach works on a wide variety of problems. o The paper makes it appear that the authors don't actually understand the semantics of the original Bourne shell. Not just my opinion; I received an email from Steve Bourne after he read your paper, and I consider him to be a subject matter expert. The paper is lacking the generally accepted form of: o What problem are you trying to solve? o What have others done? o What's our approach? o How does it do? Filtering out all of the jargon added for buzzword compliance, I think that the paper is really trying to say: o Programmable command interpreters such as those found in Unix-based systems have been around for a long time. For this paper, we're focusing on the GNU bash implementation of the POSIX P1003.2 shell. Other command interpreters predate Unix. o This implementation is used more often than many other scripting languages because it is available and often installed as the default command interpreter on most modern systems (Unix-based or otherwise). In particular, it is often the default for Linux systems. o The shell as defined above is being used in ways that are far more complex than originally contemplated when Bourne created the original syntax and semantics, much less the new features added by the POSIX standards committee. The combination of both the POSIX and bash extensions to the Bourne shell exposes a new set of limitations and issues such as performance. o While much work has been done by the bash implementors, it's primarily been in the area of expanding the functionality, usually in a backward-compatible manner. Other shells such as the original ksh and later ash and zsh were implemented with an eye towards the performance of the internals and user perspectives. o Performance optimization using modern techniques such as JIT have been applied to other languages but not to POSIX shell implementations. This paper looks at doing that. It is unsurprising that techniques that have worked elsewhere work here too. o It's hard to imagine that the application of this technique is all that's required for a 50-year life extension. The title of this paper implies that it's going to be comprehensive rather than just being a shameless plus for an author's project. Of course, this doesn't make much of a paper. Guessing that that's why it was so "bulked up" with irrelevancies. It appears that all of you are in academia. I can't imagine that a paper like this would pass muster in front of any thesis committee, much less get that far. Not only for content, but for lack of proofreading and editing. The fact that the ACM would publish such a paper eliminates any regret that I may have had in dropping my ACM membership.

3 years, 10 months

The Unix shell: a 50-year view -- second pass feedback wanted

by Jon Steinhart

Thanks to everyone who provided me feedback on the first pass, especially those who suggested "shopt -u flame-off". Here's the second version. Once again, would appreciate feedback. Thanks, Jon I read your "Unix Shell Programming: The Next 50 Years" expecting some well thought out wisdom. I was sorely disappointed. The paper is lacking the generally accepted form of: o What problem are you trying to solve? o What have others done? o What's our approach? o How does it do? Some particulars: o The paper never defines what is meant by the term "Unix shell." I think that you're using it to mean a command interpreter as described in the POSIX 1003.2 documents. o There is no 50 year old Unix shell. I started using Unix in the early 1970s, and the command interpreter at the time (Ken Thompson's shell) was nothing like later shells such as the Bourne shell (sh since research V7 Unix), Korn shell (ksh), C shell (csh), and the Bourne again shell (bash). Unix mainstreamed the notion of a command interpreter that was not baked into the system. The paper lacks any discussion of prior art. In practice, shell implementations either predate the POSIX standard or were built afterwards and include non-standard extensions. o The paper repeats a claim that the shell has been largely ignored by academia and industry. Yet, it does not include any references that support that claim. In fact, the large body of published work on shells and ongoing work on shells such as zsh shows that claim to be incorrect. o The paper applies universal complaints such as "unmaintainable" to the shell; it doesn't call out any shell-specific problems. It doesn't explain whether these complaints are against the scripts, the implementation, or both. One of the reasons for the longevity of the family of shells descended from Bourne's sh is that experienced practicioners have been able to write easily maintainable code. Scripts written in the 1980s are still around and working fine. o The paper seems to complain that the fact that the shell is documented is a problem. This is astonishing. Proper documentation has always been a key component of being a professional in my decades of experience. As a matter of fact, my boss telling me that "nobody will give a crap about your work unless you write a good paper" when I was a teenager at Bell Labs is what led me to UNIX and nroff. o The paper includes non-sequiturs such as discussions about Docker and systemd that have nothing to to with the shell. o The paper has many "nop" statements such as "arguably improved" that don't actually say anything. o Examples, such as the one on page 105 don't work as there is no input to "cut". o The single result in Figure 1 is insufficient evidence that the approach works on a wide variety of problems. o The paper gives the appearance that the authors don't actually understand the semantics of the original Bourne shell. Not just my opinion; I received an email from Steve Bourne after he read your paper, and I consider him to be a subject matter expert. o Proofreading should have caught things like "improve performance performance" on page 107 among others. I think that the paper is really trying to say: o Programmable command interpreters such as those found in Unix-based systems have been around for a long time. For this paper, we're focusing on the GNU bash implementation of the POSIX P1003.2 shell. Other command interpreters predate Unix. o This implementation is used more often than many other scripting languages because it is available and often installed as the default command interpreter on most modern systems (Unix-based or otherwise). In particular, it is often the default for Linux systems. o The shell as defined above is being used in ways that are far more complex than originally contemplated when Bourne created the original syntax and semantics, much less the new features from kash adopted by the POSIX standards committee. The combination of both the POSIX and bash extensions to the Bourne shell exposes a new set of limitations and issues such as performance. o While much work has been done by the bash implementors, it's primarily been in the area of expanding the functionality, usually in a backward-compatible manner. Other shells such as the original ksh and later ash and zsh were implemented with an eye towards the performance of the internals and user perspectives. o Performance optimization using modern techniques such as JIT compilation have been applied to other languages but not to POSIX shell implementations. This paper looks at doing that. It is unsurprising that techniques that have worked elsewhere work here too. It's hard to imagine that the application of this technique is all that's required for a 50-year life extension. The title of this paper implies that it's going to be comprehensive but instead concentrates on a couple of projects. It ignores other active work on shells such as "fish". While the issues with the paper remain, they would not have been quite so glaring had it had a more modest title such as "Applying JIT Compilation to the POSIX Shell".

3 years, 10 months

Good News:

by Caipenghui

UNIX history memoir has been published in China (publish by posts and Telecommunications Press in China) and translated into Chinese. It is also called name "UNIX legend" in China. I'm going to buy now. If you're in the computer industry, it's exciting just to understand how these buzzwords came into being. Even if you don't have a deep technical background, You can also benefit a lot from these ides that shine with genius. I have been looking forward to a Chinese book about the history of Unix development for so many years, and now I can finally see it. Thanks bwk and other guys.

3 years, 10 months

Re: [TUHS] Binary log files

by norman＠oclsc.org

Some of us have, literally for decades, been dealing with wtmp by rolling it weekly or monthly or quarterly or whatever, letting cron run something like cd /usr/adm # that's how long I've been doing this! umask 022 >wtmp.new ln wtmp wtmp.prev mv wtmp.new wtmp # also so long ago there was no seq(1) nums=`awk 'BEGIN {for (i = 12; i >= 0; i--) print i; exit}'` for i in $nums; do inext=`expr $i + 1` if [ -f wtmp.$i ]; then mv wtmp.$i wtmp.$inext fi done mv wtmp.prev wtmp.0 This really isn't rocket science. It isn't even particularly interesting UNIX history. Can we move on to something that IS interesting? Here are some things I find more interesting: 1. utmp came before wtmp: utmp(V) appears in the First Edition manual, wtmp(V) only in the Second. Apparently interest in who else is logged in right now predated interest in who has logged in recently. 2. Both files started out in /tmp. wtmp is first said to be in /usr/adm instead in the Fifth Edition manual, utmp in /etc in the Sixth. 3. The names /tmp/utmp and /tmp/wtmp appear to have been issued by the Department of Redundancy Department. I think it quite likely that Ken and Dennis would have been familiar with that joke once the recording containing it was issued in mid-1970, but I don't know whether utmp existed in some form before that. I see no sign of it in the fragments of PDP-7 source code we have (in particular init doesn't seem to use it), but what about later PDP-7 or very early PDP-11 code predating the late-1971 First Edition manual? Norman Wilson Toronto ON Not Insane

3 years, 10 months

Binary log files

by John Cowan

As long ago as the 7th Edition, several binary log files were maintained: the file generated by acct(2) (one record per process) and the utmp and wtmp files (one record per login). Both of these are defined by structs in .h files, so they are definitely not portable (int sizes, endianism, etc.)

3 years, 10 months

[tuhs] Dennis Ritchie's couch

by Nelson H. F. Beebe

The last article of the latest issue of the Communications of the ACM that appeared electronically earlier today is a brief interview with this year's ACM Turing Award winners, Al Aho and Jeff Ullman. The article is Last byte: Shaping the foundations of programming languages https://doi.org/10.1145/3460442 Comm. ACM 64(6), 120, 119, June 2021. and it includes a picture of the two winners sitting on Dennis Ritchie's couch. I liked this snippet from Jeff Ullman, praising fellow list member Steve Johnson's landmark program, yacc: >> ... >> At the time of the first Fortran compiler, it took several >> person-years to write a parser. By the time yacc came around, >> you could do it in an afternoon. >> ... ------------------------------------------------------------------------------- - Nelson H. F. Beebe Tel: +1 801 581 5254 - - University of Utah FAX: +1 801 581 4148 - - Department of Mathematics, 110 LCB Internet e-mail: beebe(a)math.utah.edu - - 155 S 1400 E RM 233 beebe(a)acm.org beebe(a)computer.org - - Salt Lake City, UT 84112-0090, USA URL: http://www.math.utah.edu/~beebe/ - -------------------------------------------------------------------------------

3 years, 10 months

Re: [TUHS] The Unix shell: a 50-year view

by Jon Steinhart

Wow. This is a terrible paper. It's full of of incorrect, unsubstiantiated, and meaningless statements. It's so bad in my opinion that I'm not sorry that I dropped my ACM membership a while ago. These folks really needed an editor. The paper annoys me so much that I've decided to write a lengthy note to the authors which I'll post here once I'm done. Jon

3 years, 10 months

Disassemblers

by Henry Bent

>From what I can gather the only way to reasonably examine the disassembly of a program in the early days of Unix was adb. Is this true? Was there a way to easily produce a full disassembly? I'll confess to being fairly ignorant of adb use since I always had dbx or the equivalent available. The first tool I'm aware of to purposefully/primarily produce a full listing is MIPS dis (ca. 1986?) but there must have been something before that for other systems, no? -Henry

3 years, 10 months

pointer disambiguation (was Re: Disassemers)

by Paul Winalski

On 7/1/21, scj(a)yaccman.com <scj(a)yaccman.com> wrote: > When PCC came along and started running on 32-bit machines, I started > thinking about algorithms for optimization. A problem that I had no > good solution for could be illustrated by a simple piece of code: > > x = *p; > > y = *q; > > q gets changed > > *q = z; > > The question is, do I need to reload x now because q might have been > changed to point to the same place as p? Yes, this is a very well-known problem in scalar optimization in compiler engineering. It's called pointer disambiguation and is part of the more general problem of data flow analysis. As you observed, getting it wrong can lead to very subtle and hard-to-debug correctness problems. In the worst case, one has to throw out all current data flow analysis of global and currently active local variables and start over. In your example, the statement "*q = z" may end up forcing the compiler to toss out all data flow information on x and z (and maybe p and q as well). If q could possibly point to x and x is in a register, the assignment forces x to be reloaded before its next use. Ambiguous pointers prohibit a lot of important optimizations. This problem is the source of a lot of bugs in compilers that do aggressive optimizations. Fortunately a bit of knowledge of just how "q gets changed" can rescue the situation. In strongly-typed languages, for example, if x and z are different data types, we know the assignment of z through q can't affect x. We also know that the assignment can't affect x if x and z have disjoint scopes. The 'restrict' keyword in C pointer declarations was added to help mitigate this problem. Some compilers also have a command line option that allows the user to say, "I solemnly swear that I won't do this sort of thing". -Paul W.

3 years, 10 months

Re: [TUHS] Using printf from Assembly Language in V6, and db.

by jnc＠mercury.lcs.mit.edu

> From: Paul Riley <paul(a)rileyriot.com> >> (I wrote a note, BITD, explaining how all this worked; I'll upload it >> to the CHWiki when I get a chance.) Now here: https://gunkies.org/wiki/PDP-11_C_stack_operation along with simple examples of args and auto variables, which are both referenced via the FP. > As a non-C consumer of printf, should I point R5 at some space for a > stack and call printf in the same manner as the C example I cited? Not necessary to do anything with R5 (you can leave it blank); the only things a PDP-11 C routine needs are: - a good stack - the arguments, and return point, on the top of the stack csv will set up the frame pointer, making no assumptions about the old contents of R5 - see the source: http://ana-3.lcs.mit.edu/~jnc/tech/unix/lib/csv.s although it does save the old R5 contents, and restore them on exit. Noel

3 years, 10 months

Re: [TUHS] Using printf from Assembly Language in V6, and db.

by jnc＠mercury.lcs.mit.edu

> From: Paul Riley > I want to use printf from an assembly language program, in V6. ... the > substitutional arguments for printf are pushed onto the stack in reverse > order, then the address of the string, and then printf is called. After > this, 6 is added to the stack pointer. This is all down to the standard C environment / calling sequence on the PDP-11 (at least, in V6 C; other compilers may do it differently). Calls to printf() are in no way special. Very briefly, there's a 'frame pointer' (R5) which points to the current stack frame; all arguments and automatics are relative to that. A pair of special routines, csv and cret (I couldn't find the source on TUHS, but it happens to be here: http://ana-3.lcs.mit.edu/~jnc/tech/unix/lib/csv.s if you want to see it), set up and tear down the frame on entry/exit to that routine. The SP (R6) points to a blank location on the top (i.e. lower address; PDP-11 stacks grow down) of the stack. To call a subroutine, the arguments are pushed, the routine is called (which pushes the return PC), and on return (which pops the return PC), the arguments are discarded by the caller. (I wrote a note, BITD, explaining how all this worked; I'll upload it to the CHWiki when I get a chance.) > I assume that the printf routine pops the address of the string off the > stack, but leaves the other values on the stack No, all C routines (including printf()) leave the stack more or less alone, except for CSV/CRET hackery, allocating space for automatic variables on routine entry (that would be at L1; try looking at the .s for a routine with automatic variables), and popping the return PC on exit. The exception to this is the stuff around calling _enother_ routine (sketched above). Another exception is alloca() (source here: http://ana-3.lcs.mit.edu/~jnc/tech/unix/lib/alloca.s again, couldn't find it in TUHS), which allocated a block of memory on the stack (automatically discarded when the routine which called alloca() returns). Note that since all automatic variables and incoming arguments are relative to the FP, alloca is _really_ simple; justs adjusts the SP, and it's done. > What troubles me is that the stack pointer is not decremented before the > first mov, in the example below. Is this some C convention? I would > assume that the first push in my example would overwrite the top of the > stack. That's right; that's because in the C runtime environment, the top location on the stack is a trash word (set up by csv). > I understand db only works on files like a.out or core dumps. If I > wanted to break the assembly language program to examine values, how can > I force a termination and core dump elegantly, so I can examine some > register values? Use 'cdb': https://minnie.tuhs.org//cgi-bin/utree.pl?file=V6/usr/man/man1/cdb.1 which can do interactive debugging. Noel

3 years, 10 months

Using printf from Assembly Language in V6, and db.

by Paul Riley

Hi, I want to use printf from an assembly language program, in V6. It seems that the Unix Programmer's Manual doesn't show how to use it from assembly, so I wrote a short C program and captured the assembler output, for some clues. Listings below. In my example, the substitutional arguments for printf are pushed onto the stack in reverse order, then the address of the string, and then printf is called. After this, 6 is added to the stack pointer. I assume that the printf routine pops the address of the string off the stack, but leaves the other values on the stack, hence the need to add 2x3=6 to the stack after calling printf in my example. What troubles me is that the stack pointer is not decremented before the first mov, in the example below. Is this some C convention? I would assume that the first push in my example would overwrite the top of the stack. Perhaps I'm not used to PDP-11 stack conventions. I understand db only works on files like a.out or core dumps. If I wanted to break the assembly language program to examine values, how can I force a termination and core dump elegantly, so I can examine some register values? Paul *Paul Riley* Email: paul(a)rileyriot.com int a, b, c; int main(){ printf("printf: start\n"); a = 1; b = 2; c = 3; printf("A = %d, B = %d, C = %d", a, b, c); printf("printf: end\n"); } .comm _a,2 .comm _b,2 .comm _c,2 .globl _main .text _main: ~~main: jsr r5,csv jbr L1 L2:mov $L4,(sp) jsr pc,*$_printf mov $1,_a mov $2,_b mov $3,_c mov _c,(sp) mov _b,-(sp) mov _a,-(sp) mov $L5,-(sp) jsr pc,*$_printf add $6,sp mov $L6,(sp) jsr pc,*$_printf L3:jmp cret L1:jbr L2 .globl .data L4:.byte 160,162,151,156,164,146,72,40,163,164,141,162,164,12,0 L5:.byte 101,40,75,40,45,144,54,40,102,40,75,40,45,144,54,40,103,40,75,40,45,144,0 L6:.byte 160,162,151,156,164,146,72,40,145,156,144,12,0 #

3 years, 10 months

demand paged 32V / Contacting Tom Raleigh

by Paul Ruizendaal

> The demand paging code for SysVR2 was written by Keith A. Kelleman and Steven J. Buroff, and in contemporary conference talks they were saying that they wanted to combine the best parts of demand-paged 32V and BSD. They may have some additional memories that could help with getting a better understanding of the final version of 32V. > > Does anybody have contact details for these two gentlemen? I’ve managed to contact Keith Kelleman and he had some interesting remarks. The paging code in SVR2 was all new code, with a focus the 3B dual processor. It does not derive at the code level from 32V and in fact he does not recall working with the 32V paging code. This kills the hope that the SVR2 code had clues about the 32V code. Keith did suggest that I tried to contact Tom Raleigh, who might have worked with the later 32V code base. Anybody with suggestions for locating him? === Besides functionality, the people that remember paged 32V all recall it being very fast. I wonder what made it fast. First to consider is “faster than what?”. Maybe Rob P. has a specific memory, but I would assume faster than 4BSD: if the comparison was with the "scatter loading, partial swapping” version of 32V people would have expected the better performance and not remember it as impressive 40 years later. Possibly the comparison is with 8th Edition which would have used the 4BSD paging code by then. If the comparison is with 4BSD, then the CoW feature in paging 32V would have been mostly matched by the vfork mechanism in 4BSD: it covers 80% of the use and it leaves the code paths simpler. If the comparison is with 8th edition, this may be the difference that people remember. The next possibility is that paging 32V had a better page-out algorithm. Joy/Babaoglu mention that the cost of the clock process is noticable. Maybe paged 32V used a VMS-like FIFO/second chance algorithm that did not need a separate kernel process/thread. Arguably this is not enough for a convincing speed difference. It is also possible that JFR found a more clever way to do LRU approximation. He remembers that his code used ‘strict LRU’, but not the algorithm. On Tenex - his conceptual reference - that was natural to do, because the MMU hardware maintains a table with 4 words of 36 bits for each frame with statistical data. With the VAX hardware it is a challenge. Considering his mathematical prowess it is maybe plausible that JFR found an efficient way. A slightly better page hit rate gives a significant speed improvement. All speculation of course: only finding the source will truly tell.

3 years, 11 months

Re: [TUHS] "The programmer must have been carried away ...

by Paul Ruizendaal

> So there you have it. Just a line of B code that wasn't updated to C. > > Cheers, > aap I love posts like this, thank you! “Sherlock Holmes and the mysterious case of the excessive parameter list"

3 years, 11 months

Code Review Request: 2.8BSD

by Warner Losh

Greetings, It has always bugged me that the bsd-family-tree file got 2.8BSD slightly wrong. It has the relationship V6 -> 1BSD -> 2BSD -> 2.79BSD -> 2.8BSD with V7 -> 32V -> 3BSD ... -> 4.1BSD -> 2.8BSD Now, as far as it goes, that's not terrible. But it's missing something. First, there was no V6 code in 1BSD or 2BSD through 2.79BSD. It was for V6 at first then for both V6 and V7, but w/o V7 code. There weren't even patches for V6 or new drivers on the early 1BSD and 2BSDs. However, starting with 2.8BSD, there's a V7 kernel. Or should I say a heavily patched V7 kernel with a ton of #ifdefs for all the fixes and enhancements collected by Berkeley and a few minor build system tweaks. Also, the code from 4.1BSD that's in 2.8 is rather minimal from what I can tell with my analysis so far, except indirectly in some of the patches to the V7 kernel appear to also be in 4.1BSD. The biggest thing that's in 2.8BSD from 4.1BSD is the job control (confined to its own directory with big warnings that basically say you'll need to update the code and even then the system is unstable). 2.9BSD has much better 4.xBSD integration, but 2.8 was quite early days for rejoining the two lines. 4.1BSD didn't have many berkeley rewrites of userland code, and the 2.8 tape has only a few of them (eg ls). So although it's not as complete as one would hope, there was a decent amount of code from 4.1BSD flowing into 2.8BSD. Now, my request. I've created a code review for FreeBSD to fix this. https://reviews.freebsd.org/D30883 is the review. We use phabricator in the FreeBSD project. Anybody can view this, but if you don't want to create an account, please send me email with a comment about the change and/or the commit message. It just adds an arc from V7 to 2.8BSD. Thanks for any time and/or insight you might have here. I'm judging the above entirely on the archived code rather than any historical knowledge... Warner

3 years, 11 months

Re: [TUHS] [EXT] "The programmer must have been carried away ...

by Angelo Papenhoff

On 23/06/21, silas poulson wrote: > I’m aware line 2238’s famous “You are not expected to understand > this.” Comment is due to odd PDP-11/45 behaviour. Actually there are two different takes on what it is exactly that you're not expected to understand. The "obvious" one in Lions' book, (i.e. saved stack being overwritten when swapping so you have to restore from the special swap-saved stack), but dmr had a different take on it, that it had to do with functions really not being happy if you switch the stack underneath them. You can find dummy variables in the interdata port that make sure the stack frames of some functions (newproc and swtch?) match. So not really a hardware thing but a consequence of the compiler. > Do you know if other sections of the C show remnants of B or the PDP? > Or is it just those spots? For the kernel I'm just aware of this one. printf was copied over a number of times (the c compiler also includes a version) but the kernel was never written in B, so i wouldn't expect any B-ness in there otherwise. aap

3 years, 11 months

"The programmer must have been carried away ...

by Angelo Papenhoff

... when he declared all the parameters for this procedure" I'm sure some of you remember this quote expressing John Lions' puzzlement over the declaration of printf in the UNIX kernel: printf(fmt,x1,x2,x3,x4,x5,x6,x7,x8,x9,xa,xb,xc) Note that parameters x2 and following are never referenced by name in the function body, only by pointer arithmetic from &x1. The historical reason for this long list of parameters is probably not well known, so I want to explain it and hope that you will find it as enjoyable as I did when I came across it some time ago while studying leftover files of B. To call a function in B, there's a little dance you have to do: first push the function address onto the stack, remember the stack pointer, push all the arguments, then restore the stack pointer and do the actual call. In diagrams, this is what happens to the B stack: push func addr: +------------+ | | <- sp +------------+ | func addr | +------------+ mark (n2): +------------+ | | <- sp +------------+ | | <- old sp (saved on native stack) +------------+ | func addr | +------------+ push arguments: +------------+ | | <- sp +------------+ | arg n | +------------+ | ... | +------------+ | arg 0 | +------------+ | | <- old sp (saved on native stack) +------------+ | func addr | +------------+ call (n3): pop old sp from stack jump to func addr on stack set ret addr and old fp +------------+ | arg n | +------------+ | ... | +------------+ | arg 0 | +------------+ | ret addr | +------------+ | old fp | <- sp, fp +------------+ The callee then sets sp, so has to know how many args and automatics! +------------+ | | <- sp +------------+ | auto n | +------------+ | ... | +------------+ | auto 0 | +------------+ | arg n | +------------+ | ... | +------------+ | arg 0 | +------------+ | ret addr | +------------+ | old fp | <- fp +------------+ So because the arguments to a functions were grouped together with the automatics and not below the ret addr and saved fp, there has to be space on the stack lest they clash. This of course means that this printf in B would probably break if called with more arguments than it expects. So there you have it. Just a line of B code that wasn't updated to C. Cheers, aap

3 years, 11 months

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

1996

1995

1994

1993

1992

1991

1990

TUHS