Digressing a bit (but only a bit) talking about IPC: Powershell and CMS PIPELINES both take the approach of more structured pipelines, where pipe contents are not just streams of bytes but can be structured records. This offers a lot of power, but it also inhibits the ability to arbitrarily compose pipe stages, because you've effectively introduced a type system. On the other hand you can certainly argue that stream-of-bytes pipes ALSO introduce a type system, it's just a completely ad-hoc, undocumented, and fragile one that relies on the cooperation of both ends of the pipe to work at all, and you'd be right. In practice...well, I'd rather use stream-of-bytes, but I am more comfortable in Unix-like environments than Powershell, and my CMS PIPELINES skills are quite rusty now. On Sat, Jul 31, 2021 at 7:21 AM Adam Thornton <athornton(a)gmail.com> wrote:

https://github.com/PDP-10/sri-nic/blob/master/files/fs/c/ccmd/ccmdmd.unx On Sat, Jul 31, 2021 at 11:46 AM Richard Salz <rich.salz(a)gmail.com> wrote:

Sorry, I remembered it from AUUGN and checked those first, then searched for "CCMD Unix Columbia.edu" and got the hit on Lar's PDP-10 sources. and note the dyslexic spelling in my earlier email of AUUGN -- sigh. On Sat, Jul 31, 2021 at 12:06 PM Richard Salz <rich.salz(a)gmail.com> wrote:

On Sat, Jul 31, 2021 at 12:18 PM Clem Cole <clemc(a)ccc.com> wrote: Frank da Cruz wrote a very nice reminiscence of the DECSYSTEM-20s at Columbia that discusses the creation of CCMD as they decommissioned the PDP-10s and switched to Unix on VAXen (and then Suns). http://www.columbia.edu/kermit/dec20.html When I was a student, we were still given accounts on the CUNIX cluster; 64-bit SPARC machines running Solaris at the time. At the time, the actress Julia Styles was a student. One day, I was walking out of Mudd (the engineering building) with a friend of mine who suddenly grabbed my arm and said, "oh my god oh my god oh my god that's Julia Styles!" Being perpetually ignorant of popular culture, I had no idea who she was referring to confusedly thought she meant Julia Child, the late host of a cooking show. "...But I thought she was dead?" "No, Dan, that's Julia Child!" We decided to look up Ms Styles in the student directory, but being a celebrity she wasn't listed. However, one could still discover her "UNI" (login name) by grepping for her in the NIS password database. We did that and sent her an email: "Christy was too embarrassed to say hi to you and Dan thought you were Julia Child." Predictably, she did not respond. In retrospect, I idly wonder how many such emails she got, most presumably of the creepy variety, but we just thought ours was funny. It appears that CUNIX still exists: https://cuit.columbia.edu/unix - Dan C. -----------------

On Sat, Jul 31, 2021 at 11:57 AM Paul Winalski <paul.winalski(a)gmail.com> wrote: There was someone posting here on TUHS a while back about leveraging a special context-sensitive `--shell-help` or similar command line program and synthesizing a protocol between the shell and a program to provide TOPS-20 like command completion. It was nowhere near what you get from the COMND JSYS, but seemed like a reasonable approximation. This is verging on COFF territory, but one of the reasons such a mechanism is unlike what you get from TOPS-20 is that, in that system, as soon as you type the name of a command, you're effectively running that command; the process model is quite different from that of Unix. With respect to command line handling in general, I think there are some attempts at making things more rational available in modern languages. Command line parsing packages for Go and the `clap` package for Rust come to mind ( https://rust-lang-nursery.github.io/rust-cookbook/cli/arguments.html) I've used clap recently in a few places and it's very convenient. - Dan C.

On Sun, Aug 1, 2021 at 1:44 PM Chet Ramey <chet.ramey(a)case.edu> wrote: Thanks for the useful background information on existing solutions. If I understood the proposal correctly, it was that the program in question would, itself, be the generator as described above. Perhaps it was coupled with a standard structured format for consumption by the shell, which seems like it would be useful for this sort of expansion. Of course, the process model in TOPS-20 was very different than in Unix, and in that system, as soon as you typed the _name_ of a command it's image was "run up" in your process. So the interactive help system was provided by a running instance of the program itself. What I gathered from the proposed model was that it involved multiple invocations of the program, but with a special option that would trigger behavior informally described as, "here's the context I've built so far; let me know what options are available here." I don't know that it's terribly "Unixy", but I can see how it would be useful for interactive use. As an aside, I maintain some older "machines" at home (even modest hardware can emulate a PDP-10 or Honeywell DPS8), and find that doing so provides me with perspective that can be very useful. Looking at other systems that were available roughly around the time of Unix (TENEX, Multics), it strikes me that the Unix was a bit of an odd-duck with the way it handled exec in terms of destructively overlaying the memory of the user portion of a process with a new image; am I wrong here? I wonder why the "one program per process and exec destroys what was running before" mechanism was implemented? I can imagine it had a lot to do with the constraints that early Unix machines must have imposed on design, not to mention implementation simplicity, but I wonder what the designers thought of other systems' process models and whether they were considered at all? Perhaps Doug and Ken might have thoughts here? - Dan C.

On Sun, Aug 1, 2021 at 5:55 PM Dan Cross &lt;crossd(a)gmail.com&gt; wrote: See dmr's paper at <https://www.bell-labs.com/usr/dmr/www/hist.html> for details, but in short exec and its equivalents elsewhere have always overlaid the running program with another program. Early versions of PDP-7 Linux used the same process model as Tenex: one process per terminal which alternated between running the shell and a user program. So exec() loaded the user program on top of the shell. Indeed, this wasn't even a syscall; the shell itself wrote a tiny program loader into the top of memory that read the new program, which was open for reading, and jumped to it. Likewise, exit() was a specialized exec() that reloaded the shell. The Tenex and Multics shells had more memory to play with and didn't have to use these self-overlaying tricks[*]: they loaded your program into available memory and called it as a subroutine, which accounts for the name "shell". So it was the introduction of fork(), which came from the Berkeley Genie OS, that made the current process control regime possible. In those days, fork() wrote the current process out to the swapping disk and set up the process table with a new entry. For efficiency, the in-memory version became the child and the swapped-out version became the parent. Instantly the shell was able to run background processes by just not waiting for them, and pipelines (once the syntax was invented) could be handled with N - 1 processes in an N-stage pipeline. Huge new powers landed on the user's head. Nowadays it's a question whether fork() makes sense any more. "A fork() in the road" [Baumann et al. 2019] < https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.… is an interesting argument against fork(): * It doesn't compose. * It is insecure by default. * It is slow (there are about 25 properties a process has in addition to its memory and hardware state, and each of these needs to be copied or not) even using COW (which is itself a Good Thing and can and should be provided separately) * It is incompatible with a single-address-space design. In short, spawn() beats fork() like a drum, and fork() should be deprecated. To be sure, the paper comes out of Microsoft Research, but I find it pretty compelling anyway. [*] My very favorite self-overlaying program was the PDP-8 bootstrap for the DF32 disk drive. You toggled in two instructions at locations 30 and 31 meaning "load disk registers and go" and "jump to self" respectively, hit the Clear key on the front panel, which cleared all registers, and started up at 30. The first instruction told the disk to start reading sector 0 of the disk into location 0 in memory (because all the registers were 0, including the disk instruction register where 0 = READ) and the second instruction kept the CPU busy waiting. As the sector loaded, the two instructions were overwritten by "skip if disk ready" and "jump to previous address", which would wait until the whole sector was loaded. Then the OS could be loaded using the primitive disk driver in block 0.

On Sun, Aug 01, 2021 at 04:49:50PM -0700, Larry McVoy wrote: s/have/have not/ called fork().... Sigh. -- --- Larry McVoy lm at mcvoy.com http://www.mcvoy.com/lm

On Sun, 1 Aug 2021, Dan Cross wrote: I've often wished I could run a free/open Bourne-type shell on 16-bit MS-DOS and OS/2. Porting to the former is next to impossible because of the lack of *any* concept of multitasking. Porting to the latter is difficult because multitasking isn't done anything like the Unix way. I actually like the spawn* functions better, though I think on Unix fork/exec is the most natural way to implement them. -uso.

On Sun, Aug 1, 2021 at 8:13 PM Andrew Warkentin &lt;andreww591(a)gmail.com&gt; wrote To start the child the parent would either In that case you don't need *any* primitive except create_empty_process(): you can do exec() by opening the file, writing to /proc/<child>/mem and then to <proc/<child>/pc-and-go.

On Sun, Aug 1, 2021 at 8:19 PM John Cowan &lt;cowan(a)ccil.org&gt; wrote: Sadly, that's not _quite_ true. You still need some primordial way to get the system going. Once you have a proc_create and a make_proc_runnable system call, it seems like it opens the door to doing all kinds of cool things like moving binary parsers out of the kernel and into user space, but consider how `init` gets bootstrapped: often, there's a handful of instructions that basically invoke `execl("/bin/init", "init", 0);` that you compile into the kernel; on creation of process 1, the kernel copies those instructions into a page somewhere in user portion of the address space and "returns" to it; the process then invokes /bin/init which carries on with bringing up the rest of the system. Now you're confronted with two choices: you either put a much more elaborate bootstrap into the kernel (in this day and age, probably not that hard), or you have a minimal bootstrap that's smart enough to load a smarter bootstrap that in turn can load something like init. I suppose a third option is to compile `init` in some dead simple way that you can load in the kernel as a special case, and invoke that. This problem isn't insurmountable, but it's a wide design space, and it's not quite as straight-forward as it first appears. As I mentioned in the email I linked to earlier, Akaros implemented the proc_create/proc_run model. It really was superior to fork()/exec() and I would argue superior to spawn() as well. - Dan C.

On 8/1/21, Theodore Ts'o &lt;tytso(a)mit.edu&gt; wrote: My OS will be microkernel-based and even the RPC channel to the VFS itself will be a file (with some special semantics). read(), write() and seek() will bypass the VFS entirely and call the kernel to directly communicate with the destination process. The call to create an empty process will return a new RPC channel and there will be an API to temporarily switch to an alternate channel so that VFS calls occur in the child context instead of the parent. All process memory, even the heap and stack, will be implemented as memory-mapped files in a per-process filesystem under /proc/<pid>. This will be a special "shadowfs" that allows creating files that shadow ranges of other files (either on disk or in memory). Environment variables will also be exposed in /proc of course. Any process that has permissions to access the RPC channel file and memory mapping shadow files in /proc/<pid> will be able to manipulate the state. The RPC channel will cease to function after the child has been started. setuid and setgid executables will not be supported at all (there will instead be a role-based access control system layered on top of a per-process file permission list, which will allow privilege escalation on exec in certain situations defined by configuration). Even though everything will use a file-based implementation underneath, there will be a utility library layered on top of it so that user code doesn't have to contain lots of open()-read()/write()-close() boilerplate.

John Cowan wrote: That's almost exactly what what ITS does. You open the USR: device and get a file descriptor (not really, but close enough) into the child process (inferior job).

On Tue, Sep 28, 2021 at 10:46:25AM -0700, Greg A. Woods wrote: I supported Xerox PARC when they were redoing their OS as a user space application on SunOS 4.x. They used mmap() and protections to take user level page faults. Yeah, there were bugs but that was ~30 years ago. In more recent times, BitKeeper used mmap() and protections to take the same page faults (we implemented a compressed, XORed file storage that filled in "pages" on demand, it was a crazy performance improvement) and that worked on pretty much every Unix we tried it on. Certainly worked on Linux first try. So what is it about mmap you don't like?

On Wed, Sep 29, 2021 at 09:40:23AM -0700, Greg A. Woods wrote: I think you didn't use SunOS 4.x. mmap() was implemented correctly there, the 4.x VM system mostly got rid of the buffer cache (the buffer cache was used only for reading directories and inodes, there was no regular file data there). If you read(2) a page and mmap()ed it and then did a write(2) to the page, the mapped page is the same physical memory as the write()ed page. Zero coherency issues. This was not true in other systems, they copied the page from the buffer cache and had all sorts of coherency problems. It took about a decade for other Unix implementations to catch up and I think that's what you are hung up on. SunOS 4.x got it right. You can read about it, I have all the papers cached at http://mcvoy.com/lm/papers ZFS screwed it all up again, ZFS has it's own cache because they weren't smart enough to know how to make compressed file systems use the page cache (we did it in BitKeeper so I have an existance proof that it is possible). I was deeply disapointed to hear that ZFS screwed up that badly, the Sun I was part of would have NEVER even entertained such an idea, they worked so hard to get a unified page cache. It's just sad.

Greg A. Woods wrote: The original post above made me think of the TENEX (later TOPS-20) primatives for fork (a noun, aka process) control: SFORK -- create an empty fork/process (halted) GET -- map executable SFORK -- start fork HFORK -- halt a running fork KFORK -- kill a fork SPJFN -- set primary file job file numbers (stdin/stdout) SPLFK -- splice a fork into tree TENEX, like UNIX was created with with knowledge of the Berkeley Timesharing System (SDS 940) and MULTICS. Like MULTICS, TENEX was designed from square one as a VM system, and I believe the 4.2BSD specified mmap call was inspired by the TENEX PMAP call (which can map file pages into a process AND map process pages into a file, and map process pages from another process). The "halted" process state was also used when a user typed CTRL/C. A halted process could be debugged (either in-process, entering a newly mapped debugger, or one already linked in, or out-of-process by splicing a debugger into the process tree). Threads were easily implemented by mapping (selected pages of) the parent process (leaving others copy-on-write, or zero-fill for thread-local stoage). Starting on small machines (an 8KW PDP-7, and a (28KW?) PDP-11) UNIX placed a premium on maximum usefulness in the minimum space. The PDP-7 source we have implements fork (implemented, as on the PDP-11 by swapping out the forking process) but not exec! The Plan9 rfork unbundles traditional Unix descriptor and memory inheritance behaviors. For all the VM generality, a sore place (for me) in TENEX/TOPS-20, a single file descriptor (job file number) was shared by all processes in a login session ("job"). "Primary" input and output streams were however per-process, but, ISTR there was nothing to stop another process from closing a stream another process was using. And like MULTICS, TENEX had byte-stream I/O, implemented day-one for disk files (I'd have to look, but system code may well have implemented it by issuing PMAP calls (monitor call code could invoke monitor calls)), and most simple user programs used it, since it was simpler to program than file mapping. refs: https://opost.com/tenex/tenex72.txt https://www.opennet.ru/docs/BSD/design-44bsd-eng/x312.html http://www.bitsavers.org/pdf/dec/pdp10/TOPS20/AA-4166E-TM_TOPS-20_Monitor_C… P.S. And on the ORIGINAL topic, TOPS-20 started with code from the TENEX EXEC (shell) that implemented command completion and incremental, and made it the COMND system call (tho it could well have been a shared library, since almost all of the COMND code called other system calls to do the work).

On Mon, Aug 2, 2021 at 1:37 PM Lars Brinkhoff &lt;lars(a)nocrew.org&gt; wrote: Should be required reading of all intro to OS students: Programming Semantics for Multiprogrammed Computations Jack B. Dennis and Earl C. Van Horn Massachusetts Institute of Technology, Cambridge, Massachusetts Volume 9 / Number 3 / March, 1966 [If your Internet search fails you and/or you find it behind an ACM paywall or the like, drop me a line, I'll forward a PDF of a scan]. ᐧ ᐧ

On 8/2/21 1:59 PM, John Cowan wrote: Original UCB source material on Genie can be found at bitsavers.

Sorry bad cut/paste -- that was from CACM if its was not obvious ᐧ On Mon, Aug 2, 2021 at 2:52 PM Clem Cole &lt;clemc(a)ccc.com&gt; wrote:

Michael Siegel writes: While I agree with you in theory, I'm dubious about how it would work in practice. Sorry if it sounds like I've lost faith with my profession but I'm trying to accept reality. Where would such documentation come from? Not wanting to reopen old flame wars, but the fragmentation of the documentation system where some things are man pages, some are info pages, some are random HTML files, some are only online, some things having no documentation at all except maybe a help message, and much of that have no actual content, is reality. Who's going to write more documentation, and how is it going to be kept consistent with other documentation? Is your help system going to be yet another fragment? Also, we seem to be well into a "Farenheight 451" world now where a vast number of people communicate only by photos and video. Writing seems to be deprecated. I don't have confidence that any documentation would be useful even if people wrote it. As a current example, I'm having trouble with a btrfs filesystem and I can't say that the btrfs-check manual page contains any useful content. Maybe since Microsoft "AI" is now going to write code for, not sure what to call them, programmers doesn't seem right any more, maybe it'll write their documentation too? I guess what I'm saying is that it sounds like you are having some good thoughts on a technical solution that I think will fail without also having a social solution. If you could somehow extract your help info from man pages without creating a whole new documentation system it might work in the few cases where there are good manual pages. Wow I sound grumpy this morning. Jon

On Sat, Jul 31, 2021 at 12:41 PM Clem Cole &lt;clemc(a)ccc.com&gt; wrote: The biggest argument (which was reasonable) was Unix command, sometimes That's modest (especially now that tar accepts "-" before the key) compared to remembering when to use -F, when to use -d, and when to use -t to specify the field separator, and when you are stuck without a field separator option; also what the default is with no option (I'd prefer "arbitrary amount whitespace" in all cases, but that's often not available at all). These inconsistencies still piss me off no end.

On Sat, Jul 31, 2021 at 1:39 PM Anthony Martin &lt;ality(a)pbrane.org&gt; wrote: What that doesn't give you is the ability to say "git <git-options> diff <git-diff-options>", which is very nice and makes the inconsistencies I just posted on less likely. Fortunately, any getopt-variant can deal with these; you just have to pass the tail of argv and a suitably reduced value for argc to another call of the options parser.

On Sat, Jul 31, 2021 at 2:58 PM Michael Siegel &lt;msi(a)malbolge.net&gt; wrote: In the old days, when the first Unix shell wars started, there was a Unix adage: *"Bourne to Program, Type with Joy"* FWIW: tcsh supports TOPS-20 autocomplete -- a little work with your search engine, you can figure out how to use its many options. That said, the GNU bash is said to do it also, but I can not say I have tried it personally since the ROMS in my fingers were long ago burned to 'Type with Joy.' Also in 50 years, it's so much that UNIX is perfect, it has lots of flaws and quirks. Thinking about them and considering 'better' solutions is often wise, particularly when capabilities (like Moore's law) give you new tools to solve them. But a level of wisdom here is not all of those quirks are worth repairing. In the case of command-line parsing, getopt(3) has proven to be 'good enough' for most things. If it was really as bad as you seem to think, I suspect one of the previous N attempts over the last 50 years might have taken root. My point in my previous message was that getopt(3) was created to solve the original UNIX problem. It did actually take root (I'll not get into if the Gnu long stuff was an improvement). But there were other attempts, including the Tops-20 scheme (which has been pointed out is quite similar to yours) that have been around for at least 35 years in the UNIX community and it did not catch on. I ask you to think about if maybe your value of that feature might be more than others have set it to be. As an analog, when I first came to UNIX and C from other systems, ideas like the open curly brace/close curly brace instead of BEGIN/END in C, and there were plenty of things in Ken's original shell that I found annoying, particularly coming from the regularity of TOPS-20 and the like. Hey, I used EMACS, TECO and DDT and none of them were in my new kit. But I forced myself to learn the new tools and new way of doing things. Since I was programming on UNIX in C, I made sure my code looked like everyone else [K&R did not yet exist -- but we would later call this 'White Book C." Why? So someone else could read it. I learned that style too and frankly have a hard time with any C code that does not follow it today. But if I am writing in a BEGIN/END style language, I adopt that style. When in Rome and all that. In time, the wonderful things I could do in the UNIX world way outpaced what I could do in the old world. In fact, by the time either TECO or EMACS bacame available for my use by then on a Vax, I never switched off the earlier UNIX tools I had learned. Like I said, I 'Type with Joy", frankly even if I'm on a Mac, Linux or Windows -- I switch the shell to be tcsh. Could I learn a new shell, sure? If I were to switch today, it would probably be zsh, but my suggestion is to learn the tools that system has really well. They keep using them. Adapt to the style of the system you are using. Anyway, that my thoughts from an old guy.

On 7/31/21 2:56 PM, Michael Siegel wrote: POSIX forbids this behavior, FWIW, so you'll probably have a hard time finding a shell -- at least one with POSIX aspirations -- that implements it. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRU chet(a)case.edu http://tiswww.cwru.edu/~chet/

On 8/1/21 3:23 PM, Richard Salz wrote: Sure. If you invest effort in building a solution, you can do just about anything. If you want, you can write a function that generates the set of aliases you use to do this. The thing is, you're going to have to build it -- you can't expect to find a shell that does a $PATH search for a pathname containing a slash. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRU chet(a)case.edu http://tiswww.cwru.edu/~chet/

On Sat, Jul 31, 2021 at 3:21 PM Jon Steinhart &lt;jon(a)fourwinds.com&gt; wrote: You mean like not using getopt and rolling your own? Shrug. while ((i = getopt(argc, argv, "xxxxx:xxxx")) != -1) switch (i) { case .... } argc -= optind; argv += optind; So I never got getopt(). One of my rules is that I don't use a library I don't know, what lines in the above are extra beyond what you write? The last two if being generous I suppose.

On Sat, Jul 31, 2021 at 5:34 PM Jon Steinhart &lt;jon(a)fourwinds.com&gt; wrote: It's how all my getopt code works. getopt is in libc and a stdlib.h so you can't count that against it :) on the other hand, your sample code didn't show arg/no-arg handling.

On Sat, Jul 31, 2021 at 3:33 PM Jon Steinhart &lt;jon(a)fourwinds.com&gt; wrote: Except for all the things this gets wrong, it's ok. The problem with inlining getopt is that you wind up with cases like -f foo'' on the command line being treated differently than '-ffoo'. Inlined code like this can be quite frustrating for the user to use. Your locality of reference is cut and paste bugs that getopt eliminates because it handles all the special cases in a uniform way. And a few more bugs... Being a geezer, I have lots of code lying around from which I can extract getopt has been standardized since the 80s and has had universal adoption since the 90s. Hardly a version chasing issue since it's in everybody's libc. The flip side to this is that libraries can be debugged once, while inline code like the above needs to be deugged over and over.... Given the number of times I've been burned by trying to roll my own getopt, I stopped trying years ago. It's harder than it looks. Warner

Warner Losh writes: Well, no. Inline code doesn't need to be debugged over and over. It doesn't have to be written from scratch every time. While in theory your point about libraries is correct, it hasn't seem to have worked out in practice. Better in C than in node.js, but there have been plenty of spectacular bugs found in old C libraries recently. Jon

Warner Losh writes: Fine by me. Never hurts to know what other people consider best practices.

On Sat, Jul 31, 2021 at 03:04:48PM -0700, Bakul Shah wrote: That last one is a gotcha waiting to happen: program --this-is-the-long-option is the same as program --this but that will break scripts (and fingers) when program gets a new option like program --this-is-the-even-longer-option We wrote our own getopt() for BitKeeper and it had long and short options but no gotcha unique prefix.

On Jul 31, 2021, at 3:14 PM, Bakul Shah &lt;bakul(a)iitbombay.org&gt; wrote: Forgot to add that whoever extends "program" should know not to create a new option that uses a longer name breaking a full form old options.

-- Bakul :-) Note that -help in MH program is far more useful as it spells out the full option name. Consider % refile -he Usage: refile [msgs] [switches] +folder ... switches are: -draft -[no]link -[no]preserve -[no]retainsequences -[no]unlink -src +folder -file file -rmmproc program -normmproc -version -help ... vs % ls -z ls: invalid option -- z usage: ls [-ABCFGHILPRSTUWZabcdfghiklmnopqrstuwxy1,] [--color=when] [-D format] [file ...]