On Wed, 4 Mar 2020, Tyler Adams wrote: I seem to recall that "sort -r" was in V6, or perhaps that was one of the programs I'd back-ported from V7 (being stuck with 11/40-class boxes). And speaking of "tac" (which I never saw), I couldn't think of a single use for "rev" (although no doubt I'll now get told). Mind you, you get some amusing output with the "man" command because of the way that the underlining works... -- Dave

On Tue, 3 Mar 2020, arnold(a)skeeve.com wrote: Ah, of course :-) And Arabic too, as I recall. -- Dave

On Wed, 4 Mar 2020, Lyndon Nerenberg wrote: (Uses for "rev") Neat! -- Dave

On Tue, Mar 3, 2020, at 13:44, Adam Thornton wrote: Sure, but "stdin is a sequence of any type, and the argument is an expression that operates on that type or the name of a property that that type has" is universal enough. The part that has to operate on a specific structure isn't the command, it's the arguments. For example, a powershell pipeline to produce a list of files sorted by modified date is: gci . | sort lastwritetime | select name all three *commands* are universal - not all objects have a "lastwritetime" and "name" property, but sort and select can operate on any property that the sequence of objects passed into it has. (gci is an alias for get-childitem... it also has aliases ls and dir, but I'm emphasizing that it's not exclusive to directories) *assuming that ls -t didn't exist*, to do this with unix tools that operate on text you would need: ls -l | [somehow convert the date to a sortable format, probably in awk] | sort | [somehow pick the filename alone out of the output - possibly with cut or sed or awk again] and it's very difficult to get tools like awk, sort, and cut to work on formats that contain more than one field that may contain embedded spaces (you can just about get away with it for ls output because the date is always three "words"). A significant portion of ls's options are related to sorting, because you can sort based on fields that are either not present in the output, or are not in a format that can be sorted textually. Maybe it would be enough to have the universal interface be "tables" (i.e. text streams in some format that supports adequate escaping of embedded row and column delimiters)... or maybe even just table rows, and let the user deal with memorizing column numbers (or let each originating command support a fully general way to specify what columns are requested, as ps alone does on modern systems) Of course, this isn't *really* different from allowing any data structure - after all, the value for any field could itself be a fully escaped table in text format. The benefit of having actual data structures with types is that when you *don't* end the pipeline with select, each object knows how to print itself [files print mode, mtime, size, and name in a human-readable format, more or less equivalent to ls -l] rather than just dumping out every single field that you might want sort or select to operate on.

On 2020-03-04 16:50:34, Random832 spake thus: [...] There are some examples of that type of thing in widely used Unix tools; my use of 'sort -k1,1n' further down is demonstrating such a use case (the 'sort' command is being told that it is operating on numbers). But beyond some lowest common denominator types ("number", "string", ...) how many commands can really usefully operate on a large number of types? For example, a program that can operate on IP addresses is probably doing something different than a program that wants to operate on email addresses. I could see where named properties of some object can be used more generally than types, but again there are widely used tools that do do that (e.g., jq(1)). IMHO, though, they are more cumbersome to use than most of the commands I need to use minute to minute. (Just nit-picking at this particular example) You could do it without ls[0]: $ stat -c '%Y %n' * | sort -k1,1n | xargs -L1 sh -c 'echo "$@"' That doesn't seem so bad to me, but if it was something I needed regularly I'd of course put it in an alias[1] or (more likely) a short script file. [...] Yes, that's often true. And when I enounter it I typically start out by seeing if I can inject and remove tokens in the data at key places in the pipeline. Beyond anything trivial, though, I then quickly start reaching for tools to put the data into some form that more easily allow for it (CSV, JSON, ...). But that invariably adds other complications (such as the need to find or build tools to marshal/unmarshal the data, and to deal with data-domain-specific notions of null-vs-empty-string). For the (more common (for me)) case where there is only one field that contains embedded spaces, I just try to get 'em at the end of the line and let the shell deal with it: $ some-command | while read -r first second rest; do ... ; done [...] Well, in some sense with byte streams you have a table of newline-delimited bytes (rows), and byte subfields separated by whitespace (columns). And anything on top of that could (in some context, and with some syntax) be considered just further escaped tables in text format. I think that's essentially the same thing that you said, only with the outermost table syntax removed. But like you said, this isn't really different from allowing any data structure. Importantly, though, it doesn't impose any particular data structure, either. I've worked at a couple of different places that had in-house tools for working with explicit table semantics in command line suites, and where they fit the data domain, that was hugely useful. Generally speaking, they were special purpose enough to warrant their own tools, but still general purpose enough to be composable (were designed for use in shell pipelines) and applicable in domains beyond the intentions of their original authors. Still, the burden of "thinking in tables" would make them too heavyweight for a lot of common use cases. Sometimes my data structure is "paragraphs of text": $ lorem -p 3 | perl -00 -wnle '2 == $. && print' | wc -w Other times I want a tree (JSON, s-expressions, ...), or even a stream of trees[2]. I consider it a feature that these more complex data structures are not assumed or imposed in contexts where they are not needed. Take care, -Al [0] You could get 'ls' to do it, too, (without '-t') but here the use of TIME_STYLE is a presumably non-portable (but handy!) GNU-ism: $ TIME_STYLE='+%s' ls -l | tail -n +2 | sort -k6,6n | xargs -L1 sh -c 'shift 5; echo "$@"' It's different from the '-t' option, though, in that it forces a predicatable date field format in the output of 'ls -l', so side-steps the need for downstream date parsing altogether and simply jumps into sorting (after chopping off the 'total N' header (groans all around)). [1] E.g., $ # read 'bmt' as: "by mtime" $ alias bmt='stat -c "%Y %n" * | sort -k1,1n | xargs -L1 sh -c '"'echo "'"$@"'"'" $ bmt [2] Probably flattened.

On Wed, Mar 4, 2020 at 11:04 PM Random832 <random832(a)fastmail.com> wrote: Another difference is that probably most people on this list are extremely familiar with the various quirks and I/O nuances of the tools many have been using every day for decades. Just as the native speakers of a natural language can't so easily see/appreciate its complexity (e.g., pronunciation in English!), I suspect many of us have internalized these idiosyncrasies. I teach occasional shell/Python courses to absolute beginners (no computing experience at all) and came to appreciate how weird the shell is (in the sense of having baked-in historical accidents that cannot / will not / should not be "corrected"). Some of my appreciation of that was due to discussions on this list (e.g., regarding comment syntax, and the : command) - so thanks! I know what follows won't be to everyone's taste, but I like Python and I love shell pipelines, so I tried to write a shell that gave you both and which allowed fairly free mixing of invoking UNIX tools and running Python. You can send anything down its pipelines - lines of text, atoms, numbers, Python objects, whatever (in the Python _ variable). Of course the receiving end of the pipeline needs to know (or figure out) what it's getting. One advantage is that you have a carefully designed programming language (no offence intended!) underlying the shell, so you can e.g., write shell functions in Python (and put them in a start-up file if you want) and just pipe regular UNIX output into them and pipe their output into whatever's next (more Python, another UNIX command, etc). Probably almost no one would actually want to regularly do the following on the command line, but you could: for f in _)] Here I've stuck a simple (DSU - see [1]) Python function in between two UNIX commands and use it to get the most recently modified files. You probably wouldn't want to do this either, but you could: Of course it also lets you do things you *would* want to do :-) More at https://github.com/terrycojones/daudin Python has fairly nice tools for reading and evaluating Python code, which meant that getting a first version of this implemented took only one evening of playing around. It's pretty simple (and still has plenty of rough edges). Apologies if this seems like self-promotion, but I very much enjoy thinking about things in this thread and about how we work with information. I'm also constantly blown away by how elegant UNIX is and how the core ideas have endured. Pipelines are really wonderful, as "natural" alternative to function composition as a mathematician or programmer would do it (see point #1 at https://github.com/terrycojones/daudin#background--thanks) and I wanted to build a shell that preserved that, while giving you Python. The overview of their history on pages 67-70 of bwk's recent book [2] is very interesting. Terry [1] https://en.wikipedia.org/wiki/Schwartzian_transform [2] https://www.amazon.com/UNIX-History-Memoir-Brian-Kernighan/dp/1695978552

On Tue, 10 Mar 2020, Dan Stromberg wrote: That sort of makes sense from a mathematical point of view, if you regard it as a matrix of side effects. I hate to think about how it affects Perl (my favourite language) though :-) I'm starting to think that if a utility requires many options then perhaps they ought to be split into filters (or at least environment variables); I despair at how *ix is drifting from "one tool, one job" to "one size fits all"... The "ls" command for example really needs an option-ectomy; I find that I don't really care about the exact number of bytes there are in a file as the nearest KiB or MiB (or even GiB) is usually good enough, so I'd be happy if "-h" was the default with some way to turn it off (yes, I know that it's occasionally useful to add them all up in a column, but that won't tell you how many media blocks are required). Quickly now, without looking: which option shows unprintable characters in a filename? Unless you use it regularly (in which case you have real problems) you would have to look it up; I find that "ls ... | od -bc" to be quicker, especially on filenames with trailing blanks etc (which "-B" won't show). I've never liked GNU's "--bloody-long-option" convention as you still have to look up which one does what, but I've never thought about that view; a lot of long options still accept a single character (subject to feeping creaturism, of course). -- Dave

On Wednesday, 11 March 2020 at 14:18:08 +1100, Dave Horsfall wrote: A good example. But you're not removing options, you're just redefining them. In fact I find the -h option particularly emetic, so a better choice in removing options would be to remove -h and use a filter to mutilate the sizes: $ ls -l | humanize But that's a pain, isn't it? That's why there's a -h option for people who like it. Note that you can't do it the other way round: you can't get the exact size from -h output. And then there's the question why you don't like the standard output. Because the number strings are too long and difficult to read, maybe? That's the rationale for the -, option. This is arguably a bug in the -B option. I certainly don't think the pipe notation is quicker. But it's nice to have both alternatives. Greg -- Sent from my desktop computer. Finger grog(a)lemis.com for PGP public key. See complete headers for address and phone numbers. This message is digitally signed. If your Microsoft mail program reports problems, please read http://lemis.com/broken-MUA

On Wednesday, 11 March 2020 at 19:14:32 -0400, Dan Cross wrote: It's certainly more work than -h. What I meant (and this is certainly my interpretation) was that somebody added the -h option because of perceived pain with piping output through another program. I didn't intend to imply that it was the only alternative. Yes, that sounds like an excellent method. And yes, I deliberately didn't mention this option, though it occurred to me. I have a couple of scripts like this, like: alias l="ls -lbL," Yes, this is an issue that I mentioned in an earlier message (I added a positional parameter to work around it). But this is in the nature of the output. mc doesn't have this issue. Heh. Never thought of that. But since -h (apparently) never produces output with 4 digits, the -, doesn't ever come into effect. I've just tried it on some big files, and the -, is effectively ignored. I don't like the standard output because things like this are hard to read: -rw-r--r-- 1 grog lemis 8234010624 22 Mar 2012 Casanova-TV-1-5 -rw-r--r-- 1 grog home 13225168900 31 Aug 2019 Movie:_Sahara_2005-2016-04-11-2028 I find this easier to read: -rw-r--r-- 1 grog lemis 8,234,010,624 22 Mar 2012 Casanova-TV-1-5 -rw-r--r-- 1 grog home 13,225,168,900 31 Aug 2019 Movie:_Sahara_2005-2016-04-11-2028 I can't speak for Dave, but this is also less painful: -rw-r--r-- 1 grog lemis 7.7G 22 Mar 2012 Casanova-TV-1-5 -rw-r--r-- 1 grog home 12G 31 Aug 2019 Movie:_Sahara_2005-2016-04-11-2028 The problem for me there is the difficulty comparing lengths, and the implicit inaccuracy. Hmm. In this particular case, so does Linux: === grog@bilbo (/dev/pts/11) ~ 2 -> touch "foo " === grog@bilbo (/dev/pts/11) ~ 4 -> l foo* -rw-r--r-- 1 grog grog 1499570 Jun 30 2012 foo -rw-r--r-- 1 grog grog 0 Mar 12 10:40 'foo ' I wonder if that's something we should emulate in FreeBSD. At the very least we should consider whether the lack of identification of trailing blanks is a bug in the FreeBSD implementation of -B. This option isn't in POSIX, and in Linux it means -B, --ignore-backups do not list implied entries ending with ~ So maybe it's a candidate for fixing. Greg -- Sent from my desktop computer. Finger grog(a)lemis.com for PGP public key. See complete headers for address and phone numbers. This message is digitally signed. If your Microsoft mail program reports problems, please read http://lemis.com/broken-MUA

On Wednesday, 11 March 2020 at 20:53:12 -0400, Steve Nickolas wrote: A brave man to defy POSIX! I wasn't so brave, which is why we have the -y option. OK, let's compare notes: Arguably -a could go too. The distinction seems arbitrary. Neither would this ls(1) be. No arguments, but I suspect that somebody had a good reason for this distinction, and removing it could cause problems. -S isn't POSIX. And to implement it without an option would mean removing -h. As I mentioned earlier, -t can't be done by a filter without significantly modifying the timestamp output. That was my rationale for the -D option, which allows sorting by an external filter. -r could work. I think this says more about your file systems than about the options. I find both incredibly useful, and there's no easy way to get the information elsewhere. stat(1) would be an option, but then that could replace ls(1) completely. -g is already obsolete in FreeBSD (accepted and ignored). -o has already been repurposed (show file flags). Agreed. Possibly. Certainly I wouldn't miss it. OK. I think that it could be replaced by -b. "?" isn't really very helpful. I can't agree with that, but I've never used it. The only sensible use would appear to be talking about disk blocks, but on FreeBSD at any rate it looks at the BLOCKSIZE environment variable, which I have set to 1048576 (so that utilities will display in MB where appropriate), and that's what -s does too: 2079 -rw-r--r-- 1 grog wheel 2,178,735,915 4 Oct 11:15 Willkommen-bei-den-Honeckers---Spielfilm,-Deutschland-2016-20191003-125200.mp4 That makes it pretty useless. So, any others? -G: Colorized output. I'd be *really* happy to get rid of this, but it's not easy to instate with a filter, so I suppose there are enough people who like it that it will have to stay. -P: Seems only to be there to cancel a -H or -L. -W: "Display whiteouts when scanning directories". I don't even understand what that is. -a: See discussion of -A. --color: Again, no thanks. -f: We haven't really discussed this one. If you want to remove -S, -r and -t, then arguably -f should become the default and be -removed. -n: Make it the default and require a filter to convert group and user numbers to IDs. -y: If we get rid of all sorting, it will no longer be needed. -,: Make the option standard: output numbers with commas every 3 digits. Then this option specification wouldn't be needed. Of course, none of this will happen. But it is interesting to think about it. In particular, options like -g and -o, which are no longer modern. Greg -- Sent from my desktop computer. Finger grog(a)lemis.com for PGP public key. See complete headers for address and phone numbers. This message is digitally signed. If your Microsoft mail program reports problems, please read http://lemis.com/broken-MUA

On Wednesday, 11 March 2020 at 23:34:46 -0400, Steve Nickolas wrote: It might have originated there, but then I would expect it to be spelt '--produce-human-readable-output'. I haven't been able to establish from the FreeBSD sources or commit logs when it was introduced. It would clearly have been a reimplementation. So it is! This was the first option that I wanted to add, back when I still had practice wheels. I asked my mentor, and he said "not the Unix way", so I let it be. Then Wes Peters came up with the idea, and I thought he committed it, but it seems that it ultimately came from Kostas Blekos in 2005, based on the same feature on NetBSD and OpenBSD. I wonder when it made it to POSIX. Yes, that was one of the possibilities I thought of. Another obvious one was time_t, which is even easier to process. And then there's ISO 8601. That's why it didn't take me long to decide "do it *your* way” with the -D option. Greg -- Sent from my desktop computer. Finger grog(a)lemis.com for PGP public key. See complete headers for address and phone numbers. This message is digitally signed. If your Microsoft mail program reports problems, please read http://lemis.com/broken-MUA

On 2020-Mar-11 20:53:12 -0400, Steve Nickolas <usotsuki(a)buric.co> wrote: How does humanize decide which column to work on? If it only works on "ls -l", then it's not useful if I want other columns as well. Maybe it could just humanize any large number it found, but you probably don't want to "humanize" the inode number or filename. Except they are reporting completely different things - consider sparse files or filesystems (like ZFS) that support compression. -- Peter Jeremy

On Thu, Mar 12, 2020, 1:37 AM Steve Nickolas <usotsuki(a)buric.co> wrote: Stat returns two values. The offset of the last byte and the number of blocks allocated to the file. Useful if you have a sparse file too... Warner (Of the filesystems I frequently work with, one of them does actually

On Thu, 12 Mar 2020, Greg 'groggy' Lehey wrote: I also had something like that in mind, except being British/Australian I'd spell it with an "s" :-) Which is why I suggested there be a means to turn it off; I'm becoming a fan of environment variables to modify the standard behaviour of tools (but I loathe the Penguin/OS default to use colours). More than likely; as I approach age 68 I notice that I'm losing some cognitive facility... I might start using "," and see if I like it, but I see that the Mac doesn't have it (my Penguin is off the air at the moment), and having it as an environment variable would be nice. Agreed; as for the bug I think it comes down to what is meant by an unprintable character. I certainly remember finding "hidden" set-uid shells with the name of ".. " etc back when I was going after the UNSW kiddies with an axe back in the late 70s... -- Dave

At Fri, 13 Mar 2020 11:36:47 +1100, Greg 'groggy' Lehey <grog(a)lemis.com> wrote: Subject: Re: [TUHS] Command line options and complexity Because of course NetBSD has chosen a different option letter: 'M' Unfortunately on NetBSD and FreeBSD the appearance of commas (or whatever is appropriate) depends on the locale being correctly configured, and this is not always so easy to do! -- Greg A. Woods <gwoods(a)acm.org> Kelowna, BC +1 250 762-7675 RoboHack <woods(a)robohack.ca> Planix, Inc. <woods(a)planix.com> Avoncote Farms <woods(a)avoncote.ca>