Re: [TUHS] Tech Sq elevator (Was: screen editors) [ really I think efficiency now ]

[Resending as this got squashed a few days ago. Jon, sorry for the duplicate. Again.] On Sun, Jan 12, 2020 at 4:38 PM Jon Steinhart <jon(a)fourwinds.com> wrote: What I understood was that, by translating into a lowest-common-denominator format like text, one loses much of the semantic information implicit in a richer representation. In particular, much of the internal knowledge (like type information...) is lost during translation and presentation. Put another way, with text as usually used by the standard suite of Unix tools, type information is implicit, rather than explicit. I took this to be less an issue of efficiency and more of expressiveness. It is, perhaps, important to remember that Unix works so well because of heavy use of convention: to take Doug's example, the total number of commands might be easy to find with `wc` because one assumes each command is presented on a separate line, with no gaudy header or footer information or extraneous explanatory text. This sort of convention, where each logical "record" is a line by itself, is pervasive on Unix systems, but is not guaranteed. In some sense, those representations are fragile: a change in output might break something else downstream in the pipeline, whereas a representation that captures more semantic meaning is more robust in the face of change but, as in Doug's example, often harder to use. The Lisp Machine had all sorts of cool information in the image and a good Lisp hacker familiar with the machine's structures could write programs to extract and present that information. But doing so wasn't trivial in the way that '| wc -l' in response to a casual query is. While that may be true, it sort of assume that this is something so common This I think illustrates an important point: Unix conventions worked well enough in practice that many interesting tasks were not just tractable, but easy and in some cases trivial. Combining programs was easy via pipelines. Harder stuff involving more elaborate data formats was possible, but, well, harder and required more involved programming. By contrast, the Lisp machine could do the hard stuff, but the simple stuff also required non-trivial programming. The SQL database point was similarly interesting: having written programs to talk to relational databases, yes, one can do powerful things: but the amount of programming required is significant at a minimum and often substantial. The flip side is that one often loses information in the conversion to text: yes, there are structured data formats with text serializations that can preserve the lost information, but consuming and processing those with the standard Unix tools can be messy. Seemingly trivial changes in text, like reversing the order of two fields, can break programs that consume that data. Data must be suitable for pipelining (e.g., perhaps free-form text must be free of newlines or something). These are all limitations. Where I think the argument went awry is in not recognizing that very often those problems, while real, are at least tractable. Many people have claimed, incorrectly in my opinion, that this model fails Certainly. I think you hit the nail on the head with the proviso that one must _understand_ the Unix model and how to use it. If one does so, it's very powerful indeed, and it really is applicable more often than not. But it is not a panacea (not that anyone suggested it is). As an example, how do I apply an unmodified `grep` to arbitrary JSON data (which may span more than one line)? Perhaps there is a way (I can imagine a 'record2line' program that consumes a single JSON object and emits it as a syntactically valid one-liner...) but I can also imagine all sorts of ways that might go wrong. - Dan C.