13 May
2024
13 May
'24
1:34 a.m.
On Sat, May 11, 2024 at 2:35 PM Theodore Ts'o <tytso(a)mit.edu> wrote:
I bet most of the young'uns would not be trying to do this as a shell
script, but using the Cloud SDK with perl or python or Go, which is
*way* more bloaty than using /bin/sh.
So while some of us old farts might be bemoaning the death of the Unix
philosophy, perhaps part of the reality is that the Unix philosophy
were ideal for a simpler time, but might not be as good of a fit
today
I'm finding myself in agreement. I might well do this with jq, but as you
point out, you're using the jq DSL pretty extensively to pull out the
fields. On the other hand, I don't think that's very different than piping
stuff through awk, and I don't think anyone feels like _that_ would be
cheating. And jq -L is pretty much equivalent to awk -F, which is how I
would do this in practice, rather than trying to inline the whole jq bit.
But it does come down to the same argument as
https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.…
And it is true that while fork() is a great model for single-threaded
pipeline-looking tasks, it's not really what you want for an interactive
multithreaded application on your phone's GUI.
Oddly, I'd have a slightly different reason for reaching for Python (which
is probably how I'd do this anyway), and that's the batteries-included
bit. If I write in Python, I've got the gcloud api available as a Python
module, and I've got a JSON parser also available as a Python module (but I
bet all the JSON unmarshalling is already handled in the gcloud library),
and I don't have to context-switch to the same degree that I would if I
were stringing it together in the shell. Instead of "make an HTTP request
to get JSON text back, then parse that with repeated calls to jq", I'd just
get an object back from the instance fetch request, pick out the fields I
wanted, and I'd be done.
I'm afraid only old farts write anything in Perl anymore. The kids just
mutter "OK, Boomer" when you try to tell them how much better CPAN was than
PyPi. And it sure feels like all the cool kids have abandoned Go for Rust,
although Go would be a perfectly reasonable choice for this task as well
(and would look a lot like Python: get an object back, pick off the useful
fields).
Adam
13 May
13 May
1:47 a.m.
New subject: [COFF] Re: On Bloat and the Idea of Small Specialized Tools
On Sun, May 12, 2024 at 12:34:20PM -0700, Adam Thornton wrote:
But it does come down to the same argument as
https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.…
And it is true that while fork() is a great model for single-threaded
pipeline-looking tasks, it's not really what you want for an interactive
multithreaded application on your phone's GUI.
Perhaps a meaningless aside, but I agree on fork(). In the last major
project I did, which was cross platform {windows,macos, all the major
Unices, Linux}, we adopted spawn() rather than fork/exec. There is no way
(that I know of) to fake fork() on Windows but it's easy to fake spawn().
--lm
2:13 a.m.
New subject: forking, Re: [COFF] Re: On Bloat and the Idea of Small Specialized Tools
It appears that Larry McVoy <lm(a)mcvoy.com> said:
Perhaps a meaningless aside, but I agree on fork(). In
the last major
project I did, which was cross platform {windows,macos, all the major
Unices, Linux}, we adopted spawn() rather than fork/exec. There is no way
(that I know of) to fake fork() on Windows but it's easy to fake spawn().
The whole point of fork() is that it lets you get the effect of spawn with
a lot less internal mechanism. Spawn is equivalent to:
fork()
... do stuff to files and environment ...
exec()
By separating the fork and the exec, they didn't have to put all of
the stuff in the 12 paragraphs in the spawn() man page into the the
tiny PDP-11 kernel.
These days now that programs include multi-megabyte shared libraries
just for fun, I agree that the argument is less persuasive. On the
third hard, we now understand virtual memory and paging systems a lot
better so we don't need kludges like vfork().
R's,
John
4:56 a.m.
New subject: forking, Re: [COFF] Re: On Bloat and the Idea of Small Specialized Tools
On Sun, May 12, 2024 at 4:14 PM John Levine <johnl(a)taugh.com> wrote:
It appears that Larry McVoy <lm(a)mcvoy.com>
said:
Perhaps, but as I've written here before, `fork`/`exec` vs `spawn` is
a false dichotomy. Another alternative is a `proccreate`/`procrun`
pair, the former of which creates an unrunnable process, the latter of
which marks it runnable. Coupled with a set of primitives to
manipulate the state of an extant, but unrunnable, process and you
have the advantages of fork/exec without the downsides (which are
well-known;
https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.…)
Similarly, this gives you the functionality of spawn, without the
downside of a singularly complicated interface. Could you have
implemented that in something as small as the PDP-7? Perhaps not, but
it does not follow that `fork` now remains a good primitive.
My spelunking in the original GENIE documentation leads me to believe
that its `fork` provided functionality similar to what I described.
- Dan C.
Perhaps a meaningless aside, but I agree on
fork(). In the last major
project I did, which was cross platform {windows,macos, all the major
Unices, Linux}, we adopted spawn() rather than fork/exec. There is no way
(that I know of) to fake fork() on Windows but it's easy to fake spawn().
The whole point of fork() is that it lets you get the effect of spawn with
a lot less internal mechanism. Spawn is equivalent to:
fork()
... do stuff to files and environment ...
exec()
By separating the fork and the exec, they didn't have to put all of
the stuff in the 12 paragraphs in the spawn() man page into the the
tiny PDP-11 kernel.
5:34 a.m.
New subject: forking, Re: [COFF] Re: On Bloat and the Idea of Small Specialized Tools
On Sun, May 12, 2024 at 06:56:35PM -0400, Dan Cross wrote:
Similarly, this gives you the functionality of spawn,
without the
downside of a singularly complicated interface. Could you have
implemented that in something as small as the PDP-7? Perhaps not, but
it does not follow that `fork` now remains a good primitive.
Our spawnvp() implmentation is 40 lines of code. Worked fine everywhere.
I can post it if you like.
7:21 p.m.
New subject: forking, Re: [COFF] Re: On Bloat and the Idea of Small Specialized Tools
On Mon, May 13, 2024 at 11:34:38AM +1000, Dave Horsfall wrote:
On Sun, 12 May 2024, Larry McVoy wrote:
/*
* Copyright 1999-2002,2004-2006,2015-2016 BitMover, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#include "system.h"
void (*spawn_preHook)(int flags, char *av[]) = 0;
#ifndef WIN32
pid_t
bk_spawnvp(int flags, char *cmdname, char *av[])
{
int fd, status;
pid_t pid;
char *exec;
/* Tell the calling process right away if there is no such program */
unless (exec = which((char*)cmdname)) return (-1);
if (spawn_preHook) spawn_preHook(flags, av);
if (pid = fork()) { /* parent */
free(exec);
if (pid == -1) return (pid);
unless (flags & (_P_DETACH|_P_NOWAIT)) {
if (waitpid(pid, &status, 0) != pid) status = -1;
return (status);
}
return (pid);
} else { /* child */
/*
* See win32/uwtlib/wapi_intf.c:spawnvp_ex()
* We leave nothing open on a detach, but leave
* in/out/err open on a normal fork/exec.
*/
if (flags & _P_DETACH) {
unless (getenv("_NO_SETSID")) setsid();
/* close everything to match winblows */
for (fd = 0; fd < 100; fd++) (close)(fd);
} else {
/*
* Emulate having everything except in/out/err
* as being marked as close on exec to match winblows.
*/
for (fd = 3; fd < 100; fd++) (close)(fd);
}
execv(exec, av);
perror(exec);
_exit(19);
}
}
#else /* ======== WIN32 ======== */
pid_t
bk_spawnvp(int flags, char *cmdname, char *av[])
{
pid_t pid;
char *exec;
/* Tell the calling process right away if there is no such program */
unless (exec = which((char*)cmdname)) return (-1);
if (spawn_preHook) spawn_preHook(flags, av);
/*
* We use our own version of spawn in uwtlib
* because the NT spawn() does not work well with tcl
*/
pid = _spawnvp_ex(flags, exec, av, 1);
free(exec);
return (pid);
}
#endif /* WIN32 */
Our spawnvp() implmentation is 40 lines of code.
Worked fine everywhere.
I can post it if you like.
Pretty please...
-- Dave 
9:29 a.m.
New subject: forking, Re: [COFF] Re: On Bloat and the Idea of Small Specialized Tools
On Sun, May 12, 2024 at 4:57 PM Dan Cross <crossd(a)gmail.com> wrote:
l.
Perhaps, but as I've written here before, `fork`/`exec` vs `spawn` is
a false dichotomy. Another alternative is a `proccreate`/`procrun`
pair, the former of which creates an unrunnable process, the latter of
which marks it runnable. Coupled with a set of primitives to
manipulate the state of an extant, but unrunnable, process and you
have the advantages of fork/exec without the downsides (which are
well-known;
https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.…)
Similarly, this gives you the functionality of spawn, without the
downside of a singularly complicated interface. Could you have
implemented that in something as small as the PDP-7? Perhaps not, but
it does not follow that `fork` now remains a good primitive.
IMO something like that is the best model (although it probably would
have been a bit complicated for a PDP-7/PDP-11). That's basically what
I'm doing in the OS that I'm writing
<https://gitlab.com/uxrt/uxrt-toplevel>. Processes will basically just
be containers for hierarchical groups of threads, and will have pretty
much no other state besides the command line. All of the context
normally associated with a process (file descriptor space,
permissions/UID/GID, filesystem namespace, virtual address space) will
instead be in separate objects that are explicitly bound to threads.
Separate APIs for creating an empty process, creating threads within
it, manipulating context objects and binding threads to them, and
starting the process will be provided (all of these APIs will use a
file-based transport underneath; this will be the first OS I know of
where literally everything is a file). The base process APIs will be
general enough to allow an efficient copy-on-write fork() to be
implemented on top of them for backwards compatibility and the
remaining use cases where forking still makes sense (since even all
process memory will be implemented with files, this will be
implemented with a special in-memory "shadow filesystem" that creates
alternate mappings of other memory filesystems).
Really I'd say there are actually several design decisions in
conventional Unix that made sense on a PDP-7 or PDP-11, but no longer
make sense in the modern world. For instance, the rather inflexible
security model with its fixed set of root-only system calls rather
than some form of role-based access control, or the use of on-disk
device nodes bound by numbers rather than something like separate
special filesystems for each driver that get union mounted together,
or the lack of integrated support for userspace filesystem servers
(yes, there's FUSE, but it's kind of a poorly integrated hack that is
rarely used for anything important).
2:43 a.m.
New subject: [COFF] Re: On Bloat and the Idea of Small Specialized Tools
On Sun, 12 May 2024, Adam Thornton wrote:
I'm afraid only old farts write anything in Perl
anymore. The kids just
mutter "OK, Boomer" when you try to tell them how much better CPAN was than
PyPi. And it sure feels like all the cool kids have abandoned Go for Rust,
although Go would be a perfectly reasonable choice for this task as well
(and would look a lot like Python: get an object back, pick off the useful
fields).
I must be an old fart then; the last language I used where white space was
part of the syntax was FORTRAN...
-- Dave
8:33 a.m.
New subject: [COFF] Re: On Bloat and the Idea of Small Specialized Tools
On Sat, May 11, 2024 at 2:35 PM Theodore Ts'o
<tytso(a)mit.edu>
wrote:
So while some of us old farts might be bemoaning the death of
the
Unix
philosophy, perhaps part of the reality is that the Unix
philosophy
were ideal for a simpler time, but might not be as good of a fit
today
Hm .... i guess it might depend on the specific use-case(s)
involved?
At one point i realised that a primary reason i enjoy using *n*x
systems is that they're fundamentally
_text-oriented_. (Unsurprisingly, of course, given the context in
which Unix was developed.) i spend a lot of my time interacting
and working with text, and *n*x systems provide me with many
useful tools for this. Quoting the old "UNIX As Literature" piece,
https://theody.net/elements.html:
"[T]he most recurrent complaint was that [Unix] was too
text-oriented. People really hated the command line, with all the
utilities, obscure flags, and arguments they had to memorize. They
hated all the typing. One mislaid character and you had to start
over. Interestingly, this complaint came most often from users of
the GUI-laden Macintosh or Windows platforms. ...
"[A] suspiciously high proportion of my UNIX colleagues had
already developed, in some prior career, a comfort and fluency
with text and printed words. ...
"With UNIX, text — on the command line, STDIN, STDOUT, STDERR — is
the primary interface mechanism: UNIX system utilities are a sort
of Lego construction set for word-smiths. Pipes and filters
connect one utility to the next, text flows invisibly
between. Working with a shell, awk/lex derivatives, or the utility
set is literally a word dance."
Perl, with its pervasive regex-based functionality and extensive
Unicode support, fits neatly into this. i find regexes an
_incredibly_ powerful tool for working with text, whether via
Perl, sed, awk, or whatever. But my experience is that many people
treat regexes as an anathema, with Zawinski's "Now you have two
problems" regularly trotted out as a thought-terminating
cliché. Sure, regexes can, and do, get used where they shouldn't
be[a]; that doesn't mean the baby should be thrown out with the
bathwater.
But if one is only working with text under sufferance, trying to
avoid it via substantially more graphically-oriented environments,
the text-based "Unix philosophy" and the tools associated with it
might feel (and actually be) much less appropriate and
useful. Fair enough. The Unix construction set will still be there
for those of us who find them very appropriate and tremendously
useful.
Alexis.
[a] It seems unlikely that anyone on this list hasn't already seen
this, but just in case:
https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xh…
i'm looking forward to that comment sending OpenAI over the
Mountains of Madness.
8:57 a.m.
New subject: [COFF] Re: On Bloat and the Idea of Small Specialized Tools
On Sun, May 12, 2024, 8:34 PM Alexis <flexibeast(a)gmail.com> wrote:
I created, years ago, a set of time legos. They were connected as a network
of producer / consumer interfaces. Each lego would do one thing and pass
the results to the next thing in the chain. A driver would read timing data
from the driver and convert it to a MI interface. Different other legos
would take time differences, compute phase or frequency differences and
these would feed into more sophisticated algorithms or output etc. All
locking was on yhe pipe's queues so all these algorithms were lock free
apart from the queueing or dequeueing of data.
Concrptually, this is just a bunch of pipe, with many to 1 or 1 to many
added. Each lego did one thing and passed the results along to the thing in
the chain... much like 'cmd | grep | awk | more'. Plus MI data
representations for almost everything so only the driver reader thread
cared about the hw. See also tty abstraction or ifnet abstraction in
unix....
So actually not a set of FDs passing data between process, but threads
doing the same sort of thing.
The whole data filtering paradigm works in lots of different ways. And it
still works really well by analogy.
Warner
ObComplaint: fork sucks for address spaces with 100s of threads. Forst
thing we created a child process we used to broker different threads
needing to run popen or system... having a create process / munge process /
start process API is kinda what we did behind the scenes though with "send
this data" and "receive that data". We iterated to this after the first
dozen attempts to closely broker fork/exec dance proved... unreliable.
At one point i realised that a primary reason i enjoy using *n*x
On Sat, May 11, 2024 at 2:35 PM Theodore
Ts'o <tytso(a)mit.edu>
wrote:
So while some of us old farts might be bemoaning the death of
the
Unix
philosophy, perhaps part of the reality is that the Unix
philosophy
were ideal for a simpler time, but might not be as good of a fit
today
Hm .... i guess it might depend on the specific use-case(s)
involved?
systems is that they're fundamentally
_text-oriented_. (Unsurprisingly, of course, given the context in
which Unix was developed.) i spend a lot of my time interacting
and working with text, and *n*x systems provide me with many
useful tools for this. Quoting the old "UNIX As Literature" piece,
https://theody.net/elements.html:
"[T]he most recurrent complaint was that [Unix] was too
text-oriented. People really hated the command line, with all the
utilities, obscure flags, and arguments they had to memorize. They
hated all the typing. One mislaid character and you had to start
over. Interestingly, this complaint came most often from users of
the GUI-laden Macintosh or Windows platforms. ...
"[A] suspiciously high proportion of my UNIX colleagues had
already developed, in some prior career, a comfort and fluency
with text and printed words. ...
"With UNIX, text — on the command line, STDIN, STDOUT, STDERR — is
the primary interface mechanism: UNIX system utilities are a sort
of Lego construction set for word-smiths. Pipes and filters
connect one utility to the next, text flows invisibly
between. Working with a shell, awk/lex derivatives, or the utility
set is literally a word dance."
Perl, with its pervasive regex-based functionality and extensive
Unicode support, fits neatly into this. i find regexes an
_incredibly_ powerful tool for working with text, whether via
Perl, sed, awk, or whatever. But my experience is that many people
treat regexes as an anathema, with Zawinski's "Now you have two
problems" regularly trotted out as a thought-terminating
cliché. Sure, regexes can, and do, get used where they shouldn't
be[a]; that doesn't mean the baby should be thrown out with the
bathwater.
But if one is only working with text under sufferance, trying to
avoid it via substantially more graphically-oriented environments,
the text-based "Unix philosophy" and the tools associated with it
might feel (and actually be) much less appropriate and
useful. Fair enough. The Unix construction set will still be there
for those of us who find them very appropriate and tremendously
useful.
Alexis.
[a] It seems unlikely that anyone on this list hasn't already seen
this, but just in case:
https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xh…
i'm looking forward to that comment sending OpenAI over the
Mountains of Madness.
11:23 a.m.
New subject: [COFF] Re: On Bloat and the Idea of Small Specialized Tools
Hoi.
On Sat, May 11, 2024 at 2:35 PM Theodore Ts'o
<tytso(a)mit.edu> wrote:
I bet most of the young'uns would not be trying to do this as a shell
script, but using the Cloud SDK with perl or python or Go, which is
*way* more bloaty than using /bin/sh.
So while some of us old farts might be bemoaning the death of the Unix
philosophy, perhaps part of the reality is that the Unix philosophy
were ideal for a simpler time, but might not be as good of a fit
today
It depends on what the Unix philosophy is seen to be. If it is
solving problems by reading text from standard in and printing to
standard out, then that might not be suitable anymore for many of
today's problems. But if it is prefering plain text to binary,
perfering simple solutions to complex ones, increasing the number
of operations one can perform by combining small generic parts,
... all because of good reasons ... Focussing on simplicity,
clarity, generality ... Omitting needless words! ... All this still
holds true, no matter if applied as shell scripts or within the
design of a new programming language or a programming interface.
It's not so much about the tools we use -- these should be suited
for the times you live in and the problems you have to solve --
but it's more about how you look at them and how you look at the
problems and what ideas for solutions you can imagine in your
mind. Here, Unix provides a continuing inspiration.
Only, like with every old book: when we read it today, we have to
read it within the background of the times back then and transfer
its message to today's times. The older the book, the more transfer
work has to be done, the more knowledgable the then younger and
more distant readers have to be, to really understand it.
Thus, in my oppinion, the Unix philosophy remains a good and very
relevant fit today, although not all of its applications from back
then still are.
meillo
12:18 p.m.
New subject: [COFF] Re: On Bloat and the Idea of Small Specialized Tools
On Sun, May 12, 2024 at 11:23 PM markus schnalke <meillo(a)marmaro.de> wrote:
It depends on what the Unix philosophy is seen to be. If it is
solving problems by reading text from standard in and printing to
standard out, then that might not be suitable anymore for many of
today's problems. But if it is prefering plain text to binary,
perfering simple solutions to complex ones, increasing the number
of operations one can perform by combining small generic parts,
... all because of good reasons ... Focussing on simplicity,
clarity, generality ... Omitting needless words! ... All this still
holds true, no matter if applied as shell scripts or within the
design of a new programming language or a programming interface.
It's not so much about the tools we use -- these should be suited
for the times you live in and the problems you have to solve --
but it's more about how you look at them and how you look at the
problems and what ideas for solutions you can imagine in your
mind. Here, Unix provides a continuing inspiration.
Only, like with every old book: when we read it today, we have to
read it within the background of the times back then and transfer
its message to today's times. The older the book, the more transfer
work has to be done, the more knowledgable the then younger and
more distant readers have to be, to really understand it.
Thus, in my oppinion, the Unix philosophy remains a good and very
relevant fit today, although not all of its applications from back
then still are.
I agree, but it seems that most Unix developers haven't really cared
since the side branches and clones effectively took over from Research
Unix in the early 80s. They've added system calls and ad-hoc socket
RPC interfaces with abandon instead of using generic filesystem-based
extensibility APIs, added options to various commands that should just
have been separate programs, and written desktop
environments/applications that have poor composability, extensibility
and modularity (I guess KDE's KParts kind of counts as a mechanism for
composing applications, but it's limited by being based on plugins
rather than an open IPC-based API). The only Unix desktop I can think
of that really tries to follow the Unix philosophy somewhat is the
now-abandoned Étoilé <http://etoileos.com/etoile/>. There's also the
desktops of the rather obscure BTRON family
<http://tronweb.super-nova.co.jp/btronproducts.html>, although those
OSes are only vaguely Unix-like. Both have an object-centric rather
than application-centric model with support for embedding applications
within each other and controlling them with RPC APIs.
IMO, the best practical realization of the Unix philosophy for the
modern era would be a QNX/Plan 9-like OS with an Étoilé/BTRON-like
desktop, hence why I'm working on one. Some of the specifics of the
original Unix philosophy may not be relevant to large parts of modern
computing, but I'd say the general ideas still are.
223
days inactive
224
days old
12 comments
9 participants
participants (9)
-
Adam Thornton -
Alexis -
Andrew Warkentin -
Dan Cross -
Dave Horsfall -
John Levine -
Larry McVoy -
markus schnalke -
Warner Losh