4 Dec
2024
4 Dec
'24
9:17 a.m.
I was looking at the question of “impact of Unix" and thought that:
initiating (Portable) Open Source Software including the BSD TCP/IP & Berkeley
sockets libraries,
the Unix -> Minix -> Linux -> Android sequence
and BSD -> NeXtstep -> OS/X -> iOS sequence,
plus running the Top 500 supercomputers and most of the Top 500 websites,
including the infrastructure for trillion dollars companies, Facebook, Amazon (Netflix
uses them), and Google
plus so many embedded Linux / NetBSD based appliances
even going into space - on small experiments or driving SpaceX’s Falcon 9 &
presumably Starship,
would be a slam-dunk for “really high impact”
- dominating everywhere important, except Windows desktops.
Unix wasn’t just a ’research project’, it was the result of years of work by a team of
very capable, professional programmers,
who weren’t looking for kudos or recognition, nor trying to add every conceivable
‘feature’ possible, but the inverse:
how _small_ could it be and still be enough.
When it left the Labs, Unix wasn’t just “Performant”, it came with a very useful set of
tools for creating other tools (‘developing’)
and while the kernel wasn’t perfect (some ‘panic’s), it was of “Production Quality”.
For v6, I believe there were patches for 50-100 bugs circulating, perhaps the first
demonstration of “no bug is intractable with ‘many eyeballs’”.
All that in under 5 years of ‘development’, with the “initial release” stable &
performant enough for cash-strapped Universities
to gamble their reputations & budgets on running it.
Imagine the grief academics would’ve got if their basic teaching systems failed
continuously!
This adoption path pushed Unix through another barrier:
’Security’ - with a lot of bright, bored & curious students banging on it as hard as
they could for bragging rights.
How long, in releases or years, did it take for other O/S’s to hit the “very stable”
benchmark?
I don’t know enough of Linux to answer that definitively, the *BSD’s grew there through
usage and contribution,
while Microsoft NT derivates widely suffered “Blue Screen of Death” for years.
Even now, MS-Windows has serious Security / compromise issues, like the highly visible,
global “Crowdstrike” event.
Not a break-in or takeover, but an own-goal from Security perimeter control.
==========
I now think Unix has had a much larger, direct and important impact
- the C language and associated tools & libraries
that begat modern toolchains and endless portability across platforms.
In 1991, Bill Plauger had a year sabbatical at UNSW in Sydney,
and happened to say :
“C is wallpaper - people expect it everywhere”.
C gained formal recognition with the POSIX standard, satisfying conservative users /
enterprises that it wasn’t the work of a bunch of Hippies or ill-disciplined Hackers.
Even Microsoft wrote 32-bit Windows NT in C, I presume starting by writing it’s own
compiler and toolchain to start.
Borland, Watcom and many others - including Microsoft - offered (Visual) C compile &
build environments for Windows,
directly responsible for creating the ’shrink-wrap’ third party software market that drove
sales of Windows and x86 machines.
Nobody had seen a market for a billion systems before, nor sold 300M+ CPU’s in a single
year.
People don’t buy Silicon Chips or nice Boxes, they buy Applications that solve their
problems:
Software drives Sales of Hardware
- something that IBM deeply understood first with first the 1401 line, then 360-series.
The other ’small’ achievement of C and Unix was creating the market for RISC chips.
MIPS in the mid-1980’s was only able to design and build the first commercial RISC chip
because it knew it could port Unix to it and find an immediate market
- not at zero-cost, but a tiny fraction of what every other Vendor had done before
reinventing the wheel from scratch to provide incompatible O/S & tools for their
hardware.
Unix on MIPS not only came with a host of proven software, that a large pool of people
knew how to use,
but it arrived as “Production Quality” - the porting team had to test their parts -
compiler, linker, libraries - hard, but could trust the existing high-quality codebase.
In "A New Golden Age for Computer Architecture”, 2019 by Hennessy & Patterson,
make an aside:
In today's post-PC era, x86 shipments have fallen almost 10% per year since the peak
in 2011,
while chips with RISC processors have skyrocketed to 20 billion.
Today, 99% of 32-bit and 64-bit processors are RISC.
i suggest this goes back to PCC followed by the MIPS R2000 - made possible by Dennis’ C
language.
The 1977 invention of ‘pcc’ and rewriting of Unix for cross-machine portability was the
first time I’m aware of this being done.
( Miller @ UoW did a one-off hack, not to devalue his work, he ported, didn’t invent a
multi-target portable compiler )
One of the effects of “portable C” was creating whole new industries for third party
software developers
or enabling niche products, like CISCO routers and the many embedded devices.
C and Unix came with the tools to create new languages and new tools.
AWK, sed (!) and shells are obvious examples, with Perl, Python & PHP very big in
Internet of 2000.
C was a new class of language - a tool to create tools.
It creates a perfect mechanism to bootstrap any new language, tool or product,
allowing to be refined & used enough to become reliable before being made
self-hosting.
Very widely used languages such as Python are written in C.
ORACLE achieved its market dominance by providing ‘portability’ - exactly the same on
every platform.
Underpinned by portable C.
The original 1127 team went on to create other systems and languages,
not the least being a new Software Engineering tool, “Go” / golang,
addressing a whole slew of deficiencies in the C/C++ approach and
We’d have no Internet today without Apache written in C and being ported to every
environment.
Also, there’s a connection between C and ‘modern’ Software Engineering
- distributed Repositories, automated builds & regression tests, and the many
toolchains and tools used.
They tended to be built in C to address problems (Open Source) developers were finding
with existing toolchains.
‘make’ arose at Bell Labs to automate builds, along with PWB and Writers Workbench.
There’s two questions / observations about 50 years of C in broad use:
- Just how much C is out there and used ‘in production’?
- C is ‘obviously’ a product of the 1970’s, not reflecting needs of modern hardware,
networks, storage and systems,
but _what_ can replace it?
There is simply too much critical code written in C to convert it to another ‘better,
modern’ language.
Any new language that is a simple 1:1 rewriting of C cannot address any of the
deficiencies,
while any incompatible language requires redesign and reimplementation of everything -
an unachievable goal.
The Linux Kernel’s “rust” project shows the extent of the problem
- even with the best team of the best developers, its a mammoth undertaking, with
uncertain payoffs, unquantifiable effort & deadlines.
My thesis is that portable, standard C:
- not only co-evolved with other tools & needs to create the Modern Software
Engineering environment, the basis for multiple Trillion dollar enterprises (FAANG)
but
- drove the biggest, most profitable software market ever seen (Wintel)
- which drove sales volume of x86 chips (& DRAM, motherboards, LAN, GPU, monitors,
peripherals…) over 2-3 decades,
- which drove Silicon Valley, paying for new generations of Fabs and lowering chip prices
further & further
- and eventually created the Fabless RISC CPU company,
which in the Post-PC era absolutely dominates chip sales.
No Software, no Silicon…
Gordon Moore, in an early comment on his 1968 startup with Robert Noyce, said:
“we are the real revolutionaries" (vs Hippies & 1967 Summer of Love).
I think Ken & Dennis [ and 1127/ Bell Labs folk ] can say the same.
==========
I’ve written some notes, with links to Programming Languages, especially Jean Sammet’s
Histories,
and would like some critiques, suggestions & corrections if people have time and
interest.
Unix and C are intimately related - neither was possible or useful without the other.
i think there’s an interesting article in there, but I’m not sure I have what it takes to
write it, not in a finite time :)
Very happy to help anyone who does!
Did-C-lang-create-modern-software-industry.txt
<https://drive.google.com/file/d/1k936sgqHc-vHBvfCdLoSxFhdT9NaijU2/view?usp=sharing>
steve jenkin
04 - dec - 2024
==========
--
Steve Jenkin, IT Systems and Design
0412 786 915 (+61 412 786 915)
PO Box 38, Kippax ACT 2615, AUSTRALIA
mailto:sjenkin@canb.auug.org.au http://members.tip.net.au/~sjenkin
4 Dec
4 Dec
7:05 p.m.
I would add one more important impact: making data a first class component
of the computing environment.
With the notion of pipes it became possible to operate on data quickly and
flexibly. There was nothing new from a fundamental capability point of
view, but the ease with which one could construct pipelines enabled rapid
experimentation and encouraged the development of pipe-able components to
add to the tool set.
I may not have articulated this as well as it deserves.
Marc
=====
nygeek.net
mindthegapdialogs.com/home <https://www.mindthegapdialogs.com/home>
On Wed, Dec 4, 2024 at 3:43 AM <sjenkin(a)canb.auug.org.au> wrote:
I was looking at the question of “impact of Unix"
and thought that:
initiating (Portable) Open Source Software including the BSD
TCP/IP & Berkeley sockets libraries,
the Unix -> Minix -> Linux -> Android sequence
and BSD -> NeXtstep -> OS/X -> iOS sequence,
plus running the Top 500 supercomputers and most of the Top 500
websites,
including the infrastructure for trillion dollars companies,
Facebook, Amazon (Netflix uses them), and Google
plus so many embedded Linux / NetBSD based appliances
even going into space - on small experiments or driving SpaceX’s
Falcon 9 & presumably Starship,
would be a slam-dunk for “really high impact”
- dominating everywhere important, except Windows desktops.
Unix wasn’t just a ’research project’, it was the result of years of work
by a team of very capable, professional programmers,
who weren’t looking for kudos or recognition, nor trying to add every
conceivable ‘feature’ possible, but the inverse:
how _small_ could it be and still be enough.
When it left the Labs, Unix wasn’t just “Performant”, it came with a very
useful set of tools for creating other tools (‘developing’)
and while the kernel wasn’t perfect (some ‘panic’s), it was of “Production
Quality”.
For v6, I believe there were patches for 50-100 bugs circulating, perhaps
the first demonstration of “no bug is intractable with ‘many eyeballs’”.
All that in under 5 years of ‘development’, with the “initial release”
stable & performant enough for cash-strapped Universities
to gamble their reputations & budgets on running it.
Imagine the grief academics would’ve got if their basic teaching systems
failed continuously!
This adoption path pushed Unix through another barrier:
’Security’ - with a lot of bright, bored & curious students
banging on it as hard as they could for bragging rights.
How long, in releases or years, did it take for other O/S’s to hit the
“very stable” benchmark?
I don’t know enough of Linux to answer that definitively, the *BSD’s grew
there through usage and contribution,
while Microsoft NT derivates widely suffered “Blue Screen of Death” for
years.
Even now, MS-Windows has serious Security / compromise issues, like the
highly visible, global “Crowdstrike” event.
Not a break-in or takeover, but an own-goal from Security perimeter
control.
==========
I now think Unix has had a much larger, direct and important impact
- the C language and associated tools & libraries
that begat modern toolchains and endless portability
across platforms.
In 1991, Bill Plauger had a year sabbatical at UNSW in Sydney,
and happened to say :
“C is wallpaper - people expect it everywhere”.
C gained formal recognition with the POSIX standard, satisfying
conservative users / enterprises that it wasn’t the work of a bunch of
Hippies or ill-disciplined Hackers.
Even Microsoft wrote 32-bit Windows NT in C, I presume starting by writing
it’s own compiler and toolchain to start.
Borland, Watcom and many others - including Microsoft - offered (Visual) C
compile & build environments for Windows,
directly responsible for creating the ’shrink-wrap’ third party software
market that drove sales of Windows and x86 machines.
Nobody had seen a market for a billion systems before, nor sold 300M+
CPU’s in a single year.
People don’t buy Silicon Chips or nice Boxes, they buy Applications that
solve their problems:
Software drives Sales of Hardware
- something that IBM deeply understood first with first the 1401
line, then 360-series.
The other ’small’ achievement of C and Unix was creating the market for
RISC chips.
MIPS in the mid-1980’s was only able to design and build the first
commercial RISC chip
because it knew it could port Unix to it and find an immediate market
- not at zero-cost, but a tiny fraction of what every other Vendor
had done before
reinventing the wheel from scratch to provide incompatible
O/S & tools for their hardware.
Unix on MIPS not only came with a host of proven software, that a large
pool of people knew how to use,
but it arrived as “Production Quality” - the porting team had to test
their parts - compiler, linker, libraries - hard, but could trust the
existing high-quality codebase.
In "A New Golden Age for Computer Architecture”, 2019 by Hennessy &
Patterson,
make an aside:
In today's post-PC era, x86 shipments have fallen almost 10% per
year since the peak in 2011,
while chips with RISC processors have skyrocketed to 20 billion.
Today, 99% of 32-bit and 64-bit processors are RISC.
i suggest this goes back to PCC followed by the MIPS R2000 - made possible
by Dennis’ C language.
The 1977 invention of ‘pcc’ and rewriting of Unix for cross-machine
portability was the first time I’m aware of this being done.
( Miller @ UoW did a one-off hack, not to devalue his work, he ported,
didn’t invent a multi-target portable compiler )
One of the effects of “portable C” was creating whole new industries for
third party software developers
or enabling niche products, like CISCO routers and the many embedded
devices.
C and Unix came with the tools to create new languages and new tools.
AWK, sed (!) and shells are obvious examples, with Perl, Python & PHP very
big in Internet of 2000.
C was a new class of language - a tool to create tools.
It creates a perfect mechanism to bootstrap any new language, tool or
product,
allowing to be refined & used enough to become reliable before being made
self-hosting.
Very widely used languages such as Python are written in C.
ORACLE achieved its market dominance by providing ‘portability’ - exactly
the same on every platform.
Underpinned by portable C.
The original 1127 team went on to create other systems and languages,
not the least being a new Software Engineering tool, “Go” / golang,
addressing a whole slew of deficiencies in the C/C++ approach and
We’d have no Internet today without Apache written in C and being ported
to every environment.
Also, there’s a connection between C and ‘modern’ Software Engineering
- distributed Repositories, automated builds & regression tests, and the
many toolchains and tools used.
They tended to be built in C to address problems (Open Source) developers
were finding with existing toolchains.
‘make’ arose at Bell Labs to automate builds, along with PWB and Writers
Workbench.
There’s two questions / observations about 50 years of C in broad use:
- Just how much C is out there and used ‘in production’?
- C is ‘obviously’ a product of the 1970’s, not reflecting needs
of modern hardware, networks, storage and systems,
but _what_ can replace it?
There is simply too much critical code written in C to
convert it to another ‘better, modern’ language.
Any new language that is a simple 1:1 rewriting of C
cannot address any of the deficiencies,
while any incompatible language requires redesign and
reimplementation of everything - an unachievable goal.
The Linux Kernel’s “rust” project shows the extent of the problem
- even with the best team of the best developers, its a mammoth
undertaking, with uncertain payoffs, unquantifiable effort & deadlines.
My thesis is that portable, standard C:
- not only co-evolved with other tools & needs to create the
Modern Software Engineering environment, the basis for multiple Trillion
dollar enterprises (FAANG)
but
- drove the biggest, most profitable software market ever seen
(Wintel)
- which drove sales volume of x86 chips (& DRAM, motherboards,
LAN, GPU, monitors, peripherals…) over 2-3 decades,
- which drove Silicon Valley, paying for new generations of Fabs
and lowering chip prices further & further
- and eventually created the Fabless RISC CPU company,
which in the Post-PC era absolutely dominates chip sales.
No Software, no Silicon…
Gordon Moore, in an early comment on his 1968 startup with Robert Noyce,
said:
“we are the real revolutionaries" (vs Hippies & 1967 Summer of
Love).
I think Ken & Dennis [ and 1127/ Bell Labs folk ] can say the same.
==========
I’ve written some notes, with links to Programming Languages, especially
Jean Sammet’s Histories,
and would like some critiques, suggestions & corrections if people have
time and interest.
Unix and C are intimately related - neither was possible or useful without
the other.
i think there’s an interesting article in there, but I’m not sure I have
what it takes to write it, not in a finite time :)
Very happy to help anyone who does!
Did-C-lang-create-modern-software-industry.txt
<
https://drive.google.com/file/d/1k936sgqHc-vHBvfCdLoSxFhdT9NaijU2/view?usp=…
steve jenkin
04 - dec - 2024
==========
--
Steve Jenkin, IT Systems and Design
0412 786 915 (+61 412 786 915)
PO Box 38, Kippax ACT 2615, AUSTRALIA
mailto:sjenkin@canb.auug.org.au http://members.tip.net.au/~sjenkin
7:40 p.m.
This alludes to an important step that should be mentioned. Plan 9 was a rethinking of
the whole Unix world. Though it didn’t spread like C, many of its ideas were adopted
elsewhere.
And its approach to multi-machine compilation and tools is clearly reflected in the
design
and implementation of golang.
ches
On Dec 4, 2024, at 8:05 AM, Marc Donner
<marc.donner(a)gmail.com> wrote:
The original 1127 team went on to create other systems and languages,
not the least being a new Software Engineering tool, “Go” / golang,
addressing a whole slew of deficiencies in the C/C++ approach and
9:02 p.m.
I would add one more important impact: making data a first class component
of the computing environment.
Related to that is getting rid of "file types". I remember at a "lessons
of
Unix" panel at a Usenix, that someone from the Labs gave an example of a
situation where the operating system treated a file as more than just a set
of bytes (think IBM, VMS, etc). Read a source file (required them to do a
bunch of file-conversion operations), write an output, perhaps changing a
variable name from 'foo' to 'bar', and then compiling the output,
which
also required a bunch of file-conversion operations.
5 Dec
5 Dec
9:08 a.m.
It appears that Marc Donner <marc.donner(a)gmail.com> said:
With the notion of pipes it became possible to operate
on data quickly and
flexibly. There was nothing new from a fundamental capability point of
view, but the ease with which one could construct pipelines enabled rapid
experimentation and encouraged the development of pipe-able components to
add to the tool set.
Pipes were invented at least three times I'm aware of, but what made them
work so well in Unix is that they looked to the program the same as a file
so any program could use them for input or output without special arrangements,
and the shell made it easy to start two programs and pipe them together.
The Dartmouth Time-Sharing System in the late 1960s had communication files
which were essentially two-way pipes, but they were asymmetrical. One end, the
slave end, looked like a file, but the other end, the master end, was different
and the program had to know it was a com file. They were mostly used to pass
terminal I/O between user programs at the slave end and SIMON at the master end,
the terminal monitor that talked to the front end computer than ran the TTYs.
They were invented again at IBM in the 1970s and described in this paper. I wrote
them a letter, which they published, saying that Unix pipes did the same thing.
https://dl.acm.org/doi/10.1147/sj.174.0383
R's,
John
9:19 p.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
On Wed, Dec 4, 2024 at 10:18 PM John Levine <johnl(a)taugh.com> wrote:
It appears that Marc Donner
<marc.donner(a)gmail.com> said:
Once you have coroutines and queues for passing data between them, a
lot of things start to look like pipes.
With the notion of pipes it became possible to
operate on data quickly and
flexibly. There was nothing new from a fundamental capability point of
view, but the ease with which one could construct pipelines enabled rapid
experimentation and encouraged the development of pipe-able components to
add to the tool set.
Pipes were invented at least three times I'm aware of, but what made them
work so well in Unix is that they looked to the program the same as a file
so any program could use them for input or output without special arrangements,
and the shell made it easy to start two programs and pipe them together. The Dartmouth Time-Sharing System in the late 1960s
had communication files
which were essentially two-way pipes, but they were asymmetrical. One end, the
slave end, looked like a file, but the other end, the master end, was different
and the program had to know it was a com file. They were mostly used to pass
terminal I/O between user programs at the slave end and SIMON at the master end,
the terminal monitor that talked to the front end computer than ran the TTYs.
Doug has written at some length on this list about communication files
and their homomorphism to the way Plan 9 presented and handled
resources.
They were invented again at IBM in the 1970s and
described in this paper. I wrote
them a letter, which they published, saying that Unix pipes did the same thing.
https://dl.acm.org/doi/10.1147/sj.174.0383
Don't forget CMS pipelines, too!
Sadly, the Morrison paper cited above is not easily accessible, though
I acquired a copy from IEEE; perhaps a sucker's game as it was not
cheap. Your subsequent letter, and part of Morrison's response to you,
however, is available, gratis.
Reading through Morrison, one gets the impression that there are some
substantial differences with Unix pipelines; or rather, the way that
Unix pipelines are usually used. In particular, he describes his
linked streams in terms of a "network", by which he appears to mean an
arbitrary directed graph. Crucially, he describes combining nodes, for
merging data from multiple streams.
Unix pipelines, on the other hand, tend to be used in a manner that is
strictly linear, without the fan-out and fan-in capabilities described
by Morrison. Of course, nothing prevents one from building a
Morrison-style "network" from Unix processes and pipes, though it's
hard to see how that would work without something like `select`, which
didn't yet exist in 1978. Regardless, Unix still doesn't expose a
particularly convenient syntax for expressing these sorts of
constructions at the shell.
As an aside, Morrison has a web page dedicated to "flow-based
programming", that he claims to have invented in the late 1960s. That
seems like a bit of a tall claim, and I'd wager Doug gives him a run
for his money on that. https://jpaulm.github.io/fbp/
- Dan C.
10 p.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
On Thu, 5 Dec 2024, Dan Cross wrote:
They also can look a lot like temporary files. Someone, probably Heinz,
did a shell for the tiny Unix that ran on floppies so this
foo | bar
actually did this
foo > tmpfile ; bar < tmpfile; rm tmpfile
to avoid having to swap programs in and out on floppies. The main
disadvantage was that the tmpfile could overflow the tiny disks of the
time.
If anyone else needs a copy, just ask.
Regards,
John Levine, johnl(a)taugh.com, Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly
Pipes were
invented at least three times I'm aware of, but what made them
work so well in Unix is that they looked to the program the same as a file
so any program could use them for input or output without special arrangements,
and the shell made it easy to start two programs and pipe them together.
Once you have coroutines and queues for passing data between them, a
lot of things start to look like pipes. They were
invented again at IBM in the 1970s and described in this paper. I wrote
them a letter, which they published, saying that Unix pipes did the same thing.
https://dl.acm.org/doi/10.1147/sj.174.0383
Don't forget CMS pipelines, too!
Sadly, the Morrison paper cited above is not easily accessible, though 
10:17 p.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
John, thanks for the reminder of the implementation
of pipes on a constrained version of UNIX in the early
days. The exact implementation is described on page 2095
of the BSTJ July-Aug 1978 for interested parties.
Heinz
On 12/5/2024 8:00 AM, John R Levine wrote:
On Thu, 5 Dec 2024, Dan Cross wrote:
They also can look a lot like temporary files. Someone, probably
Heinz, did a shell for the tiny Unix that ran on floppies so this
foo | bar
actually did this
foo > tmpfile ; bar < tmpfile; rm tmpfile
to avoid having to swap programs in and out on floppies. The main
disadvantage was that the tmpfile could overflow the tiny disks of the
time.
If anyone else needs a copy, just ask.
Regards,
John Levine, johnl(a)taugh.com, Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly
Pipes
were invented at least three times I'm aware of, but what made
them
work so well in Unix is that they looked to the program the same as
a file
so any program could use them for input or output without special
arrangements,
and the shell made it easy to start two programs and pipe them
together.
Once you have coroutines and queues for passing data between them, a
lot of things start to look like pipes.
They were
invented again at IBM in the 1970s and described in this
paper. I wrote
them a letter, which they published, saying that Unix pipes did the
same thing.
https://dl.acm.org/doi/10.1147/sj.174.0383
Don't forget CMS pipelines, too!
Sadly, the Morrison paper cited above is not easily accessible, though
11:06 p.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
I don't think files as pipes would be "transparent to the user." Reading
an
empty pipe causes a wait until the bytes requested are available, unless
the pipe is closed first. Reading to the end of a file results in an
end-of-file error. This problem is avoided if the source process completes
before the target process begins, but then there is a different lack of
transparency, which is that the processes don't run simultaneously. (I
think this is the case with the implementation that Heinz showed.)
Still, the same sort of pseudo-pipes were in MS-DOS, and they were
occasionally useful.
Marc
On Thu, Dec 5, 2024 at 9:17 AM Heinz Lycklama <heinz(a)osta.com> wrote:
John, thanks for the reminder of the implementation
of pipes on a constrained version of UNIX in the early
days. The exact implementation is described on page 2095
of the BSTJ July-Aug 1978 for interested parties.
Heinz
On 12/5/2024 8:00 AM, John R Levine wrote:
On Thu, 5 Dec 2024, Dan Cross wrote:
Pipes were invented at least three times I'm aware of, but what made them
work so well in Unix is that they looked to the program the same as a file
so any program could use them for input or output without special
arrangements,
and the shell made it easy to start two programs and pipe them together.
Once you have coroutines and queues for passing data between them, a
lot of things start to look like pipes.
They also can look a lot like temporary files. Someone, probably Heinz,
did a shell for the tiny Unix that ran on floppies so this
foo | bar
actually did this
foo > tmpfile ; bar < tmpfile; rm tmpfile
to avoid having to swap programs in and out on floppies. The main
disadvantage was that the tmpfile could overflow the tiny disks of the
time.
They were invented again at IBM in the 1970s and described in this paper.
I wrote
them a letter, which they published, saying that Unix pipes did the same
thing.
https://dl.acm.org/doi/10.1147/sj.174.0383
Don't forget CMS pipelines, too!
Sadly, the Morrison paper cited above is not easily accessible, though
If anyone else needs a copy, just ask.
Regards,
John Levine, johnl(a)taugh.com, Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly
--
*My new email address is mrochkind(a)gmail.com <mrochkind(a)gmail.com>*
11:53 p.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
There was no concurrency in mini-Unix; the first pipeline component ran to
completion, and when it exited the next component ran, and so on. In this
way a pipeline could be arbitrarily long using just two disk files.
On Thu, Dec 5, 2024, 12:07 PM Marc Rochkind <mrochkind(a)gmail.com> wrote:
I don't think files as pipes would be
"transparent to the user." Reading
an empty pipe causes a wait until the bytes requested are available, unless
the pipe is closed first. Reading to the end of a file results in an
end-of-file error. This problem is avoided if the source process completes
before the target process begins, but then there is a different lack of
transparency, which is that the processes don't run simultaneously. (I
think this is the case with the implementation that Heinz showed.)
Still, the same sort of pseudo-pipes were in MS-DOS, and they were
occasionally useful.
Marc
On Thu, Dec 5, 2024 at 9:17 AM Heinz Lycklama <heinz(a)osta.com> wrote:
John, thanks for the reminder of the
implementation
of pipes on a constrained version of UNIX in the early
days. The exact implementation is described on page 2095
of the BSTJ July-Aug 1978 for interested parties.
Heinz
On 12/5/2024 8:00 AM, John R Levine wrote:
On Thu, 5 Dec 2024, Dan Cross wrote:
Pipes were invented at least three times I'm aware of, but what made them
work so well in Unix is that they looked to the program the same as a
file
so any program could use them for input or output without special
arrangements,
and the shell made it easy to start two programs and pipe them together.
Once you have coroutines and queues for passing data between them, a
lot of things start to look like pipes.
They also can look a lot like temporary files. Someone, probably Heinz,
did a shell for the tiny Unix that ran on floppies so this
foo | bar
actually did this
foo > tmpfile ; bar < tmpfile; rm tmpfile
to avoid having to swap programs in and out on floppies. The main
disadvantage was that the tmpfile could overflow the tiny disks of the
time.
They were invented again at IBM in the 1970s and described in this
paper. I wrote
them a letter, which they published, saying that Unix pipes did the same
thing.
https://dl.acm.org/doi/10.1147/sj.174.0383
Don't forget CMS pipelines, too!
Sadly, the Morrison paper cited above is not easily accessible, though
If anyone else needs a copy, just ask.
Regards,
John Levine, johnl(a)taugh.com, Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly
--
*My new email address is mrochkind(a)gmail.com <mrochkind(a)gmail.com>*
6 Dec
6 Dec
12:05 a.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
It appears that Marc Rochkind <mrochkind(a)gmail.com> said:
-=-=-=-=-=-
-=-=-=-=-=-
I don't think files as pipes would be "transparent to the user." Reading
an
empty pipe causes a wait until the bytes requested are available, unless
the pipe is closed first. ...
You should read Meinz' paper. This was a cut down version of Unix that ran
in 40K bytes on an LSI-11 with a couple of floppy disks. It ran one program
at a time, since swapping to floppies was absurdly slow. The pseudo-pipe
ran the first program which wrote the output to a file, then ran the second
program which read it. It was close enough for most purposes so long as
the disk didn't fill up.
Still, the same sort of pseudo-pipes were in MS-DOS,
and they were
occasionally useful.
I can think of lots of places that had a way for one program to write a temporary
file that a subsequent one read. For example, in OS/360 JCL you'd write (well,
punch)
something like this for the object output of the Fortran compiler
//SYSLIN DD DSNAME=&OBJECT,DISP=(NEW,PASS),UNIT=SYSSQ
and then in the linker read it in
//SYSIN DD DSNAME=&OBJECT,DISP=(OLD,DELETE)
The & in the name said it was a temp file so make up a unique name. I probably
didn't
get the JCL exactly right since it's been about 50 years since I wrote any.
R's,
John
5 Dec
5 Dec
11:22 p.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
Regarding pipes and pipe-like interprocess communications facilities in
other operating systems, VMS has always had a pipe-like communications
pseudo-devices called mailboxes. The main difference between Unix pipes
and VMS mailboxes is that pipes have distinct read-only and write-only file
descriptors. Mailboxes do not--channels (VMS-speak for file descriptors)
assigned to a mailbox can be used for both reading and writing. This means
that it is not possible to do "broken pipe"-type detection on a mailbox.
This very much restricts the usefulness of mailboxes.
I wrote a true pipe device driver for VMS as part of the DEC Shell product
(a port of the Unix Bourne shell to VMS).
-Paul W.
6 Dec
6 Dec
12:19 a.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
I remember on MiniUnix (a stripped down kernel that would run on PDP11’s
without memory management, in our case 11/03, 11/20, and an older 11/40)
there were no kernel pipes (and limited process concurrency).
The shell used the aforementioned emulation of pipe by just running the
output of one process into a temporary file subsequently loaded as stdin
of the other.
We kind of put all that to bed when the 11/23s came out and we could run
our full up kernel on the micros.
8:29 a.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
/23+, no? The basic 23 doesn't have an MMU, I'm pretty sure.
On Thu, Dec 5, 2024 at 11:58 AM Ron Natalie <ron(a)ronnatalie.com> wrote:
I remember on MiniUnix (a stripped down kernel that
would run on PDP11’s
without memory management, in our case 11/03, 11/20, and an older 11/40)
there were no kernel pipes (and limited process concurrency).
The shell used the aforementioned emulation of pipe by just running the
output of one process into a temporary file subsequently loaded as stdin
of the other.
We kind of put all that to bed when the 11/23s came out and we could run
our full up kernel on the micros.
8 Dec
8 Dec
2:38 a.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
NO, the 11/23 has an MMU and was perflectly capable of running UNIX.
The 23+ had the extra address lines allowing it to address 4 MB like
the 11/70.
------ Original Message ------
From "Adam Thornton" <athornton(a)gmail.com>
To "Ron Natalie" <ron(a)ronnatalie.com>; "The Eunuchs Hysterical
Society"
<tuhs(a)tuhs.org>
Date 12/5/2024 9:29:24 PM
Subject Re: [TUHS] Re: Pipes (was Re: After 50 years, what has the
Impact of Unix been?)
/23+, no? The basic 23 doesn't have an MMU,
I'm pretty sure.
On Thu, Dec 5, 2024 at 11:58 AM Ron Natalie <ron(a)ronnatalie.com> wrote:
>I remember on MiniUnix (a stripped down kernel that would run on
>PDP11’s
>without memory management, in our case 11/03, 11/20, and an older
>11/40)
>there were no kernel pipes (and limited process concurrency).
>The shell used the aforementioned emulation of pipe by just running
>the
>output of one process into a temporary file subsequently loaded as
>stdin
>of the other.
>
>We kind of put all that to bed when the 11/23s came out and we could
>run
>our full up kernel on the micros.
>
5 Dec
5 Dec
10:55 p.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
On Thu, Dec 5, 2024 at 8:20 AM Dan Cross <crossd(a)gmail.com> wrote:
Unix pipelines, on the other hand, tend to be used in a manner that is
strictly linear, without the fan-out and fan-in capabilities described
by Morrison. Of course, nothing prevents one from building a
Morrison-style "network" from Unix processes and pipes, though it's
hard to see how that would work without something like `select`, which
didn't yet exist in 1978. Regardless, Unix still doesn't expose a
particularly convenient syntax for expressing these sorts of
constructions at the shell.
Rick Troth has recently published xfl, which is pretty much CMS Pipelines
for Unix.
https://github.com/trothtech/xfl
He's got a slide deck on it at
http://www.casita.net/pub/xfl/pervasive-vmws-2024.pdf .
There are a lot of really cool things you can do with fanin/fanout.
Adam
11:35 p.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
On 12/5/24 10:19 AM, Dan Cross wrote:
Unix pipelines, on the other hand, tend to be used in
a manner that is
strictly linear, without the fan-out and fan-in capabilities described
by Morrison. Of course, nothing prevents one from building a
Morrison-style "network" from Unix processes and pipes, though it's
hard to see how that would work without something like `select`, which
didn't yet exist in 1978. Regardless, Unix still doesn't expose a
particularly convenient syntax for expressing these sorts of
constructions at the shell.
Process substitution is about as close as we can get, but most programs
still process their filename arguments one at a time, beginning to end.
The canonical process substitution example is
diff <(old-program-version) <(new-program-version)
to do simple regression testing.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU chet(a)case.edu http://tiswww.cwru.edu/~chet/
6 Dec
6 Dec
2:55 a.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
Chet Ramey via TUHS <tuhs(a)tuhs.org> wrote:
On 12/5/24 10:19 AM, Dan Cross wrote:
And fanout is simply
... | tee >(pipeline1) >(pipeline2)
Arnold
Unix pipelines, on the other hand, tend to be
used in a manner that is
strictly linear, without the fan-out and fan-in capabilities described
by Morrison. Of course, nothing prevents one from building a
Morrison-style "network" from Unix processes and pipes, though it's
hard to see how that would work without something like `select`, which
didn't yet exist in 1978. Regardless, Unix still doesn't expose a
particularly convenient syntax for expressing these sorts of
constructions at the shell.
Process substitution is about as close as we can get, but most programs
still process their filename arguments one at a time, beginning to end.
The canonical process substitution example is
diff <(old-program-version) <(new-program-version)
to do simple regression testing.
3:12 a.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
On Thu, Dec 5, 2024 at 3:56 PM <arnold(a)skeeve.com> wrote:
Chet Ramey via TUHS <tuhs(a)tuhs.org> wrote:
And indeed these things are pretty nifty, but don't they generate
trees, and not arbitrary dags? They don't quite capture the full
generality of Morrison-style networks since it doesn't seem like
there's a way to connect process substitution fan-out with fan-in; at
least, not conveniently.
It is, perhaps, notable that Go allows me to do this sort of thing
with channels and goroutines, but it has `select` built into the
language.
Funny, despite using Unix almost daily for over 30 years now, I don't
think I've ever felt limited by the power of pipelines. On the
contrary, I've lost count of the times I've felt limited on systems
that do Not support pipes.
- Dan C.
On 12/5/24 10:19 AM, Dan Cross wrote:
And fanout is simply
... | tee >(pipeline1) >(pipeline2) Unix pipelines, on the other hand, tend to be
used in a manner that is
strictly linear, without the fan-out and fan-in capabilities described
by Morrison. Of course, nothing prevents one from building a
Morrison-style "network" from Unix processes and pipes, though it's
hard to see how that would work without something like `select`, which
didn't yet exist in 1978. Regardless, Unix still doesn't expose a
particularly convenient syntax for expressing these sorts of
constructions at the shell.
Process substitution is about as close as we can get, but most programs
still process their filename arguments one at a time, beginning to end.
The canonical process substitution example is
diff <(old-program-version) <(new-program-version)
to do simple regression testing.
3:50 a.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
Some of the posts in this thread confuse pipes in UNIX shells (the |
symbol) with the pipe system call.
In the shell, how two processes can be connected with a pipe is very
constrained (only one unidirectional pipe). But the pipe system call can be
used to build much more elaborate connections. Back in 1980, when I was
still at Bell Labs, I wrote a shell called 2dsh ("two dimensional shell)
that had a more complex syntax. The memo I wrote, "2DSH—An experimental
shell for connecting processes with multiple data streams", wasn't
published externally, but exists as a Bell Labs memo. I found a reference
here:
https://scholar.google.com/scholar_lookup?title=2DSH%E2%80%94An+experimenta…
.
Here are two examples from that memo:
[image: image.png]
I stumbled across another paper from 2017 titled "Extending Unix Pipelines
to DAGs," which references my un-published Bell Labs memo. I haven't read
it since I don't subscribe to IEEE Transactions on Computers.
A while ago Doug McIlroy was kind enough to send me a scan of my memo, but
I don't think I'm allowed to publish it here. In that memo, I credit Doug
for coming with a very similar idea around the same time ("A Notation for
Arboreal Plumbing").
Marc Rochkind
On Thu, Dec 5, 2024 at 2:13 PM Dan Cross <crossd(a)gmail.com> wrote:
On Thu, Dec 5, 2024 at 3:56 PM
<arnold(a)skeeve.com> wrote:
And fanout is simply
... | tee >(pipeline1) >(pipeline2)
And indeed these things are pretty nifty, but don't they generate
trees, and not arbitrary dags? They don't quite capture the full
generality of Morrison-style networks since it doesn't seem like
there's a way to connect process substitution fan-out with fan-in; at
least, not conveniently.
It is, perhaps, notable that Go allows me to do this sort of thing
with channels and goroutines, but it has `select` built into the
language.
Funny, despite using Unix almost daily for over 30 years now, I don't
think I've ever felt limited by the power of pipelines. On the
contrary, I've lost count of the times I've felt limited on systems
that do Not support pipes.
- Dan C.
--
*My new email address is mrochkind(a)gmail.com <mrochkind(a)gmail.com>*
Chet Ramey via TUHS <tuhs(a)tuhs.org> wrote:
> On 12/5/24 10:19 AM, Dan Cross wrote:
>
> > Unix pipelines, on the other hand, tend to be used in a manner that
is
> > strictly linear, without the fan-out
and fan-in capabilities
described
> > by Morrison. Of course, nothing
prevents one from building a
> > Morrison-style "network" from Unix processes and pipes, though
it's
> > hard to see how that would work without something like `select`,
which
didn't yet exist in 1978. Regardless, Unix still
doesn't expose a
particularly convenient syntax for expressing these sorts of
constructions at the shell.
Process substitution is about as close as we can get, but most programs
still process their filename arguments one at a time, beginning to end.
The canonical process substitution example is
diff <(old-program-version) <(new-program-version)
to do simple regression testing. 
4:03 a.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
On Thu, Dec 5, 2024 at 2:13 PM Dan Cross <crossd(a)gmail.com> wrote:
On Thu, Dec 5, 2024 at 3:56 PM
<arnold(a)skeeve.com> wrote:
And fanout is simply
... | tee >(pipeline1) >(pipeline2)
And indeed these things are pretty nifty, but don't they generate
trees, and not arbitrary dags? They don't quite capture the full
generality of Morrison-style networks since it doesn't seem like
there's a way to connect process substitution fan-out with fan-in; at
least, not conveniently.
It is, perhaps, notable that Go allows me to do this sort of thing
with channels and goroutines, but it has `select` built into the
language.
Funny, despite using Unix almost daily for over 30 years now, I don't
think I've ever felt limited by the power of pipelines. On the
contrary, I've lost count of the times I've felt limited on systems
that do Not support pipes.
The <() , >() syntax is a bash extension. Not all shells support it. And
I couldn't find them in POSIX Issue 8.
Warner
Chet Ramey via TUHS <tuhs(a)tuhs.org> wrote:
> On 12/5/24 10:19 AM, Dan Cross wrote:
>
> > Unix pipelines, on the other hand, tend to be used in a manner that
is
> > strictly linear, without the fan-out
and fan-in capabilities
described
> > by Morrison. Of course, nothing
prevents one from building a
> > Morrison-style "network" from Unix processes and pipes, though
it's
> > hard to see how that would work without something like `select`,
which
didn't yet exist in 1978. Regardless, Unix still
doesn't expose a
particularly convenient syntax for expressing these sorts of
constructions at the shell.
Process substitution is about as close as we can get, but most programs
still process their filename arguments one at a time, beginning to end.
The canonical process substitution example is
diff <(old-program-version) <(new-program-version)
to do simple regression testing.
4:19 a.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
On 12/5/24 5:03 PM, Warner Losh wrote:
The <() , >() syntax is a bash extension. Not
all shells support it. And
I couldn't find them in POSIX Issue 8.
Credit where credit is due: I picked them up from ksh93, and extended
them to use named pipes on systems where /dev/fd isn't available.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU chet(a)case.edu http://tiswww.cwru.edu/~chet/
5:07 a.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
I found that 2017 paper "Extending Unix Pipelines to DAGs". It's open
access:
https://ieeexplore.ieee.org/document/7903579
The open source code itself is here: https://github.com/dspinellis/dgsh
Maybe an ambitious TUHS contributor can get the code running and give us a
report.
Marc Rochkind
On Thu, Dec 5, 2024 at 3:38 PM Chet Ramey via TUHS <tuhs(a)tuhs.org> wrote:
On 12/5/24 5:03 PM, Warner Losh wrote:
--
*My new email address is mrochkind(a)gmail.com <mrochkind(a)gmail.com>*
The <() , >() syntax is a bash extension.
Not all shells support it. And
I couldn't find them in POSIX Issue 8.
Credit where credit is due: I picked them up from ksh93, and extended
them to use named pipes on systems where /dev/fd isn't available.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU chet(a)case.edu http://tiswww.cwru.edu/~chet/
2:16 p.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
On 06-Dec-24 01:07, Marc Rochkind wrote:
I found that 2017 paper "Extending Unix Pipelines
to DAGs". It's open
access:
https://ieeexplore.ieee.org/document/7903579 <https://
ieeexplore.ieee.org/document/7903579>
The open source code itself is here: https://github.com/dspinellis/dgsh
<https://github.com/dspinellis/dgsh>
Maybe an ambitious TUHS contributor can get the code running and give us
a report.
I wrote the dgsh code with my co-author Marios Fragkoulis, so I still
have it running. Doug McIlroy, who also mentioned dgsh in another
message, is too modest to say that its design owes much to his input. I
asked him for feedback when I was working on it, and over several
iterations he proposed important (and quite demanding as I recall)
improvements to its design.
The system allows the concise and readable expression of several graph
topologies I had in mind when I started working on it, and more [1].
However, it hasn't caught on. I think the main reason is that it is
based on modified versions of several existing tools (bash, cmp, comm,
cut, diff, diff3, grep, join, paste, perm, sort) [2]. The modifications
allow the tools to coordinate between them the setup of pipes when
placed in a dgsh graph according to available inputs and required
outputs. The changes (especially for bash) aren't small, which meant
that I didn't think it was realistic to push them upstream, which now
means that the modified tools are out of date and difficult to build.
Not sure what can be done to address this problem. It seems that a
widely adopted system, such as modern Unix/Linux, has too much inertia
for it to adopt potentially disrupting innovations.
In retrospect, the way we designed the pipe graph setup could also be
improved. The current design involves an initial phase where IPC
messages are circulating around the graph to communicate the I/O
requirements of each tool, for example that comm(1) should expect input
from two processes and output to three processes. The design is brittle
and difficult to troubleshoot, because coordination happens dynamically
behind the scenes. A better design (and one I think Doug was
advocating) would statically analyze the graph's topology and invoke
each tool with appropriate parameters or environment variables.
However, this design would require significantly more extensive
modifications to bash, or the implementation of a new shell. Both
approaches required work for which we didn't have the time and energy at
the time, and also had their own downsides regarding adoption potential.
[1] https://www.spinellis.gr/sw/dgsh/#examples
[2] https://www.spinellis.gr/sw/dgsh/#tools
Diomidis
6:46 a.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
Chet Ramey via TUHS <tuhs(a)tuhs.org> writes:
On 12/5/24 5:03 PM, Warner Losh wrote:
i must say i was moderately surprised to learn a few years ago
that process substitution wasn't in POSIX - i find it succinct
syntactic sugar. There's a post on Chris Siebenmann's blog about
some of the issues involved, including the /dev/fd issue:
https://utcc.utoronto.ca/~cks/space/blog/unix/ProcessSubstitutionWhyLate
Can anyone point me to any discussions about process substitution
potentially becoming part of POSIX? This search on the Austin
Group tracker didn't yield any useful results:
https://austingroupbugs.net/search.php?project_id=0&search=process+subs…
Alexis.
The <() , >() syntax is a bash extension.
Not all shells
support
it. And
I couldn't find them in POSIX Issue 8.
Credit where credit is due: I picked them up from ksh93, and
extended
them to use named pipes on systems where /dev/fd isn't
available.
7 Dec
7 Dec
3:46 a.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
On 12/5/24 7:46 PM, Alexis wrote:
Can anyone point me to any discussions about process
substitution
potentially becoming part of POSIX?
It's never come up in any serious way, maybe as a wish list item for
some people on the mailing list.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU chet(a)case.edu http://tiswww.cwru.edu/~chet/
6 Dec
6 Dec
5:07 a.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
Warner Losh <imp(a)bsdimp.com> wrote:
The <() , >() syntax is a bash extension. Not
all shells support it. And
I couldn't find them in POSIX Issue 8.
Warner
It originated in ksh93. 30 years have gone by, it's time it
was added to POSIX.
At least, IMHO.
Thanks,
Arnold
7:09 a.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
At 2024-12-05T16:07:10-0700, arnold(a)skeeve.com wrote:
Warner Losh <imp(a)bsdimp.com> wrote:
Are you sure? I think Tom Duff originated it in his "rc" shell.
https://archive.org/details/rc-shell
Regards,
Branden
The <() , >() syntax is a bash extension.
Not all shells support
it. And I couldn't find them in POSIX Issue 8.
It originated in ksh93. 
7:31 a.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
At Thu, 5 Dec 2024 19:09:26 -0600, "G. Branden Robinson"
<g.branden.robinson(a)gmail.com> wrote:
Subject: [TUHS] Re: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
[1 <text/plain; us-ascii (7bit)>]
At 2024-12-05T16:07:10-0700, arnold(a)skeeve.com wrote:
The Wikipedia entry for "Process substitution" says:
Process substitution was available as a compile-time option for
ksh88, the 1988 version of the KornShell from Bell Labs. The rc
shell provides the feature as "pipeline branching" in Version 10
Unix, released in 1990.
So, if we believe that then indeed ksh88 pioneered process substitution.
--
Greg A. Woods <gwoods(a)acm.org>
Kelowna, BC +1 250 762-7675 RoboHack <woods(a)robohack.ca>
Planix, Inc. <woods(a)planix.com> Avoncote Farms <woods(a)avoncote.ca>
Warner Losh <imp(a)bsdimp.com> wrote:
Are you sure? I think Tom Duff originated it in his "rc" shell. The <() , >() syntax is a bash extension.
Not all shells support
it. And I couldn't find them in POSIX Issue 8.
It originated in ksh93. 
8:05 a.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
On Thu, 5 Dec 2024, Greg A. Woods wrote:
The Wikipedia entry for "Process
substitution" says:
Process substitution was available as a compile-time option for
ksh88, the 1988 version of the KornShell from Bell Labs. The rc
shell provides the feature as "pipeline branching" in Version 10
Unix, released in 1990.
So, if we believe that then indeed ksh88 pioneered process substitution.
Confirming that ksh88d and ksh88i support <(command) at least.
-uso.
10:44 p.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
"Greg A. Woods" <woods(a)robohack.ca> wrote:
I'd forgotten that it was optional in ksh88. It was there out of
the box in ksh93.
Arnold
Are you sure?
I think Tom Duff originated it in his "rc" shell.
The Wikipedia entry for "Process substitution" says:
Process substitution was available as a compile-time option for
ksh88, the 1988 version of the KornShell from Bell Labs. The rc
shell provides the feature as "pipeline branching" in Version 10
Unix, released in 1990.
So, if we believe that then indeed ksh88 pioneered process substitution. 
4:05 a.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
On Dec 5, 2024, at 1:13 PM, Dan Cross <crossd(a)gmail.com> wrote:
And indeed these things are pretty nifty, but don't they generate
trees, and not arbitrary dags? They don't quite capture the full
generality of Morrison-style networks since it doesn't seem like
there's a way to connect process substitution fan-out with fan-in; at
least, not conveniently.
Isn’t that a limitation of the shell language(s)? It would be
easier in a 2D graphical shell but that would perhaps make
specifying the more common simpler pipelines harder!
A "linear" syntax would require *naming*. For example,
let-proc a = "foo -a", b = "bar -c -d", c = "zee" in
a.in1 = file1, a.in2 = file2, b.in1 = a.out, b.in2 = file3,
c.in1 = b.out1, c.in2 = a.out, c.out = file4;
Here names a, b & c are used to denote *processes*, with
connections specified in the next two lines. Once all the
pipes are created, processes a, b & c can be started giving
them the right endpoints in forked processes prior to execing.
Doable but not really worth it. At this point you may as well
use s-expressions!
8:02 a.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
According to Dan Cross <crossd(a)gmail.com>:
And indeed these things are pretty nifty, but don't they generate
trees, and not arbitrary dags? They don't quite capture the full
generality of Morrison-style networks since it doesn't seem like
there's a way to connect process substitution fan-out with fan-in; at
least, not conveniently.
You can use mkfifo to make named pipes and then plumb them as wanted. It's
not particularly beautiful but it's as general as you could want.
That's worked since 4.4BSD which was quite a while ago.
diff <(old-program-version)
<(new-program-version)
to do simple regression testing.
And fanout is simply
... | tee >(pipeline1) >(pipeline2)
8:21 a.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
On Thu, Dec 5, 2024 at 9:02 PM John Levine <johnl(a)taugh.com> wrote:
According to Dan Cross <crossd(a)gmail.com>:
And indeed these things are pretty nifty, but don't they generate
trees, and not arbitrary dags? They don't quite capture the full
generality of Morrison-style networks since it doesn't seem like
there's a way to connect process substitution fan-out with fan-in; at
least, not conveniently.
You can use mkfifo to make named pipes and then plumb them as wanted. It's
not particularly beautiful but it's as general as you could want.
That's worked since 4.4BSD which was quite a while ago.
Sure, but that's orthogonal since it's not really using shell syntax
to build the thing up in a canonical way. I mean, perhaps in some
literal sense it is (one types the commands into the interpreter...)
but nowhere near the expressiveness of `ls | grep foo | whatever`.
Fundamentally, the questions here are all about syntax. As I wrote in
my first email, the system calls are very general, and could be used
to build arbitrary directed graphs (not even limited to DAGs) of
processes connected by pipes in all sorts of configurations
(presumably those processes would be written to handle multiple input
and output streams), without having to resort to named pipes. One can
imagine some sort of DSL that describes such graphs of pipes and
processes (Bakul alluded to this), and an interpreter for that DSL
that creates the described set of pipes and processes, all
appropriately connected and running. One could even imagine such a
thing being used as part of a more traditional pipeline.
My claim is simply that extant shell syntax isn't really amenable to
this in a natural way. As Chet and Arnold pointed out, process
substitution as pioneered in ksh gets us a little closer, but it's not
quite as general (I believe any such graphs would have to be acyclic);
it's certainly not as syntactically pleasant.
There has been work along these lines; I was sent a reference off-list
to a paper by Spinellis and Fragkoulis about a DAG-oriented shell:
https://www.spinellis.gr/sw/dgsh/, which seems relevant.
- Dan C.
diff <(old-program-version)
<(new-program-version)
to do simple regression testing.
And fanout is simply
... | tee >(pipeline1) >(pipeline2)
10:46 p.m.
New subject: Pipes (was Re: After 50 years, what has the Impact of Unix been?)
Dan Cross <crossd(a)gmail.com> wrote:
My claim is simply that extant shell syntax isn't
really amenable to
this in a natural way. As Chet and Arnold pointed out, process
substitution as pioneered in ksh gets us a little closer, but it's not
quite as general (I believe any such graphs would have to be acyclic);
it's certainly not as syntactically pleasant.
There has been work along these lines; I was sent a reference off-list
to a paper by Spinellis and Fragkoulis about a DAG-oriented shell:
https://www.spinellis.gr/sw/dgsh/, which seems relevant.
In the early-to-mid 1980s, the Georgia Tech Software Tools Subsystem
for Prime Computers offered three standard inputs, three standard
outputs, and a shell with syntax to connect things as you desired.
I don't remember the syntax though. I also don't know how much
this feature was really used.
Arnold
15
days inactive
18
days old
34 comments
22 participants
participants (22)
-
Adam Thornton -
Alexis -
arnold＠skeeve.com -
Bakul Shah -
Chet Ramey -
Dan Cross -
Diomidis Spinellis -
G. Branden Robinson -
Greg A. Woods -
Heinz Lycklama -
John Cowan -
John Levine -
John R Levine
-
Marc Donner -
Marc Rochkind -
Paul Winalski -
Rich Salz -
Ron Natalie -
sjenkin＠canb.auug.org.au -
Steve Nickolas -
Warner Losh -
William Cheswick