[COFF] Unix Philosophy + Networks == Plan 9 [ LONG ]

This report [link at end ] about a security issue with VMware Vsphere, stemming from the design/ architecture, resonated with me and the recent TUHS “Unix Philosophy” thread. Many of the criticisms of Unix relate to not understanding it’s purpose and design criteria: A platform on which to develop (other) Software. Which implies ‘running, profiling, testing & debugging’ that code. Complaining that Unix tools/utilities are terse and arcane for non-developers & testers, needing a steep Learning Curve, is the same as complaining a large truck doesn’t accelerate or corner like a sports car. Plan 9, by the same core team twenty years later, addresses the same problems with modern hardware & graphics, including with Networking. The system they developed in 1990 would’ve been proof against both vSphere attacks because of its security-by-design: No ‘root’ user, hence no ’sudo’ and no complex, heavyweight RPC protocol with security flaws, instead the simple, lightweight & secure 9P protocol. It seems Eric Raymond’s exposition on the “Unix Philosophy” is the basis of much of the current understanding / view. In the ESR & other works cited on Wikipedia, I see a lot about “Userland” approaches, nothing about the Kernel, Security by Design and innovations like ’shells’, ‘pipes’ and the many novel standard tools, which is being able to Reuse standard tools and ’stand on the shoulders of giants’ [ versus constantly Reinventing the Wheel, poorly ] ESR was always outside CSRC and from his resume, not involved with Unix until 1983 at best. He’s certainly been a mover & shaker in the Linux and associated (GNU led) Open Source community. <http://catb.org/~esr/resume.html> ESR baldly states "The Unix philosophy is not a formal design method”, which isn’t strictly untrue, but highly misleading IMHO. Nor is the self-description by members of CSRC as having “good taste” a full and enlightening description of their process. There’s not a general appreciation, even in Research & Academic circles, that “Software is Performance Discipline”, in the same way as Surgery, Rocketry, Aviation, Music, Art and physical disciplines (dance, gymnastics, even rock climbing) are “Performance” based. It requires both Theory and Practice. If an educator hasn’t worked on at least one 1M LOC system, how can they teach “Programming in the Large”, the central problem of Software Engineering? [ an aside: the problem “golang” addressed was improving Software Engineering, not simply a language & coding. ] There’s a second factor common to all high-performance disciplines, why flying has become cheaper, safer and faster since the first jet & crashes in 1950’s: - good professionals deliberately improve, by learning from mistakes & failures and (perhaps) adopting better practices, - great professionals don’t just ‘improve’, they actively examine how & why they created Errors, Faults & Failures and detect / remove root causes. The CSRC folk used to hate Corporate attempts at Soft Skills courses, calling them “Charm School”. CSRC's deliberate and systematic learning, adaption and improvement wasn’t accidental or incidental, it was the same conscious approach used by Fairchild in its early days, the reason it quickly became the leader in Silicon devices, highly profitable, highly valued. Noyce & Moore, and I posit CSRC too, applied the Scientific Method to themselves and their practices, not just what their research field. IMO, this is what made CSRC unique - they were active Practitioners, developing high-quality, highly-performant code, as well as being astute Researchers, providing quantifiably better solutions with measurable improvements, not prototypes or partial demonstrators. Gerard Holtzman’s 1127 Alumni page shows the breadth & depth of talent that worked at CSRC. The group was unusually productive and influential. [ though I’ve not seen a ‘collected works’ ] <http://spinroot.com/gerard/1127_alumni.html> CSRC/1127 had a very strong culture and a very deliberate, structured ‘process’ that naturally led to a world-changing product in 1974 from only ~30 man-years of effort, a minor effort in Software Projects. perfective “iterative design”, rigorous testing, code quality via a variation of pair-programming, collaborative design with group consultation / discussion and above all “performant” code - based first on ‘correct’ and ’secure’, backed by Doug McIlroy’s insistence on good documentation for everything. [ It’s worth noting that in the original paper on the “Waterfall” development process, it isn’t "Once & Done”, its specifically “do it twice”, ] [ the Shewhart Cycle, promoted by Deming, Plan - Do - Check - Act, was well known in Engineering circles, known to be very Effective ] Unix - the kernel & device drivers, the filesystem, the shell, libraries, userland and standard tools - weren’t done in hurry between 1969 & 1974’s CACM article. It was written and rewritten many times - far more than the ‘versions’, derived from the numbering of the manuals, might suggest. Ken’s comment on one of his most productive days, “throwing away 1,000 lines of code”, demonstrates this dynamic environment dominated by trials, redesign and rewriting - backed by embedded ‘instrumentation’ (profiling). Ken has also commented he had to deliberately forget all his code at one point (maybe after 1974 or 77). He was able to remember every line of code he’d written, in every file & program. I doubt that was an innate skill, even if so, it would’ve improved by deliberate practice, just as in learning to play a musical instrument. There’s a lot of research in Memory & Recall, all of which documents ‘astonishing’ performance by ‘ordinary’ people, with a only little tuition and deliberate practice. CSRC had a scientific approach to software design and coding, unlike any I’ve seen in commercial practice, academic research or promoted “Methodologies”. There’s a casual comment by Dennis in “Evolution of Unix”, 1979, about rewriting the kernel, improving its organisation and adding multiprogramming. By one person in months.. A documented, incontestable level of productivity, 100x-1000x programmers practising mainstream “methodologies”. Surely that performance alone would’ve been worthy of intensive study as the workforce & marketplace implications are profound. <https://www.bell-labs.com/usr/dmr/www/hist.pdf> Perhaps the most important watershed occurred during 1973, when the operating system kernel was rewritten in C. … The success of this effort convinced us that C was useful as a nearly universal tool for systems programming, instead of just a toy for simple applications. The CSRC software evolution methodology is summed by perfectly in Baba Brinkman’s Evolution Rap: "Performance, Feedback, Revision” <https://www.youtube.com/watch?v=gTXVo0euMe4> Website: <https://bababrinkman.com/> ABC Science Show, 2009, 54 min audio, no transcript This is the performance Baba gave at the Darwin Festival in Cambridge England, July 2009. <https://www.abc.net.au/listen/programs/scienceshow/the-rap-guide-to-evolution/3094260> Ken also commented that they divided up the work coding, seemingly informally but in a disciplined way, so that there was only ever one time they created the same file. [ "mis-coordination of work”, Turing Award speech ] To prove they had well defined coding / naming standards and followed them, the two 20-line files were identical… ——————— There’s a few things with the “Unix Philosophy” that are critical and not included in the commentaries I’ve read or seen quoted: - The Unix kernel was ‘conservative’, not inventive or novel. It deliberately used only known, proven solutions, with a focus on small, correct, performant. “Just Worked”, not “Worked, Just”. Swapping was used, while Virtual Memory not implemented because they didn’t know of a definitive solution. They avoided the “Second System Effect” - showing how clever they were - working as professional engineers producing a robust, reliable, secure system. - Along with Unix (kernel, fsys, userland), CSRC developed a high-performance high-quality Software Development culture and methodology, The two are inseparable, IMO. - Professionals do not, can not, write non-trivial code in a “One and Done” manner. Professional quality code takes time and always evolves. It takes significant iterative improvement, including redesign, to develop large systems, with sufficient security, reliability, maintainability and performance. [ Despite 60 years of failed “Big Bang” projects using “One & Done”, Enterprises persist with this idioticy, wasting billions every year ] - Unix was developed to provide CSRC with a great environment for their own work. It never attempted to be more, but has been applied ‘everywhere’. Using this platform, members of the team developed a whole slew of important and useful tools, now taken as a given in Software Development: editors, type settings, ‘diff’ and Version Control, profile, debug, … This includes the computer Language Tools, now core to every language & system. - Collaboration and Sharing, both ways, was central to the Unix Philosophy developed at CSRC. Both within the team, within Bell Labs and other Unix installations, notably USENIX & UCB and it’s ARPA-IPTO funded CSRG. The world of Software and Code Development is clearly in two Eras, “Before Unix” and “After”. Part of this is “Open Source”, not just shared source targeted for a single platform & environment, but source code mechanically ported to new platforms. This was predicated on the original CSRC / Bell Labs attitude of Sharing the Source… Source was shared in & out, directly against the stance of the Legal Dept, intent on tightly controlling all Intellectual Property with a view of extracting “revenue streams” from clients. Later events proved CSRC’s “Source Code Sharing” was far more powerful and profitable than a Walled Garden approach, endlessly reinvesting the wheel & competing, not cooperating with others. Senior Management and the old school lawyers arguably overestimated their marketing & product capability and wildly underestimated the evolution of computing and failed to understand completely the PC era, with Bill Gates admonisment, “You guys don’t get it, it’s all about Volume”. In 1974, Unix was described publicly in CACM. In 1977, USG then later Unix System Labs was formed to work on and sell Unix commercially, locking day the I.P., with no free source code. In 1984, AT&T ‘de-merged’, keeping Bell Labs, USL and Western Digital - all the hardware and software to “Rule the World” and beat IBM. In 1994, AT&T gave up being the new IBM and sold its hardware and software divisions. In 2004, AT&T was bought by one of its spinoff’s, SBC (Southern Bell), who’d understood Mobile Telephony (passing on to customers savings from new technology), merged and rebranded themselves as “A&T”. The “Unix Wars” of the 1990’s, where vendors bought AT&T licenses, confusing “Point of Difference” with “Different & Incompatible”. They attempted Vendor lock-in, a monopoly tactic to create captive markets that could be gouged. This failed for two reasons, IMO: - the software (even binaries) and tools were all portable, the barriers to exit were low. - Unix wasn’t the only competitor Microsoft used C to write Windows NT and Intel-based hardware to undercut Unix Servers & Workstations by 10x. Bill Gates understood ‘Volume’ and the combined AT&T and Unix vendors didn’t. ================ VMware by Broadcom warns of two critical vCenter flaws, plus a nasty sudo bug <https://www.theregister.com/2024/06/18/vmware_criticial_vcenter_flaws/> VMware's security bulletin describes both of the flaws as "heap-overflow vulnerabilities in the implementation of the DCE/RPC protocol” … DCE/RPC (Distributed Computing Environment/Remote Procedure Calls) is a means of calling a procedure on a remote machine as if it were a local machine – just the ticket when managing virtual machines. ================ CHM, 2019 <https://computerhistory.org/blog/the-earliest-unix-code-an-anniversary-source-code-release/> As Ritchie would later explain: “What we wanted to preserve was not just a good environment to do programming, but a system around which a fellowship could form. We knew from experience that the essence of communal computing, as supplied from remote-access, time-shared machines, is not just to type programs into a terminal instead of a keypunch, but to encourage close communication.” ================ Ken Thompson, 1984 Turing Award paper Reflections on Trusting Trust To what extent should one trust a statement that a program is free of Trojan horses? Perhaps it is more important to trust the people who wrote the software. That brings me to Dennis Ritchie. Our collaboration has been a thing of beauty. In the ten years that we have worked together, I can recall only one case of mis-coordination of work. On that occasion, I discovered that we both had written the same 20-line assembly language program. I compared the sources and was astounded to find that they matched character-for-character. The result of our work together has been far greater than the work that we each contributed. ================ The Art of Unix Programming by ESR <http://www.catb.org/~esr/writings/taoup/html/index.html> Basics of the Unix Philosophy <http://www.catb.org/~esr/writings/taoup/html/ch01s06.html> ================ Wiki ESR <https://en.wikipedia.org/wiki/Eric_S._Raymond> Unix Philosophy <https://en.wikipedia.org/wiki/Unix_philosophy> ================ -- Steve Jenkin, IT Systems and Design 0412 786 915 (+61 412 786 915) PO Box 38, Kippax ACT 2615, AUSTRALIA mailto:sjenkin@canb.auug.org.au http://members.tip.net.au/~sjenkin