On Fri, Aug 31, 2018 at 5:36 PM Cág <ca6c(a)bitmessage.ch> wrote: Yikes -- this is like Rolls Royce cutting a deal with GM and produce a new car based on the Chevy Sububuran, painting it, adding a few details and calling it a Silver Ghost. SunOS != is not Solaris ᐧ

On Fri, Aug 31, 2018 at 3:36 PM Cág <ca6c(a)bitmessage.ch> wrote: No. Not even close. SunOS is BSD based (first 4.2 then 4.3 then with some small amount of System V code) with a written from scratch vm. Solaris is sun's do-over based on System V Release 4. It's great leap backwards. It was a huge slap in the face of the old SunOS crew by inept management. The final indignity was SunOS 4.1 being rebranded Solaris 1.0, which was pure marketing... Warner

On Fri, Aug 31, 2018 at 3:59 PM Larry McVoy <lm(a)mcvoy.com> wrote: It's reason #1 why Netflix can't use ZFS: the double copy problem... Warner

On 8/31/2018 3:19 PM, Cág wrote: Are you quite sure? I remember using BSD on a VAX in about 1978 when vi just came out. I'm pretty sure Bill Joy wrote the man page and other documentation. Maybe it depends on what you mean by "his original vi". Jon

On 8/31/2018 3:30 PM, Cág wrote: Those of us who used pre-open-source BSD probably have a different recollection of what BSD was like than those who came later. Jon

Forgot to add: thanks for the good read, Larry! -- caóc

On Fri, Aug 31, 2018 at 07:02:36PM -0400, Arthur Krewat wrote: I did some updating of lmbench [*] when I was dancing with Netflix. I need to check that in and publish that because the bits have rotted since 1995. I have no idea if that will measure what you want, it's a bunch of microbenchmarks that measure bandwidth and latency of everything I could think of: network, disks, file systems, caches, memory, etc. It's interesting but might not be so to you. If you want to know if Solaris is at the same place as Linux, I can pretty much promise it is not in the "getting out of the way of the application". Solaris was known as Slowaris for a reason. Yeah, I know it sucked harder in the beginning, but those were order of magnitude suckages. Linus always cared about performance, he had a big hand in lmbench, we both cared. Linus was about performance like I am about code size. I ran the BitKeeper dev team for almost 20 years. I loved a changeset that removed more lines than it added. We got up to about 120K lines of code in the core and then we stayed there for the next decade, added so much more stuff but always found a way to delete stuff so we didn't get bloated. But all that said, you need to be specific about what perf you care about. These days the kernel is far more complicated, NUMA etc, and you might care about parallel make (I doubt it) or you might care about Oracle or something like that. --lm [*] http://mcvoy.com/lm/papers/lmbench-usenix.pdf

I am surprised how good Sun's technical marketing was for you to think this. Linux has scaled better since the early 2000s. The Solaris x86-64 port has some real gaffes in it to this day at least as visible in the OpenSolaris derivative codebases, serialization in the locking primitives kind of things. On Fri, Aug 31, 2018 at 8:23 PM, Theodore Y. Ts'o <tytso(a)mit.edu> wrote:

below... On Sat, Sep 1, 2018 at 12:35 PM Larry McVoy <lm(a)mcvoy.com> wrote: Amen.... ᐧ

On 9/1/2018 12:27 PM, Kevin Bowling wrote: Thanks :) - I'm basing this on Oracle database performance, for the most part, and it's weird way of supporting NUMA on Linux in a bass-ackwards sort of way. Nothing I see in the latest RedHat/CentOS tells me it even cares about NUMA, but maybe that's more of their "we know better than you" mentality and it's all hidden under the covers somewhere. ak

On Sat, Sep 1, 2018 at 3:19 PM, Theodore Y. Ts'o <tytso(a)mit.edu> wrote: Sorry this is just bogus about being weak compared to Solaris. Are you looking back with rosy glasses or have you scanned the code in the past couple years? I have and there is nothing particularly special about Solaris internals here or elsewhere. In fact, there are a lot of pessimization all over the place. As Larry said, a lot of folks in the Linux community clearly cared about performance. Although the Solaris code is fairly clean It's not clear Sun valued performance at all. A stroll through arch/*/include/asm/ was enough to convince me of Larry's claims. I'm not a Linux fanboy but credit goes where it's due. Solaris has lgroups, which are a clean design but that is the extent of its NUMA support, one shot at placement and scheduling. Linux has a NUMA allocator, aware scheduler, NUMA-optimized spinlocks and mutexes, various subsystems correctly use the primitives, and can use cgroups to contain or gang things. There is a userland policy tool called numad that tries to add some additional runtime affinity and movement policy decisions. I agree that architecturally Linux NUMA is nowhere near where Sequent and especially SGI was. And the reasons you cite are valid, Linux implementation is good for maybe 8-16 sockets of modern core count with a much tighter off chip network than the big dogs were building. Keep in mind IBM wants to sell RockHoppers and E980s (4 drawers, 16 sockets, 768 threads) for dedicated Linux use which have similar north/south and east/west off chip networks. They have a lot of very talented people on the firmware, kernel, compilers to make these things work fast, including Paul. Windows' NUMA support is on par with Solaris insofar as there is domain aware memory allocator and scheduler hierarchy that takes the domain (and SMT etc) into account. What Windows lacks is the finely tuned concurrency primitives and everything else Linux has done.. which Solaris lacks as well. I'm not even talking about RCU (or epoch based reclamation or proxy collection or hazard pointers, at least one of which is not patent encumbered), I'm just talking about the quality of primitives like spinlocks and mutex and rwlock. There are big tradeoffs to the implementations of these in terms of fairness, progress guarantees, and thread scalability. Linux leads the pack by a long shot in this department. Where you start going beyond Linux-like NUMA IMO is when you get Irix-like features of page copying, migration, and multiple advanced placement policies. Agreed, this is why x86 has dominated the server market for a long time. Agreed. Those big margins must have caused them to take their eye off what mattered right at the time Linux was getting some momentum from the big HW vendors. The E10k was only a 64-core machine on a tight backplane compared to other large systems. It didn't have any of the pressing needs that Sequent and SGI did with multi-drawer interconnects to drive excellence in NUMA. These are strange times. Intel's been putting out some real doozy chips. The mesh in Skylake is a partial improvement over the dual rings of Haswell (though they did some goofy things to increase latency in undesirable ways), and they aren't going to continue to brute force it like IBM did with their 17 metal layer process node.. many SKUs in Cascade Lake will be a dual die design and cost a hilarious amount of money. AMD's EPYC is really bad in this department too, one EPYC behaves identically to a 4 socket system with extremely poor inter-die latency [1]. I think POWER9 is universally better and the high bin chips (22 core, 88 thread, mega cache) are only around $2500 compared to Skylake's absurd $12,000. POWER9 is a single on chip NUMA domain for 24 cores. Google publicly stated they are using it for GPU servers, and that all their monorepo is built for multiple ISAs. Through the grapevine I've heard gmail is running on POWER9 as well now. That is pretty competent, the reason Intel is sucking so bad is because people allowed themselves such lock in. A hyperscaler should be able to change between a couple ISAs as needed between purchasing cycles. True. There is also at least one unencumbered strategy such as epoch based reclamation which was known about around that time [2] [1] https://www.servethehome.com/amd-epyc-infinity-fabric-latency-ddr4-2400-v-2… [2] https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-579.pdf Regards, Kevin

On Sun, Sep 2, 2018 at 12:43 PM, Theodore Y. Ts'o <tytso(a)mit.edu> wrote: One would hope so, but that was the strategy that got them eaten by a grue. Another funny anecdote about this aloofness.. Linux on sparc64 uses the Relaxed Memory Order mode that the hardware offers . Solaris.. Total Store Order. There are tons of things like this in the code that blow my mind. I would have been pissed if I were on the hardware side of SPARC. Hmm, I think they've been pretty earnest at wanting to be 2+ years ahead of the general market with POWER for as long as I can see, lots of HPC money has been subsidizing that. Depends on the workload but bus and memory bandwidth right now with PCIe Gen4 and NvLink can really cut down on server sprawl. I've met with the GM/chief architect and they see OpenPOWER positioned as a full frontal competitor to Intel Xeon. I'm fairly disappointed in my contemporaries for not recognizing the value of a completely open source firmware and on chip controller stack; especially after the recent snafu where Intel changed the microcode license to disallow benchmarks and claimed it was an accident. Your statements make sense to me with respect to AIX, as Linux has been the main effort since the 2000s. GPFS looks neat, I wish it were open or at least internals documented well enough to study the implementation academically. That's probably the dirty little secret. It's long been profitable to carefully plan software interrupt handlers, user threads, and memory allocation even on pedestrian servers if they are running a fixed function. I guess Google's Borg and the new workalikes could do semi-automagic things with cgroups these days. There is evidence of people getting pretty crazy with it when we see things like Intel cache allocation features. Agreed, one of these companies is doing pretty well with a fat dividend yield, that other has basically been dismantled for all but a couple remaining desirable platform control points like Java and MySQL. Many things in tech are happy accidents and a small number of motivated people at the right place and time. A Sun engineer admitted on some video I've seen that the green light was really given for ZFS because they got stumped by some UFS bugs.. once enough of ZFS was written to test the end to end checksumming features they found out some of these heisenbugs were LSI HBA and disk firmware issues :o) Surveying some of these filesystems.. JFS2 is a decent, nowhere near the capabilities of ZFS but even today it's not in dire need of replacement.. I suspect another issue complementary to your point is the standalone storage business is many $B of revenue. ESS/DS8000 and the like are preferred revenue. IBM and HP were more in the SAN game than Sun and SGI who let the customers configure systems themselves be used as storage (Sun was using VxFS for a long time, SGI had some CXFS things IIRC). Tru64 had a pretty interesting filesystem on paper, curious if you ever looked at its design since they open sourced it. Regards, Kevin

On Tue, Sep 4, 2018, 13:40 Gilles Gravier <gilles(a)gravier.org> wrote: I feel like we've been through this before on this list, but perhaps it bears repeating: just because you can find source (or binaries, or CD images, etc.) on the internet, that doesn't make it the least bit legal. That is clearly the case here. Sadly, there are even higher profile sites like the Internet Archive that have this problem too. The concept of "abandonware" has no legal footing whatsoever. -Henry

On 9/5/2018 2:31 AM, Gilles Gravier wrote: Bash/Shellshock kinda proves that premise incorrect, although it's pretty much the worst-case example, but still...  ;) Announced in 2014, it goes back to September 1989 (according to a wikipedia article, so I'm not sure about that date's accuracy). https://en.wikipedia.org/wiki/Shellshock_(software_bug) https://www.cvedetails.com/product/47/Linux-Linux-Kernel.html?vendor_id=33 https://www.cvedetails.com/product/17/IBM-AIX.html?vendor_id=14 https://www.cvedetails.com/product/20/HP-Hp-ux.html?vendor_id=10 https://www.cvedetails.com/product/19755/Oracle-Solaris.html?vendor_id=93 It could be argued that the above CVE results are either under-reported (closed-source), or over-reported (open-source). Or vice-versa ;) ak

On 9/5/18 11:26 AM, Warner Losh wrote: It proves that if there is someone who has an idea, or who thinks about a thing in new ways, he can verify his suspicions without too much trouble. The barrier to investigation is lowered. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRU chet(a)case.edu http://tiswww.cwru.edu/~chet/

On Wed, 5 Sep 2018, Gilles Gravier wrote: One of my pet hates is people who use "Reply All" because either they are too lazy to edit the Reply (and they top-post, too), or the list is set up that way. I really don't need my own personal copy, as well as a list copy. Oh, and people who are too lazy to trim their replies too, because they are encouraged to top-post. -- Dave

When I started work at Cambridge in 2002 our central supercomputer was being replaced with a cluster of Sun Fire E15K machines with a fancy interconnect - it topped out at position 199 on the top500 list https://www.top500.org/list/2003/06/?page=2 with a 300 core configuration. It looks like they never managed to get the whole thing working as a single cluster since the other two thirds of the installation had positions 200 and 201! (The Nov. 2002 top500 list has it in 6 x 144 core shards.) Here’s a news item about it: https://www.cnet.com/news/sun-expands-supercomputer-effort/ The big benchmarks in this lovely thesis were run on one of the E15K supercomputer boxes :-) Tony. -- f.anthony.n.finch <dot(a)dotat.at> http://dotat.at