12 Feb
2020
12 Feb
'20
12:57 a.m.
---------- Forwarded message ---------
From: Adam Thornton <athornton(a)gmail.com>
Date: Tue, Feb 11, 2020 at 5:56 PM
Subject: Re: [COFF] Old and Tradition was [TUHS] V9 shell
To: Clem Cole <clemc(a)ccc.com>
As someone working in this area right now....yeah, and that’s why
traditional HPC centers do not deliver the services we want for projects
like the Vera Rubin Observatory’s Legacy Survey Of Space and Time.
Almost all of our scientist-facing code is Python though a lot of the
performance critical stuff is implemented in C++ with Python bindings.
The incumbents are great at running data centers like they did in 1993.
That’s not the best fit for our current workload. It’s not generally the
compute that needs to be super-fast: it’s access to arbitrary slices
through really large data sets that is the interesting problem for us.
That’s not to say that that lfortran isn’t cool. It really really is, and
Ondrej Cestik has done amazing work in making modern FORTRAN run in a
Jupyter notebook, and the implications (LLVM becomes the Imagemagick of
compiled languages) are astounding.
But...HPC is no longer the cutting edge. We are seeing a Kuhnian paradigm
shift in action, and, sure, the old guys (and they are overwhelmingly guys)
who have tenure and get the big grants will never give up FORTRAN which
after all was good enough for their grandpappy and therefore good enough
for them. But they will retire. Scaling out is way way cheaper than
scaling up.
On Tue, Feb 11, 2020 at 11:41 AM Clem Cole <clemc(a)ccc.com> wrote:
moving to COFF
On Tue, Feb 11, 2020 at 5:00 AM Rob Pike <robpike(a)gmail.com> wrote:
My general mood about the current standard way of
nerd working is how
unimaginative and old-fashioned it feels.
...
But I'm a grumpy old man and getting far off topic. Warren should cry,
"enough!".
-rob
@Rob - I hear you and I'm sure there is a solid amount of wisdom in your
words. But I caution that just, because something is old-fashioned, does
not necessarily make it wrong (much less bad).
I ask you to take a look at the Archer statistics of code running in
production (Archer large HPC site in Europe):
http://archer.ac.uk/status/codes/
I think there are similar stats available for places like CERN, LRZ, and
of the US labs, but I know of these so I point to them.
Please note that Fortran is #1 (about 80%) followed by C @ about 10%,
C++ @ 8%, Python @ 1% and all the others at 1%.
Why is that? The math has not changed ... and open up any of those codes
and what do you see: solving systems of differential equations with linear
algebra. It's the same math my did by hand as a 'computer' in the
1950s.
There is not 'tensor flows' or ML searches running SPARK in there. Sorry,
Google/AWS et al. Nothing 'modern' and fresh -- just solid simple science
being done by scientists who don't care about the computer or sexy new
computer languages.
IIRC, you trained as a physicist, I think you understand their thinking. *They
care about getting their science done.*
By the way, a related thought comes from a good friend of mine from
college who used to be the Chief Metallurgist for the US Gov (NIST in
Colorado). He's back in the private sector now (because he could not
stomach current American politics), but he made an important
observation/comment to me a couple of years ago. They have 60+ years of
metallurgical data that has and his peeps have been using with known
Fortran codes. If we gave him new versions of those analytical programs
now in your favorite new HLL - pick one - your Go (which I love), C++
(which I loath), DPC++, Rust, Python - whatever, the scientists would have
to reconfirm previous results. They are not going to do that. It's not
economical. They 'know' how the data works, the types of errors they
have, how the programs behave* etc*.
So to me, the bottom line is just because it's old fashioned does not make
it bad. I don't want to write an OS in Fortran-2018, but I can wrote a
system that supports code compiled with my sexy new Fortran-2018 compiler.
That is to say, the challenge for >>me<< is to build him a new
supercomputer that can run those codes for him and not change what they are
doing and have them scale to 1M nodes *etc*..
_______________________________________________
COFF mailing list
COFF(a)minnie.tuhs.org
https://minnie.tuhs.org/cgi-bin/mailman/listinfo/coff