"And the key for all of this is motivation and incentives, as any good
historian will tell you.  This is true whether probing the start of
wars, or the decline of a technical community or tradition."


I work for the Large Synoptic Survey Telescope.  I'm in the Data Management group, and specifically in the Science Quality and Reliability Engineering team.  The 50,000 foot view of what we do is try to bring software engineering to astronomical software.

In general, the thing about scientific software is that, to put it crudely, no one gets a Nobel Prize for software.  There's a very strong incentive to write a thing that will solve whatever particular problem you need solved for your paper, and no more.  There's also the (highly correlated) problem that, to an established researcher, graduate student labor is free, and your graduate students want to finish their thesis, not engineer quality software.

Whereas what I'd like to do is factor the common infrastructure--of which there is a lot--out of the various teetering stacks of special-purpose software and create some sane and maintainable infrastructure that individual researchers can easily and relatively gracefully extend to answer their specific questions.


On Wed, Jun 26, 2019 at 8:12 AM Theodore Ts'o <tytso@mit.edu> wrote:
On Tue, Jun 25, 2019 at 07:56:46PM -0700, Larry McVoy wrote:
> > It might not be, but it is definitely relevant to Unix.  Arguably the
> > drivers of Unix's development movement away from R&D-focused places and
> > toward product-oriented entities had at least a little to do with
> > Larry's topic of complaint.  Product managers gained the ammunition to
> > demand sustainable development practices, while R&D got a little leaner,
> > a little more focused on demonstrating the thesis, a little less focused
> > on who might need to run this code five years on...
> In the good old days at Sun, we were very focussed on who would run
> this code for decades to come.  I think the engineers at Sun were very
> focussed on helping people, the reason we were there was because the
> work we did helped people.  The leverage was how much work we could
> do versus how much that helped people.  That is product oriented.
> I think the reason that any engineer works is because they feel like
> their work helps someone.  As an engineer, I wanted to go to the place
> and do the work that had the best chance of helping someone.  All of
> Sun, when I was there, was like that.  We were there to help.  Yeah,
> of course, we wanted to make money, but all of us wanted to help.
> It's the dream, you do work, your work helps.

Motivations and incentives are a very big and important aspect which
is often overlooked in large scale projects.

For example, one of the really big problems with device drivers in the
embedded space is that the team that works on SOC version X gets
disbanded, and immediately reassigned to SOC verison X+1, sometimes
before product has even shipped.  Having one device driver that works
for SOC versions N, N+1, N+2, ... N+5, is really important from a
maintainability and being able to send out bug fixes for security
flaws.  However, it means that whenever you make changes, you need to
test on N different older versions.  And between the need to release
product quickly, and the fact that engineers are !@#@! expensive, and
the teams constantly getting formed and reformed, it's much easier to
do code reuse by copying, and so you have N different versions of a
device driver in a Board Support Package version of the Linux kernel
shipping by a SOC vendor.

Unfortunately, I have to disagree with Larry, there are many, many
engineers who works because they get a paycheck, and so they go home
at 5pm.  Some people might be free to improve their code on their own
time, or late at night, but corporation also preach "work/life
balance" --- and then don't fund time for making code long-term
maintainable or reducing tech debt.

Open source helps because embarassment can be a great motivator, but
more important are the fact that there are people who are empowered to
say "no" who don't work for the corporation who is trying to cut
corners, and who have a higher allegiance to the codebase than their

There is a similar related issue around publishing papers to document
great ideas.  This takes time away from product development, and it
used to be that Sun was really prolific at documenting their technical
innovations at conferences like Usenix.  Over time, the academic
traditions started dying off, and managers who came from that
tradition moved on, retired, or got promoted beyond the point where
they could encourage engineers to do that work.  And it wasn't just at
Sun; I was working at IBM when IBM decided to take away the (de
minimus) bonus for publishing papers at conferences.  But at the
Usenix board, I remember looking at a chart of the declining number of
ATC papers coming from industry over time.   And it was very depressing...

And the key for all of this is motivation and incentives, as any good
historian will tell you.  This is true whether probing the start of
wars, or the decline of a technical community or tradition.

                                                 - Ted