Re: [TUHS] PDP-11 legacy, C, and modern architectures

I agree that C is a bad language for parallelism, and, like it or not, that's what today's hardware is giving us -- not speed, but many independent processors. But I'd argue that its problem isn't that it is not low-level, but that it is not high-level enough. A language like MATLAB, whose basic data object is an N-diemsional tensor, can make impressive use of parallel hardware. Consider matrix multiplication. Multiplying two NxN arrays to get another NxN array is a classic data-parallel problem -- each value in the result matrix is completely independent of every other one -- in theory, we could dedicate a processor to each output element, and would not need any cache coherency or locking mechanism -- just let them go at it -- the trickiest part is deciding you are finished. The reason we know we are data parallel is not because of any feature of the language -- it's because of the mathematical structure of the problem. While it's easy to write a matrix multiply function in C (as it is in most languages), just the fact that the arguments are pointers is enough to make data parallelism invisible from within the function. You can bolt on additional features that, in effect, tell the compiler it should treat the inputs as independent and non-overlapping, but this is just the tip of the iceberg -- real parallel problems see this in spaces. The other hardware factor that comes into play is that hardware, especially memories, have physical limits in what they can do. So the "ideal" matrix multiply with a processor for each output element would suffer because many of the processors would be trying to read the same memory at the same time. Some would be bound to fail, requiring the ability to stack requests and restart them, as well as pause the processor until the data was available. (note that, in this and many other cases, we don't need cache coherency because the input data is not changing while we are using it). The obvious way around this is to divide the memory in to many small memories that are close to the processors, so memory access is not the bottleneck. And this is where C (and Python) fall shortest. The idea that there is one memory space of semi-infinite size, and all pointers point into it and all variables live in it almost forces attempts at parallelism to be expensive and performance-killing. And yet, because of C's limited, "low-level" approach to data, we are stuck. Being able to declare that something is a tensor that will be unchanging when used, can be distributed across many small memories to prevent data bottlenecks when reading and writing, and changed only in limited and controlled ways is the key to unlocking serious performance. Steve PS: for some further thoughts, see https://wavecomp.ai/blog/auto-hardware-and-ai