For a brief time a long time ago, I used a 4.3BSD based Mt. Xinu, MACH
microkernel, OS on the IBM-RT as an alternative to AOS. Ran well
enough, but was disk and memory constrained. We had source to much of
the system (or perhaps all of it, don't remember), but I seem to recall
that compiling it was a big pain. Something like you had to use a
specific compiler (perhaps referred to as High C?? hc command perhaps)
to compile some of the source. gcc had a backend for the ROMP
processor, but it had a hard time making usable binaries. I think that
some variation of pcc was the usual compiler. I remember it being
pretty stock 4.3BSD with NFS and minus YP/NIS. We used them mostly as X
terminal workstations.
"High C" (or perhaps "Hi C"? It's been a while...) was the name of the system compiler on AOS; I thought it was installed as `cc`. I don't recall a pcc-derived compiler, but apparently such a thing did exist and some documentation says that High C was installed as `hc`, so my memory may be off. This old post describes RT compilers:
https://groups.google.com/d/msg/comp.sys.ibm.pc.rt/u7DUwY5U9kQ/uVqLP9FhqMEJ
Hi-C was sort of an odd compiler. I gather IBM outsourced the development of it to some third party (MetaWare) which was founded by very religious people, and I have a vague memory of some of the documentation or perhaps even error messages making biblical references.
The kernel had to be built with High C, if I recall correctly, though GCC worked OK for producing userspace binaries. I don't recall what the bug was, but it was eventually found and fixed. Perhaps it had to do with incomplete register saves on function entry interacting poorly with interrupts or something.
Some RT enthusiasts kept those machines running well beyond their prime. Why? I'm not entirely sure; as you say, they were memory and disk constrained. They were also very slow. Anyway, I have some vague recollection that at some point the bug in the compiler was fixed so that GCC could produce a working kernel; nascent NetBSD and OpenBSD ports were planned, but I don't think they ever went anywhere.
https://www.openbsd.org/romp.html exists, though I don't know that the NetBSD people ever got beyond the talking stage. The OpenBSD-romp mailing list had some interesting information, but I can't find archives anymore.
Oh well. The RT was an interesting footnote in the history of computing, but it seems that, as a workstation, it was too little too late by the time it actually hit the market. Had they released it a few years earlier? Perhaps they could have cornered the market.
- Dan C.