[TUHS] Re: Origins of the SGS (System Generation Software) and COFF (Common Object File Format)

On 2/22/23, Clem Cole clemc(a)ccc.com wrote: I can't think of any reason why ar(1) would care about the file format or internal contents of any of the modules it archives. ar(1) is a general archiving tool and can archive anything. It happens that the designers of ld(1) decided to use ar(1) to provide searchable object file libraries. ranlib(1) is a different matter. In order to index global symbols it has to understand the object file format(s) of the modules it is indexing. ranlib(1) most certainly would have to be taught to understand COFF. But not ar(1). a.out was, as object file formats go, a throwback to the stone age from the get-go. Even the most primitive of IBM's link editors for System/360 supported arbitrary naming of object file sections and the ability for the programmer to arrange them in whatever order they wished. a.out's restriction to three sections (.text, .data, .bss) did manage to get the job done, and even (with ZMAGIC) could support demand-paged virtual memory, but only just. It became pretty clear in the 1980s that an object file format more powerful and flexible than a.out was needed. CMU developed their own object file format (MACH-O) for their MACH microkernel-based OS. It had up to 8 object file sections, and the section properties (e.g., read vs. read/wrkte; executable vs. data) were not tied to the section name as in a.out. A big step forward, although still primitive compared to the object formats of VAX/VMS and the IBM S/370 OSes. Apple MacOS X still uses MACH-O for object files and executables. Whatever its origins, what we now know as COFF (Common Object File Format) is, as its name implies, intended to be OS- and machine-independent. It still has a relatively small number of sections, albeit more than MACH-O. When Microsoft developed Windows NT, they needed to replace their own MZ executable format with something that could support shareable images and they decided to go with COFF for both object files and for executables. In typical Microsoft embrace-and-extend fashion, their Portable Executable and Common Object File Format (PECOFF) is a heavily modified version of COFF with lots of MS-specific extensions. When DEC's GEM back end was chosen as the optimizer and code generator for Microsoft C/C++ on Windows NT for the DEC Alpha chip, I had to add PECOFF support to GEM's existing COFF support (which was used by DEC's commercially sold compilers for Ultrix). My original idea was to put the PECOFF support under conditional compilation (#ifdef PECOFF), but the two formats were sufficiently different that I abandoned that Idea, cloned the existing COFF module, and then modified that to create a separate PECOFF module. ELF is far more flexible than either COFF, PECOFF, or MACH-O. Those three make a distinction between sections (the bits that eventually end up in memory) and the metadata pieces of an object file or executable (program headers, symbol table, debug information, etc.). In ELF, everything is a section, including the symbol table and the tables that direct the program loader in mapping shareable images into a process's memory. ELF was originally limited to 64K sections (section numbers were unsigned 16-bit), but there is now a scheme for 32-bit section numbers. The essentially unlimited number of sections is a big boon to languages such as C++, where grouped sections with a name-decoration convention provide a convenient way to support sharing of class definitions without requiring language-specific tweaks to the software development toolset. Contrast this with the Ada implementations I'm aware of, which have their own software development library systems layered on top of the conventional compiler/linker/archiver to insure that program modules are compiled and linked in the correct order. I don't know what the timeline for the invention of COFF was. It was already called COFF and in widespread use by the time I encountered it when we added Ultrix support to GEM. I think MACH-O predated COFF; it's certainly more primitive than COFF. MACH-O was probably early to mid-1980s. OS kernel bloat was a recognized problem at the time and microkernel-based OSes were all the rage. At DEC, Dave Cutler wrote a microkernel-based OS called VAXeln to replace VAX/VMS for real-time applications. A lot of concepts from VAXeln found their way into Windows NT when Cutler left DEC for Microsoft. -Paul W.