Large firms that produced documentation, such as DEC, AT&T et al., and even some universities, might own their own gear, or they might send it out to be set.
The software varied greatly, depending on the target customer. For instance, by the early 80s, the Boston Globe's input system was still terrible - even though the computers had gotten better. I had a couple of friends working there, and they used to b*tch about it. But big newspapers (and I expect many other large publishers) were often heavy union shops on the back end (layout and presses), so the editors just wanted to set strips of "column wide" text as the layout was manual. I've forgotten the name of the vendor of the typesetter they used, but it was one of the larger firms -- IIRC, it had a DG Nova in it. My sister used CompuGraphic Gear, which was based on 8085's. She had two custom editing stations and the typesetter itself (it sucked). The whole system was under $35K in late-1970s money - but targeted to small newspapers like hers. In the mid-1908s, I got her a Masscomp at a reduced price and put 6 Wyse-75 terminals on it, so she could have her folks edit their stories with vi, run spell, and some of the other UNIX tools. I then reverse-engineered the floppy enough to split out the format she wanted for her stories -- she used a manual layout scheme. She still has to use the custom stuff for headlines and some other parts, but it was a load faster and more parallel (for instance, we wrote an awk script to generate the School Lunch menus, which they published each week).