Dan Cross: "... even if they didn't use Unix directly, it was an existence proof that such a thing was possible".
Indeed it was. IBM contracted with Interactive Systems (Heinz Lycklama's company, in Santa Monica) to produce PC/IX, which was complete System 3 for the IBM PC. 8088, 4.77MHz, and 512K of RAM (if I'm remembering the numbers correctly). It was my primary development system for the first edition of Advanced UNIX Programming.
As for why "IBM" didn't do something other than MS-DOS originally: It depends what you mean by "IBM". The PC was not originally strategic, although it might have become that way after a few years. It was just a small group in Boca Raton (as I recall) that whipped it out pretty quickly. MS-DOS was a good choice within the class of what then were known as personal computer operating systems (CP/M being the leader for 8080/Z80 Intel systems).
I don't think PC/IX would have run on a floppy-only system. And, if it would, it would have been a demonstration only--entirely impractical. IBM didn't provide a PC with a hard drive until later, and that's when PC/IX came along.
Don't forget that the IBM PC completely dominated office use where personal computers were needed. A runaway success. That makes me think that the technical solutions were correct for what the project was supposed to achieve.
When I tried to write responsive software for UNIX and UNIX-like OSes running on PCs, I could never achieve good results because the display support was inadequate. Typically, you treated the display like a terminal (escape sequences). MS-DOS allowed me to write to display memory directly, which what was made PC software so responsive.
To say it another way, UNIX on a PC was always just a port. No consideration was given to providing support for the way PCs were actually used. That didn't happen until Xerox PARC started to produce PCs (at a much higher cost, of course). The first decent "PC" was the Macintosh SE. The earlier Macs were dogs.
--Marc