I finally got a chance to talk to someone who knows a hell of a lot about the i386 than I could ever hope to know. I gave him all the materials and I think he spent more time replying to my email than doing the debugging.
Basically the registers for entering protected mode with paging are backwards. This is kind of funny as the port was done by Intel of all people.
Anyway I reversed them and I now have the Mach kernel from 1988 booted under VMware.
I have to say that it's super cool to finally have chased this one down.
