My memory was that the 68000 gave the 8086 a pretty good run for its
money, but when Moto came out with a memory management chip it had some
severe flaws that made paging and fault recovery impossible, while the
equivalent features available on the 8086 line were tolerable. There were
some bizarre attempts to page with the 68000 (I remember one product that
had two 68000 chips, one of which was solely to sit on the shoulder of the
other and remember enough information to respond to faults!). By the time
Moto fixed it, the 8086 had taken the field...
I got a lot more respect for the 8086 architecture when working at
Transmeta. The instruction set encoding means that programs are small,
and that means that, for a given icache size, the cache hit rate was much
better than for our (wide word) machine. By the time we had upped the
size of our caches, the increased area and cost made our chip much less
competitive.