Not necessarily so - imagine if you may, a phone running a new Atom Processor, one that is capable of docking to an external dock with all your peripherals attached to it like it was a desktop computer. Microsoft owns a patent on this very thing:
Microsoft patents smart smartphone dock
Now tell me x86 doesn't make sense in a phone. Especially with the advances in cloud computing, and cloud storage, there is no reason why the power of a desktop computer in your pocket is not that far away. I know you wont be using photoshop, or doing CAD work on a phone - but with a simple hdmi connection, you could be off and running, or with a dock you would have a full PC, IN YOUR POCKET.
Motorola and Asus run Android - literally the same apps your running on your phone. With a dockable Windows device, you have x86 in all its glory. I run some serious software on my Dell Venue 8 Pro - AutoCAD, Sketchup, Photoshop - They run totally fine, no lag at all (depending on the file sizes im working with).
All im saying, is that this would create an entire new market for Microsoft - think about the developing world - Even a cheaper ARM version would be able to run Full WinRT apps on a large display all through the smartphone. Office applications would scale well, IE would scale really well.