I did this with a really low end phone (a used 635 with 512mb ram that I got for 20 bucks) and left impressed. Definitely get the 640 if you can. Excellent value.
I love Android and still loved the experience. Being a 512mb ram device, browsing with Edge caused a lot of crashes and the apps that accumulate a lot of data (Aeries for Twitter and Readit for Reddit) would eventually crash, but that's to be expected. I'm not sure about the display on the 640, but the 635 had a garbage display that I couldn't wait to replace with the gorgeous AMOLED on the 950 XL.
So yes I did actually make the move, swapped over to Microsoft services for most things. Ad free YouTube with much nicer interfaces (you've got a million to pick from here) is a great perk.
As a fan of Android Holo, you know I appreciate the black theming here. ☺ Material is not something I like at all. With Microsoft you'll get the lovely option of choosing color themes (either uniformly or app specific), the best SD card support of the mobile OSs, a superb music experience that only lacks social features, and very frequent updates.
W10 Mobile has not been released to the public and is currently in bug fix mode, but it's more than capable as a daily driver. I figure its worth enjoying while it's still around. If the public doesn't pick up on it within the next few years, MSFT may very well write off the mobile division and it'll be back to the stagnating iOS and Android.