I personally went from the Galaxy S3 (which I eventually rooted), to a Lumia Icon, to the Nexus 6, and finally to the 950 XL. I think both OSes have their pros and cons, but I always lean towards Windows though.
While my experience with the Nexus 6 was generally pretty good, it did have it's own annoyances. Like the 950 XL, a number of them were fixed with either OS or app updates. I ended up having one really annoying bug that locked the phone completely during calls. I could not wake the screen, increase the volume, end the call or anything. I basically had to hope the other person would hang up or hold the power button down until the phone restarted. I believe that was fixed after the Marshmallow update happened, but I had to wait almost two months for the "flagship" Nexus phone to get that update (even though it was still about 4 months sooner than any other phone).
Android has become less and less open source and much more like iOS seemingly. I really enjoy the general look and feel of notifications, the action center, and in general the material design of the newest versions of Android, but beyond that they have really lost me. There is less overall control over the device, the seemingly necessity of having the battery saver mode, the constant (yet generally slow) drain of the battery, the heavy handed forcing of Google Now on you (where as Cortana generally seems like a choice), etc.
I end up a little biased because I prefer Microsoft services like OneDrive and Office over Google Drive. Live Tiles are a huge plus for me. Android has always just seemed like putting a bunch of desktop icons on a phone screen. I've said it more than a few times on here, but the #1 feature I depend on from Windows Phone is the ability to have Cortana read out text messages through my car stereo. That is one hell of a feature and it's honestly shocking that Android and iOS don't have this feature. Lumia phones have also standardly had much better cameras and that keeps me with them too. And now, with the WaaS philosophy, the idea of not having to wait for OS updates is incredibly appealing.
Just my thoughts though!