I'm currently using a Nexus 5 running Android 4.4.2 as my daily driver. Am also using a HTC 8X running WP 8.1 PDP as my secondary device so I'm really platform agnostic. Here's my opinion..
Android 4.4 Is butter smooth for the most part. It's as fluid as iOS or Windows Phone. There's no noticeable lag. What makes Android famous for its 'lag' is the OEM's customizations. Like Touchwiz and Sense (to a lesser extent). The Nexus 5 obviously runs of vanilla android so I don't experience any significant lag.
In many ways, Android 4.4 is ahead of WP 8.1. There are more things you can do on Android then on WP. To me WP 8.1 = Android 4.1. Microsoft has done a great job with WP 8.1 but it still isn't as feature packed as Android 4.4.
Since using my Nexus 5, I've never missed any features on my HTC 8X except for the People Hub integration with FB and Twitter. Whatever I don't like on Android, I can simply download app to remedy my problem. Customization is an integral part of Android that I absolutely love.
Moving forward, I hope WP gets more customization options. e.g. setting default apps for playing music, videos web browsing etc.
Make no mistake though, I still support WP and will get one (probably the Lumia 930) in the near future. I would also urge WP fanboys to try out a Google branded devices to experience the real power of Android without any lags