I picked up a Nexus 4 for $200 when they cut the price and have been comparing it with the 1020 lately. It's shockingly superior in many ways. The OS and apps are far more mature and flexible, and everything feels much faster. Even running the same apps side by side (e.g. Words By Post, Ruzzle, Skype), stuff happens significantly faster on the Nexus 4. Switching between apps is so much faster - two quick taps vs. tap and hold ... wait ... swipe, tap. The notification list is great. There's orientation lock. It's easier to get at important settings like wi-fi and Bluetooth. Closing apps is so much easier (although it doesn't even seem to be necessary to achieve great battery life). There have been no nasty surprises or instability so far, with much less care taken than with the 1020. It's WAY easier to select text with the magnification that it does. I can disable the lock screen. Working with tabs in Chrome is WAY easier than in IE. The Kindle app actually supports Japanese books (with dictionary!). I can run a DS emulator and it works great. The list just goes on and on and on.
From a physical perspective, the 1020 has a much better camera (of course) and the display is beautiful with a better viewing angle. The design of the phone is sexier. The power button is easier to press and I can double tap the screen to wake it up. However, the Nexus 4's smaller size and lighter weight are very nice, plus it has wireless charging built in!
And of course, the 1020 and Windows Phone do have some major advantages over Android like the live tiles + app list instead of pages of icons, free offline mapping + navigation that actually works well (when it doesn't freeze or kill the battery at least), the People hub, the Metro design language, freedom from Google's aggressive data mining, etc.
Anyway I don't want to make a full WP vs. Android post but I think way too many people around here just blindly bash Android every chance they get and ignore the many ways in which Android is superior. I wish I could have the best of WP, Android, and Nokia in one phone. What's it going to take? Some major work by MS to actually invest in this OS and address all the small but important pain points instead of just arrogantly having a "we know best" and "this is how it's going to be" attitude.
If you're not very patient, you might be better off sticking with your S4. Depending on how you use your phone and what it is about Android that you're sick of, you may find it difficult to stick with the switch.