I just installed it last night, upgrading from 10536. 10536 was so slow that it was unusable, even after hard reset. 10581 is actually good and usable. Doesn't seem to be as fast as 8.1, so for a 520 daily driver, I might hold off a bit, but in my opinion, it's usable. Not sure how overall battery life is though (my 520 is a spare/test phone).
After 10536, I was worried about the future of W10M. Sure it's a low end phone from 2012, but 8.1 was still solid on it, while 10536 wasn't. The OS was buggy, and slow. Really slow. Like, I was questioning their W10M support for Snapdragon S4 as a whole. But now, thanks to 10581, I'm very pleased to see it's come a long way. Still some polish needed (mostly things like unifying app UIs), but it's looking good. Tempted to put it on my 920, but will probably keep my wife's 640 on 8.1 until RTM (I can live with minor bugs, she can't).
Honestly though, if your DD is a 520, and you live in the USA, grab a Lumia 640 for $60 and put W10M on that. It's a great phone for dirt cheap and a *huge* step up from a 520. This is, of course, unless you're waiting for a 950...
TL;DR - not as fast as 8.1, but still solid and usable if you have to have the latest/greatest.