I'm no expert so take this with a grain of salt, but on WP 8.0 at least I believe the biggest contributing factor to an app's resume time had to do with Silverlight restoring the app's UI layout. It never seemed to keep the app's layout in memory, either that or it would always re-measure and re-arrange the layout at resume, which takes time. I believe that's why with some apps you would always see "resuming..." even if you only leave the app for 2 seconds, because Silverlight would discard its layout every time you leave it. It's been like this since 7.5, and I don't think it has anything to do with whether or not devs enabled fast resume ("fast resume" in WP terms doesn't refer to resume time last time I checked).
Of course this would affect apps with more complex layouts. If you have an app that always shows "resuming..." when you return to it, try going to a page with a simpler layout (like the settings) and it would probably resume much faster. Generally, if you tap on an element that's supposed to take you to a new page in an app, and that page takes a second or two to appear instead of appearing instantly, then you will probably see the "resuming..." screen if you try to return to the app while it's on that page.
I haven't tested many WP 8.1 apps besides my own, but I know that behaviour still exists in WP 8.0 apps on 8.1, and it's worse now since as WP 8.1 has been very trigger-happy when it comes to killing app processes after a short period of non-use, even if all you did was lock the screen. This should be less of an issue as more apps are upgraded to WinRT (and maybe Silverlight 8.1?), since it uses much less memory so apps are less likely to be terminated to free up resources. I also haven't noticed my WinRT app struggling to restore its layout on resume, so maybe that's been fixed? Or maybe WinRT's always has to restore the layout at resume as well, except that it's much more efficient at doing it.
Don't forget to take this with a grain of salt!