The problem is that "browse music" and "play music" are different apps, just like the "settings" app and the individual settings are different.
So, your scenario actually involved three apps. First, you started "browse music" by tapping the Xbox Music tile. Then, you started "play music" by playing some music. Then, you pressed the Windows key to go back to the Start screen and started a third app.
When using the task switcher to bring the "play music" app to the foreground, its "back stack" came with it. Unfortunately, its back stack is only ever one page. Pressing the Back key tries to go back, but since there's only one page, the back key effectively leaves the app (with background music playing) and returns to the last app that you were in, which in this case is really the Start screen.
If you had chosen the "browse music" app while in the task switcher, you'd have been able to use the back key to navigate around it.
The fact that the app labeled "Music" is really two apps (one for browsing and one for editing) and that the one labeled "Settings" is really a bunch is a real inconsistency that wasn't visible before Mango because before it there was no way to rearrange the back stack. With task switching, this little secret came into plain view in the card view of apps.