I have the same problem, coming from an iPhone. The concept of a hardware back button seems nice, until you discover that it's not always doing what you expect it to do. And then you are so confused that you don't use it anymore.
tapping recent works but its a 2-3 step process and its not something the average user would do when the back arrow has been imprinted into our brains .