If you close IE by dragging down from the top of your screen, try holding the minimized window at the bottom of the screen until it flips over. That should erase data from your last session.
@RumoredNow : Yes, I'm closing the tabs individually before closing the browser. I do want it to remember the remaining tabs that were left open for the next session but it is also bringing back previously closed tabs.