I recently decided to use my black Lumia 800 without the cover - mostly because I discovered that the cover got the phone body dirty. I have to say though that the phone is very nice in the hand without the cover so I'm pleased. I haven't had a problem with fingerprints.
The rubber case is a neat extra feature. It'll definitley help protect the phone if you happen to drop it. The phone is tough, yes, but it's not a brick. I have had no experience with black marks.