I was told if I turn data on then it shows up but it's not working for me. I get tons of missed calls throughout the day and it's so annoying having to call voicemail to check everytime.
Nimdock you should call into AT&T and verify you are provisioned correctly. Not getting the icon on AT&T indicates you're not provisioned right.
Do you have visual voice mail? From your screen shot it does not appear to be the case.There is the voicemail icon at the bottom of the screen that identifies when you have a message or not. This is how it looks on my phone. You can see the icon before and after.
https://skydrive.live.com/redir?resid=5B700CA437579731!193&authkey=!AHGIzTpy6IlP5rM
Do you have visual voice mail? From your screen shot it does not appear to be the case.