I just tried it out. You simply add in the IP address of your server, but with no port number and no http://.
It's a little finicky though. It will open my first selection perfectly the first time, but when I go to select a different video before the first video has finished playing, it just plays the same initial one over again. I have to close out Internet Explorer and start it back up to get a different video to play. If I watch a video straight to the end, then sometimes it will allow me to play a different video. If you're in the same show, then it tends to work great, but if you switch to a different show or a movie, then not so much. So if you want to watch multiple episodes of the same show, I would suggest waiting till the episode ends by itself, then switching to the next video. If you want to watch different shows or movies, then you'll have to close out Internet Explorer and start over.
I've tried every video format I have, AVI, MP4, MKV, and they all seem to work great, even in full screen mode (the time bar on the bottom doesn't go away though). I really hope Plex either releases an HTML5 version of the web client or more ideally, releases a proper Plex app. They already have a Windows Phone app, so I don't see why they wouldn't release one for Windows 8 as well.