Circumference is irrelevant. I know that sounds strange, but the Band isn't like a normal watch band.
The display section is as flat as a board - both top and bottom. The electronics (and display section) is the same size for all sizes. It's also quite long and tends to overhang the wrist. It also means the "width" at the top is the same for all sizes. The sides of the band itself is rigid, because that's where the batteries are. The remaining parts of the band are heavy rubber that's preformed into shape.
When you tighten a normal watch band, you're shrinking the circumference and applying pressure all around the wrist. But the Band is more like a trapezoid where only the bottom is adjustable. The different Band sizes are set by the height of the sides. Tightening the Band does two things: It pulls the sides together by narrowing the "bottom" (since three sides are rigid), and it slightly pushes the sensor package into the curve of the other side of the Band, pushing the sensor into the wrist.
The dimension that really matters is the "height" from the back of the display to the top of the sensor package. Since the underside of your wrist is quite sensitive, the pressure on that part of your wrist determines what is comfortable. And that is set by the height mentioned. And that's what's different for the different sizes. MS has a guide on the ordering page.
I've got a medium. To wear it comfortably, I've got about a 1/8" gap on either side of the wrist. If I trying to tighten it anymore, the sensor digs into my wrist and after some time it will start to hurt and/or my hand will go numb. I've tried other sizes in the store, but they're trade-offs, and not any better. So in my case, the band is gripping my wrist top/bottom, but not by the sides.
Bottom line, follow the MS guide for sizes based on height, not circumference. There's a reason for it.
At this point I think MS got the hardware "right", but the ergonomics needs some work. I wouldn't be surprised to see a 1.5G version come out - same electronics just packaged a little better.