Could it be the noise cancellation?  Maybe it needs some software tweaking yet.  (Just a side note, I always thought iPhone sounded hollow, seems like their cancellation is a little aggressive to me.)
Maybe play around with your environment, or atleast notice it when someone comments on this effect?
Just a shot in the dark.