Could it be the noise cancellation? Maybe it needs some software tweaking yet. (Just a side note, I always thought iPhone sounded hollow, seems like their cancellation is a little aggressive to me.)
Maybe play around with your environment, or atleast notice it when someone comments on this effect?
Just a shot in the dark.