First of all, do you really need your phone? Is there cell service? If not, I wouldn't bring your phone. Or any equipment not rated for extreme temperatures.
I don't know if altitude has much effect, maybe someone else can help with that. But the cold will definitely affect any phone. Cold temperatures will drain a battery quickly, and big temperature extremes, like going from freezing cold to a nice warm tent or shelter, can lead to condensation. I would definitely keep your phone close to your body to keep it warm, certainly not in an outside pocket.