Depends on the app. Most are using GPS for help with location based services, some are photo apps using it to geotag ect. I click yes for them since I understand what they are using it for. But if I came across an app I wasn't sure about, I wouldn't.
Many things your phone does requires location services, so if an app is using these things, they need to know where you are in order to display the information you are looking for.