I think design plays a major role in how a phone feels in hand. I own the Nexus 6 and it feels awesome to use despite mildly large side bezels (it almost looks disproportionate because the top and bottom bezels are so close in size to the side bezels). But I also think dedication plays an even larger role. I've always been a fan of larger phones but it still took me a day or two to get used to the Nexus 6 over my Lumia 830 and start using it normally. I never had an issue using the Nexus 6 (except Android 7.0 still feels clunky to use in some places compared to Windows 10).
No, people are always going to complain. You don't see people complain about the Galaxy Note 7 because it's, well, a Galaxy, but I've seen tons of people complain about any phone that isn't Samsung or Apple having a screen size above 5.2" and I've seen tons of people complaining about 5.2" phones not being 4.5".
I actually think what happened over the last few years is starting to happen in reverse. We tried to make phones as tiny and mobile as possible at first, then they started getting bigger and bigger, and now we're trying to make them smaller and smaller again.