I don't like crowds either, but I have spent a lot of time in big cities because I like the attractions on those cities. (Mostly art and history museums.) Having said that, I'd much prefer smaller cities.
Williamstown, MA.
Fredericksburg, VA.
Portsmouth, NH.
De Smet, SD.
Santa Fe, NM.