I lived in Manhattan for 5 years during my Ph.D. studies. My wife and I lived in Harlem (135th & Lenox across from the Harlem Hospital) and we absolutely loved it.
I ran all the time through St. Nick's to the river, down Riverside passed campus, then cut over to the park by St. John's. We always tried cool restaurants (once ate in Andy Warhol's old apartment in the village) and had our favorite bars (Dive Bar on the UWS was our go-to). It was a great time in our lives.
I couldn't imagine living there now with two kids, but we always knew it was only going to be a season of our lives, so we were able to enjoy it for what it was.
Definitely not a cheap place to live, that's for sure!