Dude, take a deep breath. Your experience is contrary to someone else's experience... but the world will carry on.
Hunk's assessment is correct based on my experience (and the experience of most people I know, fwiw). If you live in or travel to the average urban center and you look like a stereotypical tourist, or like someone from the suburbs, or like someone who doesn't know their way around the city - then you are more likely to get harassed by the impoverished residents. Whether they are trying to sell you a service or a product, get a donation, rip you off, or are just menacing people - all are forms of harassment in one way or another.
As far as harassment and experience in Baltimore - my experience there was similar to my experience every other major city I've been to (having been to and spent substantial time in 24 of the US' 50 most populous cities). Like the previous posters said - Baltimore has a nice touristy area around the harbor, Camden yards is nice, Fed Hill is nice, Fells Point is nice. If you are there to see nice things and explore a new city then Baltimore is definitely a good option for a visit. Beyond those places I just mentioned I didn't spend substantial time in other neighborhoods. One day I ran 20 miles within the city limits and never covered the same pathway twice - my run took me into the low income west side and actually right past Mondawmin Station which is, from my understanding, the epicenter of the recent looting and rioting. I remember thinking that Mondawmin was not like anything I had seen before (lots of vagrant people, lots poverty, and an intense urban vibe). At a few points in and around the west side a few kids yelled stuff at me from across the street, etc, but I suffered no threats of violence and no physical harm. Unfortunately the harassment is pretty typical if you are running for exercise through any low income neighborhood - it comes with the territory of looking like (and being) an outsider. When I stopped and asked directions I got clear accurate directions from a person without any problem. When I asked if I cook take a shit in a corner deli I got a little pushback but in the end it worked out fine. When I wasn't running and was spending time in the nice neighborhoods I had no trouble and enjoyed my time in Baltimore. In my opinion the people, food, atmosphere, weather, and culture are all worth at least one visit for you to form your own opinion.