here's my 2 cents:
yes, if it's been really cold lately (20's), and then bam, you get a 50-55 degree day, sure, one can go shirtless and not be cold. But to do it in the low 40's is just silly. As the guy above put it, sure one can do it, but if you are, you are doing it mainly just to think or show that you are tough or something, not for comfort.
And the reality is: if you are going shirtless on any day below 50, you are going to look like a tool, no matter how much you tell yourself you are not. If you are super skinny, you look pathetic and people will laugh at you, and if you are really buff, then you look like you are starved for attention and trying to desperately show up and people will, again, you guessed it...laugh at you (I guess if you are in between you won't get as negative a reaction). Above 60..... you still might get laughed at but at least it won't seem abnormal. Above 70 or 80, again, it will look like you want attention, but at least it is totally acceptable.
(and don't get me started on runners who wear shorts when it is in the 30's. It' asinine. One's legs get cold, and there are a million different types of light, flexible, long running pants that will keep you warm. It's completely dumb to not wear long pants in the 30's).