Just laugh and say, "Too skinny for what?". No one has an answer for this. Girls are generally jealous or they are coming on to you. Guys are trying to alpha-dog you and are insecure. Response works either way.
No use in getting mad about it, just be grateful for what you can do with your "skinny" bod. Do you realize that most bigger, fatter people will NEVER know what it is like to run as fast and as far as you do every day? Consider yourself lucky.