lol. I was once in a similar position. I was a 21 year old virgin, a quiet guy who never went out of his way to try for women. I let the fact that I was a virgin when most guys my age weren't eat away at my self esteem. I ended up going out to a bar one evening, getting fairly drunk, and sleeping with a woman that I would not have ever asked on a date. It wasn't even that great. The next day, all my friends got a huge kick out of this and were real happy for me. I pretended to be happy but I felt ashamed. I continued this pattern for a few more years, thinking more sex would make me feel like "the man", and raise my self esteem. I ended up getting an STD (thankfully it was curable). At 26, I decided I had enough. I met the woman I would go on to marry, and she was a 22 year old virgin. I was scared that she would leave me when I told her about my past. Needless to say she wasn't thrilled.
I look back and realize how insecure I was to think that I needed to have sex to fit in. I wish I would have waited. Sure, there were good times, but a lot of baggage went along with it, and I would have liked to share my first time with my wife.
My advice to you is just go with the flow. Don't feel bad about it, just live and if you find someone that's right for you great, if not, that's okay too. It's funny to think that having sex is like joining an elite group, when it's something that the majority of the population has done.