I feel like this thread has been done a number of times...
First, I totally agree, it can be a vain sport. The bikes are expensive, so by that point I figure the cost of clothes are pretty small. Plus, I like to be comfortable on long rides.
However, those clothes have purposes. You'd be mocked for running in basketball shoes too.
Baggy clothes can get caught on things and aren't aerodynamic. It makes riding in a pack a bit more dangerous and makes you less stable in windy conditions. Nobody wants to ride with someone who might be a danger to them, and it's selfish to think otherwise. A crash can happen for the stupidest reasons...
Aside from that, the shorts have padding that make sitting on the seat a lot more comfortable. But they don't have pockets. The kits ("jerseys" or "cycling shirts" or whatever you want to call them) have pockets, so whenever I'm tempted to wear the cycling shorts and a tshirt I realize I need to carry my keys or some cash or some warm clothes just in case, so I decide it's just easier to wear the kit. I've done short rides in a tshirt and shorts (especially if I just went for a run), but it's more important for longer rides, so it eventually becomes habit.