At speeds approaching 15 MPH and beyond (basically 5k and shorter) all the loose failing jerseys and shorts are putting a significant drag on those athletes. Cycling experts would cringe at witnessing the best of the best racing in such attire. I know it sounds absurd, but having run some computational fluid dynamic simulations in simplified objects like a jersey in a 15 MPH headwind, it creates a significant resistance to moving forward when compared to a tighter fitting short or shirt.
Keep in mind, also, that the ALL TIME list for these track events consists of times where a tenth of a second lowers you by 5, 20, 50 places.