You're 100% correct, and this has been my experience:
I ran for a solid D1 program and was pretty good. I was/am thin and fit but I was never skeletal. I have always had a more "Mediterranean" physique. I would describe it as a thin Mesomorph body type. One time a tailor described me as "Mediterranean" because I have a thin waist but wear a 38 jacket in the chest. Anyway...
I have a lot of good friends who were major frat boys. Big time drinkers and general messes during their college days. So many of them looked like total sh-t during college and the years after. Somehow, they all managed to marry attractive women. Anyway, I kept running after college and remained fit. Always felt superior to them.
So once we hit, say, 30 or so, I probably put on about 10 pounds over the course of about 3 years. Nothing more than kids, work and day-to-day responsibilities injecting their presence into my life. I was still running and eating healthy, but I could feel the extra ten pounds.
At the same time, my fat buddies discovered things like X Fit and marathons. Suddenly they are melting fat off of their bodies and looking better than ever. While I was still fitter and thinner, it was hard to feel good about my fitness when I had gained weight and they had lost noticeable amounts.
I am 37 now and it has been hard. I am still running decent times and look pretty fit, but I am also a father of three and have a demanding job, so I manage about 70 mpw and maybe hit the weights 1-2 a week (plus nightly pushups and pullups). I cannot devote my free time to working out like some can, but that was a choice I made and I can live with that... It has just been a weird dynamic to see these friends bounce back with so much success.