new analysis shows super-shoes have not impacted elite men's times

What do you mean Santa doesn’t exist? He brings me gifts every Christmas.

Haha, I know what you're saying, but no worries! The 2% improvement was only added once, not as compound interest.

As soon as I saw the Excel charts I knew not to take this seriously. You've done some good work to compile the data, but you need someone with a background in sports science, or at least in applied statistics to use this data correctly. At the very least, you need meaningful controls as well as some discussion of what impact outliers have on your analysis - and in running there are definitely outliers to be discussed.

That seems backwards. The burden of proving causation is on those who say super-shoes have destroyed the integrity of the sport and made records meaningless, yet you are demanding the analysis do the work by proving that no such super-shoe effect exists (a.k.a. the you-can't-prove-a-negative fallacy). Even so, the desire for a control is why a fair amount of time was spent trying to figure out the percentage of those who did, and did not, wear super-shoes each year.

One of the great things about this set of data was its consistency and lack of outliers. That is to say, there were no DNFs or similar incidents impacting the averages, and the averages themselves were very close to median values. Also, any runner with an extraordinary year or performance could have only a minor 1/100 effect on the average, since they only appeared once in the top-100 listings.

Excel is very annoying at times when working with time values, but fortunately, the math to determine average values is straightforward. However, as mentioned in the tech note, one should not visually compare graphs to each other since the scale of the y-axis is not consistent across graphs.

There is a critical discussion point that no one seems to have mentioned already. That being, why is there a negative trend at all. The answer, of course, is that there are continual improvements in shoes, technology, training etc. I think it's fair to say that the Adidas sub2 offered a significant improvement over shoes available in the 2000's, for example.

This analysis assumes that those improvements should continue PLUS extra improvement from supershoes. But I would say, over the last 5 years, supershoes ARE those improvements.

It's not metabolic efficiency, it's biomechanical efficiency. Less energy is dissipated by the shoes.

If you want to be really pretentious, why not call it thermodynamic efficiency?

" The burden of proving causation is on those who say super-shoes have destroyed the integrity of the sport and made records meaningless"

Do these people think we should run in all leather shoes? Or maybe leather shoes destroyed the integrity of the sport 100 years ago, when they got lighter.

Or should we all run barefoot?

I noticed in 1982 that I could actually run faster in the Nike Mariah, and feel better the next day, compared to the Nike Eagle.

The Eagle was lighter, though, and it was THE racing shoe, so I usually raced in that. I'd train a lot in the Mariah, for reasons I mentioned above.

This is a very good point. One might argue that at any time in the present we should assume the trend going forward will be flat, and that whatever processes that caused times to decline in the past are not necessarily still impactful. But when one looks at the slope and variability of the average time line, the downward trend seems clear and not very random. Maybe if we looked back at the same type of data going back 40-60 years, the trend would not be as clear.

I believe the largest force impacting times downwards is simply the number of runners training and competing at an elite level. Improvements like Strava, cell phones, as well as transportation have made it easier for runners in far-flung places to figure out what it takes to train at the highest level and compete in new places like China. Or maybe there is overall less strife in countries which produce runners. But perhaps most importantly, in terms of the economics of the sport, Ross Tucker put it very well when he wrote about Kenyan marathon dominance:

"The economic factor is huge – it means, literally, that thousands of Kenyans aged 18 to 25 are training with current champions (that’s culture, and it creates a staggering “institutional memory” across generations) to break records and win big races. This drives performance more than science ever could – it is truly a high leverage input, because when you have culture plus economics, you have the two ingredients to grow knowledge through “institutional wisdom”."

These types of cultural movements are slow-moving, which is why it makes sense to assume that trends have some inertia to them. It could be some great coincidence that these and other trends ended in 2017 just as super-shoes started to lower times, but we shouldn't be asked to believe that just because times have decreased incrementally at the same rate as before, that it's the result of shoe technology.

You're not wrong that ultimately the shoes do seem to confer a biomechanical advantage, it's just not what the labs have been able to measure. The amazing original research of Hoogkamer et al. derived the 4% number from oxygen measurements of mask-wearing participants on a treadmill. However, they took mechanical measures as well and found:

"While the observed differences in energetic cost of running between shoe conditions were as substantial as 4%, the differences in our gross biomechanical measures (i.e., peak F z, step frequency, and contact time) were on the order of only 1%."

These were not hobbyjoggers but serious 31-minute 10k guys, so it's surprising this huge advantage doesn't seem to show up in the super-elites.

ncbi.nlm.nih.gov

A Comparison of the Energetic Cost of Running in Marathon Racing Shoes

Reducing the energetic cost of running seems the most feasible path to a sub-2-hour marathon. Footwear mass, cushioning, and bending stiffness each affect the energetic cost of running. Recently, prototype running shoes were...

Link:www.ncbi.nlm.nih.gov/pmc/articles/PMC5...

Yep! I'd add that these are bad, even for Excel charts. If he wants to show that his graphs tell us something about the real world, he should at least do some very basic sensistivity analysis to convince us his results are robust to his "sample selection strategy."

For example, he could: 1) include a graph of the fastest & slowest times for the top 100; include graphs showing the average time for other sample sizes (the top 5/10/25/50/250 or whatever); 3) log the results, and so on.

For the reasons you point out, more graphs won't be enough, but at least they'd give us a better picture of what is going on in the data.

That Keith didn't do any of this before sharing his "results" shows he just doesn't get Feynman's two rules of science:

1) You must not fool yourself; and,

2) You are the easiest person to fool.

Exactly.

This "analysis" has too many problems I don't know where to start.

The cultural movement you're talking about happened decades ago. Prior to the Vaporfly, marathon depth among East Africans had already peaked in 2012 (helped in part the the Boost shoes) and was trending backwards.

Yes, this. The article fits a linear trend to the data and then assumes that this trend will continue from 2016- even without super shoes. What is causing the assumed improvement? What caused the presumed linear improvement from 2001-2016? If your answer includes 'technology' then your trend line is including the effect of the shoes.

Also, there's quite clearly not a linear progression in many of these curves. Look at the women's 10,000 graph. That line is a terrible fit for the data. The men's marathon has a clear plateau to 2018 that would make the line of fit look ridiculous if not for supershoes restoring the trend from the early 2000s.

The model of 'fit a line between the largest difference in times and assume that should continue forever without contributions from shoe tech' is just not a good one to test the effect of supershoes, IMO.

Yeah, he also created this downward trendline "by subtracting the 2016 average from the slowest year in the range." So he basically assumed there's a downward trend and then cherry-picked the dates that would make it slope down.

The one thing we know for sure is that the improvement from 2001-2016 was NOT the result of super-shoes. I'm not sure why you would make that claim.

To clarify, the trend line shown on the graphs is not what was used to calculate the position of the projected 2021 time (represented by the green triangle). That projected time is based simply on the simple average annual decrease in times from the slowest year to 2016. This allows anyone to confirm the basic truth without relying on complicated statistics: that there has been a general downward trend in times over the last twenty years that could explain why we are seeing fast times now, rather than relying on the super-shoe explanation. The trend line is just a visual aid that helps account for the fact that the 2016 time may not be, due to natural variability, very representatative of the trend at that time.

In the Men's Marathon, for example, the 2016 average time was 2:07:42, and would seem to be the logical baseline time in measuring the super-shoe effect in the years following. However, in 2012 the average time was faster, 2:06:58, and is more representative, in my opinion, of what marathon runners are capable without advanced shoe tech. Should it have been considered alarming when the men reached an average time of 2:06:54 in 2018? Not at all, they had already achieved that level in 2012, and it would be silly to label such times as worryingly fast, even if the shoes had something to do with it.

As you've noted, the trend in the Women's 10000 is harder to pinpoint, but we don't even need to rely on trends analysis to see that super-shoes have not provided the 2% boost that is widely viewed as unfair to previous record holders. If we take the 2016 time of 31:41 and multiply it by 2%, that should yield an improvement of 38 seconds, or 31:03. If we then assume 8% of the runners were not wearing super-shoes, the average time would rise to 31:06. Yet, the actual average time in 2021 was 31:22, considerably slower. Throw in the fact that the women improved on average 7 seconds per year in the 10000 from 2010-2016 and there's no way to say with certainty that shoe tech has had an effect.

Proving whether the shoes actually work or not for the elites doesn't really matter. If people didn't complain about the large improvements in times in the years preceding super-shoes, they can't then complain about similar-sized improvements after their introduction. (If they are being rational, of course, which is admittedly sometimes overrated.)

Two issues I still don't think you address:

1) By fitting a line through the slowest year in your dataset you are forcing a negative trend that may just be the result of noise. See: Women's 10,000. You are assuming improvement in times a priori when some of your curves show little. Your null hypothesis is that times should continue to linearly improve regardless of shoe tech -- I don't think that's a safe assumption.

2) Any improvement from 2001-2016 is the combination of many things, including improved technology. Super shoes, being the latest technology, may be necessary for the trend to continue. Without supershoes, perhaps we wouldn't see any improvement over the last 5 years! This, I think, is a perfectly valid hypothesis: supershoes are just the next breakthrough in a long line of improvements that have contributed to faster times. They were necessary to keep the trend going.

Moore's law for transistor density has held thanks to continual improvements distributed across the entire process of semiconductor manufacturing. We could be seeing a similar thing here. Your argument could be 'Supershoes are nothing special! We expect to see improvements in technology that increase performance. This is one of them. Big Whoop.'