The one thing we know for sure is that the improvement from 2001-2016 was NOT the result of super-shoes. I'm not sure why you would make that claim.
Once again, this is just wrong. The Adidas Boost foam, used by all 3 world record holders and most top amateurs from 2011 to 2014, was shown to improve running economy.
Yep! I'd add that these are bad, even for Excel charts. If he wants to show that his graphs tell us something about the real world, he should at least do some very basic sensistivity analysis to convince us his results are robust to his "sample selection strategy."
For example, he could: 1) include a graph of the fastest & slowest times for the top 100; include graphs showing the average time for other sample sizes (the top 5/10/25/50/250 or whatever); 3) log the results, and so on.
For the reasons you point out, more graphs won't be enough, but at least they'd give us a better picture of what is going on in the data.
That Keith didn't do any of this before sharing his "results" shows he just doesn't get Feynman's two rules of science:
1) You must not fool yourself; and,
2) You are the easiest person to fool.
The sensitivity analysis is really inherent in the year to year comparisons, as well as the orange "what if" lines (that estimate what average times would have looked like had there been no super-shoes). With respect to the former, one can see that in the Women's Half Marathon, the average time was identical to the second in 2004 and 2006, 2009 and 2010, 2014 and 2016, despite lasting close to 70 minutes or 4200 seconds. It is incredible the times were identical despite each year having encompassed different runners, different courses, different weather conditions and who knows what else. Likewise, the Men's marathon in 2015 and 2016 produced identical times of 2:07:42. These matches and near matches indicate the method is a robust and accurate way to measure their running ability and not the result of random noise.
The main goal of the selection strategy was to focus on the runners who are actually setting records, whether they're world records, national records, collegiate records, etc., since those are the marks which most people are concerned about. The top-100 does a good job containing those performances while also being lengthy enough to reduce the impact of any significant variation among individuals impacting the average. Remember also these weren't the top-100 in a single race; rather, these were the podium and other top finishers from races held worldwide, who all experienced very close to their very best and most optimal performances on those days. In other words, these were not the non-responders, these were athletes who responded well to the shoes they were wearing.
If you want to see the range of fastest and slowest times, they're easy to spot on the top and bottom of the page at WA:
I loved the way Feynman demonstrated the loss of flexibility in the space shuttle rocket's rubber O-ring simply by dunking it for a time in his glass of ice water in front of the committee that investigated the Challenger disaster. Nothing fancy, just a plain simple demonstration that anyone could understand.
These types of cultural movements are slow-moving, which is why it makes sense to assume that trends have some inertia to them. It could be some great coincidence that these and other trends ended in 2017 just as super-shoes started to lower times
The cultural movement you're talking about happened decades ago. Prior to the Vaporfly, marathon depth among East Africans had already peaked in 2012 (helped in part the the Boost shoes) and was trending backwards.
That was just an example. I am sure folks understand there are new cultural movements being created all the time. Taking a superficial look at the nationalities of the top-100 women runners in the marathon in 2001 vs 2021, the number of Ethiopians seems to have risen considerably. I can't explain why there are so many more, but perhaps this new running population coming into the upper levels of the sport is one of the factors driving the trend downwards. The main point, however, is that it's reasonable to think that trends may be impactful over multiple-year periods and not something that just restarts each year. And it would be weird if times didn't gradually improve, since that's kind of the point of competition.
Based on the last 20 years of race data, the average time of the worldwide top-100 runners was calculated for men and women in each of the following events: the 10K, 10000M, Half Marathon, and Marathon. The average yearly decrease in finish times was then determined by subtracting the 2016 average from the slowest year in the range. Then, a 2021 time was projected using that trend, along with a 2% boost from super-shoes, and compared against the actual 2021 time in that event. A "best fit" linear regression trend line was also created for comparison against the actual average times over the period.
Contrary to popular belief, these graphical analyses seem to indicate the super-shoes have not improved performances for the elite men beyond prevailing trends. The women's data is a bit more ambiguous, in part because their times have been improving at a much faster rate.
This is a big result and it might be a good time to reel back in some of the aspersions cast on the record-breaking performances over the last 5 years or so. Personally, I'm not willing to give up my Vaporflys, but for an elite runner with energy and form to burn, now might be a great time to seek out a sponsorship from a non-super-shoe brand.
Haven’t read the analysis but a lot of has known this data for a long time.
For anyone who believes super shoes only work on younger runners, or only work on indoor tracks, or only work, etc etc … I have an investment opportunity for you.
The one thing we know for sure is that the improvement from 2001-2016 was NOT the result of super-shoes. I'm not sure why you would make that claim.
Once again, this is just wrong. The Adidas Boost foam, used by all 3 world record holders and most top amateurs from 2011 to 2014, was shown to improve running economy.
Nearly everyone has defined super-shoes as those carbon fiber plate and foam models that emerged after the Nike Vaporfly prototypes in 2016. The Adidas Boost foam may well have contributed to lower times, but I can't seem to find any criticism of the shoes' tech from the period, or research showing they improved running economy. If you have any links, would appreciate them.
The Hoogkamer paper I referenced here used an Adidas Boost shoe as a control against the Vaporfly, which was found to improve running efficiency by 4% against the Adidas shoe, as well as the Nike Zoom Streak.
1) By fitting a line through the slowest year in your dataset you are forcing a negative trend that may just be the result of noise. See: Women's 10,000. You are assuming improvement in times a priori when some of your curves show little. Your null hypothesis is that times should continue to linearly improve regardless of shoe tech -- I don't think that's a safe assumption.
2) Any improvement from 2001-2016 is the combination of many things, including improved technology. Super shoes, being the latest technology, may be necessary for the trend to continue. Without supershoes, perhaps we wouldn't see any improvement over the last 5 years! This, I think, is a perfectly valid hypothesis: supershoes are just the next breakthrough in a long line of improvements that have contributed to faster times. They were necessary to keep the trend going.
Moore's law for transistor density has held thanks to continual improvements distributed across the entire process of semiconductor manufacturing. We could be seeing a similar thing here. Your argument could be 'Supershoes are nothing special! We expect to see improvements in technology that increase performance. This is one of them. Big Whoop.'
Thanks, Harambe. I was careful to say the shoes have not impacted elite men's times, while the women's data are more ambiguous. But I think it goes without saying there is a negative trend in all of these events. After all, world records do fall from time to time, although they are by definition, outlier performances and not a great proxy for the average top-100 runner. Still, it's instructive to look at world record progressions. For example:
The Men's world record marathon time dropped on average 29 seconds per year over 110 years. It is clear looking at a graph of these marathon records that the line is not straight, but more of logarithmic curve, though of course as you shorten the time scale, it will appear linear. If we start instead at 1963, we can see the average improvement to 2018 was only 15 seconds per year, so the 10 second-per-year improvement calculated off my graph does not seem unreasonable.
That said, it seems to me the most important years to consider in the trend are those closest to 2017 since it will be many of the same athletes competing. One might even more heavily weight those years. Obviously, if one were to pick a slow time only a year or two prior, that would not be a long enough period to establish a trend. Perhaps the ideal length is related to the average participant's elite career length, as there were very few runners in the top-100 in both 2001 and 2021, for example.
I understand now you were referring also to shoe technology that predated the Vaporflys. Yes, there will always be some group of factors (many of which are not connected with the athlete's training and natural ability) that are affecting the trends. But for the men at least, the actual times are so far from what the shoes should have delivered, that I have serious doubts whether they are having any affect on race times at all. Still, since times are not going up, they are pretty clearly not a negative influence, which is literally half the battle in any successful endeavor.
So Kipchoge could run under 2 hours for a marathon, despite drafting help, etc., in 1978 Nike Elites. Got it.
He probably deserves more respect for it than what he has received. He probably couldn't have done it in old Nikes, but maybe in Adidas Boosts? Maybe he knew all along the shoes weren't all that special and that it was just a publicity stunt (a great one). Makes it even more impressive.
The answer is no, but thank you for the request to update my analyses.
I did do two different comprehensive all time performance analyses, one with data up to Jan 2018, and the second with data up to Jan 2019. Neither of these capture the full impact of shoes.
Based on the last 20 years of race data, the average time of the worldwide top-100 runners was calculated for men and women in each of the following events: the 10K, 10000M, Half Marathon, and Marathon. The average yearly decrease in finish times was then determined by subtracting the 2016 average from the slowest year in the range. Then, a 2021 time was projected using that trend, along with a 2% boost from super-shoes, and compared against the actual 2021 time in that event. A "best fit" linear regression trend line was also created for comparison against the actual average times over the period.
Contrary to popular belief, these graphical analyses seem to indicate the super-shoes have not improved performances for the elite men beyond prevailing trends. The women's data is a bit more ambiguous, in part because their times have been improving at a much faster rate.
This is a big result and it might be a good time to reel back in some of the aspersions cast on the record-breaking performances over the last 5 years or so. Personally, I'm not willing to give up my Vaporflys, but for an elite runner with energy and form to burn, now might be a great time to seek out a sponsorship from a non-super-shoe brand.
I'm a big fan of analyzing performance data, but I'm not really sure what to make of this one. It does raise a few questions for me:
- Does going back to 2001 (or 2004) go back too far to provide a meaningful trend for post 2016?
- Is the top-100 per year too deep?
Looking at some of these events, many of these events seem too immature to go that far back and that deep and treat the top-100 as comparable. Some reasons the depth of performance might change:
- The men's marathon times weren't really attacked until Geb started going for time trialed records, and then it transitioned from old men at the end of their careers, to young men who skipped the track part of their careers.
- Most road races shorter than the marathon weren't really run for time until recently, especially for the women.
- The 10,000m slowly disappeared from the track, which would also impact depth in the later years.
- Also, as some mentioned, in recent years the pandemic likely impacted depth of fast performances as well in the last two years or so.
I really don't understand what you are trying to prove here.
Do you not believe there has been a steady evolution of racing shoe design over many decades, designed by
1, People who study biomechanics, materials and design technology?
[Forgive my slow response, I was mistakenly auto-banned a few days ago for a post that contained the name of an advertiser. Let me try again:]
The purpose of the analysis is to prove that (1) the integrity of the sport has not been threatened by super-shoes, (2) records and times set in the shoes are as meaningful and impressive as any set previously, and (3), among super-elite men (and perhaps women as well) the shoes do not confer anywhere near a 2% performance advantage.
Most of the evolution in shoes has been due to marketing pressures rather than a true interest in helping people run faster. Old-style shoes do not sell well and manufacturers always need something new. But don't get me wrong, the tech is impressive. I'd still strongly recommend the Vaporflys et al. to anyone, and suspect they are beneficial to the average runner.
Hey Rekrunner, forgive my slow response... I found it interesting that out of the whole 20-year period, the 2001 year of these events was often the slowest. I expected more variability and that there wouldn't be such a clear chronological trend in most events. The purpose of the trend analysis is not so much a search for causality as it is to determine the natural range of variability and improvement that each event exhibits over time. For that reason, I would say the more years, the better. (It also became clear that each event's performance timeline needs to be examined on its own. It was surprising, for example, that the trajectory of the women's 10000m was quite a different shape than the 10K Road race, which I can't explain.)
That said, I do wonder if there is some artifact in World Athletics data due to sample size or some of the other factors you mentioned that explains the trend, rather than actual changes in performance. For example, if WA was regularly pulling the times of past dopers as they became aware of them, that would have the effect of dragging down the average in previous years compared to the present. But it seems like even banned runners still have their times included in these historical datasets.
The research on running efficiency showed that, although there was some variability, all the runners that tried the shoes in the lab showed relatively massive gains in performance (around 2-6% over the Adidas Boost). Thus a 2% increase in speed (from a 4% increase in efficiency) should have been the *average* boost observed in all of the top runners that wore them. I could see how a field of runners in a race of 100 participants might contain some non-responders, but these top-100 were all having good days. The common idea that it's only the record breakers and outliers that are benefitting from the shoes just isn't supported by the research, or even by anecdotes.
No doubt much of the difference in performance is due to the changed circumstances of the races as you describe. I'm not arguing runners of today are actually that much faster than ~20 years ago. But the adoption of the super-shoes was, relative to other factors, sudden and clearcut, and should have generated an obvious signal. The difference between the slowest and fastest times in the men's marathon has been about 5 minutes, or ~4% of the total time (for women, about 10 minutes or 7%), so if they were all wearing the shoes, the entire list of runners should have moved halfway up the page! The only reason that wouldn't have happened, of course, is that not everyone wore the shoes early on, but now we're over 90%.
You're right, though, that if 2021 was still not back to normal from the elites' perspective after the pandemic, that could have affected times upward significantly, and maybe we'll see big moves downward in the years to come that could be explained by the shoes. But I would bet against it.
