No statistically significant improvement in marathon results since Vaporfly introduction

It looks like the vaporflys are two to three times better than anything produced before 2008

This is a good start, nv. If you want to answer the concerns about cherry-picking or using an arbitrary 2:10 cutoff, I recommend presenting several approaches:

1. Top 50 times each year; t-test comparing 2017-2019 v. 2013-2015.

2. Same analysis for top 20, top 100, etc.

3. Same analysis restricting the sample to the top 30 times of nike-sponsored athletes each year.

Still a very simple/simplistic approach, but interesting, and more balanced.

What track and field is like without the right accessories. Only this will satisfy those against the vaporfly shoes.

https://youtu.be/-qTAeVGl_e8

nv4 wrote:

Why not sub-2:15? I couldn't get data on that big sample size but if anyone has it let me know and I'll do the calculations.

IAAF has the top marathon results by year up to 2:35. For example in 2019,

Thanks for the link to IAAF data

Based on these data I made more detailed statistical summaries of performances by year. As you will see down in table below, there is hardly any improvement in results since the introduction of Vaporfly. Most of the performances are in par with peak of years 2011-2012. Eliud and Bekele is truly just an amazing exceptions in the statistics. That is all I have to say. Also I think improvement in performances during this year is mostly related to Tokyo Olympics and the high qualifing standard. Judge for yourself.

Year with most sub-2:05 performances – 2018

Year with most sub-2:15 performances – 2012, 2019

Year with most sub-2:25 performances – 2012

Year with most sub-2:35 performances – 2011

Year with 10th fastest marathon time – 2019

Year with 100th fastest marathon time – 2019

Year with 1000th fastest marathon time – 2012

Column 1 | Year

Column 2 | Number of sub-2:25 performances (best by athlete)

Column 3 | Number of sub-2:35 performances (best by athlete)

Column 4 | 10th fastest marathon time (best by athlete)

Column 5 | 100th fastest marathon time (best by athlete)

Column 6 | 1000th fastest marathon time (best by athlete)

YEAR | 2:05 | 2:15 | 2:25 | 2:35 | 10TH | 100TH | 1000TH

-------------------------------------------------------------------------

2019 | 12 | 566 | 1596 | 3311 | 2:04:24 | 2:08:12 | 2:19:09

2018 | 14 | 515 | 1568 | 3318 | 2:04:40 | 2:08:46 | 2:20:03

2017 | 4 | 492 | 1439 | 3106 | 2:05:39 | 2:09:11 | 2:20:20

2016 | 7 | 459 | 1471 | 3094 | 2:05:21 | 2:09:28 | 2:20:19

2015 | 2 | 486 | 1478 | 3165 | 2:06:00 | 2:09:14 | 2:20:10

2014 | 8 | 478 | 1492 | 3115 | 2:05:13 | 2:09:00 | 2:20:15

2013 | 9 | 478 | 1513 | 3174 | 2:05:16 | 2:09:08 | 2:20:09

2012 | 11 | 566 | 1678 | 3368 | 2:04:54 | 2:08:33 | 2:19:04

2011 | 3 | 495 | 1610 | 3386 | 2:05:45 | 2:09:25 | 2:19:51

2010 | 3 | 448 | 1412 | 3000 | 2:06:09 | 2:09:41 | 2:20:54

https://snipboard.io/Pc1zW3.jpg

Wildhorse wrote:

7 of the top 10 marathon performances of all time have been run in Vaporflies.

Yet, only three of the 12 world marathon major course records (six courses, male and female) are in Vaporflys.

Take out Eliud Kipchoge and it's a different picture.

"Judge for yourself."

that's not a statistical analysis. You're irresponsibly muddying the waters with your junk "analysis"

see my prior post with suggestions for a proper analysis.

You haven’t controlled for other factors which have changed from pre-2016 to post-2016 (e.g. change in drug testing protocols, etc). I’m of the opinion that the outrage about the VF is probably overblown so I’m biased towards believing your results, but your analysis focused on correlation rather than causation. I’d like to see an analysis that statistically measures performance improvements among VF adopters, and how that compares to other athletes. That’s probably the most valid way to do the analysis - the most difficult part would be gathering the data.

I once got a 5 on the AP Stats test. I don't remember much of it, but I'd still like to pretend my opinion matters.

I think that the control group or rather groups are missing. Sure there is pre and post vaporfly times shown for the marathon, but if the vaporfly is what's causing records to drop, then they shouldn't be dropping in events where the vaporfly isn't being used. Take the same look at the 100, 200, 400, 800, 1500, 3000sc, 5000, 10000. Likewise, the average amount of money on the line at the top-10 or so events of the year should be included.

The focus of distance running from a financial standpoint has shifted heavily to the marathon. The 10000 is hardly even an event anymore. If you're international class in the 10000 you're a fool to waste your limited amount of time when you can make real money in the marathon.

TL;DNR: OP, you are not even a hobby data-scientist. You are at the level of a five year old dressed up as Usain Bolt who is running around screaming, "Look at me! I'm the fastest man in the world." It was cute for a second. Now it's just annoying.

OP, you are right about one thing: Only facts can tell us whether these fancy shoes mean faster races. Unfortunately, you are wrong about pretty much everything else. As has been pointed out many times above, you don't understand what facts will, and what facts won't, help us answer the question. As a result, your posts tell us nothing. Not a thing. And, as has also been pointed out, you don't have a clue about statistics. Your confusion is obvious in the title: Whether something is statistically "significant," requires doing some real math; it's not just something you feel.

The OP has pointed out his data and methods. Those of you with a deeper statistical background who question his methods, please take on the challenge of doing the work yourself the way you think it should be done. That would be a real contribution.

Kruger-Dunning in action wrote:

you don't understand what facts will, and what facts won't, help us answer the question.

If you're even asking the question, you don't understand anything about running mechanics. A bouncy heel contributes absolutely nothing.

I'm no statistic and never said I was but I'm an engineer and I understand math. You can bash me how much you want but that won't change the fact that there really isn't any significant improvement in results when you look at plain numbers from pre to post VF introduction. If VF were so much better as some claim here you'd get difference in number of best results because for the past two years most of the faster runners have switched to them. All races I did this year almost entire elite section had VF either 4% or Next%

Claim is they improve performance by 2.7% (4% less oxygen consumption). If that would be the case we would have to see almost exponential increase in number of best sub-x:xx times per year as 2.7% increase in results translates to 2-3 min marathon improvement. Yet numbers are very much consistent in all groups just look at the table. The increase in the past year or two is small and on par with 2011-2012 results.

For example, top 1000th best time per year in the last decade by athlete. Difference between slowest year (2010, 2:20:54) and fastest year (2012, 2:19:04) is 1:50 or 1.3%. And that is before VF era.

Difference between just pre and post VF years is only about a minute or 0.7% which is less increase than what happened in 2011-2012.

Number of athletes that broke sub-2:35 per year? Difference between lowest (2010, 3000) and highest (2011, 3386) year is 11%. That is an astounding increase and yet again it happened from 2010 to 2011. Difference between pre and post VF era is much more normally distributed with low (2014, 3115) and high (2018, 3318) being only 6% difference.

If you look at sub-2:25 there is the same trend. Low year (2010, 1412) and high year (2012, 1678) is 16% apart. Crazy jump versus just pre and post VF era relatively modest increase of 7% from 1478 (2015) to 1596 (2019). Even less spectacular if take decades average time as reference.

The only increase you see is in very fast marathon times still on par with year 2012 though. But these are performances on the far left side of the Gaussian (normal) distribution curve and would be discarded by scientific study.

Column 1 | Year

Column 2 | Number of sub-2:05 performances (best by athlete)

Column 3 | Number of sub-2:15 performances (best by athlete)

Column 4 | Number of sub-2:25 performances (best by athlete)

Column 5 | Number of sub-2:35 performances (best by athlete)

Column 6 | 10th fastest marathon time (best by athlete)

Column 7 | 100th fastest marathon time (best by athlete)

Column 8 | 1000th fastest marathon time (best by athlete)

YEAR | 2:05 | 2:15 | 2:25 | 2:35 | 10TH | 100TH | 1000TH

-------------------------------------------------------------------------

2019 | 12 | 566 | 1596 | 3311 | 2:04:24 | 2:08:12 | 2:19:09

2018 | 14 | 515 | 1568 | 3318 | 2:04:40 | 2:08:46 | 2:20:03

2017 | 4 | 492 | 1439 | 3106 | 2:05:39 | 2:09:11 | 2:20:20

2016 | 7 | 459 | 1471 | 3094 | 2:05:21 | 2:09:28 | 2:20:19

2015 | 2 | 486 | 1478 | 3165 | 2:06:00 | 2:09:14 | 2:20:10

2014 | 8 | 478 | 1492 | 3115 | 2:05:13 | 2:09:00 | 2:20:15

2013 | 9 | 478 | 1513 | 3174 | 2:05:16 | 2:09:08 | 2:20:09

2012 | 11 | 566 | 1678 | 3368 | 2:04:54 | 2:08:33 | 2:19:04

2011 | 3 | 495 | 1610 | 3386 | 2:05:45 | 2:09:25 | 2:19:51

2010 | 3 | 448 | 1412 | 3000 | 2:06:09 | 2:09:41 | 2:20:54

this definitely this wrote:

The OP has pointed out his data and methods. Those of you with a deeper statistical background who question his methods, please take on the challenge of doing the work yourself the way you think it should be done. That would be a real contribution.

TL;DNR: That study has already been done, by two New York Times reporters, one of whom has a graduate degree in statistics. It's unlikely anyone here has the time, inclination or ability to improve on their findings.

A bit more detail: Many people here could look at the same data set the OP looked at and do a better job of using it to show whether the Vaporfly makes runners faster. Sadly, this would not contribute to our knowledge. That's because two New York Times reporters already did a very careful study of this question. They used a bigger, more complete data set and some real statistics to give a good estimate of how much difference Vaporflys make. Their study is better for several reasons: First, they used "big data" techniques to get a lot of information about a lot of runners (including which shoes they wore). Their data set is rich enough that they can make fairly fine-grained comparisons -- they even give estimates of how much the Vaporfly improves on several different shoe models. Moreover, since one of them is a trained statistician, they were able to look at the data several different ways.

Their study is not perfect. Their data has some serious problems. Unlike the OP, they recognize these problems, and work hard to take them into account. (Interestingly, the OP does not mention the NY Times study. Reading his original post, you would not know that anyone else had used data to answer questions about the Vaporfly.)

Let me amend my analogy: The OP is like a 5 year old who dresses up as Usain Bolt and then starts giving advice on training. In the words of the Nobel Laureate Wolfgang Pauli, he is "not even wrong."

?

I read their article. This is BS. They took data of hobby jogger performances which in itself is idiotic. There is no way VF can drop 4-5% of your time, that would be 6-7% economy increase and in minutes that translates to 5-6 minutes from your marathon time. They are superb shoes buy they have no engine built in.

You implicitly assume after the introduction of VFs that all athletes are wearing the VFs. Otherwise your inferences do not follow. Right?

Grushenka wrote:

You implicitly assume after the introduction of VFs that all athletes are wearing the VFs. Otherwise your inferences do not follow. Right?

No need to assume that all have switched to VF. It is enough that 30% have switched to VF for the trend to be clearly visible in stats as this would result in 10% increase sub-x:xx marathon times.

In reality I would say that 50% or even more sub-2:40 runners have switched to VF by now. Have you been on front lines of any marathons? Almost all guys that are sub-2:40 runners there have some sort VF model shoes now. Mostly ones that don't wear VF are the ones who have sponsorship by some shoe brand that restricts them to wear Nike's due to contract. Even where I live with all the low wages all Vaprofly shoes are sold out instantly.

I applaud the effort to use real world data to support a propisition. Nitpicking whether "statistically significant" is appropriate terminology is technically not wrong, but probably not the most important criticism, unless you are a statistician.

I think there are more basic questions that need to be thought about, before determining if sub-2:10 is a good measure that would show Vaporfly impact.

How many athletes wear Vaporflys? If that number is small absolutely, say a dozen per year, we should not expect to see large absolute changes due to Vaporflys.

If many of those athletes are sub-2:10 runners without Vaporflys, the sub 2:10 performance count would not show that an expected 2:06 performance became a 2:04.

Likewise, if Vaporfly wearers are too slow to run sub-2:10 with Vaporflys, your sub-2:10 performance does not measure their improvement.

It would be better to filter out the performances that would have been sub-2:10 with or without the Vaporflies, to magnify the differences, rather than shifting the count upwards with a large number of East African performances that would break 2:10 regardless.

Likewise, you should probably filter out East Africans completely, as many of the East Africans that get invited to compete are already of sub-2:10 quality, and it is not likely that many 2:12 quality East Africans are competing often in big, fast marathons.

Showing the impact on non-African performances would be more interesting (to me) since 2:10 represents a challenging milestone, and a 2-minute improvement would be readily visible.

Letsrun wrote this about Chicago using 2:12 (although attributing one event solely (no pun intended) to the shoes is questionable):

"If you want to see how big of an impact the new shoes are making, just take a look at the US men in Chicago. More Americans broke 2:12 in one race than in the entire preceding decade in Chicago. According to Race Results Weekly, the top 18 American men across the line all ran a personal best. Of the 10 men who broke 2:12, eight of them were either wearing the Vaporflys or a secret unreleased shoe from another company (presumably their attempted copy of a Vaporfly)."

They also showed that 11 Americans broke sub-2:12 in the 13 years of 2006-2018, versus 10 Americans in 2019 alone.

OP, have you ever even heard of a Chi-squared test?

Done.