Cornell researchers: Vaporflys improve men's times by 2.1 to 4.1 minutes in marathon, women improve between 1.2 and 4 mins

DirtyT wrote:

I don’t like this study very much, and here’s why:

1) It uses a 4 year window to judge results and therefore isn’t taking into account confounding factors such as improved calorie consumption(I.e. maurten).

2) It also doesn't take into account the very strong likelihood that there is at least a little bit of a placebo effect occurring. If everyone believes they are magically going to be faster there is no way they don’t run faster. Which this could account for the huge difference in time improvement. The human brain is absolutely accounting for at least some of this improvement and to give it all to the shoes is absurd.

I don’t have as negative of a view as this post, but would still be interested in Joe’s reply to the first point. Said another way - how does the study account for broader changes in marathoning over this time period? For example better nutritional intake, more people moving up to the marathon since the Olympic Trials half standard got harder, etc. I know they modeled the difference between Vaporfly and non-Vaporfly performances but am curious how time is taken into account or if all performances are evaluated together - which wouldn’t be 100% fair given marathoning was already improving before the shoes.

I appreciate that they called out the impact of opt-in bias and that people could choose the shoes when they were in peak fitness. I expect that is a non-negligible factor and wouldn’t play it down as much as the paper’s Discussion did.

Overall this seems like the best way to analyze this without a pure test vs control lab study. The above observations are just to consider factors that could affect the true impact to try to get to the most accurate number possible.

I understand the scope for comedy here but my thought was have they tried placebo running shoes? By which I mean shoes that look exactly like Vaporflys but aren't.

Maybe you would want to check to see if the runners that switched to vaporflys made additional changes that non-switchers did not make. As an example, maybe folks that switched to vaporflys also decided to adopt GPS watches for the first time which changed their training. Or maybe there was a new refueling drink that only people that Vaporfly switchers used on the second attempt. Maybe the people that did not switch to Vaporflys could not train as well. There would be a lot of variables that go into each performance. Overall an interesting study but needs to be expanded on.

Does anyone know of any runner(s) that ran marathon(s) in vaporflys but then changed back to more more regular shoes for additional marathon(s)?

Any runner(s) that did so could provide some interesting data.

Agree that there are many many variables that could impact marathon times, in addition to shoes worn.

just noted the posted comments with the article, and someone else asked the same question back in March.

But-LRC types, anyone know of someone who ran a marathon in vaporflys and then went back to more regular shoes for other marathons?

2.1 minutes as the minimum of the improvement range seems kinda crazy.

The vapor fly was pretty widely adopted by 2018 yet only Kipchoge managed to break 2:04, and the average top 100 time was well within the standard range for the last decade (and slower than some other years like 2012). 2018 would need to be by far the worst marathon ing year in the last decade for a 2.1 minute typical improvement to make sense.

2019 was significantly faster than 2018 (which is still weird considering the vapor fly was already popular in 2018. Are the next% significantly better?), but 2.1 minutes is still kinda pushing it even then. You still have second place at 2:04 high after the adjustment, which isn't unheard of (this occurred in 2015, but 2015 was the worst marathon ing year this decade) and the top 100 times were close to 1 minute better than usual, not 2.

joeguinness wrote:

Hi Rojo, I'm one of the authors of the study and can answer any questions you or anyone else has.

We put the code and data online here:

https://github.com/joeguinness/vaporfly

One of the nice things about the researchers.one publishing platform is that experts can read the paper for free and leave their feedback, comments, and suggestions. We'd love to hear from runners and coaches.

Technical question. Why did you run the model separately for men and women? This increases your degrees of freedom than if you had gender and gender*vaporfly as predictors in the model.

One other choice I'm not sure I agree with is the variable marathon denotes the course, but due to weather and other factors, temporal variations in performance on the same course might be larger than between course performances. What would happen if "marathon" was unique course -years, e.g. if boston 2015 and boston 2016 were treated as different marathons. Presumably, since marathon is a random effect, and because you have multiple people in your data set in the same race, this would be possible.

joeguinness wrote:

Hi Rojo, I'm one of the authors of the study and can answer any questions you or anyone else has.

We put the code and data online here:

https://github.com/joeguinness/vaporfly

One of the nice things about the researchers.one publishing platform is that experts can read the paper for free and leave their feedback, comments, and suggestions. We'd love to hear from runners and coaches.

I can't get this to run. I'm following the read me, and reproduce files, but there appears to not be any files named "men_sampled_shoe.csv" etc. When I try to generate them using the generate_sampled_dataset.R, it seems to require that I scrape the data from marathon guide?

# read in the command arguments

args <- commandArgs( trailingOnly = TRUE )

input_scraped <- args[1]

input_matches <- args[2]

max_time_minutes <- as.numeric( args[3] )

output_csv <- args[4]

Can't you add the final datasets from the scraping to the github folder? Without them, this study isn't exactly reproducible, because we have no way of knowing whether these databases (marathonguide) will still exist in the future.

These files appear to be missing

men_sampled.csv men_shoe.csv men_sampled_shoe.csv

women_sampled.csv women_shoe.csv women_sampled_shoe.csv

lobster wrote:

2.1 minutes as the minimum of the improvement range seems kinda crazy.

The vapor fly was pretty widely adopted by 2018 yet only Kipchoge managed to break 2:04, and the average top 100 time was well within the standard range for the last decade (and slower than some other years like 2012). 2018 would need to be by far the worst marathon ing year in the last decade for a 2.1 minute typical improvement to make sense.

2019 was significantly faster than 2018 (which is still weird considering the vapor fly was already popular in 2018. Are the next% significantly better?), but 2.1 minutes is still kinda pushing it even then. You still have second place at 2:04 high after the adjustment, which isn't unheard of (this occurred in 2015, but 2015 was the worst marathon ing year this decade) and the top 100 times were close to 1 minute better than usual, not 2.

See my earlier post-- % improvement, which they also give, is a better way to look at it than time. It doesn't make a lot of sense for a 2:24 guy (their cutoff for men) and a 2:05 guy to improve by the same amount of TIME, because that would mean the faster you are, the bigger speed boost you get.

I do think it'd be interesting to separate out the elites from the sub-elites and see if the effect is diminished.

Jeez, what rock have you been living under? For the millionth time, the 4% number is for reduction in "energetic cost," i.e., reduced oxygen consumption to maintain a given pace (improved economy), not a 4% improvement in performance. That translates to 2% or thereabouts in race performance.

animaxcg wrote:

So assuming this is correct can't Nike be sued for false advertising of the 4%+ running efficiency benefit? or do they have some * for legal reasons on the shoes marketing? because 2.1-4min over 120+ min is less than 4%

2.1 minutes is 1.46% of 144, so the 2:05 guy would still be improving by 1.83 minutes if they get the same benefits, which is still kinda outside of what seems reasonable when looking at changes in top 10 or top 100 times.

There are a lot of great questions and suggestions here. Let me know if I miss anything

@Ford Taurus: There was no funding for the study.

@DirtyT and @Alexi Santana: On the 4 year window of the study and improved performances in marathons over time. This is a really interesting question, especially since the olympic trials operate on a 4 year cycle. If some component of the improved performances over time are due to new people entering the sport, then we are accounting for that directly by including a term for each runner in the model. If the improved performances come from improved training, calorie consumption, etc., we are only indirectly accounting for that through the terms specific to each marathon-year combination. What I mean is that, each individual race gets a term in the model, so if people generally did better in chicago 2019 than chicago 2018, that improvement can be absorbed by the terms for these two races in the model. We could try directly adding a term that varies smoothly over time to directly account for the effects you're talking about, but we haven't tried that.

Related to your point: the model assumes that runner abilities are constant over time. We would like to include an age effect in the model, to account for the fact that runners generally improve during their younger years and get slower during their older years. This next point is kind of subtle, but I think that the vaporfly effects would actually get bigger if we included age effects. This is because our sample of runners is biased towards containing runners that were older than average when the vaporflys appeared, because we required the runners to have a fast performance in 2015 or 2016. So they were of average age for elite marathoners when they were sampled (in 2015 or 2016), which makes them older than average when vaporflys started appearing in 2017 and 2018. So this sample of runners, on average, is declining in ability when the shoes appear. Hopefully I explained that somewhat coherently.

We would love to team up with someone who is willing to dig on the internet and find the ages for all the runners in our sample.

There very well could be a placebo effect of the shoes. Our study isn't capable of detecting that. I would love to see some placebo vaporflys.

@govlie1: we did consider your point about more serious runners adopting the vaporflys. This is accounted for by controlling for runner ability in the model. We also selected men under 2:24 and women under 2:45, so these are all serious runners (though maybe some are hobby joggers by letsrun MB standards).

@Ho Hum, we also considered a model in which vaporflys impart a percent improvement rather than a fixed time improvement.

@c$: the Upshot study looked into differences in training volumes, and they didn't find anything meaningful, as I recall. They did a really nice job on their study.

https://www.nytimes.com/interactive/2018/07/18/upshot/nike-vaporfly-shoe-strava.html

@MatthewXCountry: on reproducibility. We sent a message to marathonguide.com asking for permission to post the data that we scraped, but we didn't receive a response, so to be cautious, we simply posted the code used to scrape the data. If you send me an e-mail (

joeguinness@gmail.com

), I can send you the data files you need.

@MatthewXCountry: These are great questions about interactions and marathon-year effects. we also fit the interaction model and have the results in an updated version of the paper available here:

https://arxiv.org/abs/2002.06105

The original paper did include the course-year effects you describe. In the updated version, we have both course effects and course-year effects. The size of the vaporfly effects shrink towards zero a little bit, but the overall story doesn't change.

joeguinness wrote:

In our dataset, there were 71 men and 56 women who switched shoes from a non-Vaporfly to a Vaporfly shoe. However, the other people in the study are helpful for sharpening the estimates of the vaporfly effect.

Here's an example, suppose that runner A and runner B both run Boston and Chicago in the same year. Runner A wears non-Vaporfly shoes in both races, but runner B switches from a non-Vaporfly shoe in Boston to a Vaporfly shoe in Chicago. This gives us a piece of information about the effect of the shoes because we can compare how much runner B (the shoe switcher) improved in Chicago relative to how much runner A improved.

The gist of our analysis is combining a bunch of these mini comparisons in an optimal way, using a technique called generalized least squares.

Thanks Joe; I appreciate your response and your example is helpful.

That is helpful, agreed with randomcoach on appreciating the reply.

It makes sense on the point of having "average" aged elite marathoners at the start of the study vs. at the end. Is there also potential selection bias since you pick runners who were elite at the beginning of the window but not at the end? For example if you used criteria that required someone run under 2:17 by 2017, would this bias the sample since they would by definition run at least that fast early on but not necessarily towards the end? Sorry if I missed details in the article / thread about the selection criteria.

Wow all this hub bub over 2-4 mins over a freaking MARATHON?!?! Who cares.

VroomVroom wrote:

Wow all this hub bub over 2-4 mins over a freaking MARATHON?!?! Who cares.

0/10

It would be interesting to know how many runners are actually using vaporflys and then abandoning them in subsequent marathon...

My guess is there aren't many of them. (Good for Nike)

If everyone who uses them seems to concur that they help - and those running with them tend to be more serious about their running - then I don't think it's likely someone would want to take that step backward.

Good point.

Although I think in the future people may try switching to the Adidas Adizero Adios pro.

VroomVroom wrote:

Wow all this hub bub over 2-4 mins over a freaking MARATHON?!?! Who cares.

Just imagine, you could improve from your 3:42 to a blazing sub 3:40!