If a running shoe made you 25 percent faster, would it be fair to wear it in a race? What about 10 percent? Or 2 percent? The Nike Zoom Vaporfly 4% — a bouncy, expensive shoe released to the public one year ago — raises these questions like no shoe in recent distance running history.

Nike says the shoes are about 4 percent better than some of its best racing shoes, as measured by how much energy runners spend when running in them. That is an astonishing claim, an efficiency improvement worth almost 6 minutes to a 3-hour marathoner, or about 8 minutes to a 4-hour marathoner.

And it may be an accurate one, according to a new analysis by The New York Times of race data from about 500,000 marathon and half marathon running times since 2014.

Using public race reports and shoe records from Strava, a fitness app that calls itself the social network for athletes, The Times found that runners in Vaporflys ran 3 percent to 4 percent faster than similar runners wearing other shoes, and more than 1 percent faster than the next-fastest racing shoe.

We found that the difference was not explained by faster runners choosing to wear the shoes, by runners choosing to wear them in easier races or by runners switching to Vaporflys after running more training miles. Instead, the analysis suggests that, in a race between two marathoners of the same ability, a runner wearing Vaporflys would have a real advantage over a competitor not wearing them.

The advantages for runners wearing Vaporflys were consistent for slower racers and fast ones; for men and women; for runners on their second marathon or their fifth.

The Vaporflys — which retail at $250 a pair — were widely released to the public by Nike last summer. Unlike most running shoes, they have a carbon-fiber plate in the midsole, which stores and releases energy with each stride and is meant to act as a kind of slingshot, or catapult, to propel runners forward. Compared with typical training shoes, the Vaporflys are believed to wear out quickly: Some runners have said they lose their effectiveness after 100 miles or so.

The apparent effectiveness of the shoes highlights an issue that has vexed sports officials for decades: how to determine which technological advances constitute an unfair competitive advantage. Golf officials barred the use of certain balls that fly straighter, the NFL barred the use of a sticky substance that helped players catch the ball, and swimming officials barred high-tech suits that were said to have enhanced buoyancy and speed.

The swimsuits were believed to lower times by as much as 2 percent — comparable to the apparent advantage derived from Nike Vaporflys relative to the next-best-performing popular shoes in our data.

The International Association of Athletics Federations, track’s governing body, has rules about shoes, but they are vague: “Shoes must not be constructed so as to give athletes any unfair assistance or advantage.” It does not specify what such an advantage might be.

The rules also state that shoes “must be reasonably available to all in the spirit of the universality of athletics.” The Vaporflys sell out quickly; on the secondary market, a pair can cost $400 or more. Nike’s newest version of the shoe, the Elite Flyprint, was sold to a limited number of runners in London for the 2018 London Marathon at a cost of 499 pounds (about $650).

When asked whether the shoes conform to track and field’s rules, a spokesman for Nike wrote in an email that the shoe “meets all IAAF product requirements and does not require any special inspection or approval.”

Yannis Nikolaou, a spokesman for the IAAF, said that while it’s accurate to say that the Vaporflys are legal, it’s actually more accurate to say there is no evidence they shouldn’t be.

“We need evidence to say that something is wrong with a shoe,” he said. “We’ve never had anyone to bring some evidence to convince us.”

WHERE THESE ESTIMATES COME FROM

An ideal experiment to measure how much shoes matter for race performance might involve a series of marathons on a variety of courses, with runners randomly assigned different running shoes.

That experiment does not yet exist, but something like it happens around the world almost every weekend, when tens of thousands of amateur runners compete in races and upload their race data — collected on smartphones or satellite watches — to Strava.

This data usually includes statistics like a runner’s overall time, splits for every mile and a map of the runner’s route. In about one-third of the races on Strava, athletes reported data on the shoes that they wore.

Based on profiles from more than 700 races in dozens of countries since 2014, The New York Times compiled results from about 280,000 marathon and 215,000 half marathon completed races. (These race records are public, but The Times obtained permission from every athlete named in this article.)

Below, we describe the four ways we measured the shoes’ effect.

MEASURING SHOE EFFECTS USING STATISTICAL MODELS

Pros of this approach: Tries to control for race conditions, weather, gender, age, pre-race training and a runner’s previous race times.

Cons of this approach: Still not a randomized controlled trial.

We know a lot about the runners in our data set, including their age, gender, race history and, in some cases, how much training they’ve done in the months before a race. We also know about the races themselves, including the distribution of runners’ times and the weather that day. We can put all of this information into a model to try to estimate the change in runners’ time from their previous races.

After controlling for all of these variables, our model estimates that the shoes account for an expected improvement of about 4 percent over a runner’s previous time. Including the uncertainty around the estimates, the Vaporflys are a clear outlier, one of the only popular shoes we can really say makes any difference at all.

This model estimates the effect of wearing Vaporflys compared with the effect of wearing any other shoes. It estimates the expected change in race time from a runner’s previous race given the runner’s age, gender, race history, training miles (for those athletes whose training logs are public) and the weather conditions on race day.

There are several statistical approaches one could take with this data set — and we tried several of them. None are perfect, but every way we tried, the effect of the shoes was more or less consistent: whether we included training miles or omitted them; whether weather data was included or ignored; or whether we modeled the change in time after switching shoes or the change in time from a runner’s average.

Of course, these observations do not constitute a randomized control trial. Runners choose to wear Vaporflys; they are not randomly assigned them. One statistical approach that seeks to address this uses something called propensity scores, which attempt to control for the likelihood that someone wears the shoes in the first place. We tried this, too. Our estimates didn’t change.

COMPARING GROUPS OF RUNNERS WHO COMPLETED THE SAME TWO RACES

Pros of this approach: Follows athletes of similar ability who ran in identical conditions.

Cons of this approach: Runners could save their special shoes for when they expect to have a fast race.

Tony Kharitonov and Marios Athineos don’t know each other — in fact, they live on opposite coasts — but they have something in common: They both failed to get the time they wanted in the 2017 Boston Marathon. But they failed together, just 9 seconds apart (3:02:04 and 3:02:13, respectively).

That race was one of the hottest in years, with temperatures approaching 80 degrees, and both runners were off pace after the course’s famous Heartbreak Hill, around Mile 21, and slowed considerably in the race’s final miles.

Running a marathon in less than 3 hours is a sign of an elite runner, and Boston is one of the world’s most prestigious races. In April, the men returned to run it again — and both switched to a different pair of shoes. Kharitonov wore Altra Solstice shoes, and Athineos wore Vaporflys.

Once again, weather was a factor. The 2018 race featured pouring rain, strong winds, low temperatures and a big rise in dropouts.

This time, the results were different. Athineos, 42, a researcher based in San Francisco, finished with 33 seconds to spare, at 2:59:26. But Kharitonov, 49, a software engineer in Millburn, New Jersey, could not stay warm, and once again could not maintain his pace. For the fifth year in a row, he failed to break three hours at the Boston Marathon, finishing in 3:07:17.

Perhaps not surprisingly, given the conditions, neither runner identified shoes as a factor in this year’s result. And, given what we know about how different the races were, it may seem unhelpful to compare times between them at all. But there is still meaningful information in events like these in aggregate.

Instead of directly comparing performances in the two races, we can compare the net change of runners who switched to Vaporflys with the net change of similar runners who did not. Of the 1,275 runners in our data who ran Boston both years, 52 switched to Vaporflys in 2018, as Athineos did. On the whole, those runners’ times improved more than that of the group of runners who did not.

This specific comparison may be too extreme for any study of shoes, and perhaps it is. But Boston 2017 and Boston 2018 are just one pair of races, and there are hundreds of pairs of races in our data in which large groups of runners ran the same two races and in which a subset of them switched shoes. When we compare all these pairs of races among all kinds of popular shoes, we see that, on average, runners who switched to Vaporflys improved their times more than similar runners who switched to any other kind of popular shoe.

FOLLOWING RUNNERS AS THEY SWITCH TO A NEW KIND OF RACING SHOE

Pros of this approach: Accounts for runners of varying skills over several races.

Cons of this approach: Runners could save Vaporflys for when they expect to be faster than normal, or Vaporfly wearers could be different in some way from other kinds of runners.

The runners who use Strava are an enthusiastic bunch: About 1 in 4 have uploaded data for two or more marathons, and about 1 in 10 have uploaded data for three or more. The Strava data allows us to follow these repeat racers over time, and, crucially, as they change shoes.

Glenn Kasin, 49, a doctoral student at the Southern California University of Health Sciences, is one of these avid runners. He has uploaded five marathons to Strava since November 2015, and he wore a different pair of shoes in every race. In January, at the Houston Marathon, he decided to race in Vaporflys.

In Houston, Kasin improved on his Boston time by about 18 minutes, an improvement owing to much more than a pair of shoes. He attributed his result, a 2:56:00, to many things: perfect weather, a fast course and the culmination of two consecutive years of a training regimen that included, on average, about 50 miles of running per week. But he said he also thought the shoes made a difference.

“Everything aligned, and then you throw that shoe on top of it,” he said. “The pace felt so easy that you felt like questioning: What’s going on here? When is it not going to feel easy?”

Kasin is a single data point, but in our data there are almost 4,000 runners just like him — men and women who have uploaded results for five marathons or more. When we aggregate the change in race times for runners the first time they switch to a new pair of shoes, runners who switched to Vaporflys improved their times more than runners who switched to any other kind of popular shoe.

MEASURING THE LIKELIHOOD OF A PERSONAL BEST

Pros of this approach: A measure of race performance most runners know by heart.

Cons of this approach: Doesn’t account for race conditions, increased training miles or aging. Runners who switch to Vaporflys could be different from other runners.

Race times are, in many ways, a crude way to measure performance. A marathon is not like a 400-meter dash or a 100-meter freestyle swim: One marathon may be hilly or feature sharp turns; others may be flat and straight. Weather, too, is important, with higher temperatures typically resulting in much slower times.

And yet race times are how runners qualify for prestigious races, like the Boston Marathon, and most runners know their personal best times by heart, regardless of whether the race they ran was flat or hilly, on a hot day or a cold one.

We can follow the runners in our data with this measure in mind, testing whether a runner’s fastest time is more likely when he or she switches to the Vaporflys, or any other kind of shoe.

Consider two fast runners in our data: Stephanie Andre, 35, a freelance writer from Bixby, Oklahoma, and Amanda Hicks, 34, a product director in Washington, D.C. Both women have uploaded several marathons to Strava and both switched to the Vaporflys for a recent marathon. Andre ran the 2017 Chicago Marathon and Hicks the 2018 Boston Marathon.

By objective standards, both women had excellent races. Andre placed 22nd among women in Chicago with a 2:41:50, fast enough to qualify for the 2020 U.S. Olympic team trials, and Hicks finished in Boston with a 3:15:31, among the top 5 percent of all female finishers that day. Andre set her personal record, but Hicks did not.

Someone can run a personal best for all kinds of reasons unrelated to shoes. A runner may train more, execute a better strategy on race day or run an easier course. Regardless, we found that runners who switched to Vaporflys were more likely to run their fastest race than runners who switched to almost any other kind of popular shoe.

PUTTING THEM ALL TOGETHER

None of these approaches are perfect, but they all point to a similar conclusion. Wherever we look for evidence that shoes matter in a marathon or half marathon, we find Vaporflys at or near the top of that list. More than that, the shoes tend to be outliers among all popular shoes, suggesting that there is something happening in races with Vaporflys that is not happening in races with nearly any other kind of popular shoe.

Nothing is certain, of course, and we describe some reasons for continued skepticism below. But from our perspective, the most honest answer we can give to the question “Do Vaporflys really make runners faster?” is a qualified yes.

REASONS TO REMAIN SKEPTICAL

The data is self-reported — and self-typed. All of our analysis comes from runners who publicly tell Strava what shoe they’re wearing during a race, which they type in manually. The race records included about 33,000 distinct shoe descriptions, and our analysis depends on standardizing those records. (Even for a shoe as heavily marketed as the Vaporflys, Strava users found 147 different ways to spell the name.) We’ve done our best to identify misspellings, remove shoe records that are clearly incorrect and consolidate different variations of the same basic shoe. But there’s no perfect way to do so.

More problematic is when runners do not identify their shoes with enough precision for us to categorize them. Wherever possible, we’ve attempted to identify the shoe to its most specific name, differentiating between shoes within a shoe line (or “franchise”). But not all runners were this detailed. This was most common among Adidas shoes. Adidas, for example, lists more than a dozen different subfranchises belonging to its “Boost” franchise. When runners did not specify to this level of detail, we simply identified the parent franchise of a given shoe. This could mask high-performing shoes within a franchise.

These estimates compare Vaporflys only with other popular shoes. Perhaps the greatest limitation of this data set is that for us to make meaningful conclusions, it requires that large groups of people run in a pair of shoes. This necessarily omits less popular shoes that, for all we know, may be faster than the shoes mentioned in this article. The analysis includes only the 50 most popular shoes — those with roughly 500 races or more in our data.

There still may be something we’re missing that’s not captured in the data. It’s possible that runners wear Vaporflys only when they know they are going to run faster, or that the act of wearing Vaporflys correlates with other things that indicate a runner is going to run faster. We acknowledge that it’s impossible to control for every factor that affects marathon performance. But we have some evidence that it really is the shoes.

First, we identified training mileage in the months leading up to a race for runners whose training logs are public, an important indication that a runner is getting more serious in his or her race preparations. Training miles are a variable we included in our statistical approach, and when we matched pairs of runners who ran in the same groups of races, we tried to match not just race time but also training regimens, where possible.

Second, the effect of the shoe was persistent across different tiers of athletes — for 3-hour marathoners and 4-hour marathoners alike. Faster runners improved their race times at about the same rate that slower runners did.

Finally, if runners raced in Vaporflys only when they thought they might have their best race, we might expect to see fewer of them on days with suboptimal weather. But we do not see that. In the 2018 Boston marathon, in particular — a race with conditions few runners would prefer — no shoe was more popular among the runners in our data who reported what they wore.

REASONS TO BELIEVE THESE ESTIMATES

This is a large sample of runners. There’s nothing random about people so enthusiastic about running that they upload information about all their races and shoes to a social media site. But we believe the data set is broadly representative of competitive runners. Among the races known as the World Marathon Majors — the largest and most competitive races in the world — about 1 in 6 finishers uploaded race data to Strava and made it public. Not counting the Tokyo Marathon, about 1 in 5 did.

These runners are a little faster than a much larger sample of runners that other researchers have examined, but they are broadly similar.

In all, about 600 different athletes reported wearing Vaporflys in 825 marathons and half marathons. They were the 32nd-most popular shoe in this data.

The speed increases associated with Vaporflys are consistent with the increases in a Nike-funded study by University of Colorado researchers published in the journal Sports Medicine. That experiment measured the energy output of 18 male elite runners on a treadmill at various speeds and concluded that, on average, runners in Vaporflys exerted 4 percent less energy than they did in other racing shoes. They estimated such a savings would translate into a performance improvement of around 3 percent for a 3-hour marathoner. Our findings were similar but showed a smaller effect. We found Vaporflys to be 1 percent faster than the next-fastest running shoe in our data (the Nike Streak), but our sample included a much wider range of athletes, and included men and women.

Runners who improved their performance in Vaporflys and then switched to other shoes got slower. There were very few runners in our data who met this criteria: We counted just 24 such races. But these runners finished about 7 percent slower on average than they did while wearing the Vaporflys. (Of the 24 races, 17 races were slower, and seven were faster.)

© 2018 The New York Times Company