A Word on Causation and Correlation
Correlation need not imply causation. Chances are, you’ve heard this expression before, especially if you’ve ever taken an introductory statistics class. This cliché, unlike many clichés, happens to be extremely true. Despite the ubiquity of this lesson, people make the mistake of drawing false causal inferences every day. In plain English, this means that people often see that two factors are connected, and mistakenly assume that one causes the other - but feel free to use the first line at cocktail parties.
Today, we’ll explore how false causation can affect the subject of athlete retention in collegiate rowing.
How Does Retention Affect Performance?
Most would agree that retention is an important variable in team performance. More time with the same team and coach intuitively allows an athlete to maximize their physiological and technical development. Many would argue that retention also productively contributes to the establishment and maintenance of a team culture.
Data collected from the 2016 and 2017 IRA and NCAA National Championships would support this initial impression. The Sparks Consulting research team, using linear regression models in R, found that each athlete retained predicts a 2.90 point increase at the IRA National Championship Regatta and a 2.36 point increase at the NCAA National Championship Regatta. In practical terms, this means that two additional retained athletes can make the difference between 2nd and 1st place in the varsity eight at IRAs, and three additional retained athletes can make the difference between 3rd and 1st place in the varsity eight at NCAAs.
But what happens when we add another variable to the model? Will the connection hold up? If there is true causation, then it should. Let’s take a look.
What Else Could Be at Play?
A good variable to add might be points earned at last year’s regatta. It stands to reason that teams that were good last year have a better chance of being good this year, and there is a chance that this variable will pick up some of the effect.
And guess what? It does. When this variable is added to the models, it dramatically changes the predictions. In the case of the IRA model, it completely wipes out the effect of athlete retention, knocking it far outside the realm of statistical significance. And in the NCAA model, the predicted increase in points drops from 2.36 to 0.89. This means that at NCAAs, it would take seven retained athletes to bridge the gap between third and first, rather than three. Obviously, this drastically alters our impression of the role of retention in both the IRA and NCAA.
So what is the true extent of prior performance’s role in all of this? Our team built a third set of regression models, this time measuring the relationship between prior performance and retention. Simply put, we investigated whether a good championship performance leads to more athletes sticking around, and, conversely, whether a bad championship performance leads to more athletes deciding that college is full of opportunities, and perhaps another one may be a better use of their time.
The Results
Our models showed that prior performance significantly affects retention. As a regression model, this result is worded strangely. The model has an intercept term of 22.62 for IRA teams, and 22.69 for NCAA teams. This means that, at a bare minimum, a team can be expected to return either 22.62 or 22.69 athletes. Even if they earn zero points, they can be expected to return either 22.62 or 22.69 athletes.
Here’s the important thing: for every point a team earns, the predicted retention increases. In the case of the IRA, each additional point scored predicts that 0.06 additional athletes will stay. In the case of the NCAA, each additional point predicts that 0.074 additional athletes will stay.
Still unsure? Let’s put it into context. Say a team finishes right in the middle of the pack in all three eights at IRAs, 12th place. This would give them a total of 84 points (35 points in the varsity eight, 28 points in the second varsity, and 21 points in the third varsity). 84 times 0.06 is 5.13, and 5.13 plus 22.62 (our intercept term) is 27.75. Thus, we predict that a team that finishes right in the middle of the pack will return 27.75 athletes the following year.
Let’s take a look for an equivalent NCAA team. If an NCAA team gets 12th in all three boats, they will earn 66 points (33 for first varsity, 22 for second varsity, and 11 for varsity four). 66 times 0.074 is 4.884, and 4.884 plus 22.69 is 27.57. Pretty close to the IRA results.
What if a team crushes it and wins gold in the first varsity, silver in the second varsity, and bronze in the third varsity (eight for IRA, four for NCAA)? The 1V gold medal will be worth 66 points at NCAAs and 90 points at IRAs. The 2V silver medal will be worth 42 points at NCAAs and 68 points at IRAs. Finally, the 3V/V4 bronze will be worth 20 points at NCAAs and 48 points at IRAs. Plugging those numbers into our model, we get a predicted 34.98 athletes retained for the IRA team, and a predicted 32.16 athletes retained for the NCAA team. Clearly, this represents a significant improvement over the middle of the pack team.
This Has All Secretly Been a Lesson in Confounding
Remember the word at the beginning of this article on causation and correlation? This is where that comes back into play. In statistics, confounding occurs when a model shows a relationship between X and Y, but both variables are in fact influenced by a third factor, Z. Recall that our first models showed a relationship between retention and performance, suggesting that better retention could drive better team performance. But when prior performance was added to the model, we found that it absorbed most of the effect, and in fact it was driving the improved performance, not retention. Finally, we found that prior performance significantly contributed to higher retention.
In the example above, retention is factor X, this year’s performance is factor Y, and last year’s performance is factor Z. While it looks as if factor X affects factor Y, in reality factor Z is affecting them both.
It is true that better retention correlates with better performance. But this is because teams that did better last year are more likely to retain athletes, and teams that did better last year are also more likely to do well this year. More simply, the correlation between retention and performance is not due to any causation, but is due to confounding. In technical terms, it’s spurious, and you can find plenty more examples here.
Comparing International Athletes and Prior Performance
Now that we know that prior performance is a significant predictor of present performance, let’s compare the influence of prior performance to the influence of international athletes. If you missed it, we examined the role of international athletes in collegiate rowing in detail in this article.
Recall that our model shows that international athletes significantly drive collegiate rowing performance on the men’s side, but not the women’s side. Our IRA model showed that each international athlete predicts an additional 1.88 points. If we add prior performance to this same model, we find that each point scored last year predicts an additional 0.77 points this year.
You may look at this and say, “Okay, well this means that international athletes play twice as large of a role as prior performance.”
Not quite. The average IRA team in the time period between 2015 and 2017 entered the championship regatta carrying 7.88 international athletes on their roster and hoping to improve upon the 87.22 points they scored last year. This means that our model predicts that the presence of the 7.88 international athletes on the roster will lead to an increased 14.81 points, and that last years 87.22 points will lead to an increased 66.29 points. As such, the aggregate effect of prior performance is far stronger than the aggregate effect of international athletes.
This strong effect displays the dynastic tendencies of collegiate rowing. Our models suggest that teams are able to “snowball”, constantly building upon success through better retention and other factors.
To recap, our study of collegiate rowing thus far has yielded the following results:
- International athletes significantly contribute to team performance on the men’s side, but not so much on the women’s side.
- Prior performance is far more important.
- Prior performance also contributes to higher retention, which in turn correlates with better performance.
- Coaching experience is not a useful predictor of performance (see our article on that).
- As far as we know, the best way for a college team to do better this year is to do better last year.
- This is not very practical advice, but it is certainly a building block for future research.
