Double Trouble? Assessing the impact of cup competitions on league performance

Introduction

In 1955, when the Sunderland A.F.C. was in the running for both the FA Cup and the League Championship, the club’s first-team players held a meeting in which they discussed which competition they should concentrate their efforts on. When one player suggested that they should go all out for both, captain Len Shackleton dismissed the proposition.

“Can’t be done,” said the international inside-left. “Never been done.”

The players agreed that the Cup, then the competition that captured the sporting public’s imagination to the greatest extent, should be their priority. In the event, they won neither, losing their semi-final tie to Manchester City and missing out on the Championship by 4 points.

Shackleton was wrong about the past, Aston Villa and Preston North End having both won the League and Cup double in the nineteenth century; and six years later, he would be proved wrong about the future. Today, no English team in with a chance of a double would assume that its achievement was impossible; and given the money at stake in the Premier League, few if any would take the same decision as Shackleton and his colleagues. Nonetheless, his concern that a team’s efforts in one competition could damage its chances in another is alive in the modern game. With the old order of priority reversed, Premier League managers have been known to field what looked like deliberately weakened teams in cup-ties in order to keep their best players in their best condition for upcoming league matches.

It is easy to see why they may do this, even if one doesn’t like it. More games in cup competitions mean more opportunities for a team’s players to become injured or tired, reducing their availability or efficiency in league matches. Having less time between matches may interfere with a coach’s ability to prepare his players for those games, and the resultant mental fatigue may reduce their receptiveness to such preparation. But the fact that something could be true doesn’t mean that it necessarily is true; and if I were one of those coaches, I would want to know how important these effects I was trying to avoid were. Does “double trouble” exist, and how strong are its effects?

Method and Data

To answer this question, one needs to find some way of controlling for a team’s underlying ability, which is the very thing a league competition is designed to test. One cannot simply correlate league results with cup results in isolation, because any effect that the one has on the other will be crowded out by the fact that both are highly correlated with a team’s true ability.

Fortunately for this study, if not for the excitement in the game, Kuper and Szymanski (2010) have found that a team’s league position is highly correlated with its payroll. I have therefore decided to use a club’s wage bill, available on fbref.com for every Premier League team from the 2013/2014 season, as a proxy for its underlying ability. (It should be acknowledged that for much of this period, most or all of the teams’ given payrolls are estimates rather than exact figures, and there is no season in which the financial data are perfect.) The same website also provides records on these teams’ performances in all first-class cup competitions from 2014/2015. I have therefore taken data from that point up to 2022/2023, linking league performance, cup performance, and wages in a unified dataset for each season studied. An unidentified error prevented my doing the same for the 2023/2024 season. For each season, teams were ranked by their league points, by their wages, and by the number of minutes they played in cup competitions. The qualifying rounds of continental cup competitions, which are not counted by the source website when computing playing time, were not included in the latter computation; and neither was the Community Shield, for which I forgot to make a column and could not be bothered to do so once I’d started.

A financial Over-Achievement Score (OAS) was created for each team by subtracting its ranking in league points from its ranking in wages. For example, the Leicester City team of 2015/2016, which won the Premier League championship with the division’s 17th-highest payroll, has an OAS of 16. Teams could then be ranked according to how well they over or under-performed their financially-implied expectations; and the effect of cup runs on league performance could then be estimated by studying the relationship between OAS and Total Cup Minutes, using rank orders to perform correlation analysis.

For those who have not read my previous posts comparing ranking systems and tie-breakers, I will explain my methods in more detail. If you have, or if you are familiar with rank correlation analysis already, or if you just find statistical theory boring, feel free to skip ahead to the next section.

The Pearson Product-moment Correlation Coefficient r is a measure of the strength of the relationship between two variables X and Y. It is calculated using the following equation:

(1) r = C(X,Y)/(σ_X * σ_Y)

σ_X and σ_Y are the respective standard deviations of X and Y. The numerator, C(X,Y) is the covariance between them, a measure of how much they vary with one another. The standard deviation of a variable V is the square root of the mean squared deviation from the mean of all values of that variable.

(2) σ_V = √([∑[V_i-E(V)² ]/n)

In the above formula, n is the sample size, i denotes an individual value of V and ∑ is a summation symbol. E is the expectations operator, denoting the expected, or average, value of the variable. The covariance of X and Y is calculated thus.

(3) C(X,Y) = E(XY) — E(X) * E(Y)

Pearson’s formula, however, rests on the assumption that the relationship between X and Y is linear and unbounded, and that each is a continuous variable. In this case, neither assumption is true. Both TCM and OAS are whole numbers with minimum and maximum possible values, and there is no a priori reason to assume that a relationship between the two must be linear. Furthermore, league position is an intrinsically relative variable; and if cup runs have an effect on a given team’s league performance, their effect must depend in part on the effect that other teams’ cup runs have on theirs. For these reasons, it is more appropriate to use the Spearman Rank Correlation Coefficient ρ. This is calculated by applying the Pearson formula not to the absolute values of X and Y but to their ranks within their relevant sets.

(4) ρ=(C[R(X),R(Y)])/(σ_(R(X)) * σ_(R(Y)) )

Returning to our previous example of the 2016 Leicester City team, their OAS of 16 was unsurprisingly the highest in the Premier League that season. The 510 minutes they played in cup competitions was only the 15th highest. Ergo, the R(X) for that team in that season is 15 and the R(Y) is 1.

This method makes no assumptions about the distributions of the X and Y variables, and does not depend on linearity in the relationship between them in order to get an accurate estimate of ρ. It does, however, make the assumption that the relationship is monotonic: that whether Y increases or decreases in X, it will either always increase or always decrease. If cup runs are indeed bad for a team’s league performance, they must be always bad. There must be no point at which a team’s league performance starts to improve as a result of more cup-ties being played. Similarly, if cup runs are good for league performance, there must be no point at which extra cup-ties become a hindrance. Given the multifaceted ways in which cup runs could conceivably affect league performance, this assumption is a weakness. A team might gain a confidence boost from a third-round FA Cup victory which improves its performances in subsequent league matches, yet find a run to the semi-finals too much to handle and run out of energy as the season approaches its end. The assumption of monotonicity is in this case a necessary simplification, which may or may not correspond to reality, in the attempt to establish a general truth.

Once ρ has been calculated, it must be interpreted. Even if there is no true relationship between X and Y, random variation means that the observed value of ρ is unlikely to be exactly zero. What is important is whether the difference is large enough to constitute meaningful evidence that a relationship exists. This can be investigated by using the observed value to construct the test statistic:

(5) t=ρ√[(n-2)/(1-ρ² )]

In a league with 20 teams, n-2 is 18. Under the null hypothesis that the true value of ρ is zero, t can be expected to approximately follow a t-distribution with n-2 degrees of freedom. This results in a probability of 90% that ρ will be no greater in magnitude (positive or negative) than 1.734, a 95% probability that ρ will not deviate from zero by more than 2.101, and a 99% probability that the difference will not be greater than 2.8784. Comparing the observed value of ρ against these critical values gives one a good idea as to the strength of the evidence of the null hypothesis, although it is worth remembering that the probability of getting such extreme results given the null hypothesis is not the same as the probability of the null hypothesis being true given the results.

Results and Conclusion

Below are the correlations and their associated test statistics for each season studied.

From this table, three conclusions can be drawn. First, it appears that if cup runs do have a uniform effect on league performance, the effect is negative. The second conclusion is that if this effect exists, it is likely to be very small. The third is that the evidence for the effect is very weak. There are more negative correlations than positive ones, but it is only a 5–4 split. The mean correlation over the nine seasons studied is slightly negative, but only slightly. The average correlation coefficient of 0.9269 means that on average, cup minutes explain less than 1% of the variation in financial overachievement in the League. In no case does the test statistic cross even the 10% significance threshold of 1.734, let alone the 5% and 1% thresholds. Only in one season is the test statistic greater than 1.

It should be acknowledged that the low correlations may at least in part be the result of actions taken by clubs to mitigate the damage cup competitions may do to their league campaigns, such as signing more first-team players or fielding reserves in cup-ties. Other complications to be considered are the slight incompleteness of the data regarding cup competitions and the uncertainty of much of the financial data from which a team’s OAS is derived.

Nonetheless, the results suggest that Premier League clubs do not need to diminish these competitions by fielding weakened teams, at least any more than they already do at present. If cup competitions do have the potential to derail a team’s League campaign, whatever adjustments managers are already making to counteract this appear sufficient. Given the amount of money that has been poured into the Premier League, the decline in the status of the F.A. Cup as a national institution may have been inevitable, but there is little evidence to suggest that there is any advantage to be gained by a team downgrading it any further.

So…can we have our replays back, please?