Save Percentage And The Lies We Tell Ourselves
Save Percentage — in all of its forms — is currently indispensable to goaltender evaluation. No matter how knowledgeable an observer may be, we still have to have some kind of marker for results, a way to keep track of what happened to be able to compare with other times and other goalies. Save percentage has the advantage of treating every shot in a standardized way, meaning that it doesn’t favor a goalie whose look is preferred or who makes us gasp. It just counts shots. We need that.
Moreover, public sophistication concerning the statistical realities of save percentage is growing. More and more hockey fans, even casual ones, are becoming aware of just how volatile save percentage is over time. More people are talking about the need for large samples and the power of randomness. Variations on save percentage have begun to give more insight than ever before.
Yet we still live in uneasy alliance with save percentage, for one simple reason: it lies to us. Well, more accurately, it facilitates our very human tendency to lie to ourselves, which is in a way more dangerous.
Lie #1: Past save percentages allow us to predict the future.
Much of the time, belief that a particular goaltender is or is not worth relying on centers on the fact that their save percentage has hit a certain level for a period of time. Maybe that’s over a few games. Maybe it’s over a few seasons. “He put up two great seasons over the past four years.” “She’s been amazing since the first of December.” Whatever the basis is, observers are essentially using past performance to predict the future.
It doesn’t work well, unfortunately. If we have a really, really large sample of work — around 5,000 shots at even strength, or about 3 to 4 years as an NHL starter — we have some kind of indication whether that goaltender is likely to be above or below average for their career. Not for the next season. Not for the next 4 seasons. For all the years of their career, including the ones already played. By the time this sample accrues, important decisions have already been made about the goaltender in question.
And the confidence interval — the range we are confident the goalie will end up in — is still pretty large at 5,000 shots. It’s basically the difference between an elite save percentage and a below average save percentage. For instance, in 2014, Megan Richardson calculated that Corey Crawford’s “true save percentage”, given his performance over 5500 shots to that time, was somewhere between .906 and.921. That’s a huge range. That interval gets narrower the more shots the goalie sees, but it’s it still a calculation for the goalie’s entire career.
There is a way to try to forecast the future: goaltender marcels. Marcels weight recent performance more heavily than older performance. HockeyGraphs.com have introduced a marcel system that also takes into account aging and the likelihood of regression. These are more accurate than simply using career save percentage and they are not terribly difficult to create, but it does require taking a step beyond what we see on a player’s stat page. Marcels do represent, however, our current best guess for future performance.
Lie #2: We can use save percentage as an objective measure of talent.
There are numerous reasons to make us realize that save percentage is affected by subjective decisions. The biggest one is that classifying something as a shot is enormously subjective and changes at every arena.
But we should also be wary of claiming that save percentage is a measure of talent. Talent certainly makes up a portion of save percentage. Some goaltenders are better than others and have better long term save percentage. But –and this is especially true at the NHL level, where save percentages are clustered very tightly these days — that is only really visible on the extremes and after a really long period of time.
In other words, while save percentage does capture something about goaltender talent, it captures a whole lot of other things, too. Those other things mask the measurement of talent for most players most of the time. In addition, we have almost no real grasp on what those other things are. The defense in front of the goalie? Luck? Injury? Mental lapses? Coaching? Yes, all of this and more.
Is the talent portion 10% of the results? 40%? Is that level the same for every goalie or is one player dealing with more externalities than another? There is currently no way to tell.
So, yes, save percentage contains talent, but talent simply cannot be isolated from the rest of the noise that save percentage captures.
Lie #3: Talent is stable. Therefore change is mostly due to randomness and variance, which are out of human control.
Part of the reason that we’ve gotten to this point in goalie analytics is a stunningly bad assumption about what “talent” actually is. It is a notion that talent is some objective, measurable, essential, repeating characteristic located within every player regardless of environment. “True talent.” An immutable level of ability. Results are seen to vary around that unchanging core level, a level that is discoverable if the right tools are applied correctly.
This is, of course, neither possible nor, in the end, informative. Ability varies quite a bit over the course of a career. Players — both skaters and goaltenders — make technical adjustments all the time that affect their core abilities. Performances vary over time. The obvious concerns are age and injury, but ongoing technical tweaks learned throughout a career also matter to a player’s ability level.
This isn’t to say that variance and randomness don’t affect the statistics, because they do. Extremely high or extremely low save percentages will regress towards the mean given enough time and opportunity. But now that more people know about these powerful concepts, the default explanation for change is too often to shout “Variance! Randomness! Regression!”
These are not always the best explanation for change. In essence, assuming talent is stable prevents the awareness that performances vary not just because of the random processes of hockey working out over time, but also because of things that are wholly within the goaltender’s control. That makes it far too easy to conclude that there is nothing a goaltender can do to affect outcomes and that in turn makes it much easier to dismiss the evaluation of goaltending as inherently unproductive.
In other words, not only does the environment within which goalies perform change over time, their abilities do, too. That thing we are interested in measuring, the part that overcomes randomness in the long run, often changes long before save percentage can capture it. What looks like randomness may not be randomness at all, but we simply cannot tell by the numbers currently available to us.
Lie #4: All of these issues mean we should stop using save percentage at all.
It would be tempting, given all of these issues, to give up on save percentage altogether. This would be as big a mistake as putting too much weight on the statistic. It is good and necessary to have some level of objectivity to our evaluations. Save percentage — especially 5v5 Adjusted and High Danger save percentage — are some of the best current tools we have to look at goaltenders.
The other options for evaluation are Goals Against Average and Wins, neither of which make any pretense of looking at goaltender performance separate from team performance. Save percentage is leaps and bounds above either of these in giving information about goalie performance.
The key is to use save percentage responsibly, being aware of what it does and doesn’t tell us. We must temper our enthusiasm for our own conclusions when we only have a tool as troubling as this one. It’s far too easy to lie to ourselves when using save percentage. It lends itself to that, encouraging us to think we see more than we do. The number is not the answer, although it can be a hint towards asking questions.
And in the end, that is what we should be using objective measurements to do: ask new and better questions to generate new and better understanding.