New Statistics Pushing the Frontier of Goalie Analysis
“Advanced” statistics are here to stay. They have proven consistently that they add value to traditional ways of analyzing hockey by showing patterns and raising questions that are not otherwise accessible to us. Whether you are a fan or not, the fact is that possession metrics, zone entries and exits, scoring chances, and so on shed light on parts of the game that were under-appreciated before.
Goaltender metrics, unfortunately, have generally been tougher to develop for a number of reasons. For a long time, the only “advanced” goalie statistic in wide use has involved separating 5-on-5 performance from power play and penalty kill performance. It has only been in the past few years that in-depth looks at goaltender performance have begun to bear real fruit.
The key concerns for a statistic is that it be accurate and useful – that is, that it should give information that is a good reflection of what has happened and that can be used to make real decisions about the future. In the recent past, when it comes to goaltending, we have gotten better at seeing what happened. We are not getting better at deciding what to do.
There are several reasons that goaltending stats have been so hard to develop.
Perhaps one of the biggest issues is that the necessary skill set is uncommon. This work requires detailed technical knowledge about the position (from technique to biomechanical factors to equipment) plus statistical and computer coding skills and an analytical perspective. That narrows the pool of people working effectively on the problem. Not to mention the amount of time it takes to do this properly.
Second, the current big data approach uses a flawed data set: the NHL play by play files. These are the real time reports of what happened in the game, which are then combed for data points that can be compiled by computer into the stats we see on sites like NHL.com, War-on-Ice, and Puckalytics. These files are both limited in what information they contain and riddled with errors: shots marked in the wrong locations, at the wrong time stamp, or mislabelled altogether. Phantom shots get added and actual shots are not counted, and so on.
There are ways to combat some of these errors, but we really have no good understanding of the extent of the problem. In fact, the only way to truly get a grasp on it is to manually track games and compare them to the play by play files, which defeats the entire purpose of automated collection. It also inserts its own errors into the data set.
It is currently a bit of a controversy as to how unreliable the files actually are. Suffice to say that the data is known to be flawed, although it is hotly debated how bad the problem is and to what degree it affects the conclusions we’re able to draw.
And third, there is, frankly, a reluctance to move away from old methods of thinking about how to evaluate on both sides of the statistical divide. Traditionalist thinkers who have relied on their eyes for many years are often reluctant to embrace the concept of adding a statistical check on perceptions. And the people who have spent years creating and refining statistical methods are reluctant to admit that they are asking the wrong questions about goaltending and using the wrong data to answer the questions they do come up with.
This is how we end up with saws about goaltending being “voodoo” rather than it being a position played and conceived differently than the rest if the game. And as long as stats fail the people working daily to make goaltenders better, there will be reluctance to fully embrace new statistics.
The question the stats have been designed to answer is “how do we prove that Jonathan Quick is an overrated goalie?” The questions we ought to be asking are “how do we know what our goalie ought to be working on?” and “how do we set benchmarks for success?”
Eventually we want to be able to look at a goalie in their early career and have at least a moderately good grasp on whether they will be successful in the future. To do that we need to know what parts — out of all the things that goalies do — are essential and what parts are negotiable, whether those things are learnable or not, and how to measure improvement or decline.
We are getting better, even if it is at a very slow pace. But there have been some new statistics developed in recent years that illuminate previously unmeasured areas.
These measures are not perfect. They do not tell us everything that we need to know and they have not yet helped to isolate areas of goaltender skill the way that shot attempts and transition measures have for skaters. There is a long way yet to go to find a set of stats that allow observers to know what skills are important for a goalie to have and how each goalie is doing at those skills.
Nonetheless, these measures allow us to see some things that are essentially hidden by raw save percentage. And that is an important step forward.
The big development, pioneered by War-on-Ice, has been to collect and track shot location data. This is information already included in the NHL’s play by play files, so the same kind of data is available for every goalie almost immediately after each game is concluded.
By figuring out exactly how much difference location makes to the likelihood of a shot going in the net, War on Ice were able to divide the rink into zones and then compare each individual goalie’s record on shots from each zone to the league average. This allows for a more nuanced picture of goaltender results that accounts for the kind of workload they actually see.
It is constantly updated and publicly available, two enormous points in favor of it being adopted widely. And unlike earlier ways of measuring performance, location data helps to account for at least one aspect of difficulty that has a demonstrable effect on save percentage.
There are multiple new statistics that flow from the adoption of location data. (Note that I am not calling this “shot quality.” Shot quality/save difficulty is a function of a number of variables of which location is only part.)
- Adjusted Save Percentage (War-on-Ice.com) applies each goalie’s save percentage in each of the three danger zones to the league average workload. Thus Adjusted Save Percentage is more directly comparable from goaltender to goaltender than raw save percentage is. War on Ice also include some rebound and rush information in their calculations.
- High Danger Save Percentage (War-on-Ice.com) has been shown to be the part of Adjusted Save Percentage that drives most of the differences between goalies in the long term. It is also the part that seems to be the most repeatable. That means it might be the zone that gives us the most information about goaltender talent with the least noise. It’s important to note that while HD save percentage is more consistent year to year, it is not truly predictive and shouldn’t be taken as such. It is, however, an important incremental improvement on save percentage.
- Expected Save Percentage, or xSV, from DontTellMeAboutHeart and HockeyGraphs, takes danger zones and other information in the play by play file to derive a rate at which an average netminder would stop the kinds of shots seen by each goalie. This allows comparisons between what a goalie has actually done and what they were expected to do. Like the above measures, it improves to some extent on raw save percentage in predictability. This data is not updated with every game, however, and testing of its predictiveness is not complete.
- Adjusted GSAA/60, from Nick Mercadante, turns Adjusted Save Percentage into a standardized Goals Against figure showing how much each goalie is helping their team (or not) in terms of the number of goals saved above or below what an average goalie would have saved. There’s no indication that adjGSAA is predictive from year to year. It does, however, make comparisons between goalies more intuitive for many people.
With measures like these, we can see more than we used to be able to see. This helps bring more people into the conversation and serves to direct observers’ attention to the more dangerous chances a goaltender faces. When analyzing the game, it is crucial to use the best available tools for talking about performance.
However, there is no single statistic that will tell you if a goalie is elite or undistinguished or starter material. This is especially true of statistics that cover a period of time less than several years long, at which point decisions have already been made about them and the roles they will play.
Any categorical claim about goaltender talent that is based on any single statistic should be regarded with a great deal of skepticism. And most such claims based on even a set of the statistics we have now should be considered very carefully. It takes a high level of agreement between a wide range of statistics carefully chosen and interpreted to support sweeping claims about any players.
In the end, despite the gains made in the past few years, a skilled observer’s eye test is still more trustworthy than the statistics on goaltending. The information that tells us something truly actionable comes largely from understanding the position at a very deep and detailed level. Does, for instance, Ryan Miller need to be more aggressive to be successful? This is something only careful observation and deep knowledge can answer right now, even as we get better at seeing where Miller is doing well and where he might be struggling.
We need the eye test because only the eye test gives us information we can actually use. It’s still not enough.