Introduction to Hockey Analytics Part 6.3a - Luck and Random Variance: Indicators of Luck/Variance in Hockey (Part 1)

Who's the lucky dog? - Bruce Bennett

So we now understand how luck affects the game of hockey (See Part 6.1) and we've figured out what we should do when we see a player or team whose numbers are largely luck-influenced (See Part 6.2).

What we've yet to talk about is how we can tell when a team or player is getting lucky in the NHL in the first place. It's easy to talk about regression after the fact but that does us no good; what we really want to know is when a player is due for regression in the first place.

Let's explore this topic by going over a few of the more important ways to determine when luck is a major influence on results.

A. Individual Shooting %s:

We like to think about the ability of player's to score as a very skill-intensive ability. We talk about how certain players are better at getting dangerous shots on net and beating the goalie. And to an extent this is true. There are some players who will maintain high shooting %s and there are some who will maintain low shooting %s due to differences in ability.

However, there's a ton of luck involved in shooting % to the point where over small samples - or even season long samples - the statistic is more an indicator of how lucky a player has been than an actual measure of player skill. In other words, the amount of influence luck has on shooting % (for both teams and players) overwhelms the amount of influence of skill.

Why is this? Well let's think about the two types of luck we've spoken about thus far (see Part 6.1):

The first type of luck - caused by the inability of human beings to repeat actions perfectly:

If we had a pro-hockey player shoot from the slot against an empty net 100 times with no other players on the ice, there's a better chance than not that the player would miss the net entirely on a few shots. Now consider that normally a player is not facing an empty net, but is attempting to get the puck past a goalie, which requires him to alter his aim (he can't simply aim for the middle of the net!). Now consider that the player is unlikely in real-time hockey to have more than a split second to aim and they're extremely likely to be moving while they take the shot. And of course, there are defenders in the way that the player could be aiming to avoid with his shot (or to deflect off a stick).

So in a real hockey game example, even the best NHL players will fail to get a good majority of their shots on net, not because of their individual ability but because of how difficult such shots are to make and the limits of human ability - this is the first type of luck that we're talking about here. And the amount of shots they get on net is pretty much random - some players will be better at getting shots on net (these are the best players by the way) but even they will miss a good % of their shots. And remember when we're dealing with percentages, it's certainly possible that over a small sample that even events with small likelihoods (%s) can occur repeatedly. Thus if a player gets shots on net say 40% of the time, it's possible over a 10 game sample - let's say 100 shot attempts - that he might get 60 on net even though we'd expect 40.

Of course the second type of luck is extremely apparent in shooting percentage as well - players can't control whether opposing players will deflect the puck on net or off net, whether teammates will deflect the puck (remember when a teammate deflects the puck on net it's a shot on goal for the TEAMMATE, not the player, essentially negating a shot attempt), or whether the ice is such that the puck will be bouncing (particularly mid-period when it's been a while since a zamboni did its work or in certain arenas - cough MSG cough). And then of course a decent amount of shooting percentage is simply how frequently a player is used on the power play, where players get much better chances to score on goal.*

*This is why ideally we'd talk more about even strength shooting percentage than overall shooting percentage, but we can't ignore the more commonly cited and more readily available metric.*

Perhaps more important is what the goalie will do with the shot. Over a small sample it's totally possible for a player to make some amazing shots that would go in 90% of the time...only for a goalie to stand on his head and make the 10% saves. If that was to happen, we'd have a great goal scorer who seems to have a low shooting percentage - but not because of his own talent but rather because of bad goalie luck (we'd expect over time for that player to have a higher shooting %). Alternatively it's totally possible for a player to shoot the puck directly at a goalie such that he should make the save 90% of the time only for the goalie to somehow fail to make the clean save, resulting in a goal for a player who made a low percentage shot. If that happens multiple times over a small sample, it's totally possible for a player to have an inflated shooting percentage.

The end result is that while some players will have higher shooting percentages than others over their careers (assuming we're comparing players at the same position), most players will have shooting percentages over single seasons that can vary widely because of the influence of luck.

This doesn't make shooting percentage a useless statistic mind you in fact it just makes it a very useful statistic for a different purpose: figuring out when certain players are having a large amount of good or bad luck. How can we do this? Well two ways. First, over small samples it's very common to see shooting percentages that are abnormally high or low - so extreme that not a single NHL player in modern hockey has ever put up a shooting percentage that extreme over a single season. That's an obvious sign of good or bad luck and as you might expect from something that doesn't happen over an 82 game season, is something you should expect to regress hard as the season goes on.

Example: Eric Nystrom was a 4th line reject last year who was claimed by the Dallas Stars and suddenly was a huge scorer, putting up something like a 20.3% shooting percentage in his first 32 games with the Stars. No one maintains a 20% shooting percentage in the NHL which was a good sign that his sudden goal scoring ability wasn't a breakout as much as a burst of good luck for a mediocre player. Sure enough, Nystrom dropped off and had a 9.3% shooting % the rest of the way, scoring only 4 more goals in 42 games.

Of course, not every luck-affected shooting percentage is going to be extreme: it's quite easy for players to have shooting percentages that look not out of line for an NHL player but that are in fact the products of good or bad luck. This is why you should always try and compare a player's shooting percentage to his career #s. Alternatively, if it's a new player, you compare the shooting percentage to AHL statistics if you have them or to the league average amongst similar players. This is just the same thing we talked about in our post on regression - we're trying to figure out what the true shooting percentage talent is for our particular player and if our player's shooting percentage is out of line, we expect regression toward that true talent.

Example: 2 Years ago, Michael Grabner had a breakout 34 goal rookie season for the Islanders. He did this with a shooting percentage above 14%, which is a very high shooting percentage. Now, 14% shooting percentages are possible for certain players in the NHL - Matt Moulson is a good example. But they're not common and they are somewhat high.

Fortunately, as Grabner was a 24 year old Rookie, he had 3 seasons of AHL play behind him in his career, which we could use to try and figure out what his true talent shooting percentage might be. In those 3 seasons, Grabner was a 12% shooter, much lower than his percentage in 2010-2011. This was a good indicator that Grabner was due for regression and that shots would not go in as frequently in 2011-2012. Sure enough, his shooting percentage for 2011-2012 declined to 11.5%, much more in line with his AHL rate.

On Ice Shooting %:

If shooting percentage for individual players is heavily luck based, the same is also true of the shooting percentage for the team the player is on while that player is on the Ice. Yes of course we'd expect that a player has an impact on how well his team scores while he's on the ice: One would imagine, having John Tavares on the ice probably contributes to his linemates (and defensemen behind him) getting higher percentage shots.

But again, over a small sample or even a whole season, the same factors of luck we talked about above make a major impact. As a result, the shooting percentage of a team while a player is on the ice - the player's "on-ice shooting percentage" - is extremely random because the impact of luck overwhelms the impact of skill.

Why is this important? Well, because people like to cite things like +/-! If a team is being unlucky while a player is on the ice and is failing to score despite high percentage shots, the player's +/- will suffer despite the player probably being pretty effective on offense. If a team is being lucky, a player will have a better +/- than he deserves.

Example: In 2010-2011, the Islanders had an even strength on-ice shooting % of 8.1%. Jurcina's own even strength shooting % was only 2.8%. Yet somehow, Milan Jurcina had an on-ice shooting % of 11.15%. That's really high in general, and is a clear indicator that Jurcina's getting a lot of shooting luck - there's no reason why the team's shooting % should be 3% higher than normal (that's a large difference) especially as Jurcina isn't a great shooter himself. This was a decent part of the reason why Jurcina was only -4 despite playing on a team that was outscored by 35.

C. SAVE PERCENTAGE:

If there ever was a statistic that was heavily influenced by luck, it would be save percentage. On one hand this seems a little more counterintuitive than shooting percentage - goalies are often stationary and focused upon less things (the puck and nearby players) than shooters (who have to focus upon teammates and opposing players all around them and are of course moving while they shoot). Goalie statistics also have larger sample sizes than players - a goalie will see sometimes 30 shots on goal per game, whereas a player will get generally at most 5 shots on goal in a game.

Yet save percentage is hugely affected by luck, with luck generally overwhelming the role of skill in save percentage even over a sample that covers an entire season. Why is this?

First, it's because the gap between good goalies and bad goalies is much much smaller than that of good and bad shooters. A bad shooter could be a true talent 6% shooter while a high percentage shooter can be a true talent 14% shooter.* That's an 8% gap between the awful and the great shooters. A horrific goalie has an 89% SV% or perhaps even 90%, while the best goalies tend to be at 93%. That's a gap of 3-4%! In other words, while there may be more skill involved in being a good goalie than in getting your shots to go in the net, the gap between goalies is so small that even small amounts of luck can have large effects on whether we consider a goalie to be good or bad.

*These are both extremes of course, but you get the idea.

Second, it's because luck does clearly exist in the process of stopping shots from going in the net. Take the first type of luck for example: barring perhaps shots from point blank directly into the chest protector, no professional goalie will save a shot with a slim chance of getting past him 100% of the time. And if a 99% will-be-saved shot is let in by a goalie, as will happen generally 1 out of every 100 times (and remember goalies face around 1500 shots on goal per year so this isn't a small frequency), a goalie's SV% will drop a decent "amount" despite the goalie generally being elite at stopping that type of shot.

The second type of luck is huge as well - the goalie doesn't generally influence where the shots are coming from except with regards to rebounds. So if a goalie faces a lot of shots from the slot in his first 10 games, he'll have a lower shooting percentage than a guy whose team forces shots from outside the slot, because he's facing tougher shots. Then there's luck of whether a puck that a goalie has a bead on will be deflected by a defender in front of the net at the last second, screwing over the goalie who was previously perfectly placed. And even rebounds can involve a decent amount of luck: a goalie may be able to get away with a large amount of rebounds on shots for a short period of time because he's lucky enough that opponents just cant seem to get to those juicy rebounds and bang them home. If that's the case, the goalie's save % will be higher than it should be due mainly to luck (rebounds, as you might expect, are clearly higher percentage shots than non-rebounds). The reverse also holds - a goalie might find that every rebound for a few games is being pounced upon by the other team for easy goals, killing his SV% more than he deserves.

Over a long period of time we expect these effects to even out, but research has shown that such "long periods of time" tend to be longer than even a single season (I think the usual cited number is 3000 Even strength shots).

Example 1: Al Montoya had a career .904 SV% on over 5000 AHL shots prior to his Islander debut - not a small sample (I'm not using EVSV% here since AHL doesn't track that sadly). However, he put up an impressive .921 SV% for the Islanders in 2010-2011 over 21 games (18 starts). Such an increase is a very likely sign of a goalie's stats being inflated by luck and should raise concerns. Sure enough, Montoya put up an .893 SV% in 31 games in 2011-2012*

*Before anyone objects and claims "No this was injury - I'd point out that Montoya's SV% collapsed in December even before his injury."

> Example 2: Evgeni Nabokov had a .925 EVSV% through 28 games last year. That's an okay % - below average, but acceptable. In the next 14 games - his last 14 before being injured, his EVSV% dropped to .908%. What happened - did Nabby suddenly become a terrible goalie? Not likely. What is more likely is that Nabby just suffered a bunch of bad luck (Nabby's career EVSV% in San Jose appears to be .923)

There are some other ways to figure out when SV% is going to regress which we'll talk about in our section on goalies. For now, remember that SV% and related measures all are substantially affected by luck and that when a player shows a SV% way out of line with his career #s, there's a far greater chance this is caused by luck rather than the player "figuring something out". Players can of course actually improve (and "figure something out"), but most of the time we're dealing with false alarms.

D. ON-ICE SAVE PERCENTAGE:

Well this should be obvious, but if a goalie's SV%, consisting of a sample size of all the shots a goalie has faced over a season, is substantially affected by luck, the same is even more true of a non-goalie's "on-ice save percentage".

When I refer to on-ice save percentage, I mean the save percentage of the team while a player is on the ice. Again, this SV% is almost certainly affected by player skill. It's pretty hard to argue* that a player such as a defensemen can't cause opponents to take worse shots that result in easier shots for a goalie to save.

*But Possible.

But here's the problem - on-ice SV% uses an even smaller sample size than goalie SV% - since a player is on the ice for a fraction of a time of his goalie - and the same two sources of luck affects this version of SV% as goalie SV%. So with such a small sample size, on-ice SV% is incredibly affected by luck to the point where player skill in this statistic is pretty much overwhelmed by the amount of influence of luck.

So if a player has a on-ice SV% way out of line with the team's on-ice SV%, it doesn't mean that player is a great or poordefensive player. Rather, it means that the player has gotten lucky or unlucky and we should expect this to regress....and to take stats like +/- with it.

EXAMPLE: Through the first 26 games he played (between injuries), Milan Jurcina had an even +/- due to a .941 On Ice EV SV%. That's a percentage better than even the best goalies put up, which should have been a signal to many that regression hard was coming. Especially for a guy not expected to be an elite DEFENDER. In Jurcina's final 20 games? He had an on-ice EV SV% of .868. That's HORRIFIC.

Did Jurcina really go from being an elite defender to a horrible one? Of course not - what happened was simply a swing in luck. And in the 65 games he played last season, Jurcina's EV SV% was .884, again horrible and the worst of Isles' Defensemen. As a result, his +/- since those first 26 games in 2010-2011 regressed to -38 over 85 games after an EV his first 26. All due to both his on ice SH% and on ice SV% regressing.

Or to use a more positive example:

Example 2: In Hamonic's first 24 games in 2010-2011, his on-ice SV% was .884 - pretty damn lousy. Was he a horrible defender, despite looking great to the eyes and great possession #s? Did he deserve a +/- of -1? Not likely - clearly he was getting bad luck in what his goalies were doing behind him. Regression was coming to make his #s look better

In Hamonic's final 38 games? His on Ice EVSV% was .927 despite the Isles' overall EVSV% being a terrible .907 - rising his +/- from -1 to +4 despite being on a terrible team. As it is, this meant that Hamonic probably went from really unlucky to somewhat lucky, meaning he wasn't going to sustain that trend either.

Sure enough, Hamonic's On ice SV% regressed to .911 last year. But basically all of this has been luck.

CONCLUSION:

I'm going to stop here, because I've led up to the discussion of an important statistic for this topic, known as PDO, which really deserves its own post (though it might be a short one).

But I hope you get my points here: Stats like Shooting % and SV%, and their on-ice versions are only a few of the stats, though some of the most prominent, at determining when a player's statistics are greatly influenced by luck and we ought to expect regression. Keep an eye on them as this short season goes on - and don't get misled by what is not skill, but is actually luck.

The Intro to Hockey Analytics/Advanced-Hockey-Statistics Primer so far:

Part 1: - What is the field of Hockey Analytics and Why Might You be Interested?
Part 2.1: - The Importance of Context Part 1 - Time on Ice
Part 2.2: - The Importance of Context Part 2 - Evaluating the Difficulty of Certain TOI through QUALCOMP and Zone-Starts
Part 2.3: - The Importance of Context Part 3 - Evaluating (and Compensating for) the Effect of Teammates via QUALTEAM and Relative Measures
Part 2.4: - The Importance of Context Part 4: The Concept of the Replacement Level Player
Part 3 - The Perils of Sample Size
Part 4.1 - Introduction to Hockey Analytics Part 4.1: Possession Metrics (Corsi/Fenwick)
Part 4.2 - Introduction to Hockey Analytics Part 4.2 - Possession Metrics: The Various Forms of Corsi Available on Hockey Sites
Part 4.3 - Introduction to Hockey Analytics Part 4.3: Possession Metrics: Fenwick a Measure of Effective Possession
Part 4.4 -
Introduction to Hockey Analytics Part 4.4: Possession Metrics: Scoring Chances
Part 5.1 -
Introduction to Hockey Analytics Part 5.1: Evaluating Neutral Zone Play: Zone Entries
Part 6.1 - Introduction to Hockey Analytics Part 6.1: Luck and Random Variance: An Introduction.
Part 6.2 - Introduction to Hockey Analytics Part 6.2: Luck and Random Variance: True Talent and Regression

<em>Submitted FanPosts do not necessarily reflect the views of this blog or SB Nation. If you're reading this statement, you pass the fine print legalese test. Four stars for you.</em>

Trending Discussions

forgot?

As part of the new SB Nation launch, prior users will need to choose a permanent username, along with a new password.

I already have a Vox Media account!

Verify Vox Media account

As part of the new SB Nation launch, prior MT authors will need to choose a new username and password.

We'll email you a reset link.

Try another email?

Almost done,

By becoming a registered user, you are also agreeing to our Terms and confirming that you have read our Privacy Policy.

Join Lighthouse Hockey

You must be a member of Lighthouse Hockey to participate.

We have our own Community Guidelines at Lighthouse Hockey. You should read them.

Join Lighthouse Hockey

You must be a member of Lighthouse Hockey to participate.

We have our own Community Guidelines at Lighthouse Hockey. You should read them.