Goaltending 'Quality Starts' (via Hockey Prospectus) Explained and Applied to the 2010-2011 Islanders

[A Mikb and Keith Joint]

Hockey Prospectus has released their annual "Quality Starts" metric.  Often, we are fumbling through some significant gaps in statistical data in our analysis of goaltender play.  There are several confounds that can arise including PP/PK, team defense and injury.  It does not appear that the Quality Start metric exclusively accounts for 5 on 5 play (which in my opinion, it should) because let's face it, if you have a 20 shot night, but 8 of those are during three 5 on 3s, that is far different than a 20 shot night in which the other team does not have a power play at all.  The formula is somewhat a combination of the concepts Garik and Mored used in their arguments for goaltender assessment.  An explanation of the metric can be found here

After the jump, some of the key concepts, the best and worst of the league, and a statistical breakdown of this year's Islander sextet of netminders...because none made the 41 start minimum for inclusion in the sample.

The formula for the metric from Robert Vollman of Hockey Prospectus (Any bold emphasis mine):

Quality Starts are defined first and foremost by successfully blocking shots, which are usually referred to as Saves. Evaluating every single game this season, the median save percentage of a starting goalie is .912 percent. Successfully defending the net from goals .912 percent of the time can be defined as a Quality Start. The following table demonstrates that the odds of a team winning become very high when their goalie stops a higher-than-median percentage of shots, very low when the goalie fails to stop at least 88.5% of shots, and is essentially even in the middle.

Save percentage  Winning percentage

0.913 or better  0.777
0.900-0.912      0.536
0.885-0.899      0.503
0.884 or worse   0.246  

This is expanded upon to account for the fact that sometimes, goaltenders do not have the chance to expand their shot sample size through no fault of their own.

To further refine the definition in that average range, we will award a Quality Start among goalies stopping between 88.5 percent and 91.2 percent of shots, depending on how many goals they let in. Since this is based on the number of shots the goalie faced on a particular night, which is largely outside the control of a goaltender, it may seem unfair. The idea is that on nights when the goalie is not up against a lot of shots, it is still considered a Quality Start when he's stopping at least an average number of the shots on goal.

Goals allowed	Winning %
Under 3	.727
3	.140
Over 3	.397

When considering Quality Starts, the key number is 25%. Generally a team will only lose roughly 25% of the time their goalie earns a Quality Start, and likewise will somehow manage to secure the two points 25% of the time that their goalies fails.

This accounts for random times when a goaltender "played well and lost", or "played like crap and got lucky".

If you want to see something strange, consider the league leaders from Vollman on 3/25/09:

Goaltender	GS	QS	GS%
Yann Danis	21	16	.76
Tim Thomas	48	33	.69
Henrik Lundqvist	60	40	.67
Martin Brodeur	20	13	.65

This is why sample size is sometimes crucial. Yann Danis has not may or may not have duplicated this stretch of play, however, he was Brodeur's backup the next year and may have lacked the 20 game requisite to qualify...or was so middle of the pack he wasn't featured.

Here are this year's top performers (Full Article Here) and some notables measured by quality starts and ordered by their percentage of quality starts to total starts. (Minimum 41 starts. Unfortunately, that excluded all Islander goaltending this season except as a percentage of Roloson's starts).  Also, please review the section on "wasted quality starts" that shows some of the teams that are bad concurrently with a goalie's quality start, and the section immediately following "lucky goalies" who are bailed out by their teams scoring a lot when they play badly.

Rank	Goaltender	GS	QS	QS%
1	Tim Thomas	55	40	.727
2	Roberto Luongo	60	41	.683
3	Pekka Rinne	64	43	.672
4	Sergei Bobrovski	52	34	.654
5	Anti Niemi	60	39	.650
8	Marc-Andre Fleury	62	39	.629
12	Ryan Miller	65	39	.600
17	Dwayne Roloson	54	31	.574
20	Martin Brodeur	54	30	.556
22	Henrik Lundqvist	67	37	.552

This was indeed surprising in some aspects.  It could have been assumed that Bobrovski and Miller and Lundqvist would have different positioning, while Thomas and Luongo seem aptly placed. There is a bit of a correlation also as to team standing and percentage of quality start.  Here is that breakdown (by conference rank).

Thomas/Boston (T4th total points East)
Luongo/Vancouver (1st West)
Rinne/Nahville (T4th West)
Bobrovski/Philadelphia (T2nd East)

The first non-playoff goaltender to show up on the list is Cam Ward at #13.  All of the playoff team's goaltenders are represented in the top 25 with Jimmy Howard bringing up the year at 25th.(.492 QS%)

Vollman similarly came up with a "Really Bad Start" (RBS) tracking system:

where a goalie fails to stop even 85% of the shots, leaving his team barely a 10% chance of winning. This year's leader is Brian Elliott, with 15 RBS out of 51 opportunities, taking his team right out of 29.4% of their games. Steve Mason (26.4%) and Miikka Kiprusoff (19.7%) had 14 RBS, with Khabibulin (26.1%), Dan Ellis (27.0%), Craig Anderson (20.4%) and Marc-Andre Fleury (16.1%) the only other goalies to blow at least 10. By percentage, the least reliable starters were Ty Conklin (7 RBS in 20 starts, the only goalie with more RBS than QS) and Curtis McElhinney (7 RBS in 22 starts).

Now for the Islander goaltenders.  (From here on out, we're turning it over to mikb.)

Goaltender	GS	QS	QS%	LS	LS%
Rick DiPietro	26	10	.385	7	.269
Dwayne Roloson	20	12	.600	1	.050
Al Montoya	18	13	.722	0	.000
Kevin Poulin	7	5	.714	1	.143
Nathan Lawson	7	3	.429	1	.143
Mikko Koskinen	4	2	.500	2	.500

Now - with the now-standard disclaimers of Small Sample Size™ in our minds - what we see here will simply add more ammo to the burgeoning arsenals of those gunning for RDP.  He was able to give the Isles a reasonable shot at winning a scant 38.5% of the time: in his 26 starts, he only had ten games where he held opponents to two or fewer goals, or stopped better than 91% of the opposing shots.  And seven times, he pretty much melted down: he allowed four or more goals, and saved less than 85.1% of the enemy shots.  (He had another game with 17 saves on 20 shots, so he barely avoided Lousy Start #8.)

Roli was reliable 60% of the time, and dreadful only once in twenty outings.  Montoya had no Lousy Starts and a sparkling 72.2% of his starts were Quality.  (Sign him long term!!!1!eleven!)  Our three-headed emergency goalie went 10-for-18 on Quality, and half of the other eight starts qualified as Lousy.  (Mikko is our Goalie with the Curl on His Forehead - either very very good, or horrid.)

Considering our sample sizes are so unsatisfactory, I went back to DiPietro's last two full seasons.  In 2006-07, he was Quality 36 times out of 62 (.581) and Lousy only 7 times (.113); he had three games right on the border (4+ goals and .852-.857 sv%).  In 2007-08, he went 30/63 in QS (.476) with 8 Lousy Starts (.127) and 3 borderline starts.

Roloson, since leaving the Island for Tampa, has posted Quality Starts 21 of 34 times (.618) while being Lousy 7 times (.206), and borderline once.  So his QS rate was similar to his work as an Islander, both this season and last (28/50 QS, 56%, and 9/50 LS, 18%, with two on the border).

Based on these and other numbers, it's tempting to anoint El Cubano Grande and Le Poulin our new Smitty and Chico (or Melanson, or Hrudey, or Healy, depending on when you joined Islander Country).  It's just too soon, however.  There have been plenty of goalies who have looked excellent over ten or twenty games and turned out to be mortal, or worse, as full-time starters.  If you have the heart, look at this last table: 

Goaltender	GS	QS	QS%	LS	LS%
Mystery A	24	18	.750	0	.000
Mystery B	25	20	.800	2	.080
Mystery C	66	39	.591	12	.182
Mystery D	36	16	.444	7	.194
Mystery E	53	23	.434	13	.245
Mystery F	108	54	.500	25	.231

Anyone would love to have Mystery Goalies A and B... except, as you've no doubt guessed, there's a catch.  As it turns out, Mystery Goalie A is Vesa Toskala in San Jose in 2003-04.  Work like this got him the regular gig in Toronto, where he proceeded in 2007-08 to become Mystery Goalie C and Mystery Goalie E.

B/D/F?  Again, same guy - Columbus' Steve Mason, whose first 25 career starts essentially won him the Calder (line B).  His 36 remaining starts of that season are line D - in the SAME SEASON he went from giving his team a chance to win just about every time out, to doing so less than half the time.  Line F is Mason's two seasons since, and it's fairly ugly stuff.  Basically, a guy can go from Montoyan to DiPietrid seemingly overnight.  We all want to see one of our promising guys succeed; what we have above is a hope, but not a certainty.