Testing the Predictive Power of APM, PER, and WS/48

Summary

Here at GotBuckets we’ve invested a lot of time reconstructing a stat called Adjusted Plus Minus (APM). We believe that APM, while far from perfect, has a lot to say about who the real difference makers are in the NBA. The problem is not many people are calculating APM for the public and still fewer are writing about it. Three possible reasons for this are:

  1. PER is the enlightened despot of NBA statistics, dominating player comaprison discussions and often serving as crutch for NBA analysis. We really like PER. It’s an incredible summary stat, but the runaway popularity of Hollinger’s brainchild suppresses discussion of other valuable metrics like WinShares/48 (WS/48) or APM.
  2. APM is rubbish. It was a novel idea,  but the results are batshit insane. It explains nothing. Walk away slowly.
  3. APM is tedious to calculate. Bloggers are good at words, not math.

 I’m leaning towards #1 and #3. It is our duty to address #2 because, honestly, a basketball stat is only as good as how well it explains the real world. We could tell you about J.J. Hickson’s 19.7 PER last season in Portland. Does that make him equal to LaMarcus Aldridge, whose PER was barely half a point higher? Do you expect  the Nuggets to win more games now they’ve got Hicksomania patrolling the 4? Maybe, maybe not; the answer depends on a lot of factors, including the amount of stock you place in statistical tools like PER, Win Shares, or Adjusted Plus Minus. All of these metrics assign a single value to player quality. If they are actually successful at estimating a player’s intrinsic basketball-playing worth, you should expect a roster loaded with high PER’s to win lots of real NBA games. Otherwise, stats be damned! We would be wasting our time talking about them! That is why we need to examine how effective PER,  (WS/48), and APM are at predicting team success. More importantly, we need to see how APM stacks up.

How did we investigate this? We started with the assumption that, in general, NBA teams can be assessed as the sum of their parts. We don’t want any sass to the contrary, either. LeBron James + Dwayne Wade + Chris Bosh + Ray Allen equals a predictably great basketball team that wins a lot of games. A team that starts Kemba Walker, Brendan Haywood, Gerald Henderson, and Byron Mullens yields a predictably bad NBA team. The same thing applies pretty well across the league and its spectrum of rosters. Good teams have good players—there are lots of other important factors, but this one is number one.

With that principle in mind as a starting point, we developed a spreadsheet with every NBA team’s roster from the past 10 seasons, with individual minutes played for each season, and corresponding PERs from the prior two seasons.  Two-year PER is used in order to stay consistent with 2-year APM— our APM has access to two years of data, so it’s only fair to give PER the same privilege (when tested, 2-year PER improves in predictive ability).  We queried the 2012-2013 Los Angeles Lakers roster to stand as a representative case study. Considering our sum of parts guideline of team success, this team is stocked with star power. How high do the pieces add up according to PER?

The table below includes every player to suit up as a Laker last season, and how many minutes he played for them. The column on the right calculates each player’s PER from the two seasons leading up to (but not including!) the 2012-2013 season. Kobe Bryant’s PER in the 2010-2011 season was 23.9 and his PER in 2011-2012 was 21.9; weighted by 2779 & 2232 minutes played yields a two-year PER of 23.0. We did the same thing for the entire roster except for rookies, who all get the same “replacement-level” PER of 6.7—this is the bottom 10th percentile of two-year PER for that year.

 

13Lakers

Based on their actual 2012 – 2013 minutes distribution, the Lakers achieve a minute-weighted team PER of 16.47. That’s pretty good, right? How do we translate that number into an expectation of wins for the season? To figure this out we calculated the same “Team PER” for every single roster since the 2003-2004 season, and then plotted the results on a graph with a best fit line. Remember, the results of this graph are an “out of sample” prediction, using prior season’s data to predict the next year.  This is a useful proof of a number’s usefulness, as it best allows for evaluating personnel as they move into new situations. As expected, teams with higher PER’s tend to win more games. But it’s not a crystal ball—not even close–  there are some really bad misses down there! For our 2013 Lakers case-study, the red best-fit “Line of Prediction” expects a team with a 16.47 PER Rating to win about 54.5 games in an 82 games season. We know that the Lakers famously underperformed last season. This simple PER model says they underperformed by almost 10 wins.

image001Now that we’ve introduced a bit of methodology, let’s have a look at the exact same case study with WS/48andAPM. Will they provide a more accurate assessment of the 2013 Lakers than PER? The two numbers we care about are circled, and like the PER table above we applied a “replacement level” stat for players we don’t have enough data for—and like above, the “replacement levels” are the respective 10th percentiles for each metric. Win Shares Per 48 Minutes and Adjusted Plus Minus are obviously not as intuitive as Hollinger’s PER, so we should clarify that they are not on the same scale.

We included the offensive and defensive breakdown of APM in order to illustrate what the numbers are saying. An “average” APM is about zero (maybe a bit less in practice), so why the hell does Kobe Bryant have an “average” APM??? Two thoughts on this.  First, let’s repeat APM is not perfect.   Second, while still an elite offensive talent,  APM says much of that amazing impact is washed out by an eroding impact on defense. Right or wrong, it’s an unbiased calculation; don’t shoot the messenger. Eyebrow raising results like this are exactly why we’re subjecting APM to a credibility test.

What else does the APM table say about the Lakers? It says that Dwight Howard is a tremendous force on defense and slightly below average on offense. It says the inverse about Steve Nash—who is a transcendent offensive talent and a mediocre defender. Antawn Jamison has spent years building a personal legacy of creative shooting and, uh, “passive” defense. APM agrees with this evaluation, estimating that Jamison’s massive liability on defense outweighs his positive contributions on offense.

image002We compiled the same scatter-plots for 2-year WS/48 and 2-year APM. These produce two new predictions for the 2013 Lakers, both shown below.  A team WS/48 rating of 0.117 yields an expectation of 51.3 wins out of 82 games, while  APM fared slightly better.  While that is one anecdotal case intended to elaborate on the process used to validate APM compared to other stats it does fit the broad picture offered by these out of sample predictive tests. The top left corner of each plot includes an “R-Squared” correlation statistic, which provides a measure of how well you can guess team wins using the models I’ve created for this article. An R-Squared of “1.0” describes a model that never guesses team wins incorrectly; an R-Squared of “0.0” would describe a model built by my cat; in other words, totally random with no predictive value. The R-Squared for our three NBA metrics are all similar, but APM slightly edges out WS/48 and PER—the APM R-Squared is 0.542, compared to WS/48’s 0.520 and PER’s 0.492.
image004image003Success? Is the GotBuckets staff driving to Las Vegas, to bet team futures and make a fortune? No! At best this article begins to show that APM lives in the same ballpark with PER and Win Shares in terms of evaluating player contributions to winning NBA games. While brainstorming an APM-oriented blog this summer, we decided that a “sanity test” was in order for APM. Why write about an exotic basketball stat if it offers no more basketball wisdom than oracle bones or tarot card readings? That would be an incredible waste of time, and until GotBuckets gets a quality set of oracle bones, we will write articles through the lens of Adjusted Plus Minus and soon, an improved, regularized version (RAPM).

Results:

Here’s the data presented above, ordered by correlation coefficient. The other columns break down, on average (and by quartile), how many wins the prediction models missed the true totals. Single-year PER and Win Share results are also included, even though we didn’t use them for this article.  The APM stats outperformed PER and WS/48. Win Shares appear slightly more reliable than PER, and both improve with a second year of data added.

Stat

R2

Wins Error

Min

1st Q

Median

Mean

3rd Q

Max

2-Year APM

.542

8.43

0.01

3.08

5.87

6.70

9.50

25.63

2-Year WS/48

.520

8.63

0.02

2.99

5.65

6.85

9.71

24.60

1-Year WS/48

.502

8.77

0.06

2.94

5.55

6.84

9.31

35.73

2-Year PER

.492

8.88

0.03

2.87

5.79

7.04

9.95

26.87

1-Year PER

.473

9.04

0.04

2.55

5.80

6.93

9.69

32.34

Conclusion:

Hopefully this quick-and-dirty research project lays the groundwork to place APM on par with PER and Win Shares. While none approaches exceptional reliability, we can definitely conclude that APM holds its own against popular metrics. We’re not deluding ourselves! Evaluating NBA teams and players from the perspective of APM is totally defensible, if not badly needed on the internet.

More rigorous analysis is required; aging or positions aren’t factored into the models and all rookies are treated the same— Matt Dellavedova and Kyrie Irving should not get the same “replacement level” APM, PER, or WS/48 ratings as rookies.  Moving forward, GotBuckets will develop a regularized APM with a goal of achieving better levels of correlation for out of sample predictions. Hopefully that can be attained, for your reading pleasure.

2 Comments

on “Testing the Predictive Power of APM, PER, and WS/48
2 Comments on “Testing the Predictive Power of APM, PER, and WS/48
  1. Pingback: GotBuckets | An Intro to Gotbuckets and my Brief Primer on Adjusted Plus / Minus

  2. Pingback: Links to the Present, “Daniel LaRusso’s going to fight!” (maybe) edition « Cavs: The Blog

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Current ye@r *