How good are fantasy mocks? A review of last year.

Originally posted to reddit.com/r/fantasybball

I've previously tried to answer the age old question: how good are fantasy mocks? In that spirit, I thought I'd share a new analysis of last year's predictions. This taking-stock report comes with an important caveat: it's based on a single year of data, and as such, is limited. I'm not sure what the predictive power of one year of data is, but it's probably not much: how well a website does can vary quite a bit year-to-year, and this analysis misses that fact. That said, with the data already provided in neat, orderly format (with incredible thanks to @macetastic) I couldn't resist. Results below are based on top 100 to match last year.

One last important note: this year I ran two numbers. The first is the correlation number, same as last year. The second is new, and replaces variance, which I'm going to call the "Krakatoa" correlation. This number comes at the good suggestion of reddit user @krakatoadreams and accounts for the difference between being right early on in a mock and later on in the same mock.

With that out of the way, what did I find?

A. Rankings from the top:

Basketball Monster. BBM comes in first again. A repeat performance suggests that BBM may be the most predictive, but without more historical mocks I'd hesitate from saying this with any certainty. That said, BBM came in with a stunning .746 correlation and .822 Krakatoa correlation. Wow. Good job BBM.

Hashtag. Hashtag comes in second, with a .677 correlation and .809 Krakatoa correlation. One trend to note: all websites are doing far, far better this year than in prior years. This would suggest to me that last season was more stable than in years past. A future look into why might be warranted - were there less injuries than normal? Less player movement than the year before? Or are mocks just getting better overall?

Yahoo Pre. Yahoo barely misses out on second place, coming in with a .672 correlation and .802 Krakatoa correlation.

NumberFire. I'll admit that this is the first time I've even heard of this site. But it comes in with a respectable 4th place finish, with a .649 correlation and .794 Krakatoa correlation. An important note about NumberFire: I did a bit of back-of-the envelope look at where sites would have shaken out if I'd expanded the scope of the rankings beyond 100, and NumberFire was the one website that moved in any meaningful way. Under a more expansive ranking, NumberFire probably finishes as the 2nd most predictive.

RazzBall. A huge, huge improvement by Razzball compared to last year's analysis. Razzball finishes with a .599 correlation and .73 Krakatoa correlation.

ESPN. Order in the universe is restored, with ESPN falling from 2nd to 6th. ESPN had a .514 correlation and .721 Krakatoa correlation.

CBS. CBS was...bad. .479 correlation and .688 Krakatoa correlation.

B. What does it all mean?

BBM appears to have value, coming in 1st again. If I didn't have a subscription, however, I could approximate it's predictive power by combining three rankings in particular: Hashtag, Yahoo, and NumberFire.

By using a combined Hashtag, Yahoo, and NumberFire ranking the correlation (.712) and Krakatoa correlation (.826) went up significantly and were nearly identical to BBM. This conclusion aligns well with last year's look, which found that a combination of several rankings were more predictive than any one in isolation.