This some very cool information. Great work putting it together.

I'm curious about why you picked a median of the top 20. If you're doing this by hand then yea, 20's a lot. But if you're gathering this automatically then maybe taking more samples would be a helpful comparison.


Or you might argue that 20 is too many. I'm not a statistics guy, but when I pull up the top dps on Festergut World of Logs - Real Time Raid Analysis and I just look at the top 25 dps, it consists of just 5 feral druids, 7 mages, and 8 rogues. However, in the list above feral druids are listed at #7 in the dps list for Festergut. So I would presume the next 15 feral druids performed so poorly that hunters, fury warriors and ret pallies blew past them in the averages.

Oh, this topic again.

So, let's go over how the entire concept of this is wrong-- specifically, this isn't measuring what you think it's measuring.

- Sample bias. Say there's 2 rogues in every raid and 1 feral in every raid. Half of those ferals are tanks, half are cats. The 20th druid, then, is closer to the average, because there's 4x as many rogues as cats. So, you'll want to count the 20th cat against the 80th rogue. (In other words-- percentile comparisons, not absolute rank comparisons.) How many feral druids have DPSed on a lich king kill?

- Gear bias. What percentage of loot council guilds give preference on gear to the highest DPSer? If a feral gets defaulted a DPS staff but rogues have to compete, what does that mean?

- Buff bias. Gaps aren't as large as they first appear. People more likely to get TotT/Hysteria casts, for instance, will be skewed higher than their "raw" DPS. Additional damage via TotT/Hysteria also don't get allocated to their casters. Inevitably, the buff receivers are either high performers or receive the buff from social status (from being officers, being friends with the buffer, being the rumored best class for the buff, etc).

- RNG bias. Classes that are more likely to have RNG impact their final DPS will tend to have higher maximum numbers but lower average numbers. Certain best-stats also skew RNG-- ferals can get higher maximum DPS by gearing for agility (more crits), but higher average DPS by gearing for ArP.

This is just the beginning of what to take into account. Which is why Blizzard dismisses such statistics so easily-- are they actually measuring anything worthwhile? Instead of the graphs above, give me percentile stats (99th/90th/80th/50th), accomodate buff imbalance, use average performances by player X instead of maximum performances to account somewhat for RNG, and so on.

These numbers don't necessarily say anything about class/spec balance.

I'm not a statistic master so perhaps some people may be able to describe it with more details but I'm going to give a brief explaination of why the median of "top" 20 is a totaly biased estimation.

You can describe the total damage done by 1 player(class) on one given encounter as a single sample (measure) of the quantity you are trying to estimate.

The quantity beeing a Random variable it follows a Probability distribution.

When you want to compare dps among different classes you want to compare their Probability distribution. The mean and variance are parameters that describe the probability distribution (but are enough only for Gaussian/Normal distribution), which usually fit many random events in the world thanks to the Central limit theorem.

But in our case the normal distribution isn't a good model, first because normal distribution has an infinite support, (can take negative value regardless of the mean). And it's definitivly a non symmetric distribution as everyone aim for the "best dps".
More complex distributions, as it's the case for us, usually require more parameters than mean and variance to be described. And most people understand that the mean isn't what interest us here (but can be interesting for Blizzard who care about the whole playerbase).
Modeling the right distribution to fit the scenario can be a really interesting problem, but would require of course a lot of data.

Given the measurements avaiable (lets say top 200 damage by spec), you cannot estimate distribution parameters, as some people pointed top 200 damage done by 1 spec can be the whole data for one class (including the worst players), but the 1%top for an other class.

You may eventually be able to make an estimation based on top N of your samples only if you artifically reduce all your base sample to the same size.
Let's say you have 2000 samples for Arcane Mage, but 200 for feral druids, you pick 200 samples out of your 2000 Arcane Mage samples (using a uniform random distribution). Then take the top 20 out of your 200 samples for both classes. That would be a non biased estimation of the top 10%, but once again that wouldn't describe the whole distribution.

If you don't want to discard your 2000 Arcane Mage sample, you simply have to take a fixed % top. You will average the dps of the top 10% mage (200) and the top 10% of the druids (20), your estimation for the druids would of course be much more noisy in this case.

If you want to make a robust estimation of the distribution you need the whole data set (not the top N/%), having less sample for one class/spec isn't that bad in this case as you can estimate how robust your estimation was given the number of sample.

Overall the top N/% is probably extremly sensitive to the variance of the distribution.

A bit more details on the bias estimation of your measurements,

Considering you are measuring the average of the top 10% of samples. Which corresponds to the Cumulative distribution function beeing above or equal 0.9 (1-10%).

Assuming normal distribution (as I said it's a bad model but can give you an idea) :



The erf function is a bit complicated so I just gave a numerical solution you can check on wikipedia if you want more details.

What does this mean?

Class X with 9000 average dps and 1000 dps sigma, your measure will show 10281 dps.
Class Y with 9000 average dps and 100 dps sigma, your measure will show 9102 dps.

As everyone could "feel" that measurement is biased and highly favor variance, just tried to express by how much

Used as a reference for classes, you're still not getting real reference data. Again, RNG bias. Fight length bias (you get different results from a 3:30 fight versus a 5:45 length fight, and depends on individual cooldowns). The correlation between faster kill times and higher DPS given (caused by heroism/lust and effective cooldown usage) means that your number might not match. Either you're in a guild competitive with the best guilds in the game, or you won't see similar numbers. And these numbers say nothing of consistency-- player X might reach 12000 DPS one week and 9000 the next. Is he playing optimally in both cases?

The term "median of the top 20" simply means halfway between 10th and 11th place. Why is that number a better estimate of DPS than #1? Than #100? Than #1000?

The point is, you need to do a lot of extrapolation and simply unscientific analysis to get any conclusions from the data presented. This is the kind of data that is responded to by blue posts saying "we have better data, we know more than you, we don't think there's evidence to merit a change." From a game designer perspective, it's not a good summary. It's only part of picture, and a much smaller glance than you'd think.

Rogues are more likely to talk to other rogues about TotT strategy. So, they're more likely to give them to rogues. Rogues are more likely to be established DPS in a 5+ year old game than a feral in a long-standing guild because ferals haven't always been viable DPS. That gets mitigated the longer time goes on, but history doesn't disappear.

So in the end you're claiming these numbers as representative of what you should expect to see out of an elite player; but your definition of elite poor statistically speaking; they aren't representative of the individual; they aren't expected. Again, you can't draw consistent conclusions across classes. You can't draw consistent conclusions of gear. You have no indication of what those top 20 players are doing to get scores that high.

In other words, these numbers clustered together don't give you much of an indication of anything except that, for good or ill, some players got that number. Maybe they got lucky and rode threat so tightly that another crit would've pulled aggro and wiped the raid. Maybe they blew cooldowns on Blood Queen to get the first bite, giving them another minute of buffed damage (instead of letting it go to someone who'd get more benefit, but didn't use cooldowns). Maybe they're AOEing on adds instead of focus-targeting a more dangerous enemy. Maybe all of the top 20 rogues got to stay on Deathwhisper during the mana shield, but only the top 10 ferals did.

You're right in that these mean more than nothing, and I didn't say that, exactly. But in a table that went so far as to rank every single spec (and PER FIGHT rankings as well), can you really draw firm conclusions other than basic things like "locks are weak for DPS" or "good players play spec X"?