SeanDamnit

Ok, so I spec'd out of Arcane Focus, thus removing 10% arcane hit, I then spammed rank 1 AM on Dr. Boom.

Stats:
85 Crit Rating
70 Hit Rating (7.15% at level 65 on the paperdoll)
359 Intel (self buffed with AI rank 5)
Molten Armor

13.97% crit according to the paperdoll.

Results:
3053 Casts
2407 Hits
328 Crits
318 Resists

Assuming 1 roll, crit would be 10.74%. 2 roll would be 11.99%.

So once again, assuming 2 rolls, the expected crit is around 2% off of what the paperdoll says. And I even had more casts this time (I planned on 2k casts, but went on auto pilot during a vent conversation and didn't notice I hit 3k).

And I did WWS this one: Wow Web Stats

 

Amonra

Well, thanks for doing all those tests SeanDamnit (and for paying all those respec costs!).

Barring a pretty freak set of results (and it's been a while since I studied probability so I've forgotten how to work out how unlikely that would be from these results) it shows that spellcasting is based on a 2 roll system.

My reasoning is that by speccing out of arcane focus, only the hit rate should be affected, whereas the crit rate should be the same whichever system is being used, i.e. about 11.7% from the previous testing.

If we assume a 2 roll system, the crit rate is indeed unaffected, at about 11.99% - a 0.3 increase which I suspect is within the bounds of experimental error.

However if we assume a 1 roll system then then the only explanation for the crit rate reduction of 1.0 is experimental error. Given the relatively large number of casts, I suspect that the difference is large enough to make this result statistically unlikely (would be nice if someone who remembers enough probability theory could confirm this!).

 

Gaborn

Sean's data looks very conclusive to me. What we got here is a 2-roll system and a crit depression of 2% versus a level 73.

It would be nice to know crit depression against other levels to complete the theory but these data along is enough for raid environment.

 

sp00n

With my recently acquired skills in probability and statistics (), for this data set I see the following results with a 95% certainty:

Hit: 78.84% +- 1.45%
Crit: 10.74% +- 1.1%
Resist: 10.42% +- 1.08%

Conclusion: assuming a two roll mechanic with an observer 11.99% crit rate, it is a bit outside of the 95% probabilty (although I cannot tell you by how much). 11.99% + 1.1% = 13.09%, opposed to a 13.97% crit rate on the character sheet (and thus the difference is only 0.88%).

I wouldn't say a crit depression is proven yet. Need more data.
(And that is always the problem with finding out average percentages, which I noticed by over 11,000 melee swings and still inaccuracies by about 0.6%.)

 

Amonra

To my mind crit depression is proven beyond doubt.

As you say, it's rather above 95% certain for this test alone. But more than that, on every set of results the crit rate is noticeably below the paperdoll rate.

Perhaps the only thing left to eliminate is the possibility that the paperdoll rate isn't accurate. Maybe if a level 70 has a little free time they could test against Dr Boom (or maybe has some previous results they can post). I would expect the crit rate to be at the paper doll level, or even a little higher if other posts I've seen suggesting crit enhancement against lower level targets are true.


For the type of roll system I think the data shows that it is highly likely that we are looking at a 2 roll system rather than a 1 roll system. Not absolutely proven but I for one am satisfied with the results. Especially if you start to factor in the other PvP tests which indicate a 2 roll system.

 

sp00n

Well as I've said, determining the percentage is overly difficult.
As an example, in the posting I referred to earlier (here) I have 4744 melee attacks against level 70 targets with a fixed crit rate of 25.5% according to the character sheet (with no procs that could interfere).
Yet my observer crit rate is only 24.2%, which is a 1.3% difference. Over 4744 attacks!

So either the crit rate on the character sheet is wrong (which is possible having the overall lack of information from Blizzard...), or I simply had a bad string of luck and require much much much more data sets to establish a "real" percentage (somebody in this forum mentioned you'd need around 100.000 swings to be within 0.1% certainty if you are looking at an expected value of 5%).

 

PSGarak

"Certainty" in statistics is arbitrarily defined as the 95% confidence interval. No, Sean's data is not 95% certain, but it's probably 90% certain. Statistics is bunk enough* that I'm not going to split hairs over that sort of concern. The qualitative self-consistency of the data sets suggest this is behaving appropirately.
We're certainly not at the point where we can say how the crit depression works (additive or multiplicative, or a rating rescaling), nor how much per level. That's going to require, as you say, six digit sample sizes, especially since there's no reason to believe it rounds to an integer. I'm going to say that this is provisionally solid enough to update the current theorycraft thread.


*Mathematically, statistics is not an axiomatized system. This means that everything that it does, and especially what those processes mean, are based on a web of non-rigorous assumptions and heuristics. Statistics is literally incapable of proving things, in the strictest sense of the word. Its form of "proof" is providing evidence for a claim, but the interpretation of the evidence relies on an idiosyncratic method of analysis.

 

Chirality

At first, I thought that it would be unreasonable to expect Crit depression, judging from the way melee attacks work (which i'm more familiar with) to be on the order of two to three percent...

But then again, if you compare the difference in miss rate for spells (goes from 5% to nearly 15% in three levels) to miss rate for 2h-melee (5% to 9% in three levels), perhaps the Crit Suppression *would* be much larger. In melee, crit suppression is on the order of 0.12%...So a few percent crit depression vs spells isn't a *completely unexpected* result.

But I still wonder if there's another variable unaccounted for?

Perhaps spell crit rating (from gear) scales your crit percentage vs. higher level mobs differently than +spell crit % from talents? Unlikely, but there may be a problem with making too many assumptions.

On another note--testing Spell Hit/Crit in PvP is not going to give you an idea of how the mechanics in PvE work. The fact that my warrior can take white hits from a rogue that look like "Rogue CRITS you for xxx (yyy BLOCKED)" is enough to show that the mechanics drastically change in PvP. I've always assumed a two-roll system in PvP for all mechanics.

 

Soul

We're pretty certain crit depression exists.

For each test Sean has done with arcane missiles, there is a 2% probability that the actual crit rate is 13.97% and Sean just got an unlucky streak that falls outside the the bracket of where 98% of your sets of data would stand. However, the odds that both tests would fail in this way is therefore less than 1 in 2500, which makes it pretty unlikely that the crit rate versus this mob is 13.97%, which implies that the paper doll crit rate is very likely to be wrong against targets three levels above you.

Here's the math:

Let N be the number of spells landed. Let p the probability of a crit. Then X, the number of crits, will follow a binomial distribution.

The binomial distribution can be approximated by a standard distribution with a variance of N*p*(1-p) assuming that the estimated variance and mean are appropriately large (>> 10 is the general guideline).

For the first trial,
X ~ Bi(1679, 0.1397), so E(X) = Np = 282.194, Np(1-p) ~=242.77, so
X~ N(282.194, 242.77)

Standard deviation would be the square root of Np(1-p) = 15.58. 98% of all trials should fall within 2.326 standard deviations of the mean. The first trial yielded 238 crits, which is a good deal more than 2.326 standard deviations away from the mean, so 2% is a lower bound on the probability that X really does follow a binomial distribution with p=0.1397.

You can do similar math for the second set of arcane missiles... N is 2735 and p is still 0.1397. Rinse and repeat the math and you see that the observed number of crits is well outside the 2% confidence interval and 2% again forms a lower bound on the likelihood that the p=0.1397.

Both sets of trials are independent, so the product of the two lower bounds gives us a lower bound on the probability that the probability of a crit is 0.1397, so for our mage, the probability that p=0.1397 versus a mob three levels above him is less than 1/2500. The number 0.1397 is nothing special, though... it's read off the paper doll, so the likelihood that the paper doll crit rate is correct on +3 level mobs is 1/2500. So, it's exceedingly likely that the paper doll crit rate is incorrect for mobs that are three levels above you.

Also, note that this is a two-tailed test... a one-tailed test (that is, one that tests to see if p is less than 0.1397 as opposed to simply not being equal to 0.1397) would yield an even lower probability of the paper doll crit rate being correct against +3 level mobs (less than 1/10000).

As to the issue as to whether it is a one or two roll system, the evidence is not so clear.

The hypothesis you test is that for the correct roll system, the crit rate should not change, regardless of chance to hit with spells. That is, the number of crits for a given number of trials should exhibit the same statistical behaviour, regardless of what your hit rating is for the correct system. Obviously, this will only be true for either one-roll or a two-roll system, but not both. The ideal way to test this would be to run 30 trials of 1000 casts with each spec in the same gear, obtain the mean number of crits for each trial and do a test to see if the means assuming a one-roll system are the same. If it isn't the same, then it's a two-roll system. If they are the same, then it may be a one-roll system; test assuming a two-roll system. If the result is still inconclusive, get more data.

No one wants to cast 60,000 spells, so we cheat by taking the crit rate obtained from the first trial of arcane missiles casting and using the crit rate from that set of data as an estimator of the true crit rate and test against that (this is dodgy, but diligence has its limits; besides, in the course of getting 60000 casts on Dr. Boom, you're liable to level up from killing the mobs in the neighbourhood of Dr. Boom).

So, if we assume a one-roll system, p from the first set of trials is 0.1167, E(X) for the second trial is 356.35, standard deviation is 17.74, observed number of crits is 328, so you're off by 28.35 crits from the expected value, which is 1.6 standard deviations away from the mean. But you need to be 1.96 standard deviations away to be 95% certain that it is not following the estimated statistics, so we cannot quite discount a one-roll system yet. And you can do the same math with a two-roll system. So, Sean, we really, really appreciate your data gathering, but you might need to gather more data if you want to be absolutely sure. Myself, I'm pretty happy to assume a two-roll system... we are 90% sure as of now and that's all I can say :-(.

 

even_tide

The data you gathered was nice and the effort is acknowledged, but you simply do not have enough casts to conclusively prove a one roll or a two roll system. You would need well over 60,000 casts to conclusively prove it either way. Remember, 1% of 3000 is only 30 and 1% of 30,000 is 300, so the odds of your data being off are quite good.

Secondly, I am still unconvinced that blizzard uses a two roll system. From a programming stand point, it makes absolutely no sense to use a two roll system. Blizzard's server performs literally billions of random number calculations each second and it takes a sizeable amount of processing power and memory to perform all those calculations.

Let's perform a little thought experiment and create a fictitious MMO that uses a purely one roll combat system, and lets assume that the server performs 1 million random number calculations per second. Under a two roll combat system, that same server, instead of having to perform 1 million random number calculations, would have to perform more just to achieve the same results.

A two roll combat system would be harder to program, would be a longer program thereby increasing run time. It would use more clock time and memory to perform more calculations, all to achieve the same results that a one roll system does.

And the one roll system does it better and faster.

From a pure programming stand point, there is no inherent advantage to using a two roll system, but there are some mighty large disadvantages. Remember that the blizzard programmers are not stupid people and they definitely thought of that when programming the game's combat mechanics.

 

Deceitful

One advantage would be that, assuming they want stats weighted a certain way, it automatically accomplishes this. A one roll system devalues spell hit, and increases the value of spell crit. A two roll system increases the value of spell hit, and devalues spell crit.

We already see this being talked about, in the form of crit depression. In a two roll system, a certain degree of crit depression is automatic. In a one roll system, it is another calculation that must be done.

I could make the same argument that additional calculations and programming must be done with a one roll system to achieve the same results that a two roll system accomplishes.

 

Mageborn

When we refer to crit depression we should be talking about a reduction in crit due to a mob level difference, and not a +hit difference.

This has been proven?

I find that very hard to believe since bosses are a much higher level than me and WWS reports I see don't show a reduction in crit beyond what the 2 roll system can explain.

If, however, by crit depression you mean a reduction in hit due to a lower than 100% hit rate on the mob, yes this has been proven beyond doubt -- and is explained by a 2-roll system.

 

Trickykid

Not "proven" but the data in posts #58 and #76 show that with two rolls, actual crit rate is lower than expected crit rate against a mob +3 levels.

EDIT: Also, the argument that Blizzard "should" code something this way or that doesn't hold water (especially when they don't use 1-roll for everything).

 

PSGarak

60,000 trials wouldn't prove either system one way or the other either. Even an infinite series wouldn't. Proving it requires examining the code of the rolling system. Statistical arguements just say how likely something is.

Normal stat analysis is a bit blurry here because it's usually testing the validity of a single hypothesis, whereas we're comparing the validity of two hypotheses. While the one-roll system is inside the 95% confidence margin and is therefore generally not discounted in isolation, it's being compared to the two-roll hypothesis. The probability that we're working with a two-roll system is distinctly higher than the probability of a one-roll system.

Mathity bits: For a sample set S taken from a real probability distribution function F, and a hypothetical distribution function A, the analysis by stdev and whatnot gives some idea about P(A=F|S). The thing is, F is completely unknown and can be any valid probability distribution function (which is any lebegue-integrable function with an integral of 1 over its domain).
BUT what we're actually looking at is P(A=F|S, F=(AorB)). F is not any normalized lebegue-integrable function, it is only either A or B. A and B in this case are the one-roll and two-roll systems.
So basically while we're not quite confident that the two-roll system is the most likely normalized lebegue-integrable probability distribution function to produce Sean's results, we are rather certain that it's plenty more likely than the one-roll system, which is the only other normalized lebegue-integrable probability distribution function that it was actually competing against.

If someone wants to calculate P(A=F|S, F=(AorB)), it can probably be done analytically. I'm not gonna, because I hate statistics. But I guarantee you it's more than 0.5.

 

Soul

60,000 trials would give us a definite lower bound as to the probability that it is not a one-roll system, that is all I'm saying. It's a little unreasonable to expect someone to undertake 60,000 trials to get this lower bound, so what we have is merely a qualitative statement: it's more likely a two-roll system than a one-roll system. That is all.

"Crit depression", OTOH, is pretty well-established. Barring something really strange, the likelihood of there not being level-based crit depression is pretty small. I'd say vanishingly small, but my warlock did not seem to experience a large degree of crit depression during her trials. It's an oddity that might settle out with more trials. Maybe I'll copy her to the PTR and see what comes up. But if someone else wants to do a crapload of Dr. Boom testing to get other results, they are free to do so.

 

Mulgero

Migth not be relevant as discussion is about caster crit suppression but noticed with my shaman that character sheet shows different crit values with different weapons. I had mace skill maxed out and once I got new axe and went to train skill to blastedlands crit% difference on character sheet when wielding mace and axe was quite big until I maxed out skill.

If the skill is compared against target then melee should have crit suppression against level 73 bosses. Casting do have different mechanism but it doesn't prevent possibility that caster would have similar kind of mechanism (there is already level based resists, spell school resists and bosses have unremovable base resist) so why not for crit as well?

 

even_tide

Crit suppression against higher level mobs is an almost certainty. It's explanation is the simplest answer that fits the facts. For example, as a character levels up, it's crit rate drops because his/her weapon skill is no longer maxxed, and because it requires increasing amounts of crit rating, agil, and/or intelect to achieve the same crit %. However, this crit reduction is only noticed against mobs the same level as the character. If that person would go out and fight monsters lower level, they would probobly notice an increase in their crit rate back to its old value.

To my knowledge. no one has ever gone out and tested and definitively proven that statement, but its the best conjecture that fits the fact.

 

Muphrid

I just wanted to bring this result back up. It's consistent with Zaldinar's research. No one-roll theory would predict that both normal hits and crits would occur with Cloak of Shadows up unless it works by the spell resistance mechanic, which would not only be unheard of for a buff, but it's already known that spell resistance caps out at 75%.

In absence of definitive data about crit chance decay, I find this sufficiently conclusive.

 

Liar

For what it's worth, I have gotten a full resist on a Divine Favored (100% crit chance) Holy Shock atleast twice which indicates that spells are on a 2 roll system. Unfortunately I can't provide a screenshot because my combat log doesn't show temporary gains such as Divine Favor so you have to take my word for it. D:

 

PSGarak

Well, the data from seandammit seems to indicate a 2-roll system but neither of those examples in particular indicate it. Cloak of Shadows is a target buff, not a caster debuff, so it could be on a separate roll, although it seems it isn't. And misses have a higher priority than crit on the white damage one-roll combat table, so you can miss with 100% crit rate, you just can't hit without critting.

http://sig.gamerdna.com/quizzes/INFL...erks/Garak.png

 

• Joanna

I suppose the next question is whether there's a similarly observable depression in Hit when fighting higher level targets, paper doll after all lists the conversion of Hit Rating to % for level 70 targets. That casters with 16% hit experience roughly the expected 1% resist would suggests not, and the more I think about it differentiating between a depression in hit from level difference and the 1% unavoidable background resist would serve little purpose and be more trouble than it's worth, until Hit Rating goes into quadruple figures.

Nulla in Mundo pax sincera.

 

Zoner

The reason most people 'dont have data' anymore is a lot of us originally collected it before BC came out, on the Loatheb fight. Extremely easy to get your crit rate over your hit rate (90 to 100+% crit) on this fight (onyxia buff + others & the 60% from the fight itself) and observe plenty of non-crit nukes during the course of the fight.

 

tehbio

More crit than hit doesn't guarantee a crit, you need to have enough crit to push everything else off of the combat table to guarantee a crit (hit, miss, etc). Also there are skills/talents/attributes which specifically reduce crit chance which may bring hit back up on the table.