Proc Rate Determination - Page 2

Machinator · 12/29/07, 8:53 AM

Test 2 lasted twice as long before weapon broke, maybe I forgot to repair the first time.
Hits: 10056
Time: 435.76m
Procs: 467
PPM: ~1.07

Total for both tests: ~1.11 PPM

• Aldriana · 01/02/08, 1:13 AM

As promised, I have written a simple command-line utility to compute proc rate data from combat logs, which, for the moment, I have named ProcCounter. You will need python to make it work, and you may find it helpful to have a working understanding of how command line tools typically work. If there is significant demand for it I might be persuaded to write a GUI for it, but for the moment, this should be enough to get started.

The tool makes the following assumptions about the input data:

1) The proc you are investigating is buff proc - that is, it leaves specific "gain" and "fade" messages in the combat log. This program will *not* work on instantaneous procs like combat potency or sword specialization.
2) The combat log comes from the tester. So if you bring a healer along to heal you while testing, you must use *your* combat log, not there's. The program works by looking for "You gain", "You hit", etc. type messages - not "Aldriana gains", "Aldriana hits", etc.
3) On a similar note: this only works on english-language combat logs. I don't know if it's even possible to log in different languages, but assuming for the moment it is, this program will not work on those logs.

Also, it assumes that your data is reasonably "clean", defined as follows:
1) You are only attacking with one weapon.
2) You are only autoattacking (no specials)
3) You have no Haste or attack-speed modifying effects active, other than the one being tested. So if you're testing Mongoose proc rate, you can't have your Dragonspine Trophy equipped, you should be specced out of flurry, you're not somehow getting SnD up, and so on.
4) Your data set is reasonably large. It will give some answers on smaller data sets, but to get reasonable accuracy you will need at least 1000 attacks or so, and more is better.

How to use ProcCounter
First, you will need to have Python installed. If you don't, you will need to go to python.org, download python, and install it.

Second, you will want to have ProcCounter in the same directory as the combat log(s) you wish to parse.

Third, open a command line prompt and navigate to that directory.

At this point, if you're familiar with command-line tools, you can probably just type 'python procs.py -h' and get enough information to get started. For those that would like a little more direction:

The general structure of a command for ProcCounter is as follows: python procs.py <options> [files]

There are two options that must be specified in each command line. First, you must specify the base speed of the weapon you are using. This is done with the command line option --speed (or -s for short), followed by the weapon speed. Hence, if you were testing with, for instance, Talon of Azshara, you would enter the option "-s 2.7" or "--speed 2.7"

You will also need to specify which buff you're looking for. There are two ways to do this. The first is with the option --buff (-b for short), followed by the exact name of the buff generated by the proc in quotes. For instance, if you are testing the proc rate of Band of the Eternal Champion, you would type in -b "Band of the Eternal Champion". If you misspell or abbreviate the buff name, it will not work. If you fail to capitalize, it will not work. For instance, none of the following commands will work:

* -b Band of the Eternal Champion
* -b "Eternal Champion"
* --buff "Band"
* --buff "band of the eternal champion"

Also note that you are entering the buff name; so, for instance, if you were testing Mongoose, you would want to type "Lightning Speed", not "Mongoose".

As a convenience for some of the more common procs to test, I have also added special command line options "--mongoose", "--executioner", and "--dragonspine" that you can use in place of the --buff option. The option --executioner is exactly the same as the option -b "Executioner".

Additionally, there are two more options available. The first, "--haste", allows you to enter the amount of haste rating you have equipped for the test. So, if you have a Band of Devastation on and no other haste equipped, you would enter --haste 31

Also, there is a --crits option (-c for short), which indicates that the proc can only occur on critical strikes, useful for testing, for instance, Tsunami Talisman.

After entering all appropriate options, enter one or more filenames to be parsed. You can put as many different filenames as you want. Note that all files to be parsed will be checked for the same buff, and assume the same weapon speed. So you can't mix data sets taken for two different buffs or two different speeds and expect to get correct answers.

So, for instance, if you had a test of Mongoose enchant on Talon of Azshara stored in a file called "mongoose.txt", you would enter the command line:

python procs.py --speed 2.7 --mongoose mongoose.txt

If you had 3 files of data, taken with Blade of Savagery with 31 haste rating equipped, you would use

python procs.py -s 1.4 -h 31 --mongoose file1.txt file2.txt file3.txt

If you were, for instance, testing the proc rate on Tsunami Talisman with Malchazeen instead, you would enter

python procs.py -s 1.8 --buff "Fury of the Crashing Waves" --crits file1.txt file2.txt file3.txt

When you enter an improper command line, you will get an error message which will hopefully lend some insight into what you did wrong.

If you enter a correct command, you will get a brief summary of information about the buff, including internal cooldown, uptime periods, eligible attacks, and some basic statistical analysis.

I haven't fully tested every combination of options, so there may be bugs; if so, please let me know and I will look into fixing them.

Edit: Fixed bug described by Machinator in post 18.

Machinator · 01/02/08, 3:05 AM

First, thanks for making that tool, now to nitpick.

I tried using this on one of my executioner tests(2.6 speed, autoattack, no haste).
Your program counts 194 uptime periods, which is what I count. But it says 3994 eligible attacks, I'm not sure what that means but there were 5135 hits in the file. Is this just something for haste procs?

I also had this come up at the end:

Traceback (most recent call last):
  File "C:\parser\Olds\procs.py", line 221, in <module>
    if not crits:
NameError: name 'crits' is not defined

After running it through all the logs Ive been collecting I get

Proc Rate Information
---------------------
Proc Rate:              0.0508194293612
Proc Std. Dev:          0.00109112882953

Proc rate is 95% likely to be between 4.87 and 5.3 percent.

There is no output of time or total attacks which makes it hard to translate those numbers to anything PPM.

It also seems like a pretty big range for something like 38 hours of logs. Any idea how many attacks you need to narrow that down to around +- .1%?

Latito · 01/02/08, 3:08 AM

Just a note - unless I missed that option in the Python installation, you will need to manually add the python.exe location to your system Path variable.

To do this (Windows 2K / XP):
Right Click My Computer
Properties
Advanced tab
Environment Variables button
Under System Variables, find "Path"
If you found it
-select it and hit Edit
-add a semi-colon to the end of your existing variable value and then put in the python path (default was C:\Python25)
If you can't find Path
-hit New and enter the variable name as "Path"
-enter the variable value as the python path (default was C:\Python25).

You should end up with a variable name that might look like this:
%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;c:\cygwin\usr\bin;c:\cy gwin\bin;C:\Program Files\Java\jdk1.5.0_07\bin;C:\Program Files\ant\bin;C:\Python25

Note - you likely won't have the cygwin, java or ant stuff in there if you are reading this.

If you are using a different OS (Vista, Mac OSX, etc).. good luck!

Oh, and if you use the option -h (or --help) you will get a pretty handy printout of all the options Ald described above.

Latito · 01/02/08, 3:15 AM

Machinator - the error your describing is due to a typo in line 34, it should be "crits = False", not "crit = False". I PM'd Ald about it already, for now you can just manually change the procs.py file. Once you change that it will print out the PPM properly.

About the "smaller range"... get more data

In the DPS Simulator I'm currently writing.. without a lot of proc-type stuff in there I still get variances of +/- 0.5 dps in a ~500 dps scenerio over 200+ hours of simulation. Thats just from basically weapon range, crit chance and a few talent stuff specific to rogue yellow-attack cycles.

Results from the data I posted several posts up:

D:\Temp\procs>python procs.py -s 1.3 -b "Band of the Eternal Champion" sunblade.txt
Cooldown:               61.0

Proc uptime periods:    58
Eligible attacks:       2504

Proc Rate Information
---------------------
Proc Rate:              0.0231629392971
Proc Std. Dev:          0.0030060121825

Proc rate is 95% likely to be between 1.73 and 2.91 percent.
If this is a PPM effect, it is 95% likely to be between 0.8 and 1.34 PPM.

D:\Temp\procs>python procs.py -s 1.3 --mongoose sunblade.txt
Cooldown:               0

Proc uptime periods:    88
Eligible attacks:       4904

Proc Rate Information
---------------------
Proc Rate:              0.0179445350734
Proc Std. Dev:          0.00189565317646

Proc rate is 95% likely to be between 1.42 and 2.17 percent.
If this is a PPM effect, it is 95% likely to be between 0.66 and 1.0 PPM.

• Aldriana · 01/02/08, 3:23 AM

Originally Posted by Machinator

First, thanks for making that tool, now to nitpick.

I tried using this on one of my executioner tests(2.6 speed, autoattack, no haste).
Your program counts 194 uptime periods, which is what I count. But it says 3994 eligible attacks, I'm not sure what that means but there were 5135 hits in the file. Is this just something for haste procs?

This has to do with my preferred method of measuring proc rates for refreshing procs. The problem with counting all attacks is that you don't really know how many procs actually occurred due to refreshing procs. My solution to this is based on the realization that the proc rate is also equal to the number of attacks outside of the buff uptime divided by the number of uptime periods; basically, you throw out all procs and attacks that occur while the buff is already active. So for procs that have no cooldown, "eligible" means "did not happen while the buff is already active". For procs with a cooldown, it means attacks that happened while not on cooldown.

I also had this come up at the end:

Traceback (most recent call last):
  File "C:\parser\Olds\procs.py", line 221, in <module>
    if not crits:
NameError: name 'crits' is not defined

Wow, that'll teach me to make last-minute edits before uploading. I'll update the link above with a fixed version in a few minutes.

After running it through all the logs Ive been collecting I get

Proc Rate Information
---------------------
Proc Rate:              0.0508194293612
Proc Std. Dev:          0.00109112882953

Proc rate is 95% likely to be between 4.87 and 5.3 percent.

There is no output of time or total attacks which makes it hard to translate those numbers to anything PPM.

This is related to error message you got. It's supposed to do PPM calculations for you, I just made a mistake in that section of code.

It also seems like a pretty big range for something like 38 hours of logs. Any idea how many attacks you need to narrow that down to around +- .1%?

The issue with error ranges it that to narrow the answer by a factor of 2, you need 4 times as many attacks. So it takes a *lot* of attacks to get really good accuracy. To get a 95% confidence interval .1% wide for a ~5% proc rate would take in excess of 700k attacks.

Machinator · 01/02/08, 7:06 AM

Cool, Ill give it another go after my next set is finished. Since you are sorting out hits under a proc, with the number you output, the uptime of that proc would be ineligible hits / total hits?

With so many hits needed, what do people see as a reasonable range? 700k attacks is 21 days of logs and that doesnt seem very reasonable to test for all this stuff.

glowacks · 01/02/08, 8:27 AM

True, but we have the opportunity to share our data with others. Assuming that everyone collects the data correctly, we'll be able to refine our estimates. I don't know a ton about statistics as to how you might combine small amounts of similar data that don't have the exact same setup, but probability theory tells me that if you have one person's 95% confidence interval as 10-12 and another's as 11-13, you have a ~90% of it being 11-12, ~5% 10-11 or 12-13, and practically negligible outside that. That's fairly useful even if they could be combined into a better estimate statistically by combining the data.

Assuming there are enough tests with appropriate data though, the central limit theorem says they'll tend towards a bell curve about the true value regardless of their particular parameters. It shouldn't be too hard to "eye-ball" a large enough set of data and see where the middle is likely to be..

• Aldriana · 01/02/08, 4:09 PM

Well, the ideal would to have multiple people test at the same speed, whereupon combining tests becomes pretty easy. Testing at different speeds will also work, although I'd need to dig out my old statistics textbook to figure out how to do so.

Our other great advantage here is that the proc rate is unlikely to be just anything. I mean, it *could* be 1.2835 PPM... but it probably isn't. On the whole, it's much more likely to be 1.25 PPM or 1.33 PPM (aka one proc per 45 seconds). So we probably won't actually need 700k attacks to figure out the answer with some degree of confidence.

Machinator · 01/02/08, 8:53 PM

Is there a good way to coordinate logs? I have 60k executioner attacks, how can I get that to others or vise versa without being a huge mailing list? Or are final numbers of the parses enough to post and calculate off of?

• Aldriana · 01/02/08, 10:20 PM

If you cut/paste the summary generated by my program, and list weapon speed + haste rating with it, that will be enough for them to be combined.

Machinator · 01/05/08, 9:09 PM

This is what I have for executioner so far(2.6 speed)

My own crappy program counting length of procs
Total Hits:82958
Total Procs: 8248
PPM: ~1.18

Python script
Proc uptime periods: 3275
Eligible attacks: 64058

Proc Rate Information
---------------------
Proc Rate: 0.0511255424771
Proc Std. Dev: 0.000870235218222

Proc rate is 95% likely to be between 4.94 and 5.28 percent.
If this is a PPM effect, it is 95% likely to be between 1.14 and 1.22 PPM.

I would say its definitely over 1 PPM, and 1.15 or 1.2 is probably good for testing. I will probably test enough to cut it down a bit more. I would expect mongoose to probably be about the same.
I am not sure if expecting it to come out to a nice number is appropriate. I think the concept of PPM was player made, and what they designed it internally as could be anything from proc/hour to some random number they fiddle with for balance rather than the proc itself.

• Aldriana · 01/05/08, 10:04 PM

On the Mongoose front: I have done some further testing with both Blade of Savagery and Talon of Azshara. Results are as follows:

Blade of Savagery (1.4 speed):

Cooldown: 0

Proc uptime periods: 366
Eligible attacks: 13828

Proc Rate Information
---------------------
Proc Rate: 0.0264680358693
Proc Std. Dev: 0.00136507426589

Proc rate is 95% likely to be between 2.38 and 2.91 percent.
If this is a PPM effect, it is 95% likely to be between 1.02 and 1.25 PPM.

Talon (2.7 speed):

Cooldown: 0

Proc uptime periods: 289
Eligible attacks: 5249

Proc Rate Information
---------------------
Proc Rate: 0.055058106306
Proc Std. Dev: 0.00314829123641

Proc rate is 95% likely to be between 4.89 and 6.12 percent.
If this is a PPM effect, it is 95% likely to be between 1.09 and 1.36 PPM.

This is, all in all, reasonable consistent with the theory that the proc rate is the same as Executioner, and both are around 1.15-1.2 PPM. But, as always, more data is needed to pin this down with more accuracy.

Garren · 01/06/08, 9:21 AM

Mongoose data:

Merciless Gladiator's Slicer (2.6)

Cooldown: 0

Proc uptime periods: 313
Eligible attacks: 6494

Proc Rate Information
---------------------
Proc Rate: 0.0481983369264
Proc Std. Dev: 0.00265786619911

Proc rate is 95% likely to be between 4.3 and 5.34 percent.
If this is a PPM effect, it is 95% likely to be between 0.99 and 1.23 PPM.

songster · 01/06/08, 11:11 AM

Originally Posted by Machinator

I am not sure if expecting it to come out to a nice number is appropriate. I think the concept of PPM was player made, and what they designed it internally as could be anything from proc/hour to some random number they fiddle with for balance rather than the proc itself.

Aye. My personal guess is that it will turn out to be a nice round number when expressed as a percentage proc rate for a 2.4 speed weapon. My only reason for thinking that is that our best guess is that internally these things are coded as % on hit, which is subsequently adjusted for weapon speed - and we know that 2.4s is the "base" speed for weapon normalisation. If could even be that there's a subtly different proc rate for daggers, other 1h weapons and 2h weapons, each of which is then adjusted relative to the normalisation factor for each weapon type.

Or (for brainf*ck value), there could even be some real legacy effects in there. I read in another thread that Spirit originally affected proc rates. So the proc rate may be a neat number when expressed in terms of some random combination of weapon type, weapon speed, and the expected spirit total for a level 40 warrior, say...