SimulationCraft Model Development - Page 8

dedmonwakeen · 12/05/08, 7:23 AM

Originally Posted by Koshir

I started simulation several times with the same data and got:
1st:
Warlock_2_13_56 intellect=0.39 spirit=0.45 spell_power=1.16 hit_rating=1.64 crit_rating=0.53 haste_rating=0.90
2nd:
Warlock_2_13_56 intellect=0.23 spirit=0.38 spell_power=0.93 hit_rating=1.76 crit_rating=0.45 haste_rating=0.77
3rd:
Warlock_2_13_56 intellect=0.31 spirit=0.45 spell_power=1.01 hit_rating=1.81 crit_rating=0.59 haste_rating=0.85

It looks like scaling factor is random number. I think it would be useful get true (average) scaling every try

It is simulation...... so there are LOTS of random numbers.......

In general, I would not run with less than 10000 iterations when calculating scale factors. The default number of iterations is small (only 1000) because I don't want first-time users to think the tool is hung. Remember that most modern processors are dual-core, so feel free to add "threads=2" to the command line to speed things up.

Starfox · 12/05/08, 10:20 AM

Originally Posted by dedmonwakeen

It is simulation...... so there are LOTS of random numbers.......

In general, I would not run with less than 10000 iterations when calculating scale factors. The default number of iterations is small (only 1000) because I don't want first-time users to think the tool is hung. Remember that most modern processors are dual-core, so feel free to add "threads=2" to the command line to speed things up.

hehe, yea. like Yo!-simulator was for enhancement shamans in TBC.
DPS values where pretty solid for a few hundred hours of simulation, but for ratings you had to set it to serveral thousands because variance can be a bitch

edit:
Found a bug in the calcuation of the manapool, at least for moonkins
I've got 805 int in casterform unbuffed, with 15291 mana.
Buffs: Int +60, MotW +51, Furor +10%, BoK +10%
Which should put me at ~1108, or 303 intellect more when buffed, which evaluates to a increase in mana by ~4545.
But the charts resource timeline shows me a max value of 23349 mana, which is quite a difference to the expected 19841.
Are there buffs i forgot to take into account? I'm missing ~193 int to reach the simulators value of mana

dedmonwakeen · 12/05/08, 1:22 PM

Originally Posted by Starfox

Found a bug in the calcuation of the manapool, at least for moonkins
I've got 805 int in casterform unbuffed, with 15291 mana.
Buffs: Int +60, MotW +51, Furor +10%, BoK +10%
Which should put me at ~1108, or 303 intellect more when buffed, which evaluates to a increase in mana by ~4545.
But the charts resource timeline shows me a max value of 23349 mana, which is quite a difference to the expected 19841.
Are there buffs i forgot to take into account? I'm missing ~193 int to reach the simulators value of mana

Perhaps the difference is "base" mana?

The "gear_intellect" setting should be ONLY that intellect that comes from gear.

Is it possible to tell that from the in-game character profile?

Erdluf · 12/05/08, 1:48 PM

Originally Posted by dedmonwakeen

Perhaps the difference is "base" mana?

The "gear_intellect" setting should be ONLY that intellect that comes from gear.

Is it possible to tell that from the in-game character profile?

I'm pretty sure the character profiler will show (Base + Other) for Intellect. Unbuffed, in caster form, all of the "other" intellect should be from gear.

I think that the only Moonkin stats that you might want a calculator for are spellpower-from-gear (Lunar Guidance and IMkF) MP5-from-gear (Dreamstate and Intensity), and deciding how to value most on-use or on-proc effects.

Starfox · 12/05/08, 9:23 PM

Originally Posted by Erdluf

I'm pretty sure the character profiler will show (Base + Other) for Intellect. Unbuffed, in caster form, all of the "other" intellect should be from gear.

I think that the only Moonkin stats that you might want a calculator for are spellpower-from-gear (Lunar Guidance and IMkF) MP5-from-gear (Dreamstate and Intensity), and deciding how to value most on-use or on-proc effects.

Ah ok, simulating again using only the stats from gear for default_<stat> variables.
edit: Ok, now it's only off by ~9-10int, thx

Another question, do you let Earth&Moon and Master Shapeshifter increase the druids spellpower or increase the damage done by all spells like moonfury?

Kalle · 12/11/08, 7:13 AM

Originally Posted by dedmonwakeen

When it "worked right" were you ever running with multiple threads? (The sfmt package was not previously thread-safe.)

Can you try something: Re-run the experiment with "sfmt=0" in the config file. This will use the libc rand() function. Run with "threads=0" or simply not set because the mutex locking in rand() destroys any benefit you'll get from multiple threads.

I stole the guts of sfmt (with proper acknowledgment of credit, ofc) and made it thread-safe.

I'm not sure if our error-analysis is/was off because the rng package was behaving oddly..... but more unfortunately, in cases like this "more variance" is probably the "more correct" scenario.

Let me know how the rand() experiment goes. I'll do some research on how our "error calculation" is performed. BTW: I welcome your input on this. I make no claims to be being a statistics expert.

I finally had time to do some additional testing. sfmt=0 doesn't change anything. But the problem seems to exist for some time, I tested with r750 and it also showed the increased variance.

The first test I then did, was comparing the distribution of dps between one run with x iterations and x runs with 1 iteration. They were identical (apart from some rather small statistical fluctuations).

So I tested the actual scaling of the error simcraft reports (on the example of Warlock_7_0_54, r1073) and got this diagramm (I hope the link is ok... I can't check it while not at home). The red "points" is my empirical data (with 1-sigma errorbars). The green graph is the ~ 1/sqrt(iterations) scaling that we assumed for statistically independent iterations. The blue graph is a fit for a model that includes the possibility of correlated iterations (I used the formula from Standard error (statistics) - Wikipedia, the free encyclopedia). I got a sample bias coefficient of approx. 0.0011 (1.0 would be perfect correlation, 0.0 statistical independence).

So the conclusion seems to be that iterations are not fully independent. Currently I have no idea of what could be the cause of this.

The effect is rather catastrophic. At 10,000 iterations the error is ~3.5 times as large as we currently expect; at 100,000 it's 10.5 times (if extrapolation of the fittet function yields correct results).

dedmonwakeen · 12/12/08, 10:57 AM

I'm just getting back from vacation.... I'll jump on this ASAP once I get up and running.

(Currently stuck at the in-laws: Came back from vacation to an ice storm.... No power at home......)

One thing to note is that I have integrated the sfmt code, so I can't turn it off. I'll have to remove the sfmt option.

A couple questions jump to mind:

(1) me: Perhaps I should reseed the sfmt package with a call to rand() before each iteration?

(2) me: Perhaps the depth of my "pseudo-random" sequence is not large enough? I can re-implement the sfmt package with a much larger depth.

(3) you: What version of code? Did you build it yourself? If so, what compiler?

(4) you: What platform are you running on?

(5) you: Have you tried this analysis with any other implemented classes?

Kalle · 12/13/08, 7:48 AM

(1) If the period of the random number generator is not exceeded that should have no positive effect. Everything what you want is to have a "random" starting point in the generator's sequence.

(2) If you mean the length of the generator's period, then there should be no problem. According to the source the period of sfmt is 2^1279 - 1 > 10^385.

(3) I used the r1073 archive from the downloads page, compiled with gcc 4.2.2.

(4) 32bit linux on c2d

(5) Not such a detailed analysis, but I noticed a larger than expexted variance for all players of the default raid_70.txt.

Another plot indicating the problem (I think): Link
It shows the absolute value of the difference between the mean and the average of dps over 'iterations' iterations (of course what I called the 'mean' is also an average, but over a much larger number of iterations). In red: x runs of simcraft with iterations=1. Green: 1 run with x iterations. The blue graph is essentially the same as the green one, but for the green graph I used the same order as simcrafts iterations, for the blue one I put the values in a random order.

So you gain precision by randomizing the order of simcrafts iterations (that is because the number of iterations is larger than the number used to compute the average). If the sequence was already random (in the sense of random sampling from the underlying probability distribution), there would be no difference.

dedmonwakeen · 12/15/08, 1:50 PM

Originally Posted by Kalle

Another plot indicating the problem (I think): Link
It shows the absolute value of the difference between the mean and the average of dps over 'iterations' iterations (of course what I called the 'mean' is also an average, but over a much larger number of iterations). In red: x runs of simcraft with iterations=1. Green: 1 run with x iterations. The blue graph is essentially the same as the green one, but for the green graph I used the same order as simcrafts iterations, for the blue one I put the values in a random order.

So you gain precision by randomizing the order of simcrafts iterations (that is because the number of iterations is larger than the number used to compute the average). If the sequence was already random (in the sense of random sampling from the underlying probability distribution), there would be no difference.

Thanks for the detailed analysis...... this is going to take me a while to digest and dig into......

dedmonwakeen · 12/15/08, 1:52 PM

SampleOutput - simulationcraft - Google Code has been updated for patch 3.0.8

A new download is not yet available...... I'm still waiting on modeling details for JoW and Glyph of Arcane Blast. I'd also prefer to wait until the changes go live and the wowhead talent trees are updated.

dedmonwakeen · 12/18/08, 12:11 PM

SimulationCraft version r1212 for Patch 3.0.8 is available for download.

Sample output can be found here: SampleOutput - simulationcraft - Google Code

As part of the scale factor generation, I now generate lootrank/wowhead weighted links for each profile.

Grengorr · 12/18/08, 10:56 PM

I was wondering if options for the set bonuses of [Valorous Plagueheart Circlet] were in the pipeline? I had a look through the warlock source file, but it doesn't appear to be implemented yet.

Also, I used to get crazy numbers for the scaling factors (particularly for mages) when running using multithreading a while ago (last time I tried was about two weeks ago), but I tried again with r1215 today and it's running great with 8 threads (the scale factors for 10000 iterations are quite close to running the same sim singlethreaded - without any actual analysis it at least appears to be within a believable amount of error). I didn't see a fix flicking over the svn logs or I would have tried earlier, but thanks for that fix

Is there a way to only calculate scaling factors for specific actors? I'm generally only interested in seeing that information for a couple of actors in the raid.

dedmonwakeen · 12/18/08, 11:25 PM

Originally Posted by Grengorr

I was wondering if options for the set bonuses of [Valorous Plagueheart Circlet] were in the pipeline? I had a look through the warlock source file, but it doesn't appear to be implemented yet.

Also, I used to get crazy numbers for the scaling factors (particularly for mages) when running using multithreading a while ago (last time I tried was about two weeks ago), but I tried again with r1215 today and it's running great with 8 threads (the scale factors for 10000 iterations are quite close to running the same sim singlethreaded - without any actual analysis it at least appears to be within a believable amount of error). I didn't see a fix flicking over the svn logs or I would have tried earlier, but thanks for that fix

Is there a way to only calculate scaling factors for specific actors? I'm generally only interested in seeing that information for a couple of actors in the raid.

We recently picked up a developer who is focusing on the Warlock class, so hopefully soon.

The multithreading-scaling bug was stealth-fixed by Althor very recently. So recently, in fact, that the tarball had the bug while the executable download (built 10min later) was clean. He snuck it in there......

(Of course, the tarball has since been updated with the fix after he chatted with me.)

You can set "quiet=1" in any player profile...... but it won't save you very much time because it only affects reporting.

I've been considering providing "default_buff" and "default_debuff" global directives. It kind of stands against the who multi-player theme of SimulationCraft, but the convenience may be worth the apostasy.......

EDIT: By the way..... If you have a google ID, feel free to register Issues at the simulationcraft googlecode site. Issues can be for both bugs as well as feature requests. Seeing that list grow is like a thorn in my side, so you have a better chance getting the code updated sooner rather than later.

dedmonwakeen · 12/18/08, 11:35 PM

Now that the scale-factor generation has been tied into lootrank/wowhead weight item searches, I'm hoping to get some feedback on whether the scale factors from the simulation match up with experienced intuition. Feedback would be most appreciated.

Mandalay · 12/20/08, 8:56 PM

Being new to simcraft I have some questions that are not answered on either the HowToUse page or in any EJ thread that I can find.

1. Why does "blessing_of_might=550", instead of "=1" like other buffs?
2. How can I find a comprehensive list of supported buffs and procs? E.g. potions/flasks/trinkets
3. Does warlock_2_13_56 conflagrate default to casting at time "immolate duration<3sec"?
4. Can I and if so how do I model specific cast sequences? E.g. warlock_2_13_56 backlash ---> chaos bolt/immolate/incinerate.
5. Are there any plans to support doomguard and infernal?

Thanks for your effort creating and improving this great tool.

Merry Christmas