DPS Simulator - Page 2

Nite_Moogle · 07/30/08, 11:43 AM

I have one that's very far along in development but I've more or less halted work on it pending a more finalized set of talents in WLK and some real life insanity. I will be happy to release it and its source code after the next major push if I get some time to work on it.

landsoul · 07/30/08, 1:08 PM

I think this is a great idea, if done correctly. If not apporached with the right frame of mind this cuold turn into a huge nightmare, Grim. The strengths of this idea is so huge and I wish it could be easy but I fear that there are an overwhelming amount of variables to be able to build enough loops for. Is this a brute-force project or a finesse project? A combination of both?

Potential Roadblocks
An event-driven system I fear could propogate into inaccuracy inflation. For example, if the calculated swing time was 2.93 seconds but in reality was 2.9348732 via calculation, that .0048732 could add up over time to be 3 seconds over a long fight. Multiply that by 10 different mechanics, and the automatic decision making engine will start making incorrect decisions even if one mechanic is slightly off.

Computer processors are pretty fast these days. Even with a 1ms (.001s) resolution we just don't know how fast it will run. It might just be that what you think would be slow having too many instances going through too many loops, might be too fast for you to even be able to follow in a display.

Haste mechanics I fear will completely run you around. What happens when you crit on your off-hand whirlwind, proccing DST and a new flurry when your main hand is in mid-swing? You would have to take the remainder of your main hand swing's time and add the haste application to only that section.

The problem of procs is that they are not static. How could you update dynamic changes to the warrior's stat table to your calculation loops while the loops are running?

Suggestions
Give it a shot, but make the code available for the community to see, test, and comment on. This project is going to be a big deal, with lots of little small parts forming into bigger and bigger parts that will finally come together. You could probably start working on it now and have untalented mechanics working befre the final WOTLK changes are made. Build the input, and an auto attack loop and let's see how it looks!

Make it resolution based, because at least that way you can make a variable that defines time left on swing that can be modified on the fly. Procced haste while midswing? then you do "Set Swing_Left =Swing_Left/(1+Haste_Proc_Percent).

Ask for a lot of help. You sound like you may not know a lot of neat programming tricks, which is what will be needed here. I don't know very much myself, but there are plenty of people I'm sure that do who would also have interest in helping.

What I would like to see:
For a display window, have 3 seperate columns streaming events with timestamps. Column 1 would have auto attacks and heroic strikes, Column 2 would have special attack damage, and Column 3 would have buffs gaining and fading.

dedmonwakeen · 07/30/08, 1:11 PM

Heh.... Simulators abound! And why not!? They are certainly a fun project......

Feel free to check out my signature link to SimulationCraft. a multi-player event-driven simulator.
You are certainly welcome to poach anything you like.......
So far I've only implemented Druid (Balance Only), Priest, and Shaman.
I had planned on doing Hunter/Warrior next, but intense pressure from my friends over at shadowpriest has switched my priority to Mage/Warlock.

If you end up going with a cmd-line/parm-file interface first, it would be great if we could at least share the same control-file format.... If not, no biggie! Enjoy the fun!

dedmonwakeen · 07/30/08, 1:22 PM

Originally Posted by landsoul

Haste mechanics I fear will completely run you around. What happens when you crit on your off-hand whirlwind, proccing DST and a new flurry when your main hand is in mid-swing? You would have to take the remainder of your main hand swing's time and add the haste application to only that section.

Interesting. This is confirmed? I was under the (obviously false) impression that Parries were the only method of "speeding up" a swing already in motion.

• Aldriana · 07/30/08, 1:33 PM

One comment I might make about simulators is that while they're really good for some things, they tend to be not-so-good at others. So you should think a little bit about intended usage of the simulator. For instance, if the idea is "people plug in gear and see which combination does the most damage" - a simulator is fine. On the other hand, if you're interested in EP values, even a really good simulator doesn't tend to do as good a job as a more deterministic method (spreadsheets, etc.), for the simple reason that the variance in damage requires one to run the simulator for an extremely long time in order to get accurate values out.

For instance, lets assume for the moment that the standard deviation of damage for a real 6-minute fight is 50 DPS, and one wants EP that are accurate to within 1%. It can be shown that in order to get EP with that level of precision, one needs to run a couple hundred thousand hours of simulated combat per stat, which, even with a fairly well-designed simulator, takes a while to do. So I might think a little bit about what sort of information you're hoping to get out of this simulator before I spent too much time on it; for rogue modeling, I've decided that for the questions I want to investigate, even an imperfect calculator does a better job than a perfect simulator. Which is not to say that you shouldn't go ahead and do this - I'd just be clear on what you're trying to accomplish first.

Nite_Moogle · 07/30/08, 1:53 PM

Originally Posted by Aldriana

On the other hand, if you're interested in EP values, even a really good simulator doesn't tend to do as good a job as a more deterministic method (spreadsheets, etc.), for the simple reason that the variance in damage requires one to run the simulator for an extremely long time in order to get accurate values out.

Flurry is the main thing that makes this extremely difficult to represent accurately because it is a positive feedback loop. You can estimate it but it's not going to be accurate. Unpredictable rage gain and accurate modeling of rage income and expenditure make eyeballing warrior damage quite a bit more complex than most other classes that have much more static resources to lean on for determining damage cycles. It isn't to say this can't be done, but it's nearly as complex as just writing a far more flexible simulator.

What I would like to see:
For a display window, have 3 seperate columns streaming events with timestamps. Column 1 would have auto attacks and heroic strikes, Column 2 would have special attack damage, and Column 3 would have buffs gaining and fading.

This is so impractical it isn't even funny. The simulator I've written thus far takes under 30 seconds to simulate 45 hours of combat.

dedmonwakeen · 07/30/08, 2:04 PM

Originally Posted by Aldriana

For instance, lets assume for the moment that the standard deviation of damage for a real 6-minute fight is 50 DPS, and one wants EP that are accurate to within 1%..

Not to rehash entire discussions we've had in other threads on this subject, but.....

I'd like to change your sentence slightly: We would want simulation-generated EPs with greater accuracy than those generated using closed-form approximations (aka formulation). A calculator can deliver a deterministic EP value with virtually arbitrary precision. But "precision" is not the same thing as "accuracy".

In my honest opinion: If a calculator can deliver EP values with an accuracy of 1%, there really is no need to use a simulator. It might be fun to write one..... but the usability of a calculator is far superior.

If the calculator cannot deliver a requisite level of accuracy, then the question becomes: "Can simulation deliver markedly better accuracy in a reasonable amount of time?"

It may not be possible to deliver EP values accurate to 1% with simulation...... but it may be possible to deliver values more accurate than those of formulation for some class/party/raid setups.

Grim: Please note that Aldriana still makes a very, very important point. Do whatever you can to reduce variation where possible because this will significantly reduce your runtime.

• Aldriana · 07/30/08, 2:41 PM

Agreed, there is an important distinction to be made between precision and accuracy. However, I think there's also something to be said for always getting the same answer - having the relative value of items slide relative to one another depending on luck in the simulation strikes me as a bad thing, so I would argue that it's better to have infinite precision values accurate to 5% than infinite accuracy values precise to 5%. Which basically means that if your calculator is even remotely close, it's going to do a better job on EP than most simulators.

That said: I confess I don't actually know how good the warrior spreadsheets are at the moment. I'm certainly willing to believe there are complications that make it challenging, but, at the risk of sounding cynical: it's been my impression that everyone thinks the modeling of their class is harder than everyone else's. Which is not to say that modeling warriors isn't hard - I'm sure it is. But a lot of other classes also have hard problems of similar sorts and have managed to concoct highly accurate models, so it would surprise me a bit if warriors couldn't do the same. Which, again, is not to say that you should or shouldn't do that, or this - I'm just saying, you should be clear on what information you want, and how well a simulator is going to do towards that end. If your ultimate goal is good EP values, it might make more sense to invest time in a really good calculator rather than a really good simulator. If you're more interested in just a good damage estimator, a good simulator is certainly a better investiture of time. So I simply recommend that you be clear about what you're going for.

dedmonwakeen · 07/30/08, 3:03 PM

Grim, even if the simulator does not become the "go-to" tool, it would still be a very critical aid for confirming the accuracy of the formulation used by spreadsheets and calculators.

landsoul · 07/30/08, 5:32 PM

Originally Posted by dedmonwakeen

Grim, even if the simulator does not become the "go-to" tool, it would still be a very critical aid for confirming the accuracy of the formulation used by spreadsheets and calculators.

This is like comparing apples to oranges to try and determine if the orange is right.

This is so impractical it isn't even funny. The simulator I've written thus far takes under 30 seconds to simulate 45 hours of combat.

What if you wanted to personally check the results of your work for 1 minute of fight data or would you Save a log file of 45 hours of fighting? I guess you could do a log file instead of a display, but a 3 column system would make it easier on the human to verify his work.

Gruntle · 07/30/08, 6:29 PM

Originally Posted by Aldriana

Agreed, there is an important distinction to be made between precision and accuracy. However, I think there's also something to be said for always getting the same answer - having the relative value of items slide relative to one another depending on luck in the simulation strikes me as a bad thing, so I would argue that it's better to have infinite precision values accurate to 5% than infinite accuracy values precise to 5%. Which basically means that if your calculator is even remotely close, it's going to do a better job on EP than most simulators.

That said: I confess I don't actually know how good the warrior spreadsheets are at the moment. I'm certainly willing to believe there are complications that make it challenging, but, at the risk of sounding cynical: it's been my impression that everyone thinks the modeling of their class is harder than everyone else's. Which is not to say that modeling warriors isn't hard - I'm sure it is. But a lot of other classes also have hard problems of similar sorts and have managed to concoct highly accurate models, so it would surprise me a bit if warriors couldn't do the same. Which, again, is not to say that you should or shouldn't do that, or this - I'm just saying, you should be clear on what information you want, and how well a simulator is going to do towards that end. If your ultimate goal is good EP values, it might make more sense to invest time in a really good calculator rather than a really good simulator. If you're more interested in just a good damage estimator, a good simulator is certainly a better investiture of time. So I simply recommend that you be clear about what you're going for.

The current warrior spreadsheets are good but have problems with a number of things due to not being able to simulate the spikyness of rage generation. The main issue in my opinion is weapon speeds. The current spreadsheets cannot simulate one of the main strengths of fast weapons, i.e. the smooth rage generation. A simulator could also be used to study at what rage level it is "safe" to use Heroic strike. Finally, I'm not sure the flurry modeling is really doing things as accurate as you could get in a simulator. Mid-swing haste is a real effect (it has been measured by drAllcom in one of the warrior threads) and will affect different weapon speed setups differently.

For Wotlk it's probable that we will not know initially how to optimize dps. A simulator could be used to figure out which abilities to use and when. Based on the current abilities and talents it's very likely that the set cycle we've been using for tbc might not be good at all. Bloodsurge procs and slamming will make a mess of the very ordered cycle that's used in the sheets. I think a simulator will be a better tool to use (at least initially, maybe a spreadsheet/calculator using analytical models can be constructed later based on the findings of the simulator).

I've read elsewhere that people trying to make swing simulators run into problems with convergence/variance. It might be the rounding problem that landsoul is discussing that messes things up, but it feels like that type of issues should be solvable. In your simulator, Nite_Moogle, do you reach convergence after that 30 second run (45 minute combat time)? Is the variance very high between different 30 sec runs?

edit: a thong is not the same as a thing...

• Aldriana · 07/30/08, 6:40 PM

So, I can see where paying attention to variance (aka spikiness) makes things a lot harder. On the other hand, that is, for instance, almost exactly the sort of problem I'm working on for my rogue calculator right now. Figuring out how often droughts in Combat Potency procs, Ruthlessness procs, and Relentless Strikes procs conjoin to cause SnD to drop is a challenging problem. Figuring out how often unlucky streaks will occur causing Deadly Poison to drop is a challenging problem. And, honestly, we don't have perfect answer to these sorts of questions. But I think we do understand them well enough to get within 1% if that's the goal. Does the same apply for warriors? I have no idea. But I wouldn't be willing to bet much against it. Hard problems are made to be solved.

dedmonwakeen · 07/30/08, 7:22 PM

Originally Posted by landsoul

This is like comparing apples to oranges to try and determine if the orange is right.

I have to disagree pretty strongly.

If we have to use fruit analogies....
I think it is more like comparing two different ways to count apples.......

Confirming the behavior model of simulation is considerably more straight-forward than proving a reduced approximation accurately models complex game mechanics........ which is why simulation can be helpful to confirm formulation.

Nite_Moogle · 07/30/08, 10:43 PM

Originally Posted by Gruntle

I've read elsewhere that people trying to make swing simulators run into problems with convergence/variance. It might be the rounding problem that landsoul is discussing that messes things up, but it feels like that type of issues should be solvable. In your simulator, Nite_Moogle, do you reach convergence after that 30 second run (45 minute combat time)? Is the variance very high between different 30 sec runs?

You don't start to see very consistent results until you have several days of combat time per run. The strategy I took is to repeat a fight of X duration a large number of times (which makes handling things like trinkets and death wish more comparable to real fights) and use the compiled statistics from all runs to determine the averages. You can have extremely high variations between consecutive fights depending on RNG, especially where you are not capped on hit or expertise. Part of the goal was to create an efficient simulator that would be able to do very large numbers of short fights repeatedly in order to make it possible to re-run these simulations with different stats to determine stat weighting. The latter part lacks implementation which is why I've not released it; it's not terribly useful in its current form.

dedmonwakeen · 07/31/08, 6:17 AM

Examples of variance that are "fair" to model "incorrectly":
Base weapon damage: Just take the average of min/max
Direct-dmg spells: just take the average of min/max
Partial resists: Do not use a randomized bucket sytem centered around the default mitigation..... just use the default mitigation pct every time.