[Mage] Rawr Spec Shootout

MyrddinE · 06/09/08, 4:53 PM

So I just burnt three or four weeks of time (mostly while goofing off at work) running hundreds of tests with Rawr to see which Mage spec beats all.

I initially started this test to prove that Frost mages are the best DPS at lower gearings, but I found some surprising results... but why don't you look for yourself. The results are here: Mage Spec Spreadsheet

From the spreadsheet:

Overview
This spreadsheet was imported from Excel, so obviously some formatting was lost. In particular, the cell backgrounds, which were a dynamic conditional format in Excel, were manually added by me in the Google version.

The first tab, Shadowpriest, shows the maximum DPS achieved by the various talent specs at each gearing level. I cleared Rawr's item cache, and repopulated it with equipment capped at the given iLevel. Then I asked Rawr to optimize the equipment for the best gear it could, given the spec and available gear.

Other than iLevel, there are a few other differences between the gearing. iLevel 114 (pre-Kara) sockets only green quality gems, and only the basic global buffs are applied; a player at that level is most likely doing 5 man content. iLevel 125 (Kara gear) sockets only blue quality gems, and most global buffs are applied. iLevel 135 gear and above sockets purple quality gems, and all global buffs are applied. Only one group-specific buff (paladin auras, Draenai auras, Totems, etc) is ever included: Heroism. This helps eliminate discussion of group makeup on the Mage's DPS.

Obviously, the most important tab is the Shadowpriest tab. That shows the DPS of the various specs with different levels of equipment, assuming you have a Shadowpriest with you. This is the tab where where I ran a full Optimize for every single value on the list (often several) except for the very last one (164 + all buffs).

Then I took every one of those builds, no change in equipment, and saw how well they did without a shadowpriest. That's on the NoShadowpriest tab.

Finally, I took all of those builds (with Shadowpriests), and ran them through different fight lengths (with no change in equipment). That's all the Time tabs, one tab for each level of equipment. This shows the burst damage capability of a build, as well as how mana-efficient it can be for long endurance fights.

Highlighting
The highlighted boxes on each screen are in the top 10% of damage for the given row, using the 0/0/0 talentspec as the baseline.

For example, on the Shadowpriest tab, at the 125 iLevel gearing, the highest DPS spec is 18/0/51 with 1420.72dps. 0/0/0 clocks in with 587.71dps. 1420.72 - 587.71 = 833.01 dps from the highest to the lowest. So the top 10% of the specs are within 83.30dps from the top spec... or anything over 1337.42dps. That is what is highlighted in red.

Results
Frost builds did surprisingly well. Though some Fire builds were able to beat Frost at the very end gearing (this is the best possible gear in the game, assuming you have anything you want out of Sunwell, badge gear, etc), Frost was in the lead by a tiny margin the entire way up. The advantage of Frost is even more evident when you look at different fight lengths... short fights really favor Frost, where it can pop out two Elementals; long fights favor Frost because it has better mana efficiency. I expected Frost to win at low gearing levels (in fact, proving that was why I started this project) but I did not expect it to do so well at the end.

Arcane did susprisingly poorly. In fact, I was so convinced that something must be wrong with the optimizer that I worked on the 164 gearing manually, trying to find a set of gear that got Arcane to do better... I was able to beat the optimizer, but unable to get Arcane even close to Fire or Frost. This contradicts conventional wisdom that at very high gearing, Arcane's large mana pool enables it to shine. Even in very short fights Arcane is unable to beat Fire or Frost. This either indicates that Arcane proponents are wrong, or perhaps that Rawr is not modeling Arcane rotations well.

Fire did about what I expected... not so well early on, but a strong finish. No real surprises there. It's worthwhile to point out that Fire gains a larger benefit from Shaman totems than any other spec... Fire noticably pulls ahead in peak theoretical DPS when put in a group with a Shadowpriest, Moonkin, and two Shamans in order to get every possible buff in the game (note the last row on the Shadowpriest tab).

Further Improvements
In the Elitist Jerks thread, I posted all the files used to construct this data, including the slightly modified version of Rawr (I added two mage specs to the drop-down list, Fire/Meditation and Frost/Meditation). Please feel free to take those files and find better optimized sets of gear for a given spec and gearing, and I will update this spreadsheet with the best results. Make sure you use the same version of Rawr (only available in that thread), unless you feel like re-doing ALL of the DPS numbers in a more recent version of Rawr. Also make sure you use the included itemCache files... don't accidentally include higher iLevel gear on a lower-iLevel character.

If you do post a better DPS number for a given spec and iLevel, you need to provide the character file that created it. Attach it to the post or something. I will not update DPS numbers unless I can verify that the gear is of the proper level, the talents match the listed specs, and the fight and buff options were not altered.

You can download the Rawr build I used to create this data, the source code, and all the saved characters and ItemCache files for the various item levels. The data files also includes the original Excel spreadsheet that the online spreadsheet was created from.

Kavan · 06/09/08, 7:17 PM

There's a very important problem with these results. All the characters have race set to night elf. This means there are 0 base stats which hurts arcane quite a bit since it's more stats based.

MyrddinE · 06/09/08, 7:33 PM

That was deliberate, though I forgot to mention it. Some races have int bonuses, some spirit, some have haste buffs... I didn't want to tie the computation to any particular mage build, so I removed all racial benefits to mages by setting it to a race that has none. There is no other way to model all the talent specs evenly.

MyrddinE · 06/09/08, 7:40 PM

Aww crap... now I see what you mean. The base stats are not included at ALL... as in, no gear means 0 int. Well shit.

In that case, which race do you believe has the worst racial ability, or one that can be disabled/ignored? I'd really hate to have to run the numbers 5 times.

Kavan · 06/09/08, 7:43 PM

You could easily take Undead that has no mage specific racials.

Wizeowel · 06/10/08, 11:22 AM

Indeed, the file "135+200=1581.44@40-00-21.xml" jumps from 1581 dps to 1831 dps just by selecting undead

Sancus · 06/10/08, 11:48 AM

Only one group-specific buff (paladin auras, Draenai auras, Totems, etc) is ever included: Heroism. This helps eliminate discussion of group makeup on the Mage's DPS.

This doesn't make any sense. You assume Heroism but no totems? That's a nonsensically unrealistic assumption. There is absolutely no reason not to include Wrath of Air if you're going ot assume Heroism.

Somegnome · 06/10/08, 1:14 PM

Actually, it does make sense.

Heroism is a once in ten minutes push of a button, totems require the shaman to be alive and to keep them up every two minutes, unless they get destroyed for some reason which makes it even less likely they are up all through the fight.

Last but not least, heroism doesn't have a range limit, in many fights it can happen more or less easily for a mage to be seperated from the totems, in terms of range, though.

However, I'd like to inquire about how rawr models heroism in the mage module. Does it take into account that for the sake of maximizing dps of a fire mage, it's best to be used at 20%? Apologies if this seems a stupid question, I just found no mention of this in the sheet and think it's important enough to ask about.

Thank you very much for this sheet anyways, I found it very interesting to skim through.

Sancus · 06/10/08, 1:21 PM

Heroism is a once in ten minutes push of a button, totems require the shaman to be alive and to keep them up every two minutes, unless they get destroyed for some reason which makes it even less likely they are up all through the fight.

Last but not least, heroism doesn't have a range limit, in many fights it can happen more or less easily for a mage to be seperated from the totems, in terms of range, though.

False. Heroism requires the shaman to be alive when the boss hits 20-30% or thereabouts, not "once in 10 minutes", because mage-group shaman are required to use it at that time. Any assumption which assumes your shaman is dying before 2 minutes into the fight is a bad one, since no one benchmarks dps assuming their group is going to die. You may as well add death chances in for VT/VE and for YOU, to average out your dps assuming you die some of the time.

Shaman should always be in range of their group, if they're not, your raid leader is incompetent or your shaman is. Not a valid reasoning.

Any benchmark that includes Heroism but not totems is wrong and not worth looking at.

Otterpop · 06/10/08, 2:21 PM

*edit* my point was irrelevant, since Rawr spec comparisons are for high-end endgame. I'm pretty casual, as is most of my guild, and rarely get heroism when I should anyway. Mods, feel free to delete this post if need be.

MyrddinE · 06/10/08, 2:57 PM

I included Heroism because it's a core mechanic of high end raiding. Stacking Heroism + Icy Veins + etc when the boss is at 20% is such a powerful mechanic that raids will go to a lot of effort to hit that. I've seen raids swap a melee Shaman into the caster group at 25%, just so he could pop heroism for the mages.

That, plus Heroism's longer range (100 yards rather than 20ish for totems) means that Heroism is much more likely to be available than any particular totem. In addition, unlike the other totems which have a flat addition to damage, the Heroism totem's haste effect alters rotations more... Arcane specs can benefit very little, sometimes, at high ends from heroism.

Because it's a powerful, often used buff that alters the dynamics of one spec vs another, I thought it was important to include it despite being a group buff.

Lysistrate · 06/11/08, 7:02 AM

Hi and thank you for your work.

For sure you are right that for swapping, a shaman can basically sit anywhere. in high end raiding group composition is crucial. with max 3 mages it looks like this:
Shadow
Mage
Mage
Mage
Shaman

Do the calculations with a resto shaman.. I bet this is the most common raid setup.

Setup with 2
Shadow
Mage
Mage
Shaman
additional healer due to shadow support

So your statement is kind of contradictory... high endgame raiding with shadow support and heroism but nothing useful fills the free slot?

Feel free to browse the brutallus logs... I bet you will find in every mage grp a shadow and at least a resto shaman.

Ranec · 06/11/08, 10:50 AM

I have recently become rather suspicious of Rawr. Based on the results you found through an exhaustive search of viable specs for mages, I don't know that Rawr is correctly modeling certain aspects of the mage class correctly. I became convinced of this last night, when a tier 4 mage using pure arcane blast was #1 raid dps during trash, beating a tier 6 rogue (normally #1 dps) wielding the Illidan offhand. Granted, the mage had a shaman popping heroism and drums, and a shadow priest restoring mana. However, the rogue also had two shamans popping totems. Based on the above Rawr results, this should not have happened, but it was a legitimate occurrence (no random disconnects, etc). Of course, this was only during trash, arcane blast didn't seem to be as viable during long boss fights due to mana problems.

Also, I examined the source code to Rawr. Results are built using a rather long list of shot rotations that the author developed, probably borrowing from personal experience as well as the existing knowledge base. However, I was able to identify a few obvious rotations that were not present (or I at least didn't see). Hopefully the author will recognize the advantage of building an algorithm that seeks out the most efficient shot rotations, perhaps using a genetic algorithm. Given the plethora of available talents, it would be difficult to enumerate all possible shot rotations manually, and Rawr isn't there yet.

In summary, take what you get from Rawr with a grain of salt. It is extremely difficult to do what Rawr is trying to do, and it needs some additional work to get there.

Pintofbrew · 06/11/08, 11:02 AM

Originally Posted by Ranec

I have recently become rather suspicious of Rawr. Based on the results you found through an exhaustive search of viable specs for mages, I don't know that Rawr is correctly modeling certain aspects of the mage class correctly. I became convinced of this last night, when a tier 4 mage using pure arcane blast was #1 raid dps during trash, beating a tier 6 rogue (normally #1 dps) wielding the Illidan offhand. Granted, the mage had a shaman popping heroism and drums, and a shadow priest restoring mana. However, the rogue also had two shamans popping totems. Based on the above Rawr results, this should not have happened, but it was a legitimate occurrence (no random disconnects, etc). Of course, this was only during trash, arcane blast didn't seem to be as viable during long boss fights due to mana problems.

Firstly, you must be refering to T5 mage, not T4. T4 with Arcane Blast would net you a great big bowl of Fuck-all. Secondly, AB is massively powerful on Trash. I am never beaten by any rogue on trash, and this is purely because trash is so short I can manage to 100% rely on AB, which is nigh-on 2300dps. Thirdly, the way trash timing is staggered, I get one set of CDs every other pull, meaning that 2kDPS just hitched a ride to 3200DPSville for 20sec. (by stacking a 20% haste buff, a +436dmg and a global +30% damage, all with no costly consumables.)

For arcane you mention T5 as an indication of gear superiority, not realizing that it means a 20% buff to 100% of the damage out. This does not hold true on Bosses, of course, because on Bosses AB can only account for 60-80% of your output, the rest being a low-DPS rotation (~1500DPS)

Lastly, a lot of Rogue output is due to CD stacking of their own. I'm pretty positive no rogues do Haste pots on trash and I'm equally sure running around on trash is a lot more time-wasting than Tabbing. Not to mention Blade Flurry is a no-no due to Poly.

There is nothing wrong with Rawr, there is nothing wrong with your Trash DPS observations. Both are WAI and wearing a Glave OH will not help when on trash I hit every 1.3sec for 2400 ABs and every other pack with 1sec 4k ABs, before taking crits into account at 37%.

Maledict · 06/11/08, 11:10 AM

I have recently become rather suspicious of Rawr. Based on the results you found through an exhaustive search of viable specs for mages, I don't know that Rawr is correctly modeling certain aspects of the mage class correctly. I became convinced of this last night, when a tier 4 mage using pure arcane blast was #1 raid dps during trash, beating a tier 6 rogue (normally #1 dps) wielding the Illidan offhand. Granted, the mage had a shaman popping heroism and drums, and a shadow priest restoring mana. However, the rogue also had two shamans popping totems. Based on the above Rawr results, this should not have happened, but it was a legitimate occurrence (no random disconnects, etc). Of course, this was only during trash, arcane blast didn't seem to be as viable during long boss fights due to mana problems.

Why is this a surprise? Arcane spec is perfect for trash - it can blow all it's mana every pull, it has incredibly low threat thanks to talents, and requires no build up unlike fire or frost to reach maximum dps on each mob. Even tier 6 rogues have to accumulate combo points and lose them swapping targets.

The more important question is, why are you parsing and comparing trash?

For the original post, ignoring all group buffs apart from Bloodlust makes the comparisons fairly worthless I'm afraid. The advantage fire spec has over frost is that fire spec scales much better, thanks to the pet not gaining many of the mages abilities. Also, it might be best if you considered times in terms of mobs, rather than pure numbers - apart from the Shade of Akama final burn, for example, I cannot think of where I'd use a 100 second parse.

Pintofbrew · 06/11/08, 11:24 AM

It's extremely hard to make a consistent set of limitations; Having too-short time-frame imballances the encounter in favor of mana-intensive builds, provided they don't need to use mana-generation tools. Then, making the time-frame too long makes the same builds "step-up" whenever they gain another Evo CD.For time/fight conditions I think the default Rawr ones are perfectly fine as an indication

As for buffs, perhaps it's best if you try and make a series of comparisons, like "leagues".

How about one series with every buff expectable in an absolutely optimized raid; an Elemental Shaman and a SP, with a Retadin in the raid, Mal-CoS, CoE, JoW and Misery, every buff including I-Div. Spi (you may or may not include an Innervate too). Then moderate to an intermediate scenario, drop the retadin and the subsequent judgements, drop Div Spi and drop the E-shaman for an R-shaman. With a last "low" scenario, further dropping the shaman entirely, losing one of three blessings (BoK, naturally) and dropping CoE too. That should cover most realistic scenaria.

Kavan · 06/11/08, 11:46 AM

Originally Posted by Ranec

I have recently become rather suspicious of Rawr. Based on the results you found through an exhaustive search of viable specs for mages, I don't know that Rawr is correctly modeling certain aspects of the mage class correctly. I became convinced of this last night, when a tier 4 mage using pure arcane blast was #1 raid dps during trash, beating a tier 6 rogue (normally #1 dps) wielding the Illidan offhand. Granted, the mage had a shaman popping heroism and drums, and a shadow priest restoring mana. However, the rogue also had two shamans popping totems. Based on the above Rawr results, this should not have happened, but it was a legitimate occurrence (no random disconnects, etc). Of course, this was only during trash, arcane blast didn't seem to be as viable during long boss fights due to mana problems.

The above results are for boss fights, so comparing them to trash is really not relevant. And taking into account that the above results use 0 base stats it's really not surprising that it undervalues arcane. It's not a problem with Rawr, it's a user input problem.

Originally Posted by Ranec

Also, I examined the source code to Rawr. Results are built using a rather long list of shot rotations that the author developed, probably borrowing from personal experience as well as the existing knowledge base. However, I was able to identify a few obvious rotations that were not present (or I at least didn't see). Hopefully the author will recognize the advantage of building an algorithm that seeks out the most efficient shot rotations, perhaps using a genetic algorithm. Given the plethora of available talents, it would be difficult to enumerate all possible shot rotations manually, and Rawr isn't there yet.

In summary, take what you get from Rawr with a grain of salt. It is extremely difficult to do what Rawr is trying to do, and it needs some additional work to get there.

There is some truth to this, although you don't seem to see the big picture. The authored cycles are only the building blocks of the optimization scheme. They are used as input to a linear programming representation of the problem. The requirements of Rawr require that evaluating the value of a single gear set is fast, so using genetic algorithms to generate cycles is not really feasible. This would only make sense as an offline process to increase the pool of authored cycles.

There is one very important issue though that makes me think this is not a good idea. For cycles to be used in LP they have to be independent or self-reliant in a sense, this is especially hard to enforce for dynamic cycles. I've done quite a bit of genetic programming and I don't see this as a particularly useful application. I think using theorycrafted cycles is more appropriate. Although if someone comes up with an algorithm that generates cycles it's fairly straightforward to include them in. For that matter if you think you have a cycle that is missing all you have to do is ask for it to be added. Unless it is some very complex dynamic cycle I might code it up myself.

◊ manly · 06/11/08, 12:04 PM

Originally Posted by Ranec

I have recently become rather suspicious of Rawr. Based on the results you found through an exhaustive search of viable specs for mages, I don't know that Rawr is correctly modeling certain aspects of the mage class correctly. I became convinced of this last night, when a tier 4 mage using pure arcane blast was #1 raid dps during trash, beating a tier 6 rogue (normally #1 dps) wielding the Illidan offhand. Granted, the mage had a shaman popping heroism and drums, and a shadow priest restoring mana. However, the rogue also had two shamans popping totems. Based on the above Rawr results, this should not have happened, but it was a legitimate occurrence (no random disconnects, etc). Of course, this was only during trash, arcane blast didn't seem to be as viable during long boss fights due to mana problems.

Common problem. One cannot evaluate and compare 2 parses where player skills are not different. If anything, you can see a huge discrepancy across players on trash, some of it being playstyle-based and/or skills.

MyrddinE · 06/11/08, 7:25 PM

Also note that unlike some people, i'm not looking to find the theoretical maximum DPS possible for a mage. Which gear is the absolute best isn't really a priority for me. Instead, I'm looking to compare one spec to another. So while the buffs are significant in overall DPS, they are less so when comparing spec to spec.

Not to say it's completely insignificant either. But 150ish +spell damage for a fire mage becomes 170ish spell damage... and 20 spell damage extra is not going to make or break the fire spec, nor will it radically throw off comparisons between one spec and another.

At some point you have to have a cutoff between what is reasonable to model and what isn't. I ran the optimizer with all the global buffs, and one party buff (Heroism), as a reasonable approximation of what a 25 man raid will be able to put on a mage. Some mages will get more... crit from a Boomkin, totems, whatever. But few mages can rely on a specific group buff like they can rely on getting, for example, Blessing of Wisdom. Now, perhaps unrealistically, I also gave out Blessing of Kings. In my raids, I rarely had to suffer with fewer than 3 pallies, so Kings is a reasonable expectation for my TC (some other raids may not have that luxury). Kings is notable in that it helps Arcane spec more than any other. Another buff I gave out is Improved Spirit... something that our priests will re-spec to ensure is available to the raid, but perhaps other groups don't always get.

The point is that there are too many variables... to many 'what ifs' when comparing DPS. So I went with what I felt to be the closest approximation of what a typical raider can rely on when running the Optimizer to choose equipment. It's easy to just add a few buffs to a set of gear and see what the new result is... it's SLOW to re-run the optimizer for every single variation on buffs, and I think the accuracy for comparison is fine just using global buffs and Heroism.

If you want to expand upon my work and run results with different buffs, feel free! That's why I zipped up all that and put it out for others to use. I'm not trying to keep this to myself... if you feel a different set of buffs would produce notable differences in which spec comes out on top, then by all means do the work and publish the results!

For now, I'm gonna go back to recompiling the work properly, using a working race.