We are drawing near to the end of another expansion, and thus the end of another generation of theorycrafting tools. As our rotations are likely to be significantly altered by forthcoming mechanics changes, the existing tools will require significant revisions to remain useful and accurate; hence, its a natural time to look at what other changes and improvements can be made. Hence, I'd like to take this opportunity to solicit feedback about what people have and haven't liked about the current generation of tools.
That is: ShadowCraft was a significant departure for the rogue community. Previously, we'd most relied on spreadsheets; while calculator programs had existed previously, they'd never really caught on with rogues - ShadowCraft was our first successful venture into that territory. And while there are some areas where it was a significant improvement, there were certainly downsides to the move as well. So I'd be interested in hearing opinions about what people did and didn't like about ShadowCraft, and what they'd like to see changed and kept the same in Mists.
I will encourage you to be constructive in your comments - remember that ultimately what gets build is up to the people who are building it. This is merely an invitation to provide feedback about what you have and haven't liked, for them to act on (or not) as they see fit.
I will also note that I said "them" and not "us". I don't expect to have as much time to directly contribute to the authorship of these tools as I have in the past; I don't have any plans to write or significantly contribute to the Mists tool(s) at this point in time. I will give advice and guidance to the extent that it is wanted, but it is my sincere hope that some among you still step up to do the heavy lifting with modeling that I have done the last few expansions. If you are interested in doing so, I'd encourage you to speak up; whether you want to contribute to a larger project (such as ShadowCraft) or do something on your own, it will be helpful to the community to know who's is thinking about writing something.
I think in general Shadowcraft has been a fairly effective tool- in particular the armory import and auto reforger are conveniences that the old excel spreadsheets lacked (some did support armory import via macros but not all). That said, the shift to a web-hosted app brought along a few problems of its own. First thing that immediately comes to mind is that errors in modeling or plain-old bugs, even if fixed in the backend, took quite some time to migrate to the live server. Antiarc obviously isn't under any requirement to do all this, and he did great work in getting the UI up and running. But it nonetheless makes things somewhat unreliable when it's difficult for most people to be using the latest version of the tool. Along the same lines, the excel spreadsheets were always fairly conducive to modifications like adding new items or quickly applying bug fixes.
I can't necessarily comment on the accuracy of the spreadsheets vs. the simulator method of dps modeling- I'll assume that the simulators do a better job of it but don't really know for sure.
Both approaches would probably have run into the same trouble with significant cycle/rotation/priority changes like the introduction of the legendary proc and how to accurately model its effect on dps. That one is so dependent on player choice, has such high variance, and is so affected by the timing of each encounter that I'm not sure there is a way to handle it properly.
All that said, I think Shadowcraft has been a step in the right direction but I think it'd be beneficial to the rogue community if it had better support for getting backend updates to whatever frontend that players use.
The own big downside I experienced with Shadowcraft is that you're basically depending on the developers to do all of the work. The hurdles you have to pass are just too high, which severly limits the amount of candidates willing to work/help out on this project. With the excel sheet it was relatively easy, a whole lot of people were able to look at the "source code" and identify bugs or even add new features. This is now not the case anymore.
Below is a list of the top 3 things that bugged me the most:
I had severe problems installing the whole tool in one of my own virtual machines. I tried three times, I failed three times. And I'm pretty sure I'm not the only one. Please either somehow make this easier, or provide a very detailed step for step how-to (for people that have absolute zero experience with Ruby or MongoDB, or the general architectur behind the tool).
The possibility (but not necessity!) to fiddle more with the cycles/circumstances of a fight. I.e. the ability to set the value for dodge (to 0, or higher than normal for boss fights where you do have to stand in front of the boss (to emulate parry)).
The automatic reforge did produce some weird results from time to time. I'm not sure if there's an easy way to fix that though due to the nature of the reforge tool and the cost/benefit of it.
The reforge optimizer was a nice gesture, but tools like wowreforge.com do the same thing better, with changeable stat weights and stat caps. As for spreadsheet vs online tool, I much prefer spreadsheet for the reasons sp00n pointed out.
Well, the first thing that comes to mind that was as great as it has been for past spreadsheets is that people like yourselves invested huge amounts of time to provide us with something that no other class can compare to. Thank you all for that
Now for some proper feedback:
+Auto-Reforge was something I used constantly even though I knew it couldn't be 100% accurate all the time. The other reforge tool posted in the forums here was much more accurate, but took a whole while longer to print out the results - which quite often were the same that shadowcraft application gave me anyway.
+I personally liked the graph that made it easy to jump back to previous gear arrangements, something that was much more difficult to accomplish in older spreadsheets.
+It is just a convenience, but being able to compare specs within the same application was, well, more convenient than having to open up different spreadsheets, where in some special cases you had to do the work twice to accommodate certain boss situations.
-As sp00n already pointed out, it was quite a ruckus having to wait for the implementation of new items, estimated values for procs like trinkets or the new daggers, something that was much easier when you could simply change the new primary/secondary stats yourself in a spreadsheet. This is something a lot of people can do by themselves in a spreadsheet.
-The general knowledge (/salute) of excel among the community was much much bigger than the knowledge of the tools you guys used this time. It certainly had advantages, otherwise you wouldn't have decided to go for it, but it made it almost impossible for someone with a great idea to improve it to do it by themselves and release it when it was done.
-Being able to put in simple values like primary/secondary stats, parry/dodge chance or even boss armor back in TBC yourself was something I enjoyed immensely, not to mention that it was necessary back then.
All in all I'm very happy to see that the rogue community still provides a lot of insight into the game mechanics, something other classes surely lack. Personally I prefer spreadsheets over an online tool for the aforementioned reasons, I just like being able to change things myself in case that you guys have more important stuff on your hands than playing around with a game, or for things that noone is interested in but me.
The problem with Excel driven spreadsheets is that they're rather clunky when it comes to "programming" (just remember Aldriana's cycle calculation!). You just can't do everything, or everything in a reasonable way using Excel only.
Therefore the switch to an online was both a step in the right and the wrong direction, a compromise.
In the end, what would be an ideal solution, would probably something more in the line of what SimulationCraft does. Not necessarily their approach of simulating a battle, but deploying a tool that you can both run online as well as on your own machine without too much understanding of the underlying architecture.
Basically what would be optional is to have pre-populated item databases that you can edit just as easily as in a spreadsheet, and the same goes for the basic combat mechanic (dodge rating, number of mobs, etc).
I have no idea how easily this would be possible with the current Python backend and Ruby/MongoDB frontend (which I absolutely admire for its overall graphical experience, wish I could do that), but I personally would deem this as a near to optimal solution.
(This would also takes off some stress from the current hosters, both in terms of traffic and if there is a problem, like right now that Shadowcraft is only reporting a 502 error when accessing the URL).
* UI , Easy import from armory , no need of recurrent download for new content , reforges, preloaded spec templates
Would be nice :
* List of possible upgrades per slot synthesis (a report?) based on a selection of tier content (like the previous spreadsheet), with source location of the items
* Ability to add items, and even theorycrafting on a customized local profile , using the frontend UI
* The source NPC of each item, on the tooltip
* Ability to see (from the front end UI) the formulas , and to edit them in the local profile
About the ease of install of backend
* Maybe providing the framework as a pre-installed VM image ? (i expect OS licences issues there^^)
Hope we will have continuity in the quality of modeling in the future
I will use this opportunity to thank all the team for their efforts.
My ideal would be a program we download, that has a separate item db and has options that cover what most players tweak. Depending on what language is chosen, we can then also offer a source code download so for those people who want to change the rotation to experiment or add new trinket logic they can then compile or run their own version.
I utterly agree with Seliathan's post above, particularly regarding the usefulness of the auto-reforge button (and associated export tool; that, coupled with Reforgerade, made my online life oh-so-much less of a hassle) and the "history chart" over on the right that allowed for relatively quick back-and-forth comparisons of potential gear/buff adjustments.
To me, the question of what could be done to improve ShadowCraft (or its ilk) in the future is as much as anything a question of what place we see it having within the rogue community as a whole. It can't be underestimated just how massive an impact it's had to be able to tell people who ask about optimal reforges to "ShadowCraft it" instead of "spreadsheet it." There is a Grand Canyon-sized chasm between having a beautifully designed, fully functional, mostly intuitive, Web-based front-end interface and having an .xls you need to download. ShadowCraft has made rogue stat optimization accessible to a far greater number of players than ever before.
That broader accessibility raises a question of just how user-friendly the tool should be in the future, and to what extent it should cater to an increasingly casual (and arguably less knowledgeable) yet increasingly raid-experienced (thanks to Raid Finder) player base. To me, the sweet spot would have some way of catering to all. You'd have the flashy UI for the masses who mostly only want to know three things:
1) How should I reforge all this stuff?
2) How should I gem all this stuff?
3) Is this an upgrade?
The interface could be pared down to drastically reduce the number of customizable options, thus making it less of a pain in the ass to maintain.
Meanwhile, you could have the downloadable spreadsheet/program/thingie that would allow for deeper tweaking of stat weights, talents, buffs, debuffs and the like, to allow those of us who care to conduct more fine-tuned experimentation.
I've used your, Aldrianna, spreadsheets in the past, and the new tool really is an improvement over them. As much as the spreadsheets allowed for customizations, I'm sure we've gained in practicality with Shadowcraft.
I particularly enjoy the reforging tool, the right-side graph, the gear snapshots, the way talents and their calculated dps and glyphs are displayed; I like that it's nice and user-friendly. All in all, I find it the best gear optimizer out there (compare it to Female Dwarf and you'll know).
However, as some have said already, backend updates sometimes make their way to Shadowcraft too slowly. For instance, there still is this bug with Wrath of Unchaining 390 and 403. If I go browse the source code in Aldrianna's GitHub, I find an inappropriately referenced variable for LFR WoU which triggers the miscalculation. I could download the backend and make the change myself, but the way to make it work still elude me and I'm left clueless despite being a bit nerdy and tech savvy.
As a final word, I really really like Shadowcraft. Does it need though to be online? What if it was distributed as a package for us to use offline? I read that the backend's in Python and the frontend in Ruby/MongoDB: it'd easily be deployable on OS X; and I'm sure that Windows users, given a clear and didactic step by step tutorial, could deploy it as well. A local web server with an app that's tied to it to download the updated source code when available (from GitHub via git clone maybe?), access to the local cloned source code, with the same interface.
This would also allow Antiarc to stop paying for something we get free, as noble as it is.
*DPS Graph with the ability to click and jump to previous states
*Auto-Reforge with ability to export into Reforgerade
*Premade talent/glyph templates (although this will be a lot less relevant in MoP)
Suggestions for Improvements:
*Add a Known Issues link at the top, with a link to report bugs
*Add the drop location to mouseover tooltip
*Color code (or otherwise distinguish) items if their proc model is currently in "beta" stage
*Add ability to narrow items to specific item levels
*Add a pie chart (or just table) showing what % of total damage each individual ability composes. I find this more useful than the raw DPS number when trying to figure out if I'm "doing it wrong."
My feelings mirror Andeh's list above. I agree with others who would like an option to set a cap for expertise and I imagine a similar function for hit would be useful too for pvpers due to the lower requirements. In addition I think an option to export stat weights to LootRank.com would be useful and with this the addition of stat weights for weapon speed would be nice. A display for the DPS provided by set bonuses would be great too but I don't know how feasible that is.
I'm curious on whether people would rather work in/deal with Java over Python. Personally I would rather have a Java engine, but does anyone think Python still is the best language? It might be worth looking back on this topic too, but I fear most won't care so long as the end product is usable.
I would also like to see support for handling phases, and maybe even interruptions. Thinking back to Al'akir, we could estimate the phase transitions and how that effects EP weights and gearing without trying to math it out ourselves.
I wasn't a fan of the 3rd party dependency for the frontend, and I said that back at the end of t11 or something. The gui ended up looking fantastic, but I can only wonder if Antiarc's app was the only one that was really running. I did something weird to get it to load once on a VM, but have since lost it and never could get it to run again.
I'd like to see a shift to more end user control: being able to update the engine without needing a new frontend, letting people update the backend without waiting on a middle man, allow people to add whatever items they wanted, be able to turn more knobs in the engine.
I also think we could use a reforger update. I'm personally okay with (almost hoping for?) an integrated exhaustive solution. That doesn't seem feasible for a web server though, and should be left to just client side apps if those take off.
I did like the ease of making changes to the frontend though, and the abilty to revert several changes at once. The app felt quick at the same time, which is usually a big deal of mine. Most of the features that it did have felt seamless, and the availability was great for what it did.
The decision to use Python was primarily mine; it is the language I'm most comfortable with so in the interest of getting things up and running quickly, I went with what I knew. That said, its not without its advantages; I think it is among the more user-friendly languages out there, is lightweight enough to be distributed bundled with an application, and so on.
That said: I don't think we've quite hit on the right platform solution yet. The particular problem we ran into towards the end of Wrath is that we had a single point of failure on keeping the site updated - when Antiarc got busy, there really wasn't any way for anyone to help him out and do site updates. This has been an issue with past approaches as well - in previous expansions when various authors got too busy to update their spreadsheets, much the same effect occurred - but it is something that would be worth avoiding if there's an elegant way to do it.
Its also worth noting that both Antiarc and I started writing in the languages we were most comfortable with, and the end result was sort of a Frankenstein's monster of different languages. I think it would have been a lot more manageable had we written it end to end with one language in mind. For instance, with the existing library you could write a perfectly good webapp in Django using MySQL or SQLite for DB persistence, and I think that would make the integrations a lot simpler.
Ultimately, though, the problem is that everyone wants something a little different from their theorycrafting tool. Some people want an in-game addon (which has to be in Lua) while others want an app for their phone (which can't be). Some people want online, and some people want offline. No one solution is going to work for everyone, so its sort of a matter of picking which audience or audiences you want to support and letting everyone else make do. So if you'd like to do it in Java, that's fine... but there will be tradeoffs to that decision, and its worth understanding them.
I will also note that user control is sort of a myth. I mean, there are always going to be power users that are comfortable making their own edits to the code, or digging into the bowels of the spreadsheet, or whatever; but they are *vastly* the minority, even amongst EJ posters. For those of you that want to do it, I can certainly understand why; but I'm not sure it can be a major design goal. I think for the community as a whole, having something that works out of the box and is updated in a timely fashion is vastly more important than something that power users can edit themselves.
In terms of my two cents about how ShadowCraft worked as a whole: I think the major gain was in maintainability. Say what you will about how updates and bugfixes are working out in ShadowCraft, but its nevertheless an evolutionary step in being able to have things work out of the box. With a spreadsheet every new proc, no matter how simple, had to be manually implemented, and every piece of gear manually entered; in ShadowCraft this all just works. Set bonuses and new wierd procs and such still require specific effort, but the amount of work that was saved by gear imports and a general system for modeling procs cannot be underestimated.
I think the biggest shortcoming of ShadowCraft is in terms of gear recommendations. I think the spreadsheets of Wrath were much better at presenting upgrade recommendations than ShadowCraft is. As it turns out making good gear recommendations is surprisingly complicated, and something that you really need to put time and effort into to get it right; and in Cata, we didn't. And I kind of wish we had.
There is a lot of truth to "user control is ... a myth." If you take it to its extreme, though, the only thing you need to publish is a text file:
BestRogue.txt Use rotation xxx for single-target and yyy for AoE.
Use this BiS list.
Use that flask.
For users who want a bit more detail, you might add
Here are our approximate stat weights and caps to use while gearing up.
It is/isn't worth splitting 4t17 into 2t17+2t18.
The people who will get the most benefit from your tool are the ones who can use it to write BestRogue.txt, gain insight into their class, develop a creative solution for a special situation, or see a problem in the existing theorycraft. Those people will very likely need to dig into the bowels of your tool, and if they can't (or it is too hard) they will use (or create) a different one.