Item database project (for all these spreadsheets) - Page 5

mook06 · 10/31/07, 3:22 AM

Managed to get a nearly complete db of my own, i know nothing when it comes to parsing, so this contains the entire xml files from allakazham, its like 200mb uncompressed..over 20,000 files

PS: didn't scan if for virus's, but should be fine, use at own risk
Link:
7HQ6EX57

Goravan · 11/11/07, 3:32 PM

Originally Posted by mook06

anyone figured out how to parse weapon dps / dmg / speed?

good question. I tried ^$(\d+(?:|\.\d+)) damage per second$$ but i get the error "Application has generated an exception that could not be handled." after execute the WowExtractItem.exe

Tornhoof · 11/11/07, 5:14 PM

I can only suggest using the armory item data, see my equipoptimizer project for working xml parsing code etc.

The armory xml format is fairly well structured, there are not that many regexps necessary, mainly just for spelldata.

item-tooltip.xml and item-info.xml contain infornation about ilvl, stats, drop locations etc.

dr_AllCOM3 · 11/19/07, 8:30 PM

Is Allakhazam not up to date with it's stats?
I'm missing stats like Haste or Expertise.
This project would be a godsend for my own sheet, since I know nothing about parsing databases

. The armory would be better indeed.

koaschten · 11/20/07, 3:26 AM

Last time i checked Tornhoof had a pretty up-to-date dump of items from the armory for his gear optimizer. Can't find his thread at the moment though :X

Stirius · 11/21/07, 3:32 AM

I did a big update on my item extractor. I still believe using a different database than the armory has a big advantage: items from the testrealms are added before they appear on the armory.
Changes:
1. switched to wowhead.com - more up to date
2. new stats like expertise, haste etc are included
3. added weapon dps and speed

Download: ExtractItems2+v6.zip

Please read the included README.txt and feel free to report any bugs in this thread...

edit: fixed the parsing...

Demor · 11/26/07, 6:16 PM

Wowhead is now showing stats such as Crit Rating and Hit Rating in a different manner causing the parsing not to pull those particular stats.

Here's an example:

Old - "Equip: Increases your critical strike rating by 14."
New - "Equip: Increases your critical strike rating by 14 (1% @ L60)."

I've made multiple attempts to make these stats pull but I was not successful. Anybody have any thoughts or ideas around this?

Stirius · 11/27/07, 12:40 AM

Originally Posted by Demor

Wowhead is now showing stats such as Crit Rating and Hit Rating in a different manner causing the parsing not to pull those particular stats.

They changed it just after I uploaded this version. A quick fix is to remove ".$" at the end of the rating-rexexps in parse.txt. I will upload a fixed version in a few hours.

edit: fixed it, get the new download-link from above.

[DRF]Solmyr · 11/28/07, 1:21 PM

Great work! I do have a question. Socket bonuses? Too numerous to efficiently parse? Activation? /sigh

Armor:^(\d+) Armor$
Int:^\+(\d+) Intellect$
Spirit:^\+(\d+) Spirit$
Agility:^\+(\d+) Agility$
:^Requires Level (\d+)$
dps:^\(([\d.]+) damage per second\)$
speed:Speed ([\d.]+)$
Heal:^Equip: Increases healing done by spells and effects by up to (\d+).$
Heal:^Equip: Increases healing done by up to (\d+) and damage done by up to
SpellDmg:and damage done by up to (\d+) for all magical spells and effects.$
SpellDmg:^Equip: Increases damage and healing done by magical spells and effects by up to (\d+).$
SpellHaste:^Equip: Improves spell haste rating by (\d+).$
DmgFrost:^Equip: Increases damage done by Frost spells and effects by up to (\d+).$
SpellHit:^Equip: Increases your spell hit rating by (\d+).$
SpellHit:^Equip: Improves your spell hit rating by (\d+).$
SpellHit:^Equip: Improves spell hit rating by (\d+).$
SpellCrit:^Equip: Improves spell critical strike rating by (\d+).$
SpellCrit:^Equip: Increases your spell critical strike rating by (\d+).$
Hit:^Equip: Increases your hit rating by (\d+).$
SpellPen:^Equip: Increases your spell penetration by (\d+).$
Stamina:^\+(\d+) Stamina$
Strength:^\+(\d+) Strength$
:^\+(\d+) Nature Resistance$
:^\+(\d+) Shadow Resistance$
Defense:^Equip: Increases defense rating by (\d+).$
Hit:^Equip: Improves hit rating by (\d+).$
mp5:^Equip: Restores (\d+) mana per 5 sec.$
AP:^Equip: Increases attack power by (\d+).$
Crit:^Equip: Improves critical strike rating by (\d+).$
Sockets:^(\w+) Socket$
SocketsRed:^Red Socket$
SocketsBlue:^Blue Socket$
SocketsMeta:^Meta Socket$
SocketsYellow:^Yellow Socket$
hp5:^Equip: Restores (\d+) health per 5 sec.$
ShieldBlockRating:^Equip: Increases your shield block rating by (\d+).$
ParryRating:^Equip: Increases your parry rating by (\d+).$
DodgeRating:^Equip: Increases your dodge rating by (\d+).$
CritRating:^Equip: Increases your critical strike rating by (\d+).$
BlockRating:^Equip: Increases your block rating by (\d+).$
BlockValue:^Equip: Increases the block value of your shield by (\d+).$
SpellShadow:^Equip: Increases damage done by Shadow spells and effects by up to (\d+).$
SpellNature:^Equip: Increases damage done by Nature spells and effects by up to (\d+).$
SpellHoly:^Equip: Increases damage done by Holy spells and effects by up to (\d+).$
SpellHoly:^Equip: Increases the damage done by Holy spells and effects by up to (\d+).$
SpellFire:^Equip: Increases damage done by Fire spells and effects by up to (\d+).$
SpellArcane:^Equip: Increases damage done by Arcane spells and effects by up to (\d+).$
Resilience:^Equip: Improves your resilience rating by (\d+).$
DruidAP:^Equip: Increases attack power by (\d+) in Cat, Bear, Dire Bear, and Moonkin forms only.$
IgnoreArmor:^Equip: Your attacks ignore (\d+) of your opponent's armor.$
Expertise:^Equip: Increases your expertise rating by (\d+).$
Haste:^Equip: Improves haste rating by (\d+).$
RAP:^Equip: Increases ranged attack power by 34.$
Quote:^&quot;
:^Classes:
:^Requires
:^+(\d+) Fire Resistance$
:^+(\d+) Frost Resistance$
:^+(\d+) Arcane Resistance$
:^+(\d+) Nature Resistance$
:^Durability

edit: Sorry, I meant to ask: Could you please implement a way to pull socket bonuses? These are the choices I see in the parse.txt file. Am I missing something?

Stirius · 11/30/07, 4:11 AM

Originally Posted by [DRF]Solmyr

Could you please implement a way to pull socket bonuses? These are the choices I see in the parse.txt file. Am I missing something?

I haven't found a way to include them in a smart way to use them in excel later on. One option would be to write the text "Socket Bonus: +3 intellect" into a column. This can't be used for a calculation. Another option would be a column for every socket bonus type - how many are there? 6? - and put a 1 in, when it exists.
How are you trying to use the socket boni? Maybe someone has a good idea.

edit: maybe I should explain better: this exporter is only written to get a quick export in excel format for direct comparison of itemstats. Special on-use or on-equip abilities or set boni won't be included, because it doesn't work perfectly with parsing. Anyone who is interested in a perfect database should look at the alternatives like xml-parsing the armory.

SeanDamnit · 02/13/08, 3:59 PM

I'm having some issues getting this to work. After executing the WowExtractItem.exe, the data.csv file is empty except for the headers for the stats I selected. Anyone else having issues like this?

Stirius · 02/13/08, 4:42 PM

Sorry, but wowhead changed it's output again and my program is currently broken. I'm unable to fix it easily, because wowhead is no longer sending all tooltips in a single html-page.
I am out of ideas at the moment. The only options is to make a http-request for every item. Coding this will take some time...

SeanDamnit · 02/13/08, 7:37 PM

All good - we appreciate the work you've done on this thus far. Let us know if you plan on messing with the project again.

Stirius · 02/15/08, 3:29 AM

Originally Posted by SeanDamnit

All good - we appreciate the work you've done on this thus far. Let us know if you plan on messing with the project again.

Alright, I discovered wowdb.com, which transfers all tooltip-information contained in the html-page. I rewrote large parts of the parser and it is now working again. Have fun:

ExtractItems2-v7.zip

Let me know if there are problems.

Knightfall · 03/07/08, 4:13 PM

Originally Posted by Stirius

Alright, I discovered wowdb.com, which transfers all tooltip-information contained in the html-page. I rewrote large parts of the parser and it is now working again. Have fun:

ExtractItems2-v7.zip

Let me know if there are problems.

Stirius, awesome work! Could you include the cost of an item as a column? If not, are there parsers that can (tornhoof)?