Item database project (for all these spreadsheets) - Page 3

SeanDamnit · 08/17/07, 1:40 PM

Hey Stirius, you taking requests for this? Is it possible to include socket colors, socket bonus, and the item's source (what boss drops it, vendor, tailoring, etc)?

Thanks!

ImARestoDruid · 08/17/07, 3:03 PM

I've actually written a smart parser that will dynamically setup all pulls (you don't have to worry about number of circumstances). I'll have it finished this weekend, all that's left to do in it, is write the data to database / external file. I'm using PERL for the entire project, but I'm more than willing to share any code / ideas.

◊ Vontre · 08/17/07, 7:51 PM

Originally Posted by Disquette

I appreciate the code very much, but i get an error when trying to use it. i'm using Win2k (with updated 1.1 .Net framework). Here's a screenshot of the error:

Also, if it helps, I've attached my stats.txt file (i didn't change any of the others)

I'm having the same issue, except the posted solution doesn't help.

I'm definitely interested in using this to keep my spreadsheet item database up to date.

Edit: Nevermind I fiddled with it until it worked.

Bury · 08/17/07, 9:26 PM

Any chance you can post your complete folder for The Stupider Persons Amongst Us?

ps: you're in skarzog's guild haha

◊ Vontre · 08/22/07, 7:06 PM

I successfully used this project for my spreadsheet item selector, I consider it an extremely valuable tool. Thank you.

Stirius · 08/22/07, 7:42 PM

Originally Posted by Bury

Any chance you can post your complete folder for The Stupider Persons Amongst Us?

ps: you're in skarzog's guild haha

Here you are:
Version 3 - nothing new, only the fixed .txt-file and the new dl.bat:
ExtractItemv3

sp00n · 08/23/07, 7:24 AM

Originally Posted by Aadar

What I would also like is to be able to extract the text of the Socket Bonus into a column, is there a way of achieving this?

e.g. For the line "Socket Bonus: +3 Spell Crit Rating "

I would like "+3 Spell Crit Rating" written into a column.

Also knowing whether it is a green, blue, purple would be useful by reference to or and so on.

I tried to do the same thing, but unsuccessfully.
Then I looked into the source code and the line is basically skipped and only used to get the name of the item.
I would try to change the code, but unfortunately I don't have anything to compile for .NET. I assume it's only possible with VisualBasic or alike and there are no free tools to compile?

Heavenfall · 08/28/07, 8:00 AM

The socket bonus can be worked around using 1 parse line for each interesting socket bonus.

• Slake · 08/28/07, 11:57 AM

As part of the framework for a character profiler, I wrote a program to pull data from the Armory and parse it, then write it out to a custom XML format. I ran a full crawl the other day, so I should have an items.xml and sets.xml file covering most items.

I realize XML isn't the easiest thing to work with compared to a flat text file, but it's possible to write a translator to whatever flat format you want; I might include this in the data-handling tool whenever I get around to it. If anyone is interested in the two xml files let me know and I'll put them up somewhere.

mook06 · 10/27/07, 8:21 AM

wow, just stumbled across this, awesome work

anyone figured out how to parse weapon dps / dmg / speed?

Slake: If you wouldnt mind, im interested in your files

mook06 · 10/31/07, 4:22 AM

Managed to get a nearly complete db of my own, i know nothing when it comes to parsing, so this contains the entire xml files from allakazham, its like 200mb uncompressed..over 20,000 files

PS: didn't scan if for virus's, but should be fine, use at own risk
Link:
7HQ6EX57

Goravan · 11/11/07, 4:32 PM

Originally Posted by mook06

anyone figured out how to parse weapon dps / dmg / speed?

good question. I tried ^$(\d+(?:|\.\d+)) damage per second$$ but i get the error "Application has generated an exception that could not be handled." after execute the WowExtractItem.exe

Tornhoof · 11/11/07, 6:14 PM

I can only suggest using the armory item data, see my equipoptimizer project for working xml parsing code etc.

The armory xml format is fairly well structured, there are not that many regexps necessary, mainly just for spelldata.

item-tooltip.xml and item-info.xml contain infornation about ilvl, stats, drop locations etc.

dr_AllCOM3 · 11/19/07, 9:30 PM

Is Allakhazam not up to date with it's stats?
I'm missing stats like Haste or Expertise.
This project would be a godsend for my own sheet, since I know nothing about parsing databases

. The armory would be better indeed.

koaschten · 11/20/07, 4:26 AM

Last time i checked Tornhoof had a pretty up-to-date dump of items from the armory for his gear optimizer. Can't find his thread at the moment though :X

Stirius · 11/21/07, 4:32 AM

I did a big update on my item extractor. I still believe using a different database than the armory has a big advantage: items from the testrealms are added before they appear on the armory.
Changes:
1. switched to wowhead.com - more up to date
2. new stats like expertise, haste etc are included
3. added weapon dps and speed

Download: ExtractItems2+v6.zip

Please read the included README.txt and feel free to report any bugs in this thread...

edit: fixed the parsing...

Demor · 11/26/07, 7:16 PM

Wowhead is now showing stats such as Crit Rating and Hit Rating in a different manner causing the parsing not to pull those particular stats.

Here's an example:

Old - "Equip: Increases your critical strike rating by 14."
New - "Equip: Increases your critical strike rating by 14 (1% @ L60)."

I've made multiple attempts to make these stats pull but I was not successful. Anybody have any thoughts or ideas around this?

Stirius · 11/27/07, 1:40 AM

Originally Posted by Demor

Wowhead is now showing stats such as Crit Rating and Hit Rating in a different manner causing the parsing not to pull those particular stats.

They changed it just after I uploaded this version. A quick fix is to remove ".$" at the end of the rating-rexexps in parse.txt. I will upload a fixed version in a few hours.

edit: fixed it, get the new download-link from above.

[DRF]Solmyr · 11/28/07, 2:21 PM

Great work! I do have a question. Socket bonuses? Too numerous to efficiently parse? Activation? /sigh

Armor:^(\d+) Armor$
Int:^\+(\d+) Intellect$
Spirit:^\+(\d+) Spirit$
Agility:^\+(\d+) Agility$
:^Requires Level (\d+)$
dps:^\(([\d.]+) damage per second\)$
speed:Speed ([\d.]+)$
Heal:^Equip: Increases healing done by spells and effects by up to (\d+).$
Heal:^Equip: Increases healing done by up to (\d+) and damage done by up to
SpellDmg:and damage done by up to (\d+) for all magical spells and effects.$
SpellDmg:^Equip: Increases damage and healing done by magical spells and effects by up to (\d+).$
SpellHaste:^Equip: Improves spell haste rating by (\d+).$
DmgFrost:^Equip: Increases damage done by Frost spells and effects by up to (\d+).$
SpellHit:^Equip: Increases your spell hit rating by (\d+).$
SpellHit:^Equip: Improves your spell hit rating by (\d+).$
SpellHit:^Equip: Improves spell hit rating by (\d+).$
SpellCrit:^Equip: Improves spell critical strike rating by (\d+).$
SpellCrit:^Equip: Increases your spell critical strike rating by (\d+).$
Hit:^Equip: Increases your hit rating by (\d+).$
SpellPen:^Equip: Increases your spell penetration by (\d+).$
Stamina:^\+(\d+) Stamina$
Strength:^\+(\d+) Strength$
:^\+(\d+) Nature Resistance$
:^\+(\d+) Shadow Resistance$
Defense:^Equip: Increases defense rating by (\d+).$
Hit:^Equip: Improves hit rating by (\d+).$
mp5:^Equip: Restores (\d+) mana per 5 sec.$
AP:^Equip: Increases attack power by (\d+).$
Crit:^Equip: Improves critical strike rating by (\d+).$
Sockets:^(\w+) Socket$
SocketsRed:^Red Socket$
SocketsBlue:^Blue Socket$
SocketsMeta:^Meta Socket$
SocketsYellow:^Yellow Socket$
hp5:^Equip: Restores (\d+) health per 5 sec.$
ShieldBlockRating:^Equip: Increases your shield block rating by (\d+).$
ParryRating:^Equip: Increases your parry rating by (\d+).$
DodgeRating:^Equip: Increases your dodge rating by (\d+).$
CritRating:^Equip: Increases your critical strike rating by (\d+).$
BlockRating:^Equip: Increases your block rating by (\d+).$
BlockValue:^Equip: Increases the block value of your shield by (\d+).$
SpellShadow:^Equip: Increases damage done by Shadow spells and effects by up to (\d+).$
SpellNature:^Equip: Increases damage done by Nature spells and effects by up to (\d+).$
SpellHoly:^Equip: Increases damage done by Holy spells and effects by up to (\d+).$
SpellHoly:^Equip: Increases the damage done by Holy spells and effects by up to (\d+).$
SpellFire:^Equip: Increases damage done by Fire spells and effects by up to (\d+).$
SpellArcane:^Equip: Increases damage done by Arcane spells and effects by up to (\d+).$
Resilience:^Equip: Improves your resilience rating by (\d+).$
DruidAP:^Equip: Increases attack power by (\d+) in Cat, Bear, Dire Bear, and Moonkin forms only.$
IgnoreArmor:^Equip: Your attacks ignore (\d+) of your opponent's armor.$
Expertise:^Equip: Increases your expertise rating by (\d+).$
Haste:^Equip: Improves haste rating by (\d+).$
RAP:^Equip: Increases ranged attack power by 34.$
Quote:^&quot;
:^Classes:
:^Requires
:^+(\d+) Fire Resistance$
:^+(\d+) Frost Resistance$
:^+(\d+) Arcane Resistance$
:^+(\d+) Nature Resistance$
:^Durability

edit: Sorry, I meant to ask: Could you please implement a way to pull socket bonuses? These are the choices I see in the parse.txt file. Am I missing something?

Stirius · 11/30/07, 5:11 AM

Originally Posted by [DRF]Solmyr

Could you please implement a way to pull socket bonuses? These are the choices I see in the parse.txt file. Am I missing something?

I haven't found a way to include them in a smart way to use them in excel later on. One option would be to write the text "Socket Bonus: +3 intellect" into a column. This can't be used for a calculation. Another option would be a column for every socket bonus type - how many are there? 6? - and put a 1 in, when it exists.
How are you trying to use the socket boni? Maybe someone has a good idea.

edit: maybe I should explain better: this exporter is only written to get a quick export in excel format for direct comparison of itemstats. Special on-use or on-equip abilities or set boni won't be included, because it doesn't work perfectly with parsing. Anyone who is interested in a perfect database should look at the alternatives like xml-parsing the armory.

SeanDamnit · 02/13/08, 4:59 PM

I'm having some issues getting this to work. After executing the WowExtractItem.exe, the data.csv file is empty except for the headers for the stats I selected. Anyone else having issues like this?

Stirius · 02/13/08, 5:42 PM

Sorry, but wowhead changed it's output again and my program is currently broken. I'm unable to fix it easily, because wowhead is no longer sending all tooltips in a single html-page.
I am out of ideas at the moment. The only options is to make a http-request for every item. Coding this will take some time...

SeanDamnit · 02/13/08, 8:37 PM

All good - we appreciate the work you've done on this thus far. Let us know if you plan on messing with the project again.

Stirius · 02/15/08, 4:29 AM

Originally Posted by SeanDamnit

All good - we appreciate the work you've done on this thus far. Let us know if you plan on messing with the project again.

Alright, I discovered wowdb.com, which transfers all tooltip-information contained in the html-page. I rewrote large parts of the parser and it is now working again. Have fun:

ExtractItems2-v7.zip

Let me know if there are problems.

Knightfall · 03/07/08, 5:13 PM

Originally Posted by Stirius

Alright, I discovered wowdb.com, which transfers all tooltip-information contained in the html-page. I rewrote large parts of the parser and it is now working again. Have fun:

ExtractItems2-v7.zip

Let me know if there are problems.

Stirius, awesome work! Could you include the cost of an item as a column? If not, are there parsers that can (tornhoof)?