Armory Scraper

nife · 09/06/07, 12:46 AM

Does anyone have a well written c#/c++/c/java armory scrapper ? I'm looking to try to re-start what the Arena Blogger blog used to do. Post rankings every week with movers and battlegroup and server rankings. (Basically I would like to get the data into data structures I can do the rest from there.)

If the answer is that there's not one does anyone have any suggestions on the best way to get started looking at creating one ?

crimsonsentinel · 09/06/07, 2:34 AM

World of Warcraft Arena Ranking

Is this what you're talking about?

nife · 09/06/07, 2:50 AM

nope I'm talking about Arena Blogger that was linked in the blog thread a while ago.
Basically it lists the best servers and the the best battle groups by average rating.

Herrera · 09/06/07, 3:50 AM

I have a library I use for my own purposes written in C# which extracts all armory information.
I still work on finishing bits of it, but you can do a lot in it's current state.
It's a re-write of an older VB.NET library I found somewhere on the net, but I can't recall where to be honest.

• mutagen · 09/06/07, 2:24 PM

Antiarc's Armory Tools are in PHP but are available. They may make a nice starting point or maybe a reference to solving any problems you run into writing your own in your preferred language.

Andorien · 09/06/07, 4:58 PM

The word you are looking for is "scraper".

nife · 09/06/07, 11:01 PM

Yeah my spelling sucks.
I think my only option is to create my own. Php would just be too slow to crawl the site. Thanks

Herrera · 09/07/07, 4:05 AM

It's best to make one of your own if your intention is to data mine the armory.
I've uploaded the library I work on, in case you find it useful.

GigaSize.com: Host and Share your Files

Kralizek · 10/05/07, 6:45 AM

Originally Posted by nife

Does anyone have a well written c#/c++/c/java armory scrapper ? I'm looking to try to re-start what the Arena Blogger blog used to do. Post rankings every week with movers and battlegroup and server rankings. (Basically I would like to get the data into data structures I can do the rest from there.)

If the answer is that there's not one does anyone have any suggestions on the best way to get started looking at creating one ?

i released this .net library: WoWArmoryParser - Home

Herrera · 10/05/07, 8:26 AM

Originally Posted by Kralizek

i released this .net library: WoWArmoryParser - Home

That's right. I used your library as a base to what I linked above. I converted the source to C# and added few nifty features, but with so much work lately, I haven't had a time to finish up some stuff I had in mind. Anyway, great job you did there. /bearhug

Kralizek · 10/06/07, 8:06 AM

Originally Posted by Herrera

That's right. I used your library as a base to what I linked above. I converted the source to C# and added few nifty features, but with so much work lately, I haven't had a time to finish up some stuff I had in mind. Anyway, great job you did there. /bearhug

what about?

send me the code so i can insert it into the main project.

Kralizek · 10/06/07, 12:22 PM

Originally Posted by Herrera

I have a library I use for my own purposes written in C# which extracts all armory information.
I still work on finishing bits of it, but you can do a lot in it's current state.
It's a re-write of an older VB.NET library I found somewhere on the net, but I can't recall where to be honest.

according to open source's license, it would be polite to give credit to the original author :|

you renamed the library removing any reference to the author, too....

goodr · 10/09/07, 12:22 PM

For those of you who have spent some time on mining the data out of armory, what approach do you use to gather 'all' the data? Let me elaborate. For example, if I am saving snapshots of armory profiles to my database for analysis, I don't think I can be sure I've collected, for example, "All level 70 characters on Stonemaul" or something similar.

It's been a few weeks, but I believe the best I was doing when I last looked at this problem was to manually look up guilds and add whole guilds at a time. Upon updates and such, I would eventually gather more guild names as folks moved onto new guilds, but this was never very comprehensive, especially to include folks who may not be in PVE progression guilds and only in small PVP guilds.

The long term goal is to be able to comprehensively collect all level 70 characters for analysis without asking the users to add characters themselves or manually add them myself.

Thoughts on solving this?

Thanks

Herrera · 10/10/07, 11:35 AM

Originally Posted by Kralizek

according to open source's license, it would be polite to give credit to the original author :|

you renamed the library removing any reference to the author, too....

Apologies. Nothing was intentional. The VB -> C# converter I used messed up namespaces and created all kinds of weird code, so I cleaned it up. I never plan to release the modficiations publicly, which is why I didn't bother to include author files (I even mentioned it's not my work in my original post above). There's no reference to me either. I'm only using it for personal needs. I'll send you my latest modifications.

For those of you who have spent some time on mining the data out of armory, what approach do you use to gather 'all' the data? Let me elaborate. For example, if I am saving snapshots of armory profiles to my database for analysis, I don't think I can be sure I've collected, for example, "All level 70 characters on Stonemaul" or something similar.

It's been a few weeks, but I believe the best I was doing when I last looked at this problem was to manually look up guilds and add whole guilds at a time. Upon updates and such, I would eventually gather more guild names as folks moved onto new guilds, but this was never very comprehensive, especially to include folks who may not be in PVE progression guilds and only in small PVP guilds.

The long term goal is to be able to comprehensively collect all level 70 characters for analysis without asking the users to add characters themselves or manually add them myself.

Thoughts on solving this?

No real solution there. Problems are characters without guild and arena teams. Otherwise, you can start with arena bracket listing, which would produce the majority of guilds available which will give you character info for guild members (this is one of the features I added to the Kralizek's library). Doing this for all 3 brackets will ensure you have at least 98%+ of the level 70 characters.