Elitist Jerks - View Single Post

• spawnstah · 01/23/08, 12:03 PM

Hi all,
As we all know there are a couple of websites gathering information from the WOW-Armory for various reasons. The most well known site that's doing this is of course Wowjutsu. There are also a few sites gathering data for stats, class balance, server population etc etc.
All these sites and persons hosting the services run their own crawler(s) indexing the blizzard armory. I'm myself running between 15-25 crawlers for one of the sites and I can tell you, it takes days and weeks to index all the armory information and thus making the data old and inaccurate quite fast.
Blizzard does not seem to be interested in opening up any API's or similar ways to get faster access to the data than the current webqueries for each character.

Let me make one thing clear, I'm not a coder in any way, so I dont know if this would be possible or helpful but...
This morning on my way to work, I thought; Would it be possible to make a central database for all the private sites to use, and try to keep it updated more frequently than all the different sites can manage on their own now?
Either by having all the current various crawlers report to one database, or even better, make it possible for users to run a crawler and help index and keep the database fresh?

The various private sites would then be able to use the central database with up-to-date and huge amount of gathered data for their various projects. Depending on how many crawlers/users running, I guess the data could be 2-3 days old instead of weeks.

As I mentioned, I'm not a Coder but I do have a lot of experience from running high-end and critical systems, so I do understand and expect that this project would requier quite a lot of hardware, bandwidth etc but I think those things can be solved.
Though my main concern is if for example wowjutsu would be able to use a central database or if the queries and latency would be too slow. Maybe possible to just do delta replications of changed data to their own local server?

Anyway, what do you think about a project like this. Would it work (technically), would it be useful, anyone interested in doing it?

(Sorry Mods if this was the wrong part of the forum to post this in, I didnt feel it belong in the UI and Addons section).

01/23/08, 12:03 PM	#1 (permalink)
• spawnstah Von Kaiser Stitch Tauren Warrior <Horizons> Genjuros (EU)	"Private" armory database Hi all, As we all know there are a couple of websites gathering information from the WOW-Armory for various reasons. The most well known site that's doing this is of course Wowjutsu. There are also a few sites gathering data for stats, class balance, server population etc etc. All these sites and persons hosting the services run their own crawler(s) indexing the blizzard armory. I'm myself running between 15-25 crawlers for one of the sites and I can tell you, it takes days and weeks to index all the armory information and thus making the data old and inaccurate quite fast. Blizzard does not seem to be interested in opening up any API's or similar ways to get faster access to the data than the current webqueries for each character. Let me make one thing clear, I'm not a coder in any way, so I dont know if this would be possible or helpful but... This morning on my way to work, I thought; Would it be possible to make a central database for all the private sites to use, and try to keep it updated more frequently than all the different sites can manage on their own now? Either by having all the current various crawlers report to one database, or even better, make it possible for users to run a crawler and help index and keep the database fresh? The various private sites would then be able to use the central database with up-to-date and huge amount of gathered data for their various projects. Depending on how many crawlers/users running, I guess the data could be 2-3 days old instead of weeks. As I mentioned, I'm not a Coder but I do have a lot of experience from running high-end and critical systems, so I do understand and expect that this project would requier quite a lot of hardware, bandwidth etc but I think those things can be solved. Though my main concern is if for example wowjutsu would be able to use a central database or if the queries and latency would be too slow. Maybe possible to just do delta replications of changed data to their own local server? Anyway, what do you think about a project like this. Would it work (technically), would it be useful, anyone interested in doing it? (Sorry Mods if this was the wrong part of the forum to post this in, I didnt feel it belong in the UI and Addons section).