• spawnstah

Hi all,
As we all know there are a couple of websites gathering information from the WOW-Armory for various reasons. The most well known site that's doing this is of course Wowjutsu. There are also a few sites gathering data for stats, class balance, server population etc etc.
All these sites and persons hosting the services run their own crawler(s) indexing the blizzard armory. I'm myself running between 15-25 crawlers for one of the sites and I can tell you, it takes days and weeks to index all the armory information and thus making the data old and inaccurate quite fast.
Blizzard does not seem to be interested in opening up any API's or similar ways to get faster access to the data than the current webqueries for each character.

Let me make one thing clear, I'm not a coder in any way, so I dont know if this would be possible or helpful but...
This morning on my way to work, I thought; Would it be possible to make a central database for all the private sites to use, and try to keep it updated more frequently than all the different sites can manage on their own now?
Either by having all the current various crawlers report to one database, or even better, make it possible for users to run a crawler and help index and keep the database fresh?

The various private sites would then be able to use the central database with up-to-date and huge amount of gathered data for their various projects. Depending on how many crawlers/users running, I guess the data could be 2-3 days old instead of weeks.

As I mentioned, I'm not a Coder but I do have a lot of experience from running high-end and critical systems, so I do understand and expect that this project would requier quite a lot of hardware, bandwidth etc but I think those things can be solved.
Though my main concern is if for example wowjutsu would be able to use a central database or if the queries and latency would be too slow. Maybe possible to just do delta replications of changed data to their own local server?


Anyway, what do you think about a project like this. Would it work (technically), would it be useful, anyone interested in doing it?


(Sorry Mods if this was the wrong part of the forum to post this in, I didnt feel it belong in the UI and Addons section).

 

Kagall

Its a good idea in theory. I would suggest Blizzard testing this out on the PTR, using only the small set of premade character sets to register the data for a "test" armory. That being said, I doubt anyone would want this to turn into a WoW version of Facebook, where every user has 18951839 add-ons to their profile that they have to disable (having an option to have wowjutsu collect data, for example).

Good suggestion.

 

Wara

I don't see where the individual user is affected by this idea, other than having somewhat more up to date data for 3rd Party uses of Armory. This sounds like an entirely back-end project.

That being said, I'm curious as to how this would be faster or more effecient for gathering Armory data. I'm sure Blizzard would appreciate the decreased crawler load on their servers, but does it technically enable more threads to be opened?

 

Riallatar

I would imagine what he's saying is that it'd be a whole lot more efficient for each of these crawlers to hit different subsets of data and for all of that data to be synchronized up to a central store which could be cloned out much more easily/efficiently than individual character queries.

 

Silverstorm

Sounds like essentially setting up a master/slave replication, where the Armory is the master, the new DB is the slave. Since it's read-only, accessing the slave is fine. This would allow for extremely targeted crawlers from the new DB to the Armory (per battlegroup, per server, etc.).

However, once the external sites switched over...here's what I see:

- Blizzard's site becomes more responsive due to (theoretically) fewer requests, though the new DB would have more crawlers to decrease "latency" of data.
- People start crawling the new DB to replicate information to their own applications (unless you plan to open up this DB for queries)

Basically all that would happen is adding a middleman, and then you have to deal with reliability of this DB as well. Granted, crawling the new DB would be faster, as it isn't presenting web content, but raw data. I don't think we would really gain anything from such a setup.

It's a good idea, and we could probably implement it better than Blizzard, since we care more about it, but we still rely on the lacking SLA of the Armory.

 

• Kalroth

I've coded a small site that accesses most of the armory information, albeit I only use some of it. Now I don't gather it for a local database, it's merely downloaded on demand. But this means I got a fairly good idea of what goes behind armory, why it's slow and why it freezes various browsers.

Short version
The problem with the armory is it's usage of XML/XSLT and SJAX. It's simply brutal on their servers AND our browsers. It's not going to get better until they take a good look at the armory code and decides to rewrite most of it.

Long version
First off, the idea of a central database is good, but Blizzard is already way ahead of you. The armory IS already a central database. To get information about a character, guild, item, arena team or talent build, you simply request a small XML file, at worst it's like 15-20kb of XML data, which is easily zip packed down to 4-6kb when using modern browsers or modern 'leeching' software.

Blizzard recently announced breaking the 10 million player mark. If you assume that only 1% of those access the armory on a daily basis, it's still 100,000 requests.

And because the armory wants to Web 2.0, it uses SJAX. But wait? Isn't it supposed to be AJAX? SJAX doesn't make any sense! Nope, it sure doesn't, which is why it boggles me that Blizzard relies upon it. AJAX means 'Asynchronous JavaScript and XML' while SJAX means 'Synchronous JavaScript and XML'. AJAX is good, it runs in the background, so it doesn't lock up your browser. SJAX is bad, it locks your browser until everything is loaded.

When a user requests an armory page of a character, he essentially starts a whole chain of events.
* First the user downloads a simple XML document.
^-> http ://www.wowarmory.com/character-sheet.xml (15-10kb).
* Then the browser loads the XSLT data for that XML document.
^-> http://www.wowarmory.com/layout/character-sheet.xsl (10kb).
* Then the browser parses this XSLT and finds another 4 include files.
^->http ://www.wowarmory.com/layout/includes.xsl (7.5kb).
^->http://www.wowarmory.com/layout/character-header.xsl (9kb)
^->http://www.wowarmory.com/layout/character-utils.xsl (6kb)
^->http://www.wowarmory.com/layout/mini...-templates.xsl (50kb)
* Then the includes from these files are loaded too.
^->http://www.wowarmory.com/layout/language.xsl (1kb)
^->http://www.wowarmory.com/layout/template-common.xsl (6kb)
^->http://www.wowarmory.com/layout/temp...lated-info.xsl (9kb)
* Then the includes from these files are also loaded.
^->http://www.wowarmory.com/strings/en_us/strings.xml (22kb)
^->http ://www.wowarmory.com/lastsearch.xml (1kb to ??kb)
^->http://www.wowarmory.com/login-status.xml (1kb to ??kb)
^->http://www.wowarmory.com/guild-info.xml?brief=1&r=<realm>&n=<guild>&p=1 (15kb)
^->http://www.wowarmory.com/strings/en_...cter-sheet.xml (5kb)

* Now all the XML/XSLT fun is over, lets continue with the JavaScript and CSS!
^->http://www.wowarmory.com/shared/glob...y/detection.js (14kb)
^->http://www.wowarmory.com/css/master.css (170kb)
^->http://www.wowarmory.com/css/en_us/language.css (22kb)
^->http://www.wowarmory.com/shared/glob...nav/topnav.css (1kb)
^->http://www.wowarmory.com/shared/glob...buildtopnav.js (4kb)
^->http://www.wowarmory.com/js/armory.js (50kb)
^->http://www.wowarmory.com/js/dhtmlHistory.js (40kb)
^->http://www.wowarmory.com/js/en_us/strings.js (4kb)
^->http://www.wowarmory.com/js/character/pinChar.js (3kb)
^->http://www.wowarmory.com/js/character/functions.js (11kb)
^->http://www.wowarmory.com/js/en_us/character-sheet.js (15kb)
^->http://www.wowarmory.com/js/character/textObjects.js (36kb)
^->http://www.wowarmory.com/js/character/arenaTooltips.js (2kb)
^->http://www.wowarmory.com/css/mini-search-expand.css (1kb)
^->http://www.wowarmory.com/js/paging/right/functions.js (3kb)
^->http://www.wowarmory.com/js/mini-search-ajax.js (9kb)
^->http://www.wowarmory.com/js/character-info-ajax.js (4kb)
^->http://www.wowarmory.com/shared/glob...us/menutree.js (25kb)
^->http://www.wowarmory.com/shared/glob...menu132_com.js (30kb)
^->http://www.wowarmory.com/js/en_us/menus.js (1kb)
^->http://www.wowarmory.com/shared/glob...7.6/sarissa.js (28kb)
^->http://www.wowarmory.com/shared/glob...rissa_dhtml.js (7kb)
^->http://www.wowarmory.com/js/ajaxtooltip.js (4kb)

And finally the images and swf files. No, I'm not going to list them here, lets just say that any single browser page request grabs between 50-200 of them. So all in all it's probably around 100-200 requests per character request. You can multiply this by the amount of characters you browse. Or talents or arena teams. And you can multiply this again by the amount of regular visitors they get on a daily basis.

The interesting part of this is that they're doing a lot of this synchronously and not asynchronously. This means the rest of the page has to wait on a single link. Should the armory webserver decide not to serve this data, you get a timeout penalty of 50 seconds (look for fetchXmlData2() in the armory.js file). This means that your browser will do as it's told and wait up to 50 seconds for data. Multiply this by the amount of broken links it tries to fetch.

Conclusion: The crawlers aren't causing the problem.
They aren't helping, but they're a minor part of the equation.

Note: Some of the links aren't real links. It's because this forum software tries to fetch their headers and set the link title itself.
You'll have to manually paste 3-4 of the links, sorry.

 

• spawnstah

My aim with this was not really to make life easier for Blizzard and reduce the load on the Armory.
The problem as I see it, is that lets say 10 sites are running 10 crawlers each downloading the data. Since the armory and the millions of characters are quite a lot of data, it takes these individual sites like 2-3 maybe even 4-5 weeks to get all the data they need.
Wowjutsu needs to have data that is no older than a few days to be counted as accurate and up to date.
Other sites working with class balance, realm population and so on might be okay with data that is 1 month old.
But if those 10x10 crawlers could all collect the info to a central database that they all could use, the data could in theory always be max 2-3 days old.

I can also see how instead of having those 10x10 crawlers running by the different sites, we could have hundreds of players running 1 crawler each and help with the updates.

Kalroth I understand your point, and yes Blizzard do have a central database which is useful if you use it like you do. But sites that need to access every character and guildpage to gather information kind of need to download all of those small .xml files, which takes time. Having a central database that you can access and get instant answers from, do queries to or download the database localy for faster queries.

I'm not sure how "open" the database would be, but I would like to leave it as open as possible so anyone who would be inerested could query it. But I guess it depends on the bandwidth used. Maybe give access to the database for a small fee, 10-50 usd per month/year whatever just to have some kind of control and make sure it does not go out of hands.
Or like wowwebstats does, give priority queries to the ones who pays. I dont know and I'm not sure its important at this point. =)

 

Tehehe

Knowing that, wouldn't a site that simply queries the Armory for necessary data (e.g., character-sheet.xml) and then feeds the information back to the user in a proper fashion (for example, using CSS/AJAX instead of XSLT/SJAX) fix a large part of the issue with the Armory being slow/unresponsive and its tendency to crash browsers?


I think that crawlers are a very small part of the load when compared to normal users. In addition, the Armory (Blizzard operated servers) are probably going to be much better suited to handle the load (even without all the crawlers) than any of us.

 

Kewangeder

Nice post, Kalroth. Naive question: what stops a reasonably savvy browser from caching most of that XSL, JS, and CSS data after the first retrieval?

 

Cenuij

Looking at the HTTP headers returned when fetching mini-search-templates.xsl, it appears that they're setting a few headers relating to web caching:
It appears that they're doing the right thing and setting stuff like Cache-Control and Expires on that more "static" content.

I asked a friend who is a web caching guru; his response:
Given that, the browsers should be doing the right thing and caching the those particular files.


Another thought; those HTTP headers indicate that the server claims to be Apache.
Unless the armory is fronted by a transparent redirector, I'm not surprised that there are performance problems; in the past Apache has been shown to top out at about 4000 connections per second, and given each Armory request may result in up to 100-200 requests based on what Kalroth wrote, that may explain some of the performance issues we see.

 

Gj

I don't think the issue with the armory is related to the number of files required to view the page for a few reasons:

1) Armory scrapers don't download those files.
2) A browser will download the files once, and consequent requests will use the cache (as explained in the post above).
3) Simple file requests don't put much strain on a server.
4) XSLT transformations are actually performed by your browser. The only time the XSLT transform is done by blizzard is when you view the web page with a browser that doesn't support such transformations.

The bulk of any cause of slowdown would be when the blizzard site has to access it's database backend to retrieve that data, and (to a much lesser extent) the time it takes to transform that data into meaningful XML. With so many characters/items/guilds stored, it would be very difficult for Blizzard to cache any information, and as such each request would require a hit to the database to retrieve the necessary information. I would put money on the bottleneck existing here.

Bottlenecks could also appear from inefficient data queries or code. They tend to get magnified by a very large amount when you are servicing so many requests at once.

There are a number of architectural improvements that you can make to improve such speeds, but most of the time it involves having a number databases servicing requests to spread the load. It's also common to spend time optimising data queries and the data itself to speed up requests.

 

• Kalroth

What you're asking for is basically a mirror of WoW Armory, but stored in some sort of SQL database where you can do something like "select * from characters as c where c.realm='Stormrage'". This is a nice idea, but it's not going to happen because the armory was never intended as a statistics site, it's meant as an on-demand site. You follow links to characters, you load their data and gasp at their mad epics that they afk farmed in AV.

A site like Wowjutsu will never be 100% accurate until you get a full database dump from Blizzard. There is no other API they can offer to make it accurate, nor will any outside parties be able to offer it. In my opinion anyways, knowing my luck Blizzard will probably go and prove me wrong now. But I don't think it's possible in its current state.

I wish I could say that the caching works well, but on my machines it doesn't. If I open IE7 and load an armory page, it takes 5-20 seconds to load. If I visit the same link again, it takes 2-3 seconds which is basically the character xml loading.

Now if the browser cached everything, I should be able to close the browser, open it again and visit the exact same page where I only need to load the character xml. But when I open the browser and visit the same cached page, it takes just as long as the first time I loaded the page. I experience the same with Firefox, so it's not an IE7 thing. I haven't changed any cache settings on either browser either, they're using default settings.

Consider this, what's fastest; to transfer 1 100kb file or to transfer 100 1kb files? It's the former of course. So no, simple file requests don't put much strain on the server. Unless a lot of those requests fail and are re-requested. And this is exactly where XSLT transformation with synchronous JavaScript code is bad. An XSL file loads a included XSL file that loads javascript code that loads another XSL file that loads a file and fails. Now the file request in the bottom of the chain is broken, it's not responding and the javascript in the middle of the chain is using long timeouts. All of this might be happening while the browser itself tries to reload a image, because that also failed the first time.

WoW Armory is most likely built on an intranet and that's where it belongs. It's not suited for the internet in it's current state.

 

◊ Yes

For me, with Mozilla (not firefox) and IE7 that only happens if I kill the mozilla/IE process. I haven't tested it with restarts.

My other main | My PvP Blog

 

Tanoh

I'd be quite surprised if they're even hosted on the same machine as the rest. It's pretty trivial to put up a HTTP proxy that split requests based on path to different machines and load balance it.

Pure speculation (as almost everything else in this thread) of course, but it makes no sense to put static and dynamic content on the same server if you have these numbers of visitors.

 

• Galatea

That is not correct. HTTP defines different types of caching, and different policies. What you described sounds like the policy is set to no-store, which specifies caching is allowed in ram, so long as it does not hit any sort of persistent store. That should give pretty good performance after the initial origin loads occur, and for most people would have basically as good interactive experience as caching to disk. It certainly seems weird to be used in this case, but just because you are not caching things to disk does not mean there is not caching going on. For all the details checkout RFC2616 section 14.9 (well, most of the details, there are some extensions like etags). Note that this does not mean the site performance is good, but rather that caching between sessions would only speed up the initial load, not subsequent browsing.

 

• Kalroth

I'm well aware how HTTP caching works, thank you for the RFC link anyways. Theory is one thing, real world is another.

What I'm saying is that the browsers I've tested are inconsistent in how they caches pages. In Firefox you can type about:cache and see the contents of the disk cache, how long it's supposed to survive and how many times it's been used.
The armory images should get a 3-4 hour timeout on the disk cache, it's visible from the header data, but it's just not always happening. Now I don't particular care much to find out why, but considering it does it on several computers AND totally different browsers I'm guessing it's a feature from the armory server and not on my side.

 

Okoloth

I have been trawling the Armory periodically since May last year and I have evolved my data collection and storage techniques all along the way. As Kalroth pointed out, a lot of the delay in accessing the Armory is the huge amount of processing being put onto the browser. When you are just fetching XML sheets this is not an issue.

Believe it or not, the Armory as a server is pretty good at sending back the raw XML sheets. Each request made by my crawlers is timed and a typical character sheet request is on average less than 80ms. There are noticeable busy spells at certain times of the day where the response time is longer and occasional oddities that can stall a single request for 60+ seconds until it’s timed out – but that’s normal for most heavily used websites.

Back in the early days I could easily achieve hundreds of character sheets per second on a single crawler, but mostly I kept them throttled back to keep a low profile. Recently I noticed that the Armory servers have wizened up to the requests they are receiving, above a certain threshold of requests per second they stop playing ball and reply with HTTP response headers and zero-length content. Throttling your request rate back resolves this in a matter of seconds. To achieve higher request rates you just need to have requests coming from more / different IP addresses, i.e. through several proxies.

Like everyone else has said having a bulk-data API onto the Armory data would be ideal - whether it’s done by Blizzard or someone else.

 

Okoloth

Some of the issues that a private Armory database would face:

1). Character information

The information stored for each character used to be in a single XML sheet but since the Armory changes in August 2007 this is now split across 5 XML sheets (summary, skills, reputation, talents, and arena teams). This requires 5 separate HTTP requests per character to retrieve all of the information; naturally if your project doesn’t need the information from the other pages they aren’t retrieved. For example WowJutsu only needs to see the gear guild members are wearing and probably just retrieves a single sheet per character.

Storing 5 separate sheets per character is also an issue, for my own crawling operation I chose to amalgamate the separate XML sheets into one for each character.

2). Storage

XML as a storage format is very verbose, thankfully it compresses incredibly well - I used ZLIB data compression to store each character sheet. Even so, one million characters equates roughly to 10Gb of storage space.

3). Filesystem

Choice of filesystem is important if you are going to have millions of files, directory structure is also very important.

a. Some filesystems store directory contents as a ‘linked list’, when you have thousands of files in a single directory finding files can be slow. In my own research I experimented with the EXT3 and ReiserFS filesystems on my Linux servers and found ReiserFS to perform far faster.

b. As for directory structure I used “(Region US, EU, TW) / (Battlegroup Name) / (Realm Name) / (Character Name).xml.z” with ReiserFS this proved fast enough for random character access.

c. Filesystems also have limits on the maximum number of files they can store, for Linux filesystems these are the i-nodes used to point to files, they are fixed in number and the limit is set when the filesystem is created.

4). Database

Storing all of the information in a database is infeasible; ‘digesting’ the XML out into a database-friendly structure is also hugely complicated. Instead, using the database as an index to all the character information stored on the filesystem is far more efficient. Storing some meta data about the characters (level, class, race, faction) in the database allows for quick selection of a data set without the need to parse all stored characters.

 

Fringe_Worthy

On the other hand... What sort of resources do you need for your serving machine?

It may be that actually creating that 100kb page would be a nastier hit on the server?

After all, lets say my (imaginary) pvp alt Noobtastic wears an Armory concealment set on logout that is composed of all those items that get DE'ed from RFK, Mauradon, Gnomeran, and Black Temple.

The server might have to read each one of those, because they aren't cached, so this file/task that's slowly going from 0k ... 100k is sitting on server for 15s, waiting for all those data accesses to finish?

This almost sounds like your 50s timeout situation... but instead of doing it in our comp, it's done on the server. Which has 5000 other tasks building 100k records?

We may be better off having your computer sitting around 15s, waiting for the server to look up a 1k file holding the data it needs for that foo of blah item? Less task/memory lookdown and churn? That and things are easier to diagnose when you see the pant server is no longer serving pants, instead of just noticing the rate of serving characters has dropped 90%?

(I'm not denying that they can do better. I'm just wondering if there are places where one's intuition can go badly wrong when one finds out that we're dealing with larger sets of data)