I've coded a small site that accesses most of the armory information, albeit I only use some of it. Now I don't gather it for a local database, it's merely downloaded on demand. But this means I got a fairly good idea of what goes behind armory, why it's slow and why it freezes various browsers.
Short version
The problem with the armory is it's usage of XML/XSLT and SJAX. It's simply brutal on their servers AND our browsers. It's not going to get better until they take a good look at the armory code and decides to rewrite most of it.
Long version
First off, the idea of a central database is good, but Blizzard is already way ahead of you. The armory IS already a central database. To get information about a character, guild, item, arena team or talent build, you simply request a small XML file, at worst it's like 15-20kb of XML data, which is easily zip packed down to 4-6kb when using modern browsers or modern 'leeching' software.
Blizzard recently announced breaking the 10 million player mark. If you assume that only 1% of those access the armory on a daily basis, it's still 100,000 requests.
And because the armory wants to Web 2.0, it uses SJAX. But wait? Isn't it supposed to be AJAX? SJAX doesn't make any sense! Nope, it sure doesn't, which is why it boggles me that Blizzard relies upon it. AJAX means 'Asynchronous JavaScript and XML' while SJAX means 'Synchronous JavaScript and XML'. AJAX is good, it runs in the background, so it doesn't lock up your browser. SJAX is bad, it locks your browser until everything is loaded.
When a user requests an armory page of a character, he essentially starts a whole chain of events.
* First the user downloads a simple XML document.
^-> http ://www.wowarmory.com/character-sheet.xml (15-10kb).
* Then the browser loads the XSLT data for that XML document.
^->
http://www.wowarmory.com/layout/character-sheet.xsl (10kb).
* Then the browser parses this XSLT and finds another 4 include files.
^->http ://www.wowarmory.com/layout/includes.xsl (7.5kb).
^->
http://www.wowarmory.com/layout/character-header.xsl (9kb)
^->
http://www.wowarmory.com/layout/character-utils.xsl (6kb)
^->
http://www.wowarmory.com/layout/mini...-templates.xsl (50kb)
* Then the includes from these files are loaded too.
^->
http://www.wowarmory.com/layout/language.xsl (1kb)
^->
http://www.wowarmory.com/layout/template-common.xsl (6kb)
^->
http://www.wowarmory.com/layout/temp...lated-info.xsl (9kb)
* Then the includes from these files are
also loaded.
^->
http://www.wowarmory.com/strings/en_us/strings.xml (22kb)
^->http ://www.wowarmory.com/lastsearch.xml (1kb to ??kb)
^->
http://www.wowarmory.com/login-status.xml (1kb to ??kb)
^->http://www.wowarmory.com/guild-info.xml?brief=1&r=<realm>&n=<guild>&p=1 (15kb)
^->
http://www.wowarmory.com/strings/en_...cter-sheet.xml (5kb)
* Now all the XML/XSLT fun is over, lets continue with the JavaScript and CSS!
^->
http://www.wowarmory.com/shared/glob...y/detection.js (14kb)
^->
http://www.wowarmory.com/css/master.css (170kb)
^->
http://www.wowarmory.com/css/en_us/language.css (22kb)
^->
http://www.wowarmory.com/shared/glob...nav/topnav.css (1kb)
^->
http://www.wowarmory.com/shared/glob...buildtopnav.js (4kb)
^->
http://www.wowarmory.com/js/armory.js (50kb)
^->
http://www.wowarmory.com/js/dhtmlHistory.js (40kb)
^->
http://www.wowarmory.com/js/en_us/strings.js (4kb)
^->
http://www.wowarmory.com/js/character/pinChar.js (3kb)
^->
http://www.wowarmory.com/js/character/functions.js (11kb)
^->
http://www.wowarmory.com/js/en_us/character-sheet.js (15kb)
^->
http://www.wowarmory.com/js/character/textObjects.js (36kb)
^->
http://www.wowarmory.com/js/character/arenaTooltips.js (2kb)
^->
http://www.wowarmory.com/css/mini-search-expand.css (1kb)
^->
http://www.wowarmory.com/js/paging/right/functions.js (3kb)
^->
http://www.wowarmory.com/js/mini-search-ajax.js (9kb)
^->
http://www.wowarmory.com/js/character-info-ajax.js (4kb)
^->
http://www.wowarmory.com/shared/glob...us/menutree.js (25kb)
^->
http://www.wowarmory.com/shared/glob...menu132_com.js (30kb)
^->
http://www.wowarmory.com/js/en_us/menus.js (1kb)
^->
http://www.wowarmory.com/shared/glob...7.6/sarissa.js (28kb)
^->
http://www.wowarmory.com/shared/glob...rissa_dhtml.js (7kb)
^->
http://www.wowarmory.com/js/ajaxtooltip.js (4kb)
And finally the images and swf files. No, I'm not going to list them here, lets just say that any single browser page request grabs between 50-200 of them. So all in all it's probably around 100-200 requests per character request. You can multiply this by the amount of characters you browse. Or talents or arena teams. And you can multiply this again by the amount of regular visitors they get on a daily basis.
The interesting part of this is that they're doing a lot of this synchronously and not asynchronously. This means the rest of the page has to wait on a single link. Should the armory webserver decide not to serve this data, you get a timeout penalty of 50 seconds (look for fetchXmlData2() in the armory.js file). This means that your browser will do as it's told and wait up to 50 seconds for data. Multiply this by the amount of broken links it tries to fetch.
Conclusion: The crawlers aren't causing the problem.
They aren't helping, but they're a minor part of the equation.
Note: Some of the links aren't real links. It's because this forum software tries to fetch their headers and set the link title itself.
You'll have to manually paste 3-4 of the links, sorry.