Armory Scraping

Abbi · 10/23/08, 10:37 AM

Has anyone figured out how to scrape the XML for Armory calendars yet? Using Firefox as the user-agent and going through the login process works fine, but once I get to the actual calendar page (https://www.wowarmory.com/vault/char...en+Ring&n=Abbi, e.g.) I get a very barebones XML page with only my basic character information on it.

Examining the XSLT stylesheet leads me to think that http://www.wowarmory.com/calendar/js/calendar.js has the meat of the calendar stuff, but my JavaScript is not good enough to figure out how to get event data. Given event data, I'd be writing perl to generate .ICS files for iCal.

xergio · 10/23/08, 11:29 AM

The calendar use JSON instead XML to retrieve the information. You can use Firebug and inspect the Net tab.

codeguy · 03/13/09, 8:26 PM

I'm wondering if you're still working on a project of this sort? I have the same desire, to pull armory data and store locally. Over the past couple days I have been working on something to do this, and it's near complete. The only problem with the armory calendar data is that you need to login. This is not a problem to program, however it will deter many users as they will need to supply their login information to the script. I imagine the only users that would be comfortable in doing so would be those that can execute this locally and have complete control over that information, as to automate it the script needs a way to retrieve the login information without it being supplied each execution. This works very well for me as I host my own Solaris 10 box, I know the login information isn't going to be seen by anyone but me, and if it is- well then I have much larger problems than a compromised wow account.

To make a long post short, I wanted to share some useful information regarding access this data via the armory mechanically.

Rather than create some fancy perl code to connect to the armory to download the calendar, I choose to use wget. An initial wget call logs into the armory and stores a cookie on my server, the subsequent calls pass the stored cookie. The actual calendar call is as simple as:

/usr/sfw/bin/wget --keep-session-cookies --user-agent "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1" --load-cookies="/vdsk/projects/wow/armory/tmp/calcookie.tmp" --output-document "/dsk/projects/wow/armory/tmp/calholidayWeekly.tmp" "http://www.wowarmory.com/vault/calendar/month-world.json?type=holidayWeekly&month=3&callback=calendar&year=2009&loc=1&loginType=com " > /dev/null 2>&1

As an example of the calendar data returned from the armory, below is the Darkmoon schedule for March 2009.

calendar({"now":1236989274033,"month":3,"events":[{"summary":"Darkmoon Faire","calendarType":"darkmoon","start":1236596400033,"calendarTypeId":6,"end":12372 01200033,"icon":"Calendar_DarkmoonFaireElwynn","description":"The Darkmoon Faire is here, this time in idyllic Elwynn Forest.\r\n\r\nMeet Silas Darkmoon and his troupe, play games that test mind and nerve, and behold exotic sights from the four corners of Azeroth... and beyond!","priority":1}],"calendarType":"darkmoon","year":2009,"tz":-25200000});

Obviously the script would need to parse through this information to produce something meaningful, but again that's not a problem at all.

You can also craft messages to the armory site to get the detail list from the high level calendar entry, and even accept or decline the invitation. I, however, don't see a lot of use in accepting or declining mechanically, some user decision would need to be made, and personally I make the decision in game. My script will simply be used to download calendar activity and send email notifications, to me, when an event as been added, removed or is about to occur.

Stauf · 04/23/09, 12:14 AM

I've been able to authenticate to Battle.Net and scrape calendar data (turning it into iCalendar data for iCal / Google Cal) -- Armory Calendar to ICS

The issue is, at the moment to do it, I need to collect Battle.Net login info. The code is at http://aftermath.cx/wow/armory-tools/Calendar.zip if you want to try it yourself (requires PHP with JSON support, that cookies.tmp be writable, and the HTTP auth may not work on your server (hack to make it work with php-cgi))

♦ Slackie · 05/20/10, 8:07 AM

(thread necro)

Blizzard has released an API that allows access to the auction house. Here is some technical data to aid in writing things to make use of this functionality.

Get JSON results: http://www.wowarmory.com/auctionhouse/search.json
Get XML results: http://www.wowarmory.com/auctionhouse/search/index.xml (be sure to include rhtml=false or you will get HTML data back!)

note: There is a maximum of 200 results returned per query.

Search parameters you can use:

Key	Description	Values
filterId	maps to ah categories	(see below)
maxLvl	maximum level required	any valid character level
minLvl	minimum level required	any valid character level
qual	item quality	0=poor, 1=common, 2=uncommon, 3=rare, 4=epic
reverse	reverse the sort output	true, false
sort	column to sort by	rarity, quantity, level, time, buyout

Filter categories:

Id	Description
0	Weapon
1	Armor
2	Container
3	Consumable
4	Glyph
5	Trade Goods
6	Projectile
7	Quiver
8	Recipe
9	Gem
10	Miscellaneous

note: You can filter into sub-categories and sub-sub-categories by separating the id's with a comma. filterId=0,78 would show one-handed axes. I haven't figured out how to gather up what all the sub-category id's are short of searching by hand, if you have any ideas, send me a PM.

Data that is returned:

Key	Description
auc	Auction ID
bid	Bid price
buy	Buyout price
charges	Charges
date	Date posted
icon	Icon
id	Item ID
ilvl	Item Level
n	Item name
nbid	Next minimum bid
ppuBid	Price per unit (bid)
ppuBuy	Price per unit (buyout)
qual	Item quality
quan	Quantity
req	Required level
seller	Seller
time	Time left (1=short,2=medium,3=long,4=very long)

• malthrin · 05/21/10, 12:52 PM

All done! If anyone cares to fill in the missing categories, that'd be nice.

FilterId result

0 Weapon

1 Armor

2 Container

3 Consumable

4 Glyph

5 Trade Goods

6 Projectile

7 Quiver (all)

8 Recipe

9 Gem

10 Miscellaneous

11 Quest

12

13 Two-Handed Maces

14 Cooking Recipe

15 Herb Bag

16 Dagger

17 Orange Gem

18 One-Handed Maces

19 Bullets

20 Enchanting Bag

21 Arrows

22 Thrown Weapon

23 One-Handed Swords

24

25 Purple Gem

26 Soul Shard Bag

27 Meta Gem

28 Warlock Glyph

29 Leatherworking Bag

30 Mage Glyph

31 Reagent

32 Trade Goods (Elemental)

33 Fishing Poles

34 Armor (Misc)

35 Fist Weapons

36

37 Inscription Bag

38 Holiday

39 Junk

40

41 Bow

42 Wand

43 Armor (Cloth)

44 Leatherworking Recipe

45 Alchemy Recipe

46 Food & Drink

47 Two-Handed Swords

48

49 Crossbows

50 Blue Gem

51 Jewelcrafting Recipe

52 Yellow Gem

53 Totem

54 Enchanting Recipe

55

56 Druid Glyph

57 Recipe (Book)

58 Container (Bag)

59 First Aid Recipe

60 Idol

61 Staff

62 Misc (Other)

63 Polearm

64 Gun

65 Rogue Glyph

66 Warrior Glyph

67 Shield

68 Ammo Pouch

69 Mount

70 Blacksmithing Recipe

71 Death Knight Glyph

72

73 Red Gem

74 Libram

75 Tailoring Recipe

76 Quiver

77 Armor (Leather)

78 One-Handed Axes

79

80 Paladin Glyph

81 Quest

82 Two-Handed Axes

83 Armor (Mail)

84 Priest Glyph

85

86

87 Armor (Plate)

88 Hunter Glyph

89 Prismatic Gem

90 Trade Goods (Enchanting)

91 Pet

92 Misc. Weapons

93 Shaman Glyph

94 Engineering Recipe

95 Mining Bag

96 Green Gem

97 Jewelcrafting Bag

98 Simple Gem

99 Cloth

100 Potion

101 Leather

102 Elixir

103 Metal & Stone

104 Flask

105 Meat

106 Bandage

107 Herb

108 Item Enhancement

109 Scroll

110 Consumable (Other)

111 Jewelcrafting

112 Inscription

113 Devices

114 Explosives

115 Trade Goods (Materials)

116 Trade Goods (Other)

117 Armor Enchantment

118 Weapon Enchantment

119 Armor (Misc Head)

120 Armor (Neck)

121 Armor (Shirt)

122 Armor (Ring)

123 Armor (Trinket)

124 Armor (Offhand)

125 Armor (Cloth Head)

126 Armor (Cloth Shoulder)

127 Armor (Cloth Chest)

128 Armor (Cloth Waist)

129 Armor (Cloth Legs)

130 Armor (Cloth Feet)

131 Armor (Cloth Wrists)

132 Armor (Cloth Hands)

133 Armor (Cloak)

134 Armor (Leather Head)

135 Armor (Leather Shoulder)

136 Armor (Leather Chest)

137 Armor (Leather Waist)

138 Armor (Leather Legs)

139 Armor (Leather Feet)

140 Armor (Leather Wrists)

141 Armor (Leather Hands)

142 Armor (Mail Head)

143 Armor (Mail Shoulder)

144 Armor (Mail Chest)

145 Armor (Mail Waist)

146 Armor (Mail Legs)

147 Armor (Mail Feet)

148 Armor (Mail Wrists)

149 Armor (Mail Hands)

150 Armor (Plate Head)

151 Armor (Plate Shoulder)

152 Armor (Plate Chest)

153 Armor (Plate Waist)

154 Armor (Plate Legs)

155 Armor (Plate Feet)

156 Armor (Plate Wrists)

157 Armor (Plate Hands)

Antiarc · 05/21/10, 1:37 PM

I've discovered that by using a Keep-Alive header on my connections, I can pull data at approximately one request per 0.55 sec, rather than waiting that 0.55 sec + 1 sec per HTTP connection. If you can reuse connections in conjunction with keepalives, it'll speed you up a lot.

thefool808 · 05/21/10, 4:00 PM

Here's a quick and dirty ruby implementation to get people started. You can get the cookie string straight from your cookies, or use firebug to look at the headers of a normal request after logging into the Armory.

#971617 - Pastie

Antiarc · 05/21/10, 4:37 PM

I've got a Ruby project up and running already, actually. It uses Mechanize with keepalives to optimize throughput, and stores data in MongoDB for fast analysis and reporting.

http://github.com/cheald/AuctionBoss/

• malthrin · 05/21/10, 9:18 PM

Another search parameter, f, specifies which auction house to query.

Value	Description
0	Alliance AH
1	Horde AH
2	Neutral AH

There are other valid values - probably corresponding to the various in-game auction house NPCs? But this is all you need.

edit:
Tested these sorting parameters from the javascript: 'sort' and 'reverse'. Reverse is a bool for ascending/descending. Sort can have the following values:

Value	Description
rarity	epics first
name	alphabetical
level	required level, highest first
ilvl	item level, highest first
bid	stack bid, lowest first
buyout	stack buyout, lowest first
unitbid	price per unit, lowest first
unitbuyout	price per unit, lowest first
time	time remaining, shortest first

• malthrin · 05/22/10, 12:43 PM

Bids and buyouts are done via bid.json. The parameters are:

Parameter	Description
auc	AuctionId
money	Amount to bid
f	AH to buy from

If your bid amount is less than the buyout, you place a bid on the item. If it is equal to the buyout, you buy out the item. If it's more than the buyout... someone else can test what happens then!

Here's the relevant javascript:
http://www.wowarmory.com/_js/auction/auction.js

• malthrin · 05/22/10, 1:32 PM

Some other functions:
http://www.wowarmory.com/auctionhouse/inventory.json
http://www.wowarmory.com/auctionhouse/mail.json
http://www.wowarmory.com/auctionhouse/money.json
http://www.wowarmory.com/auctionhouse/faction.json This one can get the value for f. Guess f stands for faction!

luvmachine · 05/23/10, 9:54 PM

Using bid.json and bidding for more than the buyout results in a buyout of the item for the listed price, no extra money is spent.

mail.json is used to list the mail in your box while takeMail.json is used to move it from your mailbox to your inventory.

Any help on getting the search results in XML, I can only seem to find them in JSON.

EDIT: Also, changing 'pageSize' in the search parameters will give you more results (up to 50), and changing the 'start' parameter can be used to get further down the list after what the initial results show.

♦ Slackie · 05/24/10, 12:16 AM

Originally Posted by luvmachine

Any help on getting the search results in XML, I can only seem to find them in JSON.

Check a few posts up for another one by me. It has the XML URL at the top.

How to post a new auction:

Step one: You need a "ticket", you get this by sending a GET request to http://www.wowarmory.com/auctionhouse/deposit.json

Options:

Key	Description
id	Item ID
duration	Length of auction (0=12h, 1=24h, 2=48h)
quan	Number of items
stacks	Number of stacks

You will get a response that contains a few variables:

Key	Description
ticket	Ticket
deposit	Cost to create this auction (per item)
totalDeposit	Cost to create this auction (overall)
suggestedPrice	Unknown

Step two: You use the ticket you have from step one and send a GET request to http://www.wowarmory.com/auctionhouse/create.json

Options:

Key	Description
bid	Bid price for this item
buyout	Buyout price for this item
id	Item ID
duration	Length of auction (0=12h, 1=24h, 2=48h)
quan	Number of items
stacks	Number of stacks
ticket	Ticket you got from deposit.json
guid	This will be blank for a single-stack auction

note: If you are creating a multi-stack auction, create.json will return data in the guid field. Use the data from the guid field to populate the ticket field of your next request.

♦ Slackie · 06/02/10, 12:34 AM

Anyone seeing a large amount of timeouts via the new AH API over the last 36 hours or so? I tried increasing my polling delay but still get errors which leads me to believe this is an actual problem not just polling too quickly.

• malthrin · 06/02/10, 12:32 PM

It's not the same message as the query throttle. You also run into the same error while browsing the web interface, so I'd wager it's a problem on their end.

edit: looks like it's mostly fixed this afternoon

Stack · 06/05/10, 10:30 PM

I was able to get the main summary page http://www.wowarmory.com/auctionhouse/index.xml#summary into my program as XML but has anyone found the JSON equivalent link, it would be a lot simpler if I could route all the responses from the armory thru JSON.

EDIT: Found it http://www.wowarmory.com/auctionhouse/status.json

 {"auctionStatus":{"factionId":0,"mailInfo":{"current":2,"max":50,"total":2,"fullPercent":4},"activeHighBids":0,"activeOutBids":0,"auctionsActive":0,"auctionsCancelled":2,"auctionsExpired":0,"auctionsSold":0,"bidsCancelled":0,"bidsWon":0,"goldEarned":0},"command":{"f":0,"cn":"charactor","r":"realm"},"pra":{"status":"active"}}

Another http://www.wowarmory.com/auctionhouse/cancel.json

{"success":true,"command":{"auc":901314668,"f":0,"cn":"charactor","r":"realm"},"pra":{"status":"active"}}

Another important one http://www.wowarmory.com/vault/chara...ct-submit.json

Takes a CN=charactor&r=realm as post data which returns.

{"success":true}

♦ Slackie · 06/06/10, 2:38 AM

Thanks for finding status.json, I hadn't seen that before. Does anyone know of a way to find current transactions remaining without actually posting a new auction or bidding on something? That would be useful.

Here are all the various URLs I know about, just so everyone is on the same page (I didn't add status.json from above yet)

root: "http://www.wowarmory.com/auctionhouse/"
status: "http://www.wowarmory.com/login-status.xml"

# Miscellaneous
money: "http://www.wowarmory.com/auctionhouse/money.json?%s"
alerts: "http://www.wowarmory.com/vault/calendar/alerts-user.json?%s"
inventory: "http://www.wowarmory.com/auctionhouse/inventory.json?%s"

# Calendar
calendar: "http://www.wowarmory.com/vault/calendar/month-user.json?%s"
calendar_world: "http://www.wowarmory.com/vault/calendar/month-world.json?%s"
calendar_details: "http://www.wowarmory.com/vault/calendar/detail.json?%s"

# Mailbox
mailbox: "http://www.wowarmory.com/auctionhouse/mail.json?%s"
mailbox_move: "http://www.wowarmory.com/auctionhouse/takeMail.json?%s"

# Auctionhouse related
auction_post: "http://www.wowarmory.com/auctionhouse/create.json?%s"
auction_browse: "http://www.wowarmory.com/auctionhouse/search.json?pageSize=50&rhtml=false&%s"
auction_active: "http://www.wowarmory.com/auctionhouse/auctions/index.xml?rhtml=false&%s"
auction_cancel: "http://www.wowarmory.com/auctionhouse/cancel.json?%s"
auction_deposit: "http://www.wowarmory.com/auctionhouse/deposit.json?%s"
auction_buy_get: "http://www.wowarmory.com/auctionhouse/bid.json?%s"
auction_buy_post: "http://www.wowarmory.com/auctionhouse/bid.json"

Stack · 06/06/10, 8:59 PM

I've been working with the takeMail.json all day. Based on what I've watched while using the wowamory.com site, ive concluded that you can only claim money remotely and can only see items in your mail box that you could possibly relist.

what is odd is that wowarmory doesn't supply a auc id when it does its call -9223372036854775808 is the value I get when I omit it in my post data, including it doesn't get you mail that is not just money however.

sample rejection

{
  "error": {
    "code": 112,
    "error": true,
    "message": "Mail not found"
  },
  "command": {
    "auc": -9223372036854775808,
    "mailIds": "2119124985",
    "f": 0,
    "cn": "Nullberri",
    "r": "Bonechewer"
  }
}

sample success

{
	"mailIds":[2120517468],
	"money":950000,
	"command":
		{
			"auc":-9223372036854775808,
			"mailIds":"2120517468",
			"f":0,
			"cn":"Nullberri",
			"r":"Bonechewer"
		},
	"pra":{"status":"active"}
}

Psy7th · 06/14/10, 8:27 AM

For what it's worth: -9223372036854775808 is really -(2^63) or signed long 0x8000 0000 0000 0000.

• malthrin · 07/25/10, 9:02 AM

Has anyone been getting logged out periodically since this week's maintenance?

♦ Slackie · 07/25/10, 9:59 AM

Yes. I had to modify my library to raise an error when I get the "You must log in." message so that scripts know they need to authenticate again.

• malthrin · 07/26/10, 2:42 PM

Yeah, I did the same. Kind of a pain for anyone trying to do this on an authenticator account, though.

dozens · 10/15/10, 11:10 AM

looks like I am little late to the party

The auction browsing is based off your primary toon, does anyone know the url to select a different primary toon?

Rinji · 11/22/10, 4:37 PM

To Login (w/o Authenticator):

Post to URL: https://us.battle.net/login/en/login.xml
Arguments:

accountName: <account name>
password: <password>

Afterwards, I do a get request to http://www.wowarmory.com/auctionhouse/index.xml, which should build the correct cookies. To validate a successful login, check the cookies for (perl regular expression):

/auction_sk=([\w-]*?);/

Note: I do a match in the cookies for the 'auction_sk' code, as it appears to be passing this param 'sk' around during POSTs, but currently it does not seem to be needed for a successful response.

To List all Characters:

Get URL request: http://www.wowarmory.com/vault/character-select.xml

XML response should be easily queried by xpath (ex.):

/page/characters/character/@name

Note: The first name listed (I believe) is the default character.

To Switch the Default Character:

Originally Posted by dozens

does anyone know the url to select a different primary toon?

Post to URL: http://www.wowarmory.com/vault/character-select-submit.json
Arguments:

cn: <character name>
r: <server>

Successful response:

{"success":true}