|
Armory Scraping
Has anyone figured out how to scrape the XML for Armory calendars yet? Using Firefox as the user-agent and going through the login process works fine, but once I get to the actual calendar page (https://www.wowarmory.com/vault/char...en+Ring&n=Abbi, e.g.) I get a very barebones XML page with only my basic character information on it.
Examining the XSLT stylesheet leads me to think that http://www.wowarmory.com/calendar/js/calendar.js has the meat of the calendar stuff, but my JavaScript is not good enough to figure out how to get event data. Given event data, I'd be writing perl to generate .ICS files for iCal. |
The calendar use JSON instead XML to retrieve the information. You can use Firebug and inspect the Net tab.
|
I'm wondering if you're still working on a project of this sort? I have the same desire, to pull armory data and store locally. Over the past couple days I have been working on something to do this, and it's near complete. The only problem with the armory calendar data is that you need to login. This is not a problem to program, however it will deter many users as they will need to supply their login information to the script. I imagine the only users that would be comfortable in doing so would be those that can execute this locally and have complete control over that information, as to automate it the script needs a way to retrieve the login information without it being supplied each execution. This works very well for me as I host my own Solaris 10 box, I know the login information isn't going to be seen by anyone but me, and if it is- well then I have much larger problems than a compromised wow account.
To make a long post short, I wanted to share some useful information regarding access this data via the armory mechanically. Rather than create some fancy perl code to connect to the armory to download the calendar, I choose to use wget. An initial wget call logs into the armory and stores a cookie on my server, the subsequent calls pass the stored cookie. The actual calendar call is as simple as: /usr/sfw/bin/wget --keep-session-cookies --user-agent "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1" --load-cookies="/vdsk/projects/wow/armory/tmp/calcookie.tmp" --output-document "/dsk/projects/wow/armory/tmp/calholidayWeekly.tmp" "http://www.wowarmory.com/vault/calendar/month-world.json?type=holidayWeekly&month=3&callback=calendar&year=2009&loc=1&loginType=com " > /dev/null 2>&1 As an example of the calendar data returned from the armory, below is the Darkmoon schedule for March 2009. calendar({"now":1236989274033,"month":3,"events":[{"summary":"Darkmoon Faire","calendarType":"darkmoon","start":1236596400033,"calendarTypeId":6,"end":12372 01200033,"icon":"Calendar_DarkmoonFaireElwynn","description":"The Darkmoon Faire is here, this time in idyllic Elwynn Forest.\r\n\r\nMeet Silas Darkmoon and his troupe, play games that test mind and nerve, and behold exotic sights from the four corners of Azeroth... and beyond!","priority":1}],"calendarType":"darkmoon","year":2009,"tz":-25200000}); Obviously the script would need to parse through this information to produce something meaningful, but again that's not a problem at all. You can also craft messages to the armory site to get the detail list from the high level calendar entry, and even accept or decline the invitation. I, however, don't see a lot of use in accepting or declining mechanically, some user decision would need to be made, and personally I make the decision in game. My script will simply be used to download calendar activity and send email notifications, to me, when an event as been added, removed or is about to occur. |
I've been able to authenticate to Battle.Net and scrape calendar data (turning it into iCalendar data for iCal / Google Cal) -- Armory Calendar to ICS
The issue is, at the moment to do it, I need to collect Battle.Net login info. The code is at http://aftermath.cx/wow/armory-tools/Calendar.zip if you want to try it yourself (requires PHP with JSON support, that cookies.tmp be writable, and the HTTP auth may not work on your server (hack to make it work with php-cgi)) |
(thread necro)
Blizzard has released an API that allows access to the auction house. Here is some technical data to aid in writing things to make use of this functionality. Get JSON results: http://www.wowarmory.com/auctionhouse/search.json Get XML results: http://www.wowarmory.com/auctionhouse/search/index.xml (be sure to include rhtml=false or you will get HTML data back!) note: There is a maximum of 200 results returned per query. Search parameters you can use:
Filter categories:
note: You can filter into sub-categories and sub-sub-categories by separating the id's with a comma. filterId=0,78 would show one-handed axes. I haven't figured out how to gather up what all the sub-category id's are short of searching by hand, if you have any ideas, send me a PM. Data that is returned:
|
All done! If anyone cares to fill in the missing categories, that'd be nice.
|
I've discovered that by using a Keep-Alive header on my connections, I can pull data at approximately one request per 0.55 sec, rather than waiting that 0.55 sec + 1 sec per HTTP connection. If you can reuse connections in conjunction with keepalives, it'll speed you up a lot.
|
Here's a quick and dirty ruby implementation to get people started. You can get the cookie string straight from your cookies, or use firebug to look at the headers of a normal request after logging into the Armory.
#971617 - Pastie |
I've got a Ruby project up and running already, actually. It uses Mechanize with keepalives to optimize throughput, and stores data in MongoDB for fast analysis and reporting.
http://github.com/cheald/AuctionBoss/ |
Another search parameter, f, specifies which auction house to query.
There are other valid values - probably corresponding to the various in-game auction house NPCs? But this is all you need. edit: Tested these sorting parameters from the javascript: 'sort' and 'reverse'. Reverse is a bool for ascending/descending. Sort can have the following values:
|
Bids and buyouts are done via bid.json. The parameters are:
Here's the relevant javascript: http://www.wowarmory.com/_js/auction/auction.js |
Some other functions:
http://www.wowarmory.com/auctionhouse/inventory.json http://www.wowarmory.com/auctionhouse/mail.json http://www.wowarmory.com/auctionhouse/money.json http://www.wowarmory.com/auctionhouse/faction.json This one can get the value for f. Guess f stands for faction! |
Using bid.json and bidding for more than the buyout results in a buyout of the item for the listed price, no extra money is spent.
mail.json is used to list the mail in your box while takeMail.json is used to move it from your mailbox to your inventory. Any help on getting the search results in XML, I can only seem to find them in JSON. EDIT: Also, changing 'pageSize' in the search parameters will give you more results (up to 50), and changing the 'start' parameter can be used to get further down the list after what the initial results show. |
Quote:
How to post a new auction: Step one: You need a "ticket", you get this by sending a GET request to http://www.wowarmory.com/auctionhouse/deposit.json Options:
You will get a response that contains a few variables:
Step two: You use the ticket you have from step one and send a GET request to http://www.wowarmory.com/auctionhouse/create.json Options:
note: If you are creating a multi-stack auction, create.json will return data in the guid field. Use the data from the guid field to populate the ticket field of your next request. |
Anyone seeing a large amount of timeouts via the new AH API over the last 36 hours or so? I tried increasing my polling delay but still get errors which leads me to believe this is an actual problem not just polling too quickly.
|
| All times are GMT -4. The time now is 10:48 PM. |
Forum Infrastructure by vBulletin 3.6.12 ©2000-2007, Jelsoft Enterprises Ltd.