Armory Scraping - Page 3

♦ Slackie · 12/06/10, 8:21 AM

I've put a lot of the changed stuff up online for my personal use at: https://www.ef.net/remote_auctionhouse.

Just some of the changes with the new Armory and for ease of having everything in one place rather than spread around in multiple places. I'll try to keep it updated with new stuff we discover as time permits.

immense · 12/29/10, 2:08 PM

Has anyone figured out how to change the default character in the "new" armory?

It looks like the browser sends a post request with the following parameters but I cannot get my app to work with them, I just get a (404) Page not found error:

Post url: http://us.battle.net/wow/en/pref/character
Variables:
index=3 (Character index)
xstoken=478e4589-5337-49db-b5ff-039743710268 (Set in cookie)

Thanks in advance for the help!

♦ Slackie · 12/30/10, 5:25 PM

Of the people I collaborate with on the Armory stuff, all of us are still using the "old" mechanisms to interact with the Remote Auction House. Shortly after Blizzard rolled out the new Armory they disabled the "old" URL's, but the new Armory didn't support any easy way to gather data (it returned actual HTML, not XML/JSON).

Due to popular demand from people who actually parsed Armory data, they re-enabled the old URL's while they work on finalizing an API for accessing the new Armory.

This was a thread on the subject: WTB XML Feeds! Offering lunch at Javier's! - Forums - World of Warcraft

When interacting with World of Warcraft, you can specify the character with the 'cn' key, like this:
GET /auctionhouse/search.json?pageSize=50&rhtml=false&cn=Charname&f=1&r=Mal%27Ganis

It also works for creating an auction, bidding, etc. All of the actions will accept 'cn' (character name), 'f' (faction) and 'r' (realm).

In short, don't use the new Armory URL's (yet) unless you just like parsing HTML. If you want XML/JSON just use the old URL's until they publish the new API.

Oldertarl · 01/04/11, 9:54 AM

Got some nice browser error yesterday showing json data rather than the new armory HTML page.
Perhaps ( keeping hope up ) they are in the test phase now and will release new data feeds soon.

♦ Slackie · 01/14/11, 11:42 PM

So I have a question that isn't directly related to Armory scraping, but is related to scraping... specifically, I am looking for a programmatic way to get the current patch level of Warcraft. Anyone have ideas? I poked around a bit with Wireshark looking at how the launcher works (pasted below), but I don't see anything that looks like a patch version in the plaintext portion of what it does.

The important part for me is a programmatic, reliable way to get the data. I can't look inside WTF files. I need to be able to get it off the wire, preferably from Blizzard directly. If there are other options, I'm open to those as well.

Here is the Wireshark of the launcher starting up if anyone is curious:

IP: 172.223.232.130
PORT: 80

GET /wow-pod-retail/NA/config_recommended_na_2.xml HTTP/1.1
Host: ak.worldofwarcraft.com.edgesuite.net
User-Agent: Launcher/4.0.0 CFNetwork/454.11.5 Darwin/10.6.0 (i386) (MacBookPro6%2C2)
Accept: */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: keep-alive

HTTP/1.1 200 OK
Server: Apache
ETag: "3173b46d78ca246332f8be5915acbbe3:1291056533"
Last-Modified: Mon, 29 Nov 2010 18:48:53 GMT
Accept-Ranges: bytes
Content-Length: 525
Content-Type: application/xml
Date: Sat, 15 Jan 2011 04:21:14 GMT
Connection: keep-alive

<config>
  <versioninfo type="pod">
    <version product="WoW">
      <servers>
        <server id="akamai" url="http://ak.worldofwarcraft.com.edgesuite.net/wow-pod-retail/NA/12911.streaming.2/"/>
      </servers>
      <thresholds>
        <threshold speed="1000000" red="5" yellow="5" />
        <threshold speed="1000001" red="1" yellow="5" />
      </thresholds>
     <settings>
        <setting name="patchapplicationstage" value="Recommended"/>
     </settings>
    </version>
 </versioninfo>
</config>

♦ Slackie · 01/17/11, 7:59 PM

Anyone have experience interacting with the remote AH (specifically search.json) using a character with UTF-8 symbols in the name?

I've tried to escape them but I just keep getting the This account does not have any characters who are eligible to use the Auction House error message.

If I change cn to another character on the same account with no UTF-8 symbols it works fine.

Blup · 01/29/11, 5:55 PM

Has anyone found a way to get the guild tabard form the guild summary page? It seems to be built using javascript and I can't seem to find a way to actually get it from the html I'm trying to parse.

immense · 02/07/11, 11:22 AM

The undermine journal came back up today, does that mean that XML data feeds are back up too? I have always parsed HTML but I am interested in using XML for obvious reasons!

Jakobud · 02/09/11, 6:37 PM

I know this is an old thread, but I had a question about the search.json URL. When I perform a basic search like this:

http://www.wowarmory.com/auctionhouse/search.json?pageSize=50&rhtml=false&%s

It returns results like this:

{"auctionSearch":{"start":0,"end":200,"total":200,"auctions":[{"auc":11788343... etc....

See the "total" of 200? That means it only returns the first 200 results. How do I change this to return more results? How are websites like the Undermine Journal able to pull EVERY auction and not just the first 200?

Arrielle · 02/10/11, 1:01 AM

Like Slakies said before, there are a limit of 200 results per query

♦ Slackie · 02/10/11, 2:10 AM

The pageSize key in the URL defines the maximum number of auctions you want returned for a single query. The maximum you can set this value to is 200. Well, you can set it higher, but the maximum number of auctions you will get back from a single query is 200 even if you set the number higher.

That being said, if you want more than 200 results, you can actually change the start and end values to do your own pagination. If you queried for start=0&end=200 to get the first 200 results for "Cloth" and then used start=200&end=400 for the same query, you should get 200 more auctions that are different from the ones in your first result set.

♦ Slackie · 02/10/11, 7:43 PM

Originally Posted by Blup

Has anyone found a way to get the guild tabard form the guild summary page? It seems to be built using javascript and I can't seem to find a way to actually get it from the html I'm trying to parse.

It looks like Blizzard is using <canvas> tags with JavaScript to generate the tabards. If you take a look at the data from the guild page as well as [1] that will probably get you started. You would just parse the data from the JavaScript on the guild page and use [1] to figure out how to craft a URL that will point to your particular guild tabard.

On your guild page Thuggernaut @ Mal'Ganis - Game - World of Warcraft look for the following code:

<canvas id="guild-tabard" width="240" height="240" style="display: inline;">
	<div class="guild-tabard-default tabard-horde"></div>
</canvas>

<script type="text/javascript">
	//<![CDATA[
			$(document).ready(function() {
				var tabard = new GuildTabard('guild-tabard', {
					'ring': 'horde',
					'bg': [ 0, 41 ],
					'border': [ 5, 14 ],
					'emblem': [ 92, 14 ]
				});
			});
	//]]>
	</script>

This results in the JS function retrieving the following URLs:

http://us.battle.net/wow/static/imag...ring-horde.png
http://us.battle.net/wow/static/imag...ards/bg_00.png
http://us.battle.net/wow/static/imag.../border_05.png
http://us.battle.net/wow/static/imag.../emblem_92.png

You can find the little "hooks" that are on the top here: http://us.battle.net/wow/static/imag...ards/hooks.png

There are also some color maps for the background color and shadows but I didn't dig much into how that all works. Looking at [1] will help in that respect.

[1] http://us.battle.net/wow/static/js/c...d-tabard.js?v7

Originally Posted by immense

The undermine journal came back up today, does that mean that XML data feeds are back up too? I have always parsed HTML but I am interested in using XML for obvious reasons!

The pre-Cataclysm XML/JSON feeds never went away (well, maybe for a couple days right after the new Armory launched). They are still in the same location. Blizzard mentioned in a forum post after the new Armory launched that they were working on a new API for the Armory. Until they are finished with the new API most folks are just using the XML/JSON data from pre-migration or just parsing HTML.

Ramides · 02/23/11, 11:28 AM

Hello,

I'm not sure if you can help me, but I try. I want to write an application (C#) to search in the auction house.

With "FireBug" I looked into the Battle.net login process and tried to reproduce it with C#. Unfortunately, this has not worked.

Perhaps it would help if I first show you the source code: the Battle.Net login (not perfect, but enough for a start).

public string Login(string AccountName, string Password)
{
    string URL = "https://eu.battle.net/login/de/login.frag?secureOrigin=true";
    string Post = "accountName=" + AccountName + "&password=" + Password;

    string ServerResponse = SendPOSTRequest(URL, Post);

    if (ServerResponse.IndexOf("success") > 0 && ServerResponse.IndexOf("loginTicket") > 0)
    {
        int Start = ServerResponse.IndexOf("loginTicket")+16;
        ServerResponse = ServerResponse.Substring(Start);

        int End = ServerResponse.IndexOf("\"}")-1;
        ServerResponse = ServerResponse.Substring(0,End);

        LoginKey = ServerResponse;

        return ServerResponse;
    }

    return String.Empty;
}

private string SendPOSTRequest(string URL, string Post)
{
    HttpWebRequest Request = (HttpWebRequest)WebRequest.Create(URL);
    Request.Method = "POST";
    Request.ContentType = "application/x-www-form-urlencoded";
    Request.CookieContainer = new CookieContainer();

    Byte[] byteArray = Encoding.UTF8.GetBytes(Post);
    Request.ContentLength = byteArray.Length;

    Stream DataStream = Request.GetRequestStream();
    DataStream.Write(byteArray, 0, byteArray.Length);
    DataStream.Close();

    HttpWebResponse Response = (HttpWebResponse)Request.GetResponse();
    DataStream = Response.GetResponseStream();
    cookies = Response.Cookies;
    StreamReader reader = new StreamReader(DataStream);
    string ServerResponse = reader.ReadToEnd();
    reader.Close();
    DataStream.Close();
    Response.Close();

    return ServerResponse;
}

After the execution I have the "Login Key" in JSON format (picked out via substring) and 4 cookies (JSESSIONID, BA Tassadar, login.key, cl) in a cookie collection.

After Battle.Net Login I want to retrieve data from the auction house. This is my code:

public string Test()
{
    if (!LoginOK) return String.Empty;
    string URL = "https://eu.battle.net/wow/de/vault/character/auction/alliance/?ST=" + LoginKey;
            
    string ServerResponse = SendGETRequest(URL);

    return ServerResponse;
}
private string SendGETRequest(string URL)
{
    HttpWebRequest Request = (HttpWebRequest)WebRequest.Create(URL);
    Request.Method = "GET";
    Request.CookieContainer = new CookieContainer();
    Request.CookieContainer.Add(cookies);

    HttpWebResponse Response = (HttpWebResponse)Request.GetResponse();
    Stream DataStream = Response.GetResponseStream();
    cookies = Response.Cookies;
    StreamReader reader = new StreamReader(DataStream);
    string ServerResponse = reader.ReadToEnd();
    reader.Close();
    DataStream.Close();
    Response.Close();

    return ServerResponse;
}

But every time I try this code, I get a 404 error at this line:

HttpWebResponse Response = (HttpWebResponse)Request.GetResponse();

Does anyone have any idea what I'm doing wrong?

Thank you in advance for help.

alcaras · 02/23/11, 5:41 PM

Anyone know the json url for returning active auctions, ala https://us.battle.net/wow/en/vault/c...horde/auctions ?

(I can scrap that by parsing the xhtml, but it's always logged into the same character -- whatever I choose on battle.net, not what I logged in as -- i.e. see Armory Scraping above for the issue of switching characters with the new armory login)

♦ Slackie · 02/23/11, 6:06 PM

Originally Posted by alcaras

Anyone know the json url for returning active auctions, ala https://us.battle.net/wow/en/vault/c...horde/auctions ?

(I can scrap that by parsing the xhtml, but it's always logged into the same character -- whatever I choose on battle.net, not what I logged in as -- i.e. see Armory Scraping above for the issue of switching characters with the new armory login)

There is no JSON URL. You must parse XML or HTML to get active auctions.