How to get XML with PHP from armory - Page 5

Martin240 · 03/12/10, 5:57 AM

This is getting me some headache right now. I'm trying to get a script running for our guild page but no matter what I do, the european armory almost instantly bans my ip. The us version is working fine and I can start several huge requests in a short period of time, the results are perfect, but the eu version is getting mad at me after like <10 requests, thus giving me a 503. The only reason for this that comes to my mind is my shared hosting but I don't think there are too many people hosted on it trying to read armory data. Maybe I should ask a friend if I could get his root server for this. It would be a lot easier to cronjob this and don't need to worry about a script time-out.

Oldertarl · 10/14/10, 5:57 AM

Same problem here.
Instant errorhtml type="503".
starting seeking for the resolution now.
Must be something simple like the last time i hope.

[edit] Seems to be resolved, getting clean data back again from the armory [/edit]

tsigo · 11/05/10, 2:49 PM

The new Armory design is no longer returning XML whatsoever. This is going to break hundreds of libraries and sites parsing the armory for data.

Does anyone know if they plan to offer some kind of API of their own? Or if someone with an active WoW account could ask on the feedback forum that'd be nice as well.

frozndevl · 11/05/10, 3:47 PM

Originally Posted by tsigo

The new Armory design is no longer returning XML whatsoever. This is going to break hundreds of libraries and sites parsing the armory for data.

Does anyone know if they plan to offer some kind of API of their own? Or if someone with an active WoW account could ask on the feedback forum that'd be nice as well.

Refer to question #1 in the linked post.

New armory display - Forums - World of Warcraft

sokie · 11/11/10, 12:37 AM

After running into the vagaries of Blizzard's rate limiting policies, I decided to set up a proxy to the Armory that would enforce a safe request rate.

This has a couple advantages. You can hit the proxy with as many different clients (or instances of your app) as you want without having to coordinate rate limits between them since all the rate limiting is enforced by the proxy. This also (hopefully) eliminates the possibility of incurring a lengthy IP ban, the proxy simply delays the request until it complies with the allowed rate. Alternately it can be configured to immediately return a 503 to any requests that exceed the rate but resume serving requests the instant you are back down to the allowed rate. Lastly, this is theoretically scalable in that additional proxies (on distinct IP addresses) could be added with a load-balancing proxy in front of them.

It ended up being pretty trivial to configure with Nginx using a combination of the proxy_pass and limit_req directives. I'd be happy to provide more detail if there is interest in that.

Also, I'm looking for a little help testing my proxy setup with multiple distinct clients hitting it so if you have a fairly low volume client that you could use to send some test data my way, send me a PM and we can set it up.

tangedyn · 11/11/10, 1:53 AM

Originally Posted by tsigo

The new Armory design is no longer returning XML whatsoever. This is going to break hundreds of libraries and sites parsing the armory for data.

Does anyone know if they plan to offer some kind of API of their own? Or if someone with an active WoW account could ask on the feedback forum that'd be nice as well.

XML does make it somewhat easier, but you can still parse stuff out of the HTML itself. I have updated Mew's parser for the new armory BattleNetToonBuilder.java - mew-wow-druid-model - Project Hosting on Google Code

The reforge code was a bitch to reverse engineer though.

Hinalover · 11/12/10, 7:42 AM

I was refereed to this posting but someone has been able to reverse engineer the reforging code as well. It appears to be an 8x8 grid that shows what is reforged and into what stat.

TotemSpot • View topic - [4.0.1] Elemental Rotation Simulator

"green ones are confirmed, remaining ones are "extrapolated" after the scheme was obvious."

Yellowsix · 12/21/10, 3:45 AM

I have been updating the armory parsing for Mr. Robot - World of Warcraft (the armory features won't be enabled for another week or so unfortunately, so you can't try it out quite yet), but I found this latest post on reforges helpful, and thought that I'd contribute some findings for anyone trying to do something similar.

The reforging codes that I found on the armory html pages (e.g. Yellowsix @ Suramar - Game - World of Warcraft) are buried in the anchor tags of the child divs of div with id "summary-inventory". They have a data-item attribute that looks something like:

data-item="i=66936&g1=1695&re=154&s=2127391488&d=29"

The "re" parameter is the reforge id. I will be doing some debugging and further confirmation, but it seems to follow the pattern described in the previous post... but the values are different now than what was previously posted -- I found it to be the following:

	spi	dodge	parry	hit	crit	haste	exp	mastery
spi	X	113	114	115	116	117	118	119
dodge	120	X	121	122	123	124	125	126
parry	127	128	X	129	130	131	132	133
hit	134	135	136	X	137	138	139	140
crit	141	142	143	144	X	145	146	147
haste	148	149	150	151	152	X	153	154
exp	155	156	157	158	159	160	X	161
mastery	162	163	164	165	166	167	168	X

For the time being, I am still using the old armory xml feeds (e.g. wowarmory.com/character-sheet.xml). They work just fine for most stuff. I simply layered a call to the new armory html on top to get the reforge ids.

It'll be nice when they put out new armory xml feeds... the old item-tooltip feeds lack socket bonuses, which was always a pain... I actually go out to wowhead to pull the socket bonus for items still.

Aevitas · 01/17/11, 12:53 PM

Since the latest Armory update, and the absence of XML in it, I've been trying to parse the HTML source to obtain a somewhat reliable list of all character statistics.

The below is experimental proof of concept code, but should provide a solid base for anyone looking to adapt the new armory in their tools:


			
<?php



/***

 * Aevitas' Armory Crawler

 * 

 * @author:         Aevitas

 * @version:         0.1.1

 * @description:     Provides a generic wrapper for the Battle.net Armory.

 */



class ew_armory

{

    var $region, $realm, $character;

    

    protected $errormessage = '';

    

    /**

     * Contains the entire source code of this character's armory page.

     */

    protected $html_source, $remote_uri;

    

    /**

     * Temporary storage variables.

     */

    var $hit, $haste, $agility, $mastery, $crit;

    

    /**

     * Constructs this class.

     *

     * @param int         $region     eu for EU, us for US.

     * @param string     $realm         The realm this character resides on.

     * @param string    $character    The name of the requested character.

     */

    public function __constructor($region, $realm, $character)

    {

        $this -> region = $region;

        $this -> realm = $realm;

        $this -> character = $character;

        

        if ($this -> region != null && $this -> realm != null && $this -> character != null)

            $this -> GetArmoryData();

    }

    

    /**

     * Returns the last errormessage.

     */

    public function GetErrorMessage()

    {

        return $this -> errormessage;

    }

    

    /**

     * Fetches the data for the specified character from the armory.

     *

     * @returns    true if successful, otherwise false.

     */

    public function GetArmoryData()

    {

        // We're fetching the advanced layout for our purposes.

        $this -> remote_uri = 'http://'.$this -> region.'.battle.net/wow/en/character/'.$this -> realm.'/'.$this -> character.'/advanced/';

        

        if (file_get_contents($url))

        {

            $this -> html_source = file_get_contents($url);

            return true;

        }

        

        $this -> errormessage = 'Unable to fetch the data for the specified character from the armory.';

            

        return false;

    }

    

    /**

     * Returns the statistics block of a character as fulltext.

     *

     * @returns    string Textblock representing the character's statistics area.

     */

    private function GetStatsBlock()

    {

        // These two values mark the start and end of the statistics block.

        $start = 'new Summary.Stats({';

        $end = '"foo": true';

        

        // Determine the starting position and the end position of the stats block within the HTML source.

        $st_pos = strpos($this -> html_source, $start);

        $en_pos = strpos($this -> html_source, $end);

        

        return substr($this -> html_source, $st_pos, ($en_pos - $st_pos));

    }

    

    /**

     * Fetches an array containing all statistical elements of a character.

     *

     * @returns    array Character statistics.

     */

    public function FetchStatsArray()

    {

        $ret = new array();

        $stats_block = split(',', $this -> GetStatsBlock());

        

        // And write all data to an array. Nice and easy.

        for ($i = 0; $i < count($stats_block); $i++)

            $ret[$i] = $stats_block[$i];

            

        return $ret;

    }

}



?>

If anyone is aware of a simpler (or perhaps the old?) method of pulling stuff from the Armory, then by all means, let me know and I'll update this class.

sean111 · 01/20/11, 1:22 PM

I cam across a nice PHP script to do this. https://github.com/Kastang/No-XML-WoW-Armory-Parser

Torn · 02/08/11, 11:06 AM

The scripts listed here seem to focus on character or guild data. Has anyone managed to scan the auction house?

baytar · 03/08/11, 10:26 AM

Originally Posted by Torn

The scripts listed here seem to focus on character or guild data. Has anyone managed to scan the auction house?

WE've improved the mentioned php script to get item data to use in the first days of ReforgeMaster.com (wow reforge optimizer). But then realized using our own database built on wowhead made more sense.

But in general that script can be modified to get any data on any page of armory. It's a very good start.

elitist_prick · 03/19/11, 12:25 PM

Originally Posted by sean111

I cam across a nice PHP script to do this. https://github.com/Kastang/No-XML-WoW-Armory-Parser

thanks for this. these classes are pretty great (i just had to change the opening php tag from just "<?" to "<?php" to get it to work on my end.

quick question for the seasoned pros:
whats the best way to scan big chunks of the armory? i saw earlier in the thread, someone suggested using arena teams as a way to jump from guild to guild on a server. is this the best way?

baytar · 03/23/11, 11:21 AM

Originally Posted by elitist_prick

thanks for this. these classes are pretty great (i just had to change the opening php tag from just "<?" to "<?php" to get it to work on my end.

quick question for the seasoned pros:
whats the best way to scan big chunks of the armory? i saw earlier in the thread, someone suggested using arena teams as a way to jump from guild to guild on a server. is this the best way?

<? vs. <?php is about your php version me thinks.

I don't know which method is better for scanning but the one you mentioned seems to be the most organized one

thefool808 · 03/23/11, 12:02 PM

Originally Posted by baytar

<? vs. <?php is about your php version me thinks.

Which tag you can use depends on the short_open_tag setting in the php.ini configuration file. You can see what setting you currently have (and which config file php is reading) using the phpinfo() function.

PHP: Description of core php.ini directives - Manual

maketas · 03/29/11, 2:56 PM

Originally Posted by thefool808

Which tag you can use depends on the short_open_tag setting in the php.ini configuration file. You can see what setting you currently have (and which config file php is reading) using the phpinfo() function.

PHP: Description of core php.ini directives - Manual

the opening tag <? is removed as of PHP 5.3, and was disallowed before that version. Yes, you can use just <?, but you shouldnt as it is bad programming using something deprecated, and any PHP compilers of 5.3 or higher will not understand the <? tag.

thefool808 · 03/29/11, 4:03 PM

Originally Posted by maketas

the opening tag <? is removed as of PHP 5.3, and was disallowed before that version. Yes, you can use just <?, but you shouldnt as it is bad programming using something deprecated, and any PHP compilers of 5.3 or higher will not understand the <? tag.

Short tags are neither removed nor deprecated. This probably isn't the place to discuss the merits of that decision, but I would agree that it's best to avoid using them for portability's sake (e.g. trying to run your code on a server where you can't set the directive).

chaud · 04/01/11, 4:50 PM

It appears the API is finally closer to being a reality, although I have no idea why they started with Realm Status.

WTB XML Feeds! Offering lunch at Javier's! - Forums - World of Warcraft

Furizaa · 04/04/11, 11:54 AM

Originally Posted by chaud

It appears the API is finally closer to being a reality, although I have no idea why they started with Realm Status.

WTB XML Feeds! Offering lunch at Javier's! - Forums - World of Warcraft

Now this is interesting. Can't wait to see what this new API is about. I bet it's something similar to the remote auction JSON API. This would make a lot of sense since their own battle.net pages could load the info through AJAX directly into a native JS object. If Blizz opens the X-Site policy we could finally delegate the API request to the end-user, thus bypassing the timed IP ban.

chaud · 04/14/11, 1:25 AM

They finally added the API forum, Realm Status API - Forums - World of Warcraft

Registration of your app looks promising, hopefully they will specify what the request limit is finally.

Furizaa · 04/15/11, 2:06 PM

Seems like they have some sort of Cross Site Policy set. Surprisingly this works:

Simple Realm Status with JQuery & JSONP - Forums - World of Warcraft

Can someone with access to the US forums ask what we can expect for the future? If they plan to allow X-Site-Ajax requests to fetch data from the API, i ditch my current PHP scripts and write this stuff from scratch in JavaScript.

BinaryMuse · 04/15/11, 7:05 PM

Originally Posted by Furizaa

Seems like they have some sort of Cross Site Policy set. Surprisingly this works:

Simple Realm Status with JQuery & JSONP - Forums - World of Warcraft

Can someone with access to the US forums ask what we can expect for the future? If they plan to allow X-Site-Ajax requests to fetch data from the API, i ditch my current PHP scripts and write this stuff from scratch in JavaScript.

This is specifically what JSONP was designed to do: allow loading JSON data from other domains. See JSONP - Wikipedia, the free encyclopedia.

zork · 04/29/11, 9:48 AM

I set up a small testfile in PHP that gets data from the new WoW Armory API via CURL.

The result is delivered in JSON and then converted into a PHP array.

<?php

  //function and variables

  function json_code($json) {
    return json_decode($json, true);
  }

  //html output
  function p($output) {
    $eol = "\n";
    echo $output.$eol;
  }

  //html header stuff
  header('Content-Type: text/html; charset=utf-8');

  //html output
  p('<html>');
  p('<head>');
  p('  <title>WoW Armory API: Realm Status Test</title>');
  p('</head>');
  p('<body>');
  p('  <h1><a href="test1.php">WoW Armory API: Realm Status Test</a></h1>');

  //url to ask for
  $url = 'http://us.battle.net/api/wow/realm/status?realm=Medivh&realm=Blackrock';

  //curl init
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL,$url);
  curl_setopt($ch, CURLOPT_VERBOSE, 1);
  curl_setopt($ch, CURLOPT_POST, 0);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
  $json = curl_exec($ch);
  curl_close ($ch);


  $json_orig = $json; //temp
  //$json = utf8_encode($json); //in case utf8 encoding is needed
  $arrayed_json = json_code($json); //generate php array from json response

  //Output
  p('  <h3>Output</h3>');
  //print the json code
  print_r($json_orig);

  p('');

  p('  <hr>');

  //print the php array
  print_r($arrayed_json);

  p('');
  p('  <hr>');

  //print result in loop
  $realms = $arrayed_json['realms'];

  for ($i=0;$i<count($realms);$i++) {
    p('  <h3>Realm: '.$realms[$i]['name'].'</h3>');
    p('  <p>Type: '.$realms[$i]['type'].'</p>');
    p('  <p>Status: '.$realms[$i]['status'].'</p>');
    p('  <hr>');
  }

  p('</body>');
  p('</html>');

?>

Once the data is retrieved and converted into an PHP array it can be extracted easily.

Result of the code above:

<html>
<head>
  <title>WoW Armory API: Realm Status Test</title>
</head>
<body>
  <h1><a href="test1.php">WoW Armory API: Realm Status Test</a></h1>
  <h3>Output</h3>
{
  "realms":[
    {
      "type":"pvp",
      "population":"high",
      "queue":false,
      "status":true,
      "name":"Blackrock",
      "slug":"blackrock"
    },
    {
      "type":"pve",
      "population":"medium",
      "queue":false,
      "status":true,
      "name":"Medivh",
      "slug":"medivh"
    }
  ]
}
  <hr>
Array
(
    [realms] => Array
        (
            [0] => Array
                (
                    [type] => pvp
                    [population] => high
                    [queue] => 
                    [status] => 1
                    [name] => Blackrock
                    [slug] => blackrock
                )

            [1] => Array
                (
                    [type] => pve
                    [population] => medium
                    [queue] => 
                    [status] => 1
                    [name] => Medivh
                    [slug] => medivh
                )

        )

)

  <hr>

  <h3>Realm: Blackrock</h3>
  <p>Type: pvp</p>
  <p>Status: 1</p>
  <hr>
  <h3>Realm: Medivh</h3>
  <p>Type: pve</p>

  <p>Status: 1</p>
  <hr>
</body>
</html>