Server locations, Latency & Australia - Page 7

faces · 11/26/07, 9:57 PM

Originally Posted by Drunkmunky

Our last prime minister also promised he would not introduce a GST and we all know what happened there so I'm remaining skeptical. It doesn't matter who you vote for, they are all politicians

Pipe networks has had it's Project Runway project in the planning and preparation stages for a while now (latest update oct 26, http://www.pipenetworks.com/docs/med...Update%205.pdf) so regardless of how Rudd does things theres definitely some good prospects for the future.

I don't particularly trust Rudd on the whole world class broadband promise, but at least I don't think it's going to get worse, and it actually looks like it'll gradually get better during his term (regardless of how involved in it they are).

Calantus · 11/27/07, 11:08 AM

Originally Posted by Drunkmunky

Our last prime minister also promised he would not introduce a GST and we all know what happened there so I'm remaining skeptical. It doesn't matter who you vote for, they are all politicians

He said that for one election, and during that term they did not put in GST. You can't really expect more than that. We elect for one term and you're accountable for your promises for that one term. New term, new promises. My memory is a little hazy but I'm pretty sure the term they brought in GST it was actually an election issue where they said "we will bring in GST". I can half-picture the scare adds from the opposition saying a vote for liberal is a vote for GST, etc, etc.

EDIT: I'm not trying to argue politics here, just pointing out that it wasn't a blatant lie. Blatant lies in election promises are pretty rare nowdays because it's easy to point them out and just as easy to keep them vague and half-heartedly fulfill them. I think we'll get at least a token gesture in the area of improved broadband network from Rudd.

jayowen · 11/27/07, 8:09 PM

Originally Posted by Calantus

He said that for one election, and during that term they did not put in GST. You can't really expect more than that. We elect for one term and you're accountable for your promises for that one term. New term, new promises. My memory is a little hazy but I'm pretty sure the term they brought in GST it was actually an election issue where they said "we will bring in GST". I can half-picture the scare adds from the opposition saying a vote for liberal is a vote for GST, etc, etc.

EDIT: I'm not trying to argue politics here, just pointing out that it wasn't a blatant lie. Blatant lies in election promises are pretty rare nowdays because it's easy to point them out and just as easy to keep them vague and half-heartedly fulfill them. I think we'll get at least a token gesture in the area of improved broadband network from Rudd.

The point is Howard emphatically said "never ever", then he introduced it. This is a blatant lie.

Labor has committed to FTTN 12 meg for basically everyone who doesn't live in the bush. A watered down version of this would be a broken promise, we can only wait and see.

Drunkmunky · 11/27/07, 9:01 PM

I'm glad you're not trying to start a political debate, I am so over the goddamn elections

The money really needs to be spent on infrastructure as I think that is where we are lacking, I wish they'd make up their mind on what broadband tech they are going to use as well. Telstra proposes 50Meg VDSL, Labor promises FTTN 12Meg (not sure what protocol that actually is?) and the G9 havn't proposed anything yet because they suck.

jayowen · 11/27/07, 10:42 PM

FTTN is fiber to the node. Basically it means a lot of small sub exchanges which service a small number of residences. Each sub-exchange (node) is fed by fiber then the "last mile" into your house is still copper, probably still some form of dsl. With the limiting factor on dsl speeds most often being distance from the exchange this in essence makes everyone be close to their exchange.

Spazmo · 12/05/07, 12:35 AM

Just wanted to highlight an interesting development on the Whirlpool forums. Representatives from one of Australia's most respected and top notch ISPs (Internode) have recently stated that they may have found an issue with a bug in some protocols that may be causing our ping to be higher. They're not releasing any solid details about it but it sounds like examining the results achieved by lowerping.com may have somehow tipped them off about it. They say they are now communicating with Blizzard to implement a possible fix. All very secretive at the moment but it sounds promising. I trust the guys at Internode to know what they're talking about.

Starts to get interesting from page 10 onwards (linked):
Another WoW latency whinging post - Internode - Whirlpool Broadband Forums

Cyn · 12/05/07, 4:07 AM

Wow, those posts get me very excited. Especially when im allready with Internode.

A real solution to the problem? rather than $7 a month to some random on the horizon?

• Intermission · 12/05/07, 5:00 AM

Originally Posted by Spazmo

Just wanted to highlight an interesting development on the Whirlpool forums. Representatives from one of Australia's most respected and top notch ISPs (Internode) have recently stated that they may have found an issue with a bug in some protocols that may be causing our ping to be higher...

..Starts to get interesting from page 10 onwards (linked):
Another WoW latency whinging post - Internode - Whirlpool Broadband Forums

Wow... very interesting read!

Good news on the horizon it seems.

♦ Cadfael · 12/05/07, 5:00 AM

Originally Posted by Cyn

Wow, those posts get me very excited. Especially when im allready with Internode.

A real solution to the problem? rather than $7 a month to some random on the horizon?

From the insinuations of the Internode guy, it seems to be some sideeffect of the TCP parameters on Windows systems. (He specifically mentions Redmond/Windows). If that is true, sticking a Linux router with two NICs between your gaming machine and the Internet may already be enough. It's all speculation though until someone says what's causing the problem.

It may just be sufficient to lower your TCP MTU, change the window size, enable/disable SACKs, etc. There's a ton of TCP parameters that can be juggled. It may be that a simple Registry Key on Windows fixes it.

But without an explanation what the problem really is, it remains guesswork. If you have time, start playing with your Windows' TCP parameters (if you know how to do that, that is) and see if something helps.

• Intermission · 12/05/07, 9:49 AM

Update for people who haven't read the Whirlpool thread:

Internode have set up their own 'lowerping/PingFu' style service (for Internode customers). It's reportedly working almost as good or just as good as Lowerping, depending on a few factors, mainly location as the test server is in Adelaide.

From what I could gather, the Internode solution isn't a SSH tunnel though.. they do something with the network protocol or along those lines. Over my head anyway, and they seem a little hush hush about the exact details of the problem until they are convinced they have solved it completely.

A bunch of quotes from various Internode employees:

"We've spent some time doing some quality investigating and we know that it's a WOW protocol/client related issue."

"I standby my comment that it's client/protocol related as I've done extensive testing and I KNOW what the problem is."

"We're talking to Blizzard to help them understand the problem and hopefully fix the problem properly. If not, then we know what the problem is and we'll then look at trying to work out a better way to solve it."

"I will say this again: What we seem to see here is a third party protocol bug/interaction, not an Internode fault, so its hardly surprising that we couldn't magically fix it without the new information that is now available."

"I think we'd prefer Blizzard to implement a fix so that it isn't needed anymore. Think of this as a temporary workaround."

Drunkmunky · 12/05/07, 6:08 PM

Holy mother of awesomesauce Batman! This is great news for Australian Wow players, that unexplained 200ms extra now makes sense and Internode think they can fix it <3

If Blizzard don't comply and fix the TCP stack issue in the client (which is what Internode think is the problem) then Internode are promising:

if this does 'work' (once we understand what 'works') means, we're happy to see if we can offer our customers a similar TCP stack tuning service directly as an Internode supported thing for this purpose.

So Internode might be receiving a complete flood of wow players churning to them :O

Ragnor · 12/05/07, 6:25 PM

Well they have released hardly any actual details about what they have discovered. Smells like more of a Internode marketing exercise on the whirlpool forums to me.

Personally, I'd still put my money on it being:
a) Blizzard/AT&T having a crappy route back to Australia/NZ/Asia (tracert only shows your route to wow, not back) PLUS b) Some level of priority for local US traffic by AT&T

I know people in NZ, AU, Singapore and Japan all have higher latency then you'd expect from a US hosted server and given they all vary in geographic distance from the US and internet capacity/infrastructure. You'd expect people closer to the US with better infrastructure to have less of a problem but this doesn't seem to be the case.

.. but hey I'll freely admit I could be wrong.

• Intermission · 12/05/07, 6:33 PM

Internode WoW Gateway Trial - Internode - Whirlpool Broadband Forums

More up-to-date link.

Drunkmunky · 12/05/07, 6:43 PM

Originally Posted by Ragnor

Well they have released hardly any actual details about what they have discovered. Smells like more of a Internode marketing exercise on the whirlpool forums to me.

Personally, I'd still put my money on it being:
a) Blizzard/AT&T having a crappy route back to Australia/NZ/Asia (tracert only shows your route to wow, not back) PLUS b) Some level of priority for local US traffic by AT&T

I know people in NZ, AU, Singapore and Japan all have higher latency then you'd expect from a US hosted server and given they all vary in geographic distance from the US and internet capacity/infrastructure. You'd expect people closer to the US with better infrastructure to have less of a problem but this doesn't seem to be the case.

.. but hey I'll freely admit I could be wrong.

Well I've just read through the whingepool threads about it and it seems like it's real to me, they've already set up a test server in adelaide which is showing great results. Users have been testing it and are reporting that it works so I will take that as proof. Internode reps claim that they have contacted blizzard with a full report of their findings and are awaiting reply.

Technical information:
So far as I could tell from what Internode have said is that it is a bug in the Wow client and how it creats TCP/IP stacks, so It's a flaw in the protocol. The reason that services like Lowerping.com curcumvent this is that you tunnel your TCP connection through the ssh protocol and I think it is also re creates the stack (the dodgy TCP stack is re done properly) at the Lowerping proxy. All the Internode server does is re order the stack properly as it passes through, no tunneling required.

♦ Cadfael · 12/05/07, 6:55 PM

Well whatever it is they do, it's strange. I just took a good look at my network traffic and I didn't see anything even remotely in the land of the strange.

It's just a single TCP connection, with no special header options. Lots of small packets though in the ballpark of 100 to 300 bytes. Generally the one my clients send with PSH flag set which is normal behaviour if you flush the pipe to send out packets fast and not letting the stack accumulate more data. It may be however that the combination of the many small packets, window size and sending behaviour is bad on long haul connections. Clamping the window size down or some such might help there. I'd really need to see Australian packet dumps before and after the proxy to see what they do.

However it's cool that the problem is found and you have some options and probably a final resolution of the situation pending.

♦ pewsey · 12/05/07, 9:59 PM

The responses have been very interesting. I'm a bit surprised at the latencies reported by Internode customers. I'm on Telstra Cable and have really never had much greater than about a 450-500 ping, and it sits on 350-400 "most" of the time. Heck, I'd be thrilled for a sub 200ms ping.

I'm all for this study, and I applaud the Internode guys. Network Theorycrafting FTW !

Given that they've done it with nothing more than just a gateway, it's very likely that it must be relating to packet size, fragmentation, assembly or header. It also presumably is relating to number of hops and/or the specific router configurations that form the Aus/US link doing "something" - because otherwise we'd see the same effect on our US bretheren.

Londo Molari · 12/06/07, 3:25 AM

Having read the Whirlpool forums mentioned above, decided to check it myself.

My lowest ping so far (non raid) has been just on 270ms - bombing run in Ogri'la - wandering as high as 410ms. Given that my 'normal' range is 480 - 600 ms I'm currently happy. Will have more information later tonight Melbourne time post raid.

Internode admit to it been in test/pre production mode. Should be interesting if they roll out the production version (what kind of word is 'productionised'??), placing servers into each capital.

• Falk · 12/06/07, 3:59 AM

I for one would instantly agree with anyone saying the WoW connection protocols have some screwy or inefficient shit up with them. You can't really otherwise explain how certain people disconnect on, say, Razorgore, when WoW barely pushes 5kbps without any addons.

Here's to hoping Internode's solution works, even though I'm not in Australia.

♦ Cadfael · 12/06/07, 4:49 AM

Originally Posted by falkon2

I for one would instantly agree with anyone saying the WoW connection protocols have some screwy or inefficient shit up with them. You can't really otherwise explain how certain people disconnect on, say, Razorgore, when WoW barely pushes 5kbps without any addons.

Well Razorgore's Room caused traffic in the downstream above 7.2kbit/sec. I'm very certain of this, because we had at that time a raider with ISDN (european, which is 64kbit downstream bandwidth). He always lagged out and disconnected. Once diagnosed, I worked with him to enable channel bonding, so he always disconnected, switched configuration and reconnected with 128kbit which then went fine. For Vael he dropped back to one connection again and for Lashlayer he had to go back to dual channel, if I remember correctly.

There isn't that much wrong with the connection protocol, it's just too much data for a euro-ISDN with 64kbit due to the whelp's movement and presence data. It's especially too much for US-ISDN with inband signalling which reduces the available bandwidth to 56kbit and for any 56kbit modem. Everyone else should actually have been fine, unless they had a sucky net connection and didn't notice it until there.

However, as I've seen last night, WoW sends a lot of small packets. It flushes data reguarly which means that the TCP packets are not sent "full" but almost empty. This is done to reduce latency, as otherwise the TCP stack would wait to accumulate a bit more data before sending out the packet. Now if the average data size is in the ballpark of 100 bytes per packet, then we have a protocol overhead of ~40%, which means that if ingame it tells you that it is receiving 5kbit/sec, the total data received is more like 7kbit/sec due to the rather large overhead.

Also TCP has a lot of mechanisms built in to adapt to the traffic condition, it will slow down if it detects problems along the path such as packets that take too long to arrive or are being dropped. Over high latency paths, I can imagine that sending a lot of small packets over TCP is probably one of the worst things that can happen. It should still not almost double the overall latency due to that, though. There are mechanisms that should in theory adapt to the situation and then provide steady latency. This seems to fail here. Blizzard does not send anything special - at least in my case, but I'm not in a high latency situation - it is really an absolute normal TCP data stream, but with rather small average packet size. Now what they could do to help rectify the situation is to provide hints to the TCP stream in high-latency cases that tells the two TCP stacks involved to better react to the situation, as the standard mechanism seems to worsen the situation instead of helping it.

This can be done by an intermediary proxy as well, though it really best be done at either source or destination and as such optimizations are generally rather poor for high throughput (bulk data transfer) it really should be done at Blizzard's side for those connections only.

Here's to hoping Internode's solution works, even though I'm not in Australia.

It seems to. I'm still interested in hearing their exact technical analysis of the problem to see if I were thinking in the same direction though.

Elhindenburg · 12/06/07, 6:40 AM

I'm an Internode customer and can confirm that it does indeed work, went down from a usual 450-500 ping to hovering around 270 all day.

Will be great when they add a server on the east coast however because I live in Melbourne its going to Adelaide then back again adding perhaps another 50ms.

I knew there was a reason I choose Internode. (Besides the great service and excellent file server farm :P)

Tinkerfizzle · 12/06/07, 1:11 PM

Originally Posted by Cadfael

Well Razorgore's Room caused traffic in the downstream above 7.2kbit/sec. I'm very certain of this, because we had at that time a raider with ISDN (european, which is 64kbit downstream bandwidth). He always lagged out and disconnected. Once diagnosed, I worked with him to enable channel bonding, so he always disconnected, switched configuration and reconnected with 128kbit which then went fine. For Vael he dropped back to one connection again and for Lashlayer he had to go back to dual channel, if I remember correctly.

I had a problem completely identical to this -- it happened for me over the summer while I was on very limited bandwidth: just about every trash pull during Hyjal(and the summer before that, during Vael), my bandwidth usage shot up to cap, and I disconnected shortly thereafter, without fail.

Why the heck there's such sheer bandwidth usage from just the number of NPCs present in both these scenarios, I really have no idea.

Baalzaman · 12/08/07, 4:03 AM

Just to bump this with some very interesting info posted in the official forums (technical support): WoW Forums -> International Lower Ping HOWTO + Observations

It's fairly lengthy, apologies in advance.

Originally Posted by Yarre

After reading about lowerping.com and also reading about an ISP that is providing a similar service to it's customers I decided to do a bit of investigation into how the wow server/client communicate.

There is a definite issue, no it isn't a bug as such, just a limitation of TCP/IP. it could be worked around at the blizzard end!

But, I've found out how to work around it. This is information I believe benefits everyone, so basically hey lowerping.com, eat a bowl of **** for not sharing it.

Anyway! HERE GOES.

SOMEWHAT DUBIOUS TECHNICAL EXPLANATION (I'm a software developer who likes to play at network sysadmin).
-------------------------------------------------------------------------------------------------------

OK to sum up. I'm far from a network god, but!

Here is the technical explanation at least as to *why* this occurs.
Boosting Socket Performance on Linux

I'm not entirely concerned with the technical details, but here's what I do to fix it.

Basically on a higher latency connection (trans-pacific for example) your TCP ACKS have an inherent delay. This can be up to .2-.5 of a second. I myself live in New Zealand and I get 150ms flat to the wow servers, and 300-400 in game.

Here is a BAD ACK (for wow traffic).

11:28:00.686393 IP 12-129-225-21.attens.net.3724 > myip.cable.telstraclear.net.1230: P 14746:14824(78) ack 162 win 14600
11:28:00.857213 IP myip.cable.telstraclear.net.1230 > 12-129-225-21.attens.net.3724: . ack 14824 win 17520

0.170820 (170ms+) ack reply.
This sucks because while our client is twiddling its thumbs holding on to that ack, the wow server
can't/won't send us any more data. This artifically inflates our ping. For me it seems to be in the realm of 170ms. WHen you're already
pinging at 150 to the US this obviously isn't good as it creates aping of at LEAST 320ms+.

Here is a GOOD ACK (for wow traffic, for other kinds of traffic this would probably be suboptimal).

14:19:36.001116 IP 12-129-225-21.attens.net.3724 > myip.cable.telstraclear.net.58484: P 7468:8463(995) ack 53 win 4537 <nop,nop,timestamp 286604871 1342539>
14:19:36.001301 IP myip.cable.telstraclear.net.58484 > 12-129-225-21.attens.net.3724: . ack 8463 win 4006 <nop,nop,timestamp 1342554 286604871>

0.000185 sec ack reply, letting the WoW server know we can handle more data ASAP! So it will send us this data ASAP!

What is happening is your client is hoping that the wow server will send it some more data in that timeframe, which it will only have to ACK once
for multiple packets. BUT, the wow people have actually done a great job of implementing what is a stunningly low bandwidth network engine. So in reality
for pretty much every packet the wow server sends you your client can send that ACK back right away, instead of waiting for more data. There's really very few packets
being sent, so waiting for more to arrive just artifically inflates the latency.

HOW TO FIX
----------

There may be other ways to skin this cat. As I said, it works for me (well) and I'm not persuing it further.

You need a linux gateway, using iptables.

You need the following software installed on the gateway and the knowledge of how to use it, as well as superuser privileges.
(although if stuff is already installed it'll probably work fine).

socat
iptables

Step 1. You need to redirect all wow traffic to your gateway localhost, using iptables.
You need to know the IP address of your wow server to do this. In my case this is blackrock. The wow port appears to be 3724.

In a terminal: sudo iptables -t nat -A PREROUTING -p tcp -d your.wow.server.ip.address --dport 3724 -j REDIRECT --to-ports 3724

This will break wow, as all traffic outgoing on 3724 is now going to the loopback interface.

Step 2. We take that traffic coming to the loopback interface, modify the way it communicates with the WoW server and forward it to the wow server.
Wow will no longer be broken.

So on our gateway we start socat. Socat is an amazing tool and I recommend it to anyone that likes futzing with tcp and other streams. I picked on it in this
cause because it can modify the way client/server tcp communications behave, but that's only the tip of the iceberg as to what this tool is capable of. My hat
off to the author.

In this case, socat is accepting incoming tcp traffic on 3724 (where we redirected our wow traffic to) and forwarding that traffic

In a terminal: sudo socat -d -d -d TCP4-LISTEN:3724,nodelay,fork,reuseaddr,su=nobody TCP4:12.129.225.21:3724,nodelay
(could use -lmlocal2 to log to syslog after connection is accepted, could also use & to tie it off from the terminal)

I've set it up here so it will accept more than one connection, should you have more than one wow using PC on your local lan.
I only have one, and haven't tested this festure, but it should work. You could also set up a SOCKS server for external access
and it should work fine.

Now, you can fire up WoW on a machine on your local LAN and in the terminal you started socat in, it should start printing out
a bunch of debugging information (you can lessen this later by removing the -d switches).

All going well wow will connect to your server and you'll be off. My latency typically sits at about 250ms in Shattrath and 180-200
in other zones. Which is a stellar improvement on the 350-450 I used to get.

BUT HOW DOES IT WORK??!
-----------------------

socat creates a socket connection to the wow server and sets the TCP_NODELAY option. Your ACK packets will now be sent instantly
instead of being held in case more data arrives. It's that simple. Linux doesn't have a TCP switch to make this happen so I use
socat. It's better that way anyway as it doesn't affect my applications globally, just WoW.

Mac users, I believe in OSX there is a kernel option for ACK delay (net.inet.tcp.delacktime). I don't know about this myself not
owning a mac but lowering it may produce similar results. Note: try at own risk and it will probably apply globally to your OSX
install and therefore hurt high bandwidth applications and downloads.

WHY DON'T BLIZZARD IMPLEMENT THIS IN THE CLIENT?
------------------------------------------------

You do have to wonder. I basically researched and implemented this over the course of 4-5 hours. It is a simple option that can be
set in any socket capable application. Blizzard could and should make this a switch in the wow client.

The answer is that probably the majority of their customers live in the USA, and that basically it wasn't worth their time to
investigate. That said their network code guys should have been more proactive about this.

P.S. Blizz you should hire me, I just helped you out. My rates are very competitive!

Note: If your ping is absolute rubbish this wont fix that (but it will help), you need to get a real ISP.

Binkenstein · 12/09/07, 3:48 PM

Also, using the following regkey should help (I'm usually 500+ on a good day, was just short of 300 after using it).

From the second page of the thread Baalz listed above http://support.microsoft.com/kb/328890/

Drunkmunky · 12/09/07, 8:55 PM

Yeah so it turns out that the problem is that the wow client doesn't use TCPNoDelay which is freaking retarded imo... Anyway if you change that registry setting your wow latency will be fixed BUT: It will have an adverse effect on other internet services you use (mostly downloading). The bit about the linux box and the iptables is a complicated way of making this happen for only your wow traffic, I think I will try this when I get home (I havn't had any pleasant experiences with linux so far).

It's worrying that people are reporting bans from using lowerping.com, I think until Blizzard releases their fix (even a console command that you can enable if you want to would do the job) I will have to sweat it out and make this linux box work.

buddhasevil · 12/09/07, 9:25 PM

I have been posting on the Blizzard forums to try and lobby for some Hawaii based servers, as we are still in the USA (in name only). Still being in the US will eliminate the Aus etherwebpolitiks issues, and I think this is a good step to bridging the latency gap. I don't see why Blizzard wouldn't set up a few servers on Oahu, and make "truer" oceanic servers. Maybe someone on these forums has some insight on how I can personally continue to lobby for this.

The distance between Brisbane, Australia and Honolulu, Hawaii is 4687 miles.
The distance between Los Angeles, California and Honolulu, Hawaii is 2551 miles.

So maybe just the pure geographic location will cut back the ping? I am by no means a wizard with networking or internet issues, it just seemes to make sense that the closer you are to a server, the less lag you have.