Diabolic

Blizzard outsources their hosting and networking to AT&T. AT&T has a specific branch of operations dedicated to on-line gaming customers, including Sony, Blizzard, and so on.

Does anyone know specifically what type of hardware is currently being used for WoW servers? I have been unable to find specs on the web but was hoping that perhaps someone else could.

 

Anaxo

I doubt anyone will be able to find on the web what type of hardware Blizzard hosts the servers/databases on. If it's not proprietary information, I'd bet it's considered confidential or held under some sort of NDA.

Maybe someone knows someone in the networking department at AT&T?

 

Kerruul

It's clear from some of their job postings (e.g. this one) that the database back-end is Oracle running on Red Hat Linux. They're clearly a UNIX shop.

It's unclear from the SAN engineer posting what storage system they use, but I'd bet on either EMC, IBM or maybe Hitachi based arrays with probably a Brocade fabric. (Those being the most common I've run across.) Possible NetApp, but that's more of a NAS solution and most serious DB people prefer to use FC-AL over iSCSI or NAS-based solutions.

It looks like they may have some solaris in-house as well, though it's not clear in what capacity. (iostat and vmstat being the indication in this listing that makes me think that.) While Linux has those tools, they tend to be the first resort of an old-school Solaris admin. (whereas my inclinations run to other tools for Linux.)

It's also not clear what vendor their networking gear is on (amazingly the network engineer job listed doesn't ask for Cisco/Juniper/Other experience.) If I had to bet, I'd go Cisco, but every now and again you see someone preferring Juniper (for example.)

It's not clear what software runs the WoW Servers beyond Oracle (and clearly there's something more to it than the database.) And it's impossible to suss out a deployment architecture from this meagre info.

A little digging around AT&Ts publicly available info would probably provide further info.

 

Valdamir

I'd venture to say this is something that a TON of people would love to know -- myself included. That said, I can't help but agree with Blizzard on keeping as tight a lid on this as they can. Whatever it is they have/use, you can be certain once it was known the critics and criticisms would be never ending. They get plenty of that already.

 

Neuro

Your best bet would be to either ask them directly, or find a script / program that detects hardware of a target IP. I'm sure there's something like that out there, would just take a bit of digging around.

I'd be surprised if it were confidential information (and it's certainly not under any form of NDA), so with some asking / nosing around, you should be able to dig something up.

 

Kerruul

Unless you can break through the firewall this will probably be of limited use. nmap is the defacto tool for this sort of thing and most networks that are internet exposed use a lot of tricks to prevent precisely this sort of peeking. (I know we shield a lot of servers at work from precisely this sort of attack.)

For example, ICMP is usually largely disabled to prevent using ping to find live hosts. TCP ports that are not in use are often set to drop traffic rather than send an RST packet. (This kills the performance of most port scanners.) Even if you do manage to find a valid server IP (and, I know, some are listed in various files in the WoW install) and are able to get your scanner working against it, you're probably looking at some sort of load balancer or hardened front-end that may not really represent the servers in the cluster behind it. I can almost guarantee some sort of three-tier deployment architecture where the bulk of the hardware is in a relatively protected and private network that is only reachable from the realm servers and from the management networks.

Which isn't to say this sort of grey-hat investigation can't be done. I've even done stuff like this in a prior life with different (legal) targets. But anyone who isn't particularly good at this is likely to set off an IDS with clumsy probes. (I'm assuming Blizzard's networks are professionally secured with the usual firewall, IDS, routing and other such tricks that you use to secure an internet exposed application.) Ultimately, to get the information you'd want, you'd need to penetrate these private networks. And that's where you start breaking federal and state laws.

Oh, I'd be surprised if much of it was public information. While security through obscurity is a bad thing, you don't want to give your enemies clues as to your weaknesses.

 

Anaxo

Hm, I'd swear the CMs were asked about about Blizzard's server hardware a long time ago, and their response was something along the lines of 'confidential info'. My memory may be faulty, and I am certainly no networking engineer to be able to infer hardware usage based on job postings or technical write-ups.

 

Nite_Moogle

I vaguely remember a comment about Blizzard purchasing a large amount of Itanium Blades at some point in early 2006, that may have been idle speculation.

 

Fex

Hah. That's too bad they're only considering applicants in CA. I'd apply (and would fit the requirements perfectly) in a heartbeat if it was local. Then again, maybe not. I could see myself never sleeping on a Monday or Tuesday night for the rest of my employment tenure.

I think the reference to iostat and vmstat might just have been a general indicator of familiarity with UNIX tools. They do exist on Linux, despite there being alternatives. However, I am the aforementioned old-school Solaris admin. :/

Note to my employer snooping my web traffic: I like my job and I'm not leaving.

As for the security at Blizzard, I would assume they would be top-notch. You don't run a MMO and not expect to be a #1 target for intrusion attempts.

 

Haldane

If by "Itanium Blades", you mean "HP Opteron-based servers", you'd be correct.

 

Kerruul

That's consistent with both Redhat Linux and HP-UX, actually. HP-UX has an ia64 personality if memory serves, and Redhat obviously does. I've never actually had the "pleasure" of working with the ia64 gear, but it's a well-respected choice for big databases. My guess would be that they use Oracle RAC (9i definitely, maybe 10g now that we're on WoW 2.0) for scaling since it's cheaper than buying big SMPs.

Possibly, although vmstat is usually a tip-off. It wasn't until relatively recently that Linux picked that command up, and any relatively experienced Linux admin tends to go elsewhere for their tools. In fact, I'd be willing to bet that the vmstat implementation on Linux came out of some frustrated Solaris admins spare time. (Or was it iostat... I always forget...)

It used to be you could tell what flavor of admin you were talking to just by asking about tools. (Still can for the most part, but Linux has gradually adopted so many idioms and tools... It's the magpie of the UNIX world.)

Yeah, I'd be absolutely shocked if they didn't have nearly air-tight networks. I could guess at probable network topologies and basic firewall rulesets just based on what I know to be best practices. But, that's getting so deep into wild speculation it's more suited for something like slashdot. (The WoW Forums equivalent for this discussion...)

 

Diabolic

Things that I have discovered so far: Blizzard purchases and provides AT&T the server hardware. AT&T then provides installation, service, monitoring, bandwidth, and otherwise comprehensive 24/7 support from a fairly large team of individuals.

Turbine software delivered 1,000 Dell blade servers to AT&T as part of a recent scaling move for their on-line games. Rumor was that the servers were PowerEdge 1855's but I can't confirm that. I also could find no specific information on the internal setup, such as memory, hard drives, etc. That many blade servers would be about a 6 million USD investment - not really that big of an expense if we were talking about Blizzard.

 

Nite_Moogle

Yeah, now that you mention it I do remember part of the blurb saying something about AMD processors.

 

Shlomi

Backend server software is likely to be custom C or C++, because those are still the language of choice for games programming, and Blizzard doubtless needs to wring every last ounce of performance out of their kit. They also want people who have 5 years C++: http://www.blizzard.com/jobopp/core-...ogrammer.shtml

The credits inside the manual in the TBC expansion credit Brian Fitzpatrick as the network programming lead (which actually sounds like a super fun job, they have lots of stuff going on to cram a 40-man MC raid into 2.5KB/sec downstream). This might be the same Fitz who designed the Subversion repository architecture, but then again he might still be at Google.

 

ildon

I remember an anonymous post on slashdot describing the setup by someone who claimed to have first hand knowledge.

Shakey ground, I know.

Still was an interesting read. I can't manage to find it, though. :/

 

Diabolic

As a programmer with rather extensive experience myself, I would say that C++ is all but a certainty for the language. Virtually any other choice would be unable to meet the performance needs.

As for the network programming, I'm also fairly confident that Blizz reduces everything that it possibly can to a bit or byte(s), with minimal overhead for anything outside of TCP. I have done very similar work in the past and although it is not easy, it is indeed fun.

Lastly, could you elaborate briefly on what the Subversion Repository Architecture is? I would appreciate it.

 

Mercuria

http://subversion.tigris.org/

It's an open-source configuration management tool, like cvs and such.

 

spinal

I do recall seeing EVA experience explicitely listed in the storage engineer requirements back when wow launched. Unfortunately I don't remember if it was just listed amongst others but it struck my mind back then because we were doing some tests with some of these aswell at that time.

 

Foxery

Also worth noting, for those who haven't thought about it: It's believed that each "realm" physically consists of 4-5 actual computers. Kalimdor, Eastern Kingdoms, Outlands, Instances.

edit: Ever since Battle Groups were implemented, PvP must also be a seperate machine -- a pretty beefy one that hosts all of the realms in a paticular hosting site.

Hence why performance problems on one don't affect the others, and why a single continent can crash without the whole realm dying.

 

Magunsson

Interesting thread. The chat server is almost certainly seperate as well, as often the realms can die but chat will still be working until they shut down the realm properly. There will be a number "management" servers as well, that monitor status of the other servers and allows remote reboots etc. These may be per realm, or maybe for groups of realms, or maybe there is just one (with some redundency) per continent. Speculation here obv.

Realms are also grouped into physical sites, we know this as it has been said. Who is in your Battlegroup is dependant upon who is in your physical site.

The authentication server architecture seems to be something that has caused them a lot of bother. It seems like they have only one Authentication server per Continent, which causes problems when mass numbers of realms are restarted as the Authentication server bottlenecks. I suspect now they have some kind of distributed architecture for the authentication server as this has been improved in recent times, but the login-spikes are sufficiently huge sometimes to still cause problems.

(edit: typos)

 

Magunsson

Other pages, including the "how to submit a resume" page mention C++. This add here also mentions STL and Boost, so this presumes they are from the Herb Sutter / Scott Meyers school of hard knocks.

From a project management point of view, it talks about Scrum, and other Agile development technologies which makes a lot of sense. WoW especially seems to release things with a "timeboxing" concept, where the release date is (semi) fixed at the start, and they cram in features until that release date, moving whatever features they can't fit into future patches.

 

Magunsson

That they have an ex-Googler is interesting, although I suspect like most of the games industry, they would use Alien Brain for config control.

 

Meep

For art assets, perhaps. I've never heard of a games company using Alienbrain for code assets, and Avid seem rather reluctant to name any on their web site. Perforce and Subversion would be more likely choices.

 

Snowcrasher

I personally think this is the more interesting aspect of their operations to learn about. Which server platform/processor/router brand etc. doesn't really say too much without an understanding of exactly how many servers comprise a realm and how they are partitioned.

 

Ukerric

A friend of my manager was customer sales manager at HP and involved in the processing of the dozens of hundreds or so DL580 that Blizzard purchased in august 2004 in preparation of the original launch.

I used to joke that they probably got free shipping with their order