Blizzard's Server Hardware

Diabolic · 02/20/07, 3:10 PM

Blizzard outsources their hosting and networking to AT&T. AT&T has a specific branch of operations dedicated to on-line gaming customers, including Sony, Blizzard, and so on.

Does anyone know specifically what type of hardware is currently being used for WoW servers? I have been unable to find specs on the web but was hoping that perhaps someone else could.

Anaxo · 02/20/07, 3:35 PM

I doubt anyone will be able to find on the web what type of hardware Blizzard hosts the servers/databases on. If it's not proprietary information, I'd bet it's considered confidential or held under some sort of NDA.

Maybe someone knows someone in the networking department at AT&T?

Kerruul · 02/20/07, 3:52 PM

It's clear from some of their job postings (e.g. this one) that the database back-end is Oracle running on Red Hat Linux. They're clearly a UNIX shop.

It's unclear from the SAN engineer posting what storage system they use, but I'd bet on either EMC, IBM or maybe Hitachi based arrays with probably a Brocade fabric. (Those being the most common I've run across.) Possible NetApp, but that's more of a NAS solution and most serious DB people prefer to use FC-AL over iSCSI or NAS-based solutions.

It looks like they may have some solaris in-house as well, though it's not clear in what capacity. (iostat and vmstat being the indication in this listing that makes me think that.) While Linux has those tools, they tend to be the first resort of an old-school Solaris admin. (whereas my inclinations run to other tools for Linux.)

It's also not clear what vendor their networking gear is on (amazingly the network engineer job listed doesn't ask for Cisco/Juniper/Other experience.) If I had to bet, I'd go Cisco, but every now and again you see someone preferring Juniper (for example.)

It's not clear what software runs the WoW Servers beyond Oracle (and clearly there's something more to it than the database.) And it's impossible to suss out a deployment architecture from this meagre info.

A little digging around AT&Ts publicly available info would probably provide further info.

Valdamir · 02/20/07, 4:04 PM

I'd venture to say this is something that a TON of people would love to know -- myself included. That said, I can't help but agree with Blizzard on keeping as tight a lid on this as they can. Whatever it is they have/use, you can be certain once it was known the critics and criticisms would be never ending. They get plenty of that already.

Neuro · 02/20/07, 4:12 PM

Your best bet would be to either ask them directly, or find a script / program that detects hardware of a target IP. I'm sure there's something like that out there, would just take a bit of digging around.

I'd be surprised if it were confidential information (and it's certainly not under any form of NDA), so with some asking / nosing around, you should be able to dig something up.

Kerruul · 02/20/07, 4:29 PM

Originally Posted by Neuro

Your best bet would be to either ask them directly, or find a script / program that detects hardware of a target IP. I'm sure there's something like that out there, would just take a bit of digging around.

Unless you can break through the firewall this will probably be of limited use. nmap is the defacto tool for this sort of thing and most networks that are internet exposed use a lot of tricks to prevent precisely this sort of peeking. (I know we shield a lot of servers at work from precisely this sort of attack.)

For example, ICMP is usually largely disabled to prevent using ping to find live hosts. TCP ports that are not in use are often set to drop traffic rather than send an RST packet. (This kills the performance of most port scanners.) Even if you do manage to find a valid server IP (and, I know, some are listed in various files in the WoW install) and are able to get your scanner working against it, you're probably looking at some sort of load balancer or hardened front-end that may not really represent the servers in the cluster behind it. I can almost guarantee some sort of three-tier deployment architecture where the bulk of the hardware is in a relatively protected and private network that is only reachable from the realm servers and from the management networks.

Which isn't to say this sort of grey-hat investigation can't be done. I've even done stuff like this in a prior life with different (legal) targets. But anyone who isn't particularly good at this is likely to set off an IDS with clumsy probes. (I'm assuming Blizzard's networks are professionally secured with the usual firewall, IDS, routing and other such tricks that you use to secure an internet exposed application.) Ultimately, to get the information you'd want, you'd need to penetrate these private networks. And that's where you start breaking federal and state laws.

I'd be surprised if it were confidential information (and it's certainly not under any form of NDA), so with some asking / nosing around, you should be able to dig something up.

Oh, I'd be surprised if much of it was public information. While security through obscurity is a bad thing, you don't want to give your enemies clues as to your weaknesses.

Anaxo · 02/20/07, 4:30 PM

Hm, I'd swear the CMs were asked about about Blizzard's server hardware a long time ago, and their response was something along the lines of 'confidential info'. My memory may be faulty, and I am certainly no networking engineer to be able to infer hardware usage based on job postings or technical write-ups.

Nite_Moogle · 02/20/07, 4:32 PM

I vaguely remember a comment about Blizzard purchasing a large amount of Itanium Blades at some point in early 2006, that may have been idle speculation.

Fex · 02/20/07, 4:35 PM

Originally Posted by Kerruul

It's clear from some of their job postings (e.g. this one) that the database back-end is Oracle running on Red Hat Linux. They're clearly a UNIX shop.

...

It looks like they may have some solaris in-house as well, though it's not clear in what capacity. (iostat and vmstat being the indication in this listing that makes me think that.) While Linux has those tools, they tend to be the first resort of an old-school Solaris admin. (whereas my inclinations run to other tools for Linux.)

...

Hah. That's too bad they're only considering applicants in CA. I'd apply (and would fit the requirements perfectly) in a heartbeat if it was local. Then again, maybe not. I could see myself never sleeping on a Monday or Tuesday night for the rest of my employment tenure.

I think the reference to iostat and vmstat might just have been a general indicator of familiarity with UNIX tools. They do exist on Linux, despite there being alternatives. However, I am the aforementioned old-school Solaris admin. :/

Note to my employer snooping my web traffic: I like my job and I'm not leaving.

As for the security at Blizzard, I would assume they would be top-notch. You don't run a MMO and not expect to be a #1 target for intrusion attempts.

Haldane · 02/20/07, 4:43 PM

Originally Posted by Nite_Moogle

I vaguely remember a comment about Blizzard purchasing a large amount of Itanium Blades at some point in early 2006, that may have been idle speculation.

If by "Itanium Blades", you mean "HP Opteron-based servers", you'd be correct.

Kerruul · 02/20/07, 4:46 PM

Originally Posted by Nite_Moogle

I vaguely remember a comment about Blizzard purchasing a large amount of Itanium Blades at some point in early 2006, that may have been idle speculation.

That's consistent with both Redhat Linux and HP-UX, actually. HP-UX has an ia64 personality if memory serves, and Redhat obviously does. I've never actually had the "pleasure" of working with the ia64 gear, but it's a well-respected choice for big databases. My guess would be that they use Oracle RAC (9i definitely, maybe 10g now that we're on WoW 2.0) for scaling since it's cheaper than buying big SMPs.

Originally Posted by Fex

I think the reference to iostat and vmstat might just have been a general indicator of familiarity with UNIX tools. They do exist on Linux, despite there being alternatives. However, I am the aforementioned old-school Solaris admin. :/

Possibly, although vmstat is usually a tip-off. It wasn't until relatively recently that Linux picked that command up, and any relatively experienced Linux admin tends to go elsewhere for their tools. In fact, I'd be willing to bet that the vmstat implementation on Linux came out of some frustrated Solaris admins spare time. (Or was it iostat... I always forget...)

It used to be you could tell what flavor of admin you were talking to just by asking about tools. (Still can for the most part, but Linux has gradually adopted so many idioms and tools... It's the magpie of the UNIX world.)

As for the security at Blizzard, I would assume they would be top-notch. You don't run a MMO and not expect to be a #1 target for intrusion attempts.

Yeah, I'd be absolutely shocked if they didn't have nearly air-tight networks. I could guess at probable network topologies and basic firewall rulesets just based on what I know to be best practices. But, that's getting so deep into wild speculation it's more suited for something like slashdot. (The WoW Forums equivalent for this discussion...)

Diabolic · 02/20/07, 4:47 PM

Things that I have discovered so far: Blizzard purchases and provides AT&T the server hardware. AT&T then provides installation, service, monitoring, bandwidth, and otherwise comprehensive 24/7 support from a fairly large team of individuals.

Turbine software delivered 1,000 Dell blade servers to AT&T as part of a recent scaling move for their on-line games. Rumor was that the servers were PowerEdge 1855's but I can't confirm that. I also could find no specific information on the internal setup, such as memory, hard drives, etc. That many blade servers would be about a 6 million USD investment - not really that big of an expense if we were talking about Blizzard.

Nite_Moogle · 02/20/07, 4:51 PM

Originally Posted by Haldane

If by "Itanium Blades", you mean "HP Opteron-based servers", you'd be correct.

Yeah, now that you mention it I do remember part of the blurb saying something about AMD processors.

Shlomi · 02/20/07, 7:52 PM

Backend server software is likely to be custom C or C++, because those are still the language of choice for games programming, and Blizzard doubtless needs to wring every last ounce of performance out of their kit. They also want people who have 5 years C++: http://www.blizzard.com/jobopp/core-...ogrammer.shtml

The credits inside the manual in the TBC expansion credit Brian Fitzpatrick as the network programming lead (which actually sounds like a super fun job, they have lots of stuff going on to cram a 40-man MC raid into 2.5KB/sec downstream). This might be the same Fitz who designed the Subversion repository architecture, but then again he might still be at Google.

ildon · 02/20/07, 10:25 PM

I remember an anonymous post on slashdot describing the setup by someone who claimed to have first hand knowledge.

Shakey ground, I know.

Still was an interesting read. I can't manage to find it, though. :/