As a programmer with rather extensive experience myself, I would say that C++ is all but a certainty for the language. Virtually any other choice would be unable to meet the performance needs.

As for the network programming, I'm also fairly confident that Blizz reduces everything that it possibly can to a bit or byte(s), with minimal overhead for anything outside of TCP. I have done very similar work in the past and although it is not easy, it is indeed fun.

Lastly, could you elaborate briefly on what the Subversion Repository Architecture is? I would appreciate it.

I do recall seeing EVA experience explicitely listed in the storage engineer requirements back when wow launched. Unfortunately I don't remember if it was just listed amongst others but it struck my mind back then because we were doing some tests with some of these aswell at that time.

Also worth noting, for those who haven't thought about it: It's believed that each "realm" physically consists of 4-5 actual computers. Kalimdor, Eastern Kingdoms, Outlands, Instances.

edit: Ever since Battle Groups were implemented, PvP must also be a seperate machine -- a pretty beefy one that hosts all of the realms in a paticular hosting site.

Hence why performance problems on one don't affect the others, and why a single continent can crash without the whole realm dying.

Interesting thread. The chat server is almost certainly seperate as well, as often the realms can die but chat will still be working until they shut down the realm properly. There will be a number "management" servers as well, that monitor status of the other servers and allows remote reboots etc. These may be per realm, or maybe for groups of realms, or maybe there is just one (with some redundency) per continent. Speculation here obv.

Realms are also grouped into physical sites, we know this as it has been said. Who is in your Battlegroup is dependant upon who is in your physical site.

The authentication server architecture seems to be something that has caused them a lot of bother. It seems like they have only one Authentication server per Continent, which causes problems when mass numbers of realms are restarted as the Authentication server bottlenecks. I suspect now they have some kind of distributed architecture for the authentication server as this has been improved in recent times, but the login-spikes are sufficiently huge sometimes to still cause problems.

(edit: typos)

That they have an ex-Googler is interesting, although I suspect like most of the games industry, they would use Alien Brain for config control.

For art assets, perhaps. I've never heard of a games company using Alienbrain for code assets, and Avid seem rather reluctant to name any on their web site. Perforce and Subversion would be more likely choices.

I personally think this is the more interesting aspect of their operations to learn about. Which server platform/processor/router brand etc. doesn't really say too much without an understanding of exactly how many servers comprise a realm and how they are partitioned.

Blizzard originally went live with JAAS, a Java-based auth service. Sadly I can't find the original reference for that info. It didn't scale wonderfully and I believe they're now on their own custom solution.

The problem is pretty hard. "Given a million logged in North American users on hundreds of realms spread over multiple geographical locations, ensure no-one can log in more than once, expired users can't log in, etc etc". Again, fun stuff to work on.

Someone mentioned Chat as being seperated from continent servers... Which makes me think that perhaps there is some sort of "Master" machine (or process on one of the others, at least) which controls Chat, the Character Selection screen, and the item database. This one would keep track of which continent your character is supposed to be on and when you zone to another. It would also help explain why you're able to stay connected when other continents crash. [NOTE: this is only an educated guess based on observed behavior over the years.]

Realizing these aspects of server structure makes it all the more amusing when the General forums whine about "fix my server" or "upgrade my slow realm"... It's a lot more involved than sitting down at one keyboard and pressing Control-Alt-Delete. Maintenence on "a realm" involves checking the software configuration and wiring for 4-5 boxes... It's no wonder upgrading the hardware last year required 24-hour downtimes - a swarm of technicians had to reorganize something like 40 machines per hosting site, ensuring that each one was configured for the correct function and wired to its sister boxes!

Back to hardware specifics, the processor types and OS don't seem nearly as interesting as wondering what sort of RAM and storage requirements it really takes. Keep in mind that the server doesn't need ANY graphics processing whatsoever, so these figures are probably strikingly small compared to what professional servers are capable of having in them. (I don't know where to start guessing, though.)

All a server really needs to keep in memory is a standard data structure listing characters, mobs, resources, and the coordinates of each. The item database is... well, just an ordinary database. I am oversimplifying here - of course the server has to track interactions between players and mobs, but the math happening inside the CPU is a lot simpler than the sexy 3D graphics at home make it seem.

Food for thought.

My educated guess on what the application actually "does" is pretty simple. There's a back-end database which contains your character data, as well as all the other data in the world: mob types, individual mob data, items, saved raid instances, etc. Connecting to that is a "simple" (relatively speaking) application which tracks character and mob movement. It interfaces directly to the clients which run on our desktops. Figuring out exactly which interactions are server-side and which are client-side I believe is mostly a guessing game, though there are some predictable actions that you can assign to each one. Anything which affects the entire world (i.e. things which multiple people can interact with) is server-side. Anything that you control or manipulate is triggered from the client. For instance: Two characters ride up to an herb spawn, and dismount, one gets the tap, the other doesn't. What happens? The server reports to the client that there's an herb at x,y on the map. Your herb radar pops up on your client based on this information which is sent to you (once you're in a certain radius.) The other player gets the same information. Network latency and system load determine who gets the "blip" first. You both run towards it -- this is pure client control. Does the server need to know you're moving? Not really. When you both get there, and dismount, you click the node. Client tells server, "hey, player is at x,y trying to pick up the weed." Server determines if you're in range of it, sends client "OK" and you begin harvesting. Based again on network latency which we all know and love, when the actual event is processed by the server, and sent to the client, is variable. Your bar fills up, client tells server "done picking the weed" and whoever got their "finished" packet back to the server first gets the herbs.

What this all really means is that system load in different parts of the systems affect our experience in different ways. Mobs warping around is usually a network latency problem, or "server" load. (I use "server" to mean the app that interfaces with our desktop clients on the front end, and the database on the back end.) I imagine some component of mob movement is server and database related. If the server can't update positions to the client often enough, they warp. My guess about how much interaction takes place for mob movement is that the server sends a position, a direction, and a speed every once in a while to the client, based on mobs in some arbitrary range. It then lets the client render the movement of that mob until it gets a newer update on where it should be going. This is what causes warping when there is network latency, or server lag. Your client thinks a mob is in a certain spot, but it's really not. So you either aggro it from where you don't expect, or you can't enter combat because you get out of range errors. Loot lag is a database problem. Most likely, the server process made a call to update your character data based on clicking an item in a loot window, and the success response was not forthcoming. Why does it wait? Well, you want to make SURE that when the user feedback indicates the item is looted (the window closes, or some other visual indication) that it ACTUALLY has been looted and the database is updated and saved. Otherwise, you might have lost items. ("HAI GUYZ I LOOT EPIX AND ITZ GONE NOW!!1?!?")

A lot of this seems pretty obvious to me because I work with applications like this every day, just not nearly on this scale. I hope it's given what I can assume to be a more educated audience than the WoW forums crowd at least a little insight, though it really is only an educated guess after all.

Wall of text crits you for 45212.
You die.

During the first week or two of TBC we did actually see this happening quite a lot on Blackrock. Outland, as expected, was horribly overcrowded and regularly dropping people. Some of the time it was Outland as a whole, a lot of the time however it was just Hellfire Peninsula with those of us that had escaped to Terrokar especially/Zangarmarsh to some degree able to watch others fall offline on a regular basis.