The problem with that however, is that WSG/AB/EOTS will have even more people disappearing mid match, which is yet another problem people will complain about.

Wow, I hope I never use software that you program.

If it were *just* a question of buying more servers and plugging them into power and the net, I'm sure it would be done by now. There are things that take time no matter how much money you have, such as reconfiguring networks to support the additional traffic, upgrading cooling systems in the data center that houses it all, or maybe even expanding to another center if the current one is full.

We don't know that it's NOT a matter of buying more servers. That may very well fix it. It just may not make financial sense to do that right now, or ever. Regardless, we don't know until someone that does know tells us. So please, for the love of god, stop speculating about it every other post.

My point about scalability of 'instance' servers is mostly guess work since I don't know how their stuff really works. Here's what assumptions I'm basing it on, along with what I know about various web architectures.

Avair's Wild Assumptions about Blizzard architecture
* Instance servers are essentially a pool of capacity waiting to be used. No instances means under utilized capacity.
* Each instance takes certain amount of resources (i.e. CPU/Memory). A 2v2 instance is probably about the same as 5v5.
* Since instances are not shared by many people, the problem isn't the amount of communication between clients, its the overhead of each instance. Ironforge or AQ gate opening 'lag' has to do with having one server location needing to keep X thousands of clients updated.
* When players match for a 2v2 arena, their characters are 'moved' to the new instance, on a different server. I would expect this roughly similar to how WC3/SC did games, except for those RTS's the 'instance server' was basically one of the players computer.
* Each instance server shouldn't need to know about any other instance server, so there is no communication overhead of adding 100 more servers.
* You should be able to scale capacity nearly perfectly with the amount of instances you need. If each instance server holds 100 2v2 instances, and 'demand' is 1000, you should be able to buy/setup 10 servers. Obviously, there is a fixed cost/server, as well as an ongoing cost for maintenance for each server you add.
* Unlike a stateless protocol like HTTP/Web servers, you should't need load balancing to determine which servers to send requests, so there shouldn't be much overhead beyond just adding servers.

My expertise comes more in the web architecture, so I'm probably completely wrong here. But I'm still guessing that Blizzard hopes they can get away with not having to spend the money to add more servers, which are going to be underutilized 50% of every day. If their architecture can't just easily scale when they instance capacity, that itself may reveal problems, since that would require a code rewrite they don't want to undergo

But if it doesn't get better soon, Bliz is going to be giving users reasons to do other things. Queues need to be 2-3 minutes tops, since you can't do anything much but farm inbetween longer queues.

Besides personal attacks, what exactly do you have to offer to disprove my assertion? It is basically a fleshed out version of what Blizzard themselves said at Blizzcon. They said specifically that the issue is that they can only make so many matches in a given time. Whether it is the queue itself or the mechanism that transfers you to your instance, somewhere there is an issue.

If their architecture was well-designed it would literally take hours to plug in a few new servers. Configuring a network is not rocket science. Even if they had to build the entire datacenter from scratch they could have done that by now, since we have had queues for months.

Judging from past experience, their architecture is not well-designed or scalable. They have been slow to react to increased demand every time it occured. All this is not unexpected, it is their first MMO and rookie mistakes are boud to happen.

Those were not scalability issues really. Scalability implies that it is capable of being brought up to a larger scale. Two solid examples of scalable vs non-scalable solutions in a massively multiplayer online environment-

World of Warcraft. Scalable from 50,000 players to 9 million players over 3 years. Current game stability and sever expandability are still at release day standards (much better in fact). It, however, launched with X servers and X printed copies of the game assuming a popularity in line with other MMOs of the time period. Game sold out and was wildly more popular than expected. They had to bring the servers they had in reserve for the first year of the game online within days of release to accommodate the crowds (insane scalability) before they ran out of hardware and data center space. It is a game that is probably capable of supporting double or triple the current population- as long as you feed it enough servers and bandwidth. the world is compact and literally hundreds of thousands of people can be sitting at Light's Hope Chapel on any given day across all their servers- and advances in hardware make even AQ opening day crowds possible and mostly stable.

Second Life. Developed to have a single "world" for all players- but developed originally to handle about 10,000 concurrent log ins. They have had to jury rig the game engine to support the current 30-50k concurent log ins and every day is an adventure in playing with a notoriously unstable platform. The only way to scale the world is by a game engine overhaul (currently in progress) and adding in more physical space in the world for those additional people to go. Imagine if you could only have a maximum of 20-30 people in an area the size of Silithus before the server destabilized and users started crashing, and at 40 people the region was full. It's absurd. No amount of additional servers will give any one area in the game the ability to support more than 100 people (and those 100 only if they're naked and not moving around much). Technically they've scaled up to 50k concurent log ins, but the world itself will always be limited- and they've already admitted it was never designed to be a scalable solution.

Second life will never be able to handle a million concurrent users. Not even if you feed it all the hardware in the world.

WoW can scale, assuming hardware is bought, pretty much forever. There might be a limit eventually- but I doubt it's findable. I doubt they've developed a non-scalable arena solution. It's just not their style. They're simply choosing not to expand the hardware (which is pretty reasonable since they also want to encourage people to move up a bracket or two to even to load). They underestimated demand, but are content with the outcome.

Your misguided explanation of a race condition, and you assertion that the only way out is to have only one CPU doing all the matchmaking refute themselves.

Really? I'd be surprised if they could get the site selected and the permits issued in that time. I think it's sort of moot because I think they don't run their own data centers - but interfacing with another company can introduce delays of its own.

You confuse a simplified explanation with an incorrect one. Two entities doing the same thing at once, screwing each other over is a textbook race condition. The single CPU isn't the solution, it is the problem. The point is that having a billion servers ready to accept players is not useful when a single one of them has to parse the list of waiting players to find matches. Yes, there are ways to multi-thread matchmaking, but they are not easy. There are dissertations written on this stuff, and major companies invest huge $$$ to find solutions since it is one of the most pressing issues in computing today. So my guess is, they simply put a giant lock around the whole matchmaking infrastructure and called it a day. This works extremely well and is easy to implement as long as you stay below a certain threshold. Once you get past it, things get very slow. Sound familiar?

Have you ever had to wait in a queue to get your Google search results? Me neither. Why? Because they plan ahead. Blizzard did not. That is the bottom line, the rest is fluff.

Sorry, I totally disagree as well. I dont think a single CPU doing the matchmaking is the bottleneck at all. CPUs these days are fairly powerful, I could write an application to do this sort of matchmaking and it could match teams VERY fast with no problem.

It is vastly more likely that the problem is related to something else, i.e. the resources needed to generate 1,000 instances that contain only 2 players than it is to think that the matchmaking itself is the bottleneck.


Lets say your BG has 10 servers in it (huge BG!). Each server has 100 teams waiting to play at peak time. So it needs to match 1,000 teams to 500 pairs. I could write an app to do this well in a very very very short space of time. This is such a simple calculation, and it definately does not explain to me how we could have 15 minute queues last Friday night when I saw maybe... 100 players total waiting for all BG queues and ALL arena team queues.

And Aphyrax, you made this statement:
To me this means they cant create enough instances to host the matches due to limitations in their hardware (most likely). The same sentence could also mean they cant make enough matches between teams, or do it fast enough (i.e. can't match team A to team B).

I would suggest that you make an app that does this:
Checks the first team rating.
checks the list and matches to any team that has the same rating, or the first team within 2 points above or below.
Checks and matches against teams 5 points above and below.
Checks and matches teams 10 points above and below.

Increment it by 5 till a match is found.

Now that is a HORRIBLE way of doing it, but it would be a lot faster than me taking the first team rating and going through a list of printed sheets with possible team matches, right? But I bet I could manually match teams faster than some peoples 30minute queue times! It for sure is not a CPU limitation on the matchmaking of the teams.

Now reread your quoted sentence and think which is more likely.

You know how the Ancient Egyptians built a pyramid? I do. They put one block down. Then they put another block down. Then they kept putting blocks down until it was done. Simple.

In other words: You've boiled down the problem so much that the solution ignores too many factors to be of any value.

I recently spoke with a friend of mine (who will remain un-named) who works at blizz, and can confirm that there is code in place that modifies that wait time of arena's based on your /played. The longer your /played, the longer your que. This sounds insane, but here is the reasoning: if you have played this long, let's keep you playing just a bit longer. It is assumed that veteran players have a higher tolerance to waiting around, and newer subscribers arent yet used to the long wait times of the game.
He showed me some of this source code, it is reall obscure stuff. As a test he logged on to his level 19 twink and got an instant match, then got on his level 70 with 124 days played and sat there for over 3 hours. For some people I guess that would suck, but for me it is just more oppurtunity to watch Princess Mononoke on repeat.

So, this is the QueueQing thread?

Matchmaking is more than comparing two numbers. You have to transfer the character to the new instance. During that time the queue has to remain in lockdown (or so Blizzard said, none of us have seen the code). Maybe I explained it poorly, but the problem is not neccessarily that the matchmaking server is completely overloaded, but that it has to sit around idly waiting for the other pieces to do its work - which in terms of end result is the exact same thing as long as you cannot repurpose the server in the mean time, which apparently Blizzard cannot. But I stand by my statement. Having one billion arena servers does not help if only one of them can spin up an instance at a time. As such you reach a point of saturation where additional hardware has exactly zero benefit.

When you rapidly zone in and out of an instance you still get a sigificant load time, even though by the second or third time everything should be in memory, especially when you have lots of memory. So, what is the holdup? Their backend logic that is doing the transfer (or whatever else is going on). From what Blizzard said it seems they are doing the transfer while remaining under the matchmaking lock, meaning that if a transfer for example takes three seconds, the most they could spin up is 20 instances a minute, regardless of the number of instance servers.

This was never a problem in the past because you generally don't need that many instances since you spend more than two minutes in most of them. And when people started exploiting instances by creating new ones all the time they put an hourly cap on it. That way they prevented the system from becoming overloaded. They chose to not put a cap on arena matches, instead they went with the queue approach. But the end result is the same.

I really shouldn't spend any more time then this on CS 101, but given the nature of the problem, you don't need to lock the entire queue. You just need to lock individual teams. If you needed to make guarantees about the matchup - such as teams getting games in the exact order they entered the queue, or teams getting the absolute closest matchup available, then you might need to lock the whole queue. But that's not the case, so there's no reason you can't have several threads making matchups. Who knows, they may have given this problem to an intern who fails at multithreading, it wouldn't be the biggest mistake they've made.