illfittingshoes

All the Nighthaven Defenders I saw were jerks than just kinda chilled in a large semicircle a ways behind eranikus :angry:

 

XI-

I understand your point gurg, and its a good one, but if you are only transmitting to each player the data for 40 other players, instead of 300 other players, it cuts down on the network data transferred, but since we don't know where the bottleneck is I guess we can really assess the benefit or loss from these things.

 

Slug

I've played pretty much everything except L2 and I'd say, not really. Xi put it well enough. The different server architecture has handled event update choke differently and to different levels of success, but it remains a problem in some form in every game. Even the 3d games with less demanding hardware requirements seem to hold the same tendancy to crap themselves when large numbers of people gather in the exact same place at the exact same time.

And keep in mind that these periodic event calculations take place on top of the pre-existing calculations and updates that occur in most cases more frequently than every second. As I understand the game, it polls your client for updates and your client also actively sends. I'm not sure how often the serverside polls occur or what they're tied to, but the user events seem to be sent directly. It would lag even if there were no mobs at all and the movement was minimized, strictly from standard polling and updating to your PC of every individual within "pre-loading" range of you. (ie. In WoW it seems your client absolutely must know the health, mana, and state of everyone within pre-loading range at all times.) Adding the extra events on top just makes it exponentially worse.

World events have always been excellent ways to crash servers in MMO's, which is a shame. They're the best sort of event you could have in a game of this sort. There were a number of games where we would hold guild meetings in chat, alone, because trying to get everyone together physically would lag the whole zone we did it in, etc.

It has always seemed to me that the updating is the problem, not the graphical lag. Most machines don't have a huge problem displaying the gobs of polygons correctly at low detail levels, but the pipeline can only support so much information at any given second. When you go from 50 people/NPC's all updating every second to 300 people updating every second, it becomes massively problematic to follow those events and return user-specific versions of them to each individual involved.

If all 300 people stood perfectly still, didn't interact with anything, and just had their characters logged on in the same location of a game such as this; it would still be laggy. The noticable difference between "pre-event" and "during-event" is the fact that you typically quadruple+ the number of individual events you're causing during anything that's not simply movement. The game would be polling for updates and transmitting them, whether you're actually doing anything or not. When it doesn't have to update much and you're only sending movement-state updates to the server, it won't be as slow as when it's having to update 10 things related to your character on top of those events, obviously.

Take the Eranikus event as an example. Every single person in range of that event who was a simple observer, standing there and doing nothing at all, was causing a tiny bit more choke on the pipeline and updates. Even if they were out of range of the AE's, your personal client was still needing and requesting updates from them, as I understand the networking in WoW. They weren't getting hit, many weren't moving, even, but you had to know their status and they had to know every one of ours.

I kept getting feared during the event only to find myself staying right where I had been standing or running 2 paces and stopping every time. The server and mobs I had agroed on me seemed to lose track of where I was during the fear, since so many events were happening at once. I ended up not moving at all and getting a complete agro wipe because they got confused. This suggests some sort of an overflow valve in their coding, allowing for skipped "lesser" updates in matter of emergency and imminent server crash, but who's to say. If that's the case, I'd think they're heading in the proper direction to handle the issues.

When we were still in the lesser-modem days, I could understand the concept of the update choke and thought it would get better over time, but it seems like one of those problems that is more of a "nature of the beast" networking issue than an issue that a simple faster connection on all end-users would solve.

The only thing I could think of offhand to reduce lag of that sort would be to add a filter of some sort for slower overall client-updates that turns down the number of events-per-second allowed to display on your home screen relative to the people around you. If you could eliminate the need for frequent non-vital graphical updates (animations, PC locations) during periods of high strain and only update the vitals (health, debuffs, mob location) in real-time, it could fix some of the 2+ minute casts on instant spells. The trade-off would be in trying to click or interact with a PC who turned out to not be where you thought they were, so that manner of fix would create all sorts of new problems in solving the slow-down.

 

Raylen

I hate to say it but EQ actually was pretty good for this. However, the world was broken up into zones and wasn't seamless in the "outside" world like WoW is which helped with the server load.

Edit: an example would be the Kithicor event back in the early days of retail where the Plane of Hate was first opened. I remember Brell Serilis had over 350 people in Kithicor to watch it happen.

 

◊ Maniq

Another thing I was going to add, is the limiting factor of clients.

For each connection, the server has to wait for a reponse from the client, now under normal circumstances, these will be fairly minimal, but you are now in the situation where the server is waiting for a "What did you do in the last 0.5secs" from the client, if each client is lagged then you will slightly increase the time server latency, as it has to wait a max of y seconds for a response.

Now, most client <-> server relationships will account for this, but, in this case where literally every client has to know what everything else is doing, this delay becomes more appreciable, as you need to parse whats going on around you and also send out the updated information.

The point I was trying to make regarding only resolving information for players based on proximity wasn't for graphics purposes, it was for shear "what can the client afford to not know about, so I dont have to encode this data and ship it halfway across the world." - The client renderer can at this point make something up for that person to display graphically, a tauren with it's thumb up its ass, then, when its determined that the client needs to resolve that data, warp them/display them with the casting animation.

Ultimately this is about streamlining Client<->Server communications and not passing any slow points onto all the other clients involved. This by no stretch of the imagination is very hard to do, having maintained a large platform with millions of concurrent connections in the past, I can most definetly say that it's not as easy as we would all think it is.

 

Demitrius

I'm pretty sure 'Mal'ganis' is entire server rack among many in a very expensive data center. However, saying "oh well WoW has to do X calculations" doesn't really prove much of a point because many real world systems do calculations several orders of magnitude greater while having to maintain 100% uptime. Credit card servers, banks, major data hubs to name a few, have to process billions of bytes per second and reliably work with each other. To them, one minute of downtime is equivalent to $1 million dollars of lost business.

I think WoW's techinical issues stem a little bit from its hardware infastructure and a little bit with Blizzard's lack of experience in coding MMO's. Game companies do not have the vast resources necessary to design and implement redundant server environments nor do they have the techinical knowledge to code them properly.

 

hamlet

alot of those servers are basically doing the same thing over and over again. WoW is a bit more complex.

http://i16.photobucket.com/albums/b3...mai_hime37.gif

 

◊ Zoid

WoW has two fundlemental engineering problems that make this an issue.

1. The server is multithreaded, but the threads are based on area, not connection. This means everyone in Orgrimmar is being served by the same thread. The people off in Tanaris have a seperate thread and aren't lagged. The amount of "area" a thread covers changes based on population movement (Orgrimmar might be one thread, but Tanaris+Un'Goro+Silithus might all be another thread) but its still promixity based, which means when everyone gathers together, its the worst case.

2. AI is done on the same server, so its contesting with players for the same thread slice in the area.

There are what I think two ways to solve this:

1. Thread based on connection, not by location. This means that if you have 3000 people connected to a server on a quad-cpu machine, you get 750 connections served by each thread. It's not quite this simple and there are some lock contention to deal with between interactions, but its much more scalable.

2. Have the AI run as a seperate process or entirely seperate thread. AI should work sort of like a client that's just controlling thousands of mobs.

Obviously, I've been giving a lot of thought to designing something like this lately.

 

◊ Maniq

Unfortunately, both systems are only comparable so far.

In the case of a credit card system, I guarantee you, that the companies know exactly how many transactions will be occuring, at any time of the day or night, my guess would be that they know this literally down to the very thousand. I used to admin a large email platform, handling 20million+ emails a day and at any given point, I could probably guess down to the ten thousand how many mails had been sent in the last hour, for any day of the week.

The credit card companies have stuff like this down to an exact science.

Blizzard will be able to do the same, to a certain extent, they have the demographics of who will be playing when and what load they can expect. When things differ from this their model breaks down, as it's not designed to be that adaptive. A case in point would be farmers, generating multiple instances to dupe items etc.

For the most part, blizzards infrastructure copes OK, we are not for the most part plagued by lag spikes, I'm guessing that of late we have been seeing lag due to 2 things, Christmas and a greater number of people farming for the war effort.

The servers crashing and not being able to cope with world events without a metric tonne of lag are exceptional cases, its very hard to prescribe for these occurences, I think that a fundamental planning mistake was made in having the event on one of the world servers themselves, it should have been instanced (be it seemlessly on the flight path or through a portal similar to the way battlegrounds are handled).

The problems in these cases are the games such as WoW aren't comparable, you can have a server application, be it web server or database, where you can move 'things'[1] about and due to the nature of concurrency it be seamless, with games that require constant updates, its not as easy.

The clustering in something like 60%[2] of Fortune 500 companies, use a product by Veritas, this is an amazing piece of software. I would be shocked if Blizzard didn't employ it frankly, either that or Sun's equivalent.

Un-noticeable redundancy is hard to acheive when you have a model containing multiple connections, each time you want data, you open a new socket, that's infinetly easier to do, than if you rely on one single connection for everything.

You have one piece of the cluster fail over to a different machine, it's very very hard to make that failover transparent, it's possible but when you are talking about say 7k connections, it gets hard.

You are right, I don't think Blizzard had as much experience in the MMO market as we all would have liked, but, they have also not let us down in a huge amount, WoW is over a year from retail and they are still working out the problems, I think things such as the problems they saw at release and the frequent server crashes on this world event will be rectified.

[1] Without getting to involved into clustering technology notation, you have groups which comprise resources, one group could be for Kalimdor, which would have a game resource, a chat resource and a database.

[2] Or so they claim :)

 

XI-

Well here's the thing maniq. Your hardware was probably designed with maximum thoroughput designed I would assume because you ever want it to fail and if it did there would be BIG problems. Blizzard's hardware is designed down to the average and gradually strains until it becomes unplayable and crashes. But for blizzard there's really no reprecussions from this action, its merely lol we're rebooting the server maybe we'll tell you about it some time next week if at all.

 

Fignuts

Hey, I thought the original few posts in this thread were excellent, so I just wanted to weigh in with my own dumb ideas:

I think a truly great MMORPG would provide an infrastructure robust enough to allow users to add their own content, and even extend on the infrastructure. Yes, we're on our way to Stephenson's Metaverse. WoW is another big step forward, they have designed a fanstatic infrastructure based on great standards (LUA, XML, etc.), but obviously they stop short at allowing users to modify the game. The infrastructure isn't quite ready for it. Things will get quite interesting when it is ready, and users can not only extend the game and infrastructure, but profit from doing so.

One of the best games I ever played was Natural Selection, the mod for half-life. It allowed random PuG's to play a very strategic real-time FPS game, a sort of Starcraft-meets-Half-Life. The main ways it enabled PuG fun were via integrated voice chat, and with the integrated kick/ban voting systems. AV (and BG's in general) could be "fixed" if Blizzard would integrate voice chat into their game, and add a robust player vote/kick moderation system to allow teams to organize themselves, and to prevent the plague of reputation whores that sit in the game just to earn rep without playing.

 

Fignuts

PS - I agree, empty timesinks detract from the game experience. They are obviously Blizzard's chosen solution for keeping the user paying subscription fees for as long as possible, in a scalable, easy manner. Endless gathering/killing quests, reputation grinding, etc. are all big slaps in the face, and only serve to highlight the limitations in the gaming infrastructure. Their tactic is an endless cycle of dangling the carrot (or in this case, the latest round of epic gear) in front of the players, in return for their continued subscriptions, offsetting it just enough with "new" content based on their madlib system of grinding X mobs for Y BoP items to get to the next step in the questline. Again, the only way around this that I see is allowing users a robust system for generating additional content and improving the game itself. I look forward to that very much, until then, I'll continue to amuse myself with trying to get to that carrot.

 

phyra

Excellent comments Fignuts. Let me try to take this opportunity to try and steer this back towards the original discussion with my promised ruminations on dynamic content generation and the lofty goal of 'infinite' content :ph34r:

I'm reluctant to criticize WoW for its design decisions constrained by present technical limitations (such as the return of truly epic-scale world PvP, functional world events, etc), since as an outsider I can only scarcely imagine the variety of technical issues in play with any accuracy. Besides, even if we accept the de-facto technical limitations inherent in Blizzard's designs (about a hundred active players in one small area at any given time, anything too far beyond that and things start to go downhill quite quickly), it's still definitely possible to imagine plenty of fun game-playing experiences. Since Moore's Law will always guarantee us an eventual solution to technological limitations, what still stands out most in my mind is the lingering problem of content generation.

The very obvious dearth of new endgame content and their failure to provide a consistent stream of new challenges both to the casual and the hardcore raider alike has been the shortcoming we all seem to agree upon. Blizzard set an incredibly high standard for themselves with the incredibly expansive overworld and the wealth of midgame instances and once this was all eaten away by the customer base, they found themselves pushed back into a corner with nowhere to turn. Timesinks such as Battlegrounds reputation rewards, the Honor system, and random drops off raid instance and outdoor bosses are Blizzard's chosen way of stretching out existing content in a "scalable, easy manner" (as noted by Fignuts) in order to make their extensive development time worthwhile. After all, under a subscription-based system which purports to deliver a consistent stream of fun and engaging game-playing experiences to its loyable customers, it would be quite inviable if Blizzard's design team spent three months constructing all of Blackwing Lair and it was consumed and made obsolete by a majority of its playerbase in a few short weeks.

So from this point of view, what else could possibly be done in this situation? What are the alternatives to generate the illusion of 'infinite' content to hungry consumers frothing at the mouth?

One answer is to give the consumers the power to produce their own content, and to structure this community-oriented production in such a way as to improve the game itself in the process. The tremendous success and incredible longevity of Half-Life is thanks in large part to its thriving 'mod' community, from which amazing content such as Counterstrike were generated. Plenty of earlier text-based MUDs also thrived on this model.

I don't see user-generated content as the only possibly way to go here. Another method of supplying in-house content is to simply develop more novel systems for reusing existent content in new ways - or rather, by placing dynamic, scalable factors in the designs of the content itself (preferably ones that favor some lingering element of skill, creativity and coordination over stale, rote performance and mindless repetition). Something that makes you want to go back and attack that dungeon again a second, third, fourth time with the same intensity, fervor and fear as the first time you stood up against the boss. Perhaps each successive time you clear Molten Core (regulated by reputation increases perhaps), each boss gets a little buff and the rewards scale along with the risk, so guilds can boast about having killed Scale 13 Ragnaros, and there will always be something greater for you to bang your head against as the performance of your raid group comes ever closer to perfection.

The buffs can be simple, such as a linear increase in attack power or somesuch, or they could be roughly tuned as separate tiered bosses - perhaps at Scale 5 the encounters stop scaling linearly, but instead the boss gains a fresh new ability that must be dealt with? What I'm getting at is that these adjustments don't require brand new locations, artwork, design, lore, heaps of new programming, etc. in order to be implemented, but they still offer the possibility of fresh, new, exciting content for a relatively cheap investment.

As another possible scaling system, perhaps an instance scales in difficulty in inverse proportion to the amount of time that has elapsed. So maybe a six-plus-hour Nefarian battle would be relatively tame, while a three-hour clear would give you a shot at something never-before-seen-ferocious, with the corresponding chance to have a victory go down in history as something truly legendary. How about that for an incentive for everyone to stay focused and engaged during your weekly instance runs?

Finally, something as simple as a high score list or performance chart is a formidable source of an endless supply of dynamic content. Social recognition itself can be an incredibly valuable and desirable reward, even moreso than the most powerful item drops. This aspect of game design is overlooked far too often. The monumental success of the 1978 game "Asteroids" was thanks to the invention of the high score table which literally spawned an entire arcade community centered around the systems of social recognition and intense competition that these three initials generated. Again, feedback of any sort can be a very powerful device, and WoW would only improve if it replaced or complemented its 'grind rankings' with systems of feedback-based competition generated and sustained by the player's interests themselves. We see a strong spontaneous desire for much of this competition in message board posts such as "10man Onyxia kill, can anyone do it with less?", or "1.5 hour MC clear, can anyone do it faster?" - creating an incentive by standardizing some of these ad-hoc competitions and providing some mechanism of official recognition would go a long way towards generating a large amount of reusable, dynamic content from existing material.

On that note, was anyone else as pleasantly surprised as I was with the Red Shard component of the AQ quest chain (five or so hours to kill Vael through Nef)? Astonishingly close to what I have been hoping to see all along. I hadn't seen my guild so pumped and excited to run BWL in ages, and from what I heard our 3.5-hour clear was our fastest and most focused by far, since we actually had an incentive to perform well. (I was on standby the entire time but unable to get a chance to participate, since my guild's regular raiding routine has excluded me ever since I opted out of the tired weekly instance grinds.) Anyway, let's hope this isn't a one-time-only thing and that we see this as a shift away from the old passive raiding ethic towards something more challenging, competitive, and fun.

 

phyra

Oh, and I forgot to mention that PUG moderation sounds like an interesting idea and could certainly improve the quality of pick-up Battlegrounds groups. Nothing is more frustrating than being helplessly stuck with a bunch of noobs or AFKs in a game. Although it would have to protect against individual griefing and guild bullying, I think it would be possible. I would certainly like to see voice chat integrated into the WoW client as an extension of group and raid functions, but I could also see existing third-party voice chat programs becoming more widely used with an increased caliber of moderated, competitive PUG play, so this is more of a welcome afterthought than a necessity.

As others have mentioned, a more pressing issue with improving the quality of Battlegrounds play is more transparency in the queuing system such as the ability to choose your opponents for arranged matches or simply to avoid generating obviously lopsided matches. Penalizing AFKers is a band-aid solution to a deeper design issue - did Blizzard even think to consider the reasons why AFKing in the middle of a losing battle or against an obviously superior opponent was the smart and/or most enjoyable thing to do?

Take an example from one of the oldest and most celebrated of PvP designs. Most games of Go are never played out to the bitter end, they are halted as soon as the losing player acknowledges his inability to regain control of the lead and he immediately resigns. Beyond that point the rest of the moves would simply be a formality, a waste of time and effort. If this clear superiority is recognized at the beginning due to the relative strengths of the players, there would be no reason to even start the match. Hence the game's elegant handicap system that encourages cooperative play and provides a constant source of balanced challenge and competition. The Battlegrounds designers could learn much from this.

 

• Greybone

Computer science is a marvelous thing. I could write a sorting algorithm that would bring the finest of computers to their knees when it would try to sort your grocery list alphabetically. (atleast if you had a lot of groceries to buy!)

I'd be surprised if blizzard didn't compute their runspeed based on the number of entities in a given event, but it's clearly not a linear function. As you add more people, more things needs to be crossreferenced, LoS checks have to be made, health and damage checks, a billion things like that. Doing it for 8 people might take 16 times the calculations as doing it for 4 does.

It's not a easy problem to solve, and sometimes I think the easiest thing would be just to cheat:

Once you hit a certain threshold, skip stuff. Send event triggers to every client that a shadowbolt is heading for them, but don't bother updating the hp for more than half of them. Calculate how much damage a AOE does to the clients, but stop at X clients. Skip LoS checks when things starts taking too long, assume the rest is out of LoS for damage taken.

It's not right, it's messy, but atleast you'll end up with something that seems fluent to the client.

 

Jo_

very interesting discussion. as a computer science geek myself I see two different reasons why big events a lĂ* eranikus fails:

1. the event in itself

2. the power grid problem: how to handle peaks in demand (in this case cpu/connection)

number one is the fundamental "how much resources is required?", if we have 400 players, all these npc's that needs pathing, agro lists and everyone needs to be informed about what everyone else is doing, how much cpu does it require to calculate this and inform everyone about the results? since we don't know the design it's hard to guess but the question is how many % of the total resources of a server would a eranikus event with 400 people require to run smooth (from the server side perspective). 1%, 5%, 20% 5330% ? depending on how they designed it the cpu required to do a 400 people eranikus event might not even exist on the planet (I'm sure they made it better at scaling than that but you get the point hopefully)

number two, how to allocate resources and be able to handle peaks is alot about trying to get away with as little unused capacity as possible at all times. even if the event itself could be done with 400 people with enough cpu the guy saying that they should scale the servers according to a 1 per server event would get fired. I think smart design is the key here to keep both players and investors happy, more about that beneth.

thoughts about solutions:
at this stage it's naive to think that big fundamentals can be changed in this game but since this is also a general problem with some eranikus raid in the game wow serving only as one example I'll try anyways.

-smart design chooices for resource allocation: shared resources used only when needed. these peaks are big but lasts for a short period of time, having one shared resource for many servers to handle 1h each month of absurd activity wouldn't have to be cost-inefficient if done right. presents design issues but nothing they couldn't deal with.

-the event itself: Someone said cheating and that's absolutly a good solution if you can isolate what's important and what is not crucial, kindof like a priest in our raids. better communication between the visionaries and the developers would also help, they shouldn't implement an event that will crash the server most of the times. everything can't be tested but they really should be able to forsee how far from a "ok" solution they are.

-workarounds: some games have tried isolation and it's allready in this one in the form of different continents and instances (which is on a shared server which means sharing resources between servers might not be be that far away after all). maybe they should just consider having a cap on zones? make wind riders protest when you want to go to a zone with 500 people in it or/and put up a blue wall in the cave to moonglade to keep people out if there's too many people. these are all compromises but that's what computer science is all about alot of the time.

anyways, big world events like this presents both the most epic experiences for players and the toughest design issues for the developers, it will be very interesting to see how they deal with it. thank you for your time.

 

Fignuts

Wow, these are all great comments and ideas phyra, it sounds like you picked the perfect PhD program! You've certainly convinced me that it's moot to discuss the technological limitations of a game and its infrastructure, especially when one can contribute outstanding, implementable improvements like these. I especially liked the reference to Go, in regards to the BG's AFK syndrome (and their "band-aid" deserter debuff "solution"). Perhaps allowing a team concession vote would be a good addition, or maybe just another band-aid.

I think integrating voice chat would be an important step in preventing lopsided BG's games by making sure PUGs have access to the same organization capabilities as organized groups. Assuming a more robust BG queuing/ranking system existed that would make each game as balanced as possible in all other respects, the team using voice-chat would still have a clear advantage over the team that wasn't. The only way to prevent this lopsidedness is to either prevent PUGs from playing organized teams, or to integrate voice chat into the game to allow equal footing. (Hmm, possibly a good idea for an addon--allowing Ventrilo or Teamspeak users to quickly coordinate PUGs onto their servers). Currently, it is not easy to quickly organize a PUG team into using a third-party chat client. The necessary task-switching, program installation, reconfiguration, etc. are all too much hassle for your average PUG to even consider doing in the 90 seconds before a BG begins. Another problem with using third-party VOIP clients is that is that Ventrilo or TS servers cost money to run, and represent limited, privately owned resources--server owners generally don't want to give access to their servers to random people. The only feasible way around this (that I see) is via a centrally managed, integrated infrastructure (perhaps available to clients for an extra fee to cover its operational costs).

That PUG moderation idea came from my experiences playing Quake and Half-life mods, which usually evolved their own moderation systems as they matured, if they weren't initially designed with one. They usually had their own set of problems, but they definitely improved the games, and robust systems worked pretty darned well, using a two-thirds (or higher) consent requirement for passing a vote, to prevented bullying and abuse, along with other abuse-prevention strategies.

 

hamlet

bumping an old thread to put something relavent in it.

http://www.ucomics.com/foxtrot/2006/01/29/

http://i16.photobucket.com/albums/b3...mai_hime37.gif