I'm sure many raiding guilds go through this but how do you determine what classes you take for a specific 25 man instance?

Do you take at least 2 of each class? Do you not take a particular class?

I've always been against raid stacking and wanting a class balance to a raid long as the classes are available to be taken.

We take our best players - simple. If an encounter is a hell of a lot easier with less melee DPS, then we may sub out some rogues for competent ranged DPS. But on the other hand, I would rather have one of our really good rogues on a melee unfriendly fight then a sub-par Mage/Lock.

We do often sub out a healer for DPS or vice versa depending on if the encounter is healing intensive or DPS race. But I wouldn't call that "Raid stacking" as its making maybe one or two changes for an encounter.

2 prot warriors
4 dps warriors
2 resto druids
1 feral druid
2 elemental shm
1 resto shm
4 mages
3 locks
5 paladins
3 holy priests
3 shadow priests
2 hunters
3 rogues

We basically pick our group from there. There's always some differential in skill but our guild is small enough to avoid too big of falloffs - it's mostly apparently in gear when deciding between a veteran rogue that's decked in kara/gruul/mag/ssc gear vs one of our new draenei shadow priests who just dinged 70 a few weeks ago.

It varies depending on fights, but we usually rock 8 healers, 2 prot warriors + 1 feral druid, and then 14 dps from the mix above (almost always 2 shadow priests if we can manage it).

I believe Gurgthock posted in another thread, something like this, in regards to an (almost) ideal setup :

[3] Druids (2 Feral, 1 Resto)
[2] Hunters
[3] Paladins (all Holy)
[3] Mages
[4] Priests (2 Shadow, 2 Holy/Disc)
[2] Rogues
[2] Shamans (1 Resto, 1 Ele or Enhance)
[3] Warlocks
[3] Warriors (1 prot, 1 hybrid, 1 pure DPS)

We usually follow a similar if not exact template, mostly bringing one more warrior to the expense of a feral druid and sometimes an extra warlock for a mage, or an extra rogue/mage for a hunter. Healing wise, we use anywhere from 7 to 9 healers depending on the nature of the encounter, more specifically including 3 paladins and 2 shamans, filling the rest with restoration druids and/or holy priests.

I'm not at all a fan of raid stacking myself, in fact I consider it quite terrible in terms of having to be forced into it, but truth be told, most of the encounters in TBC are so tightly tuned in their current form, there is just no room for compromise, if you desire some solid and steady progression that is.

As for maintaining class balance, I think its quite clear that some classes are simply more important, so to speak, than others, usually because of their added utility, e.g shadowpriests. This of course creates a problem in the long run (more of a social one), leaving several people from classes that were considered "a must have" pre-TBC but not as good today, on the bench, but hopefully it will be addressed by better tuning the existing encounters in addition to talent reviews and of course creative content design.

Paladin buffs are too big a deal to justify not having 3 in every raid group, despite how good their healing is.

Well that depends on your point of view and exactly how "skilled" your healers are in terms of reaction time. Then there is overall utility to consider. Let me clarify. The reason you normally use 2-3 paladins besides their healing capability (which is one of the best if not the best before the patch arrives), is of course the blessings. They are just too good to be ignored if you have a reasonable paladin availability. Shamans on the other hand are excellent raid healers (or even MT healers) and their totems and heroism only add value to their efficiency. That's about it when it comes to "hard numbers", other than that you will have to find a good balance that works for your particular raid group and most importantly for the specific encounter you are attempting.

Well, you seem to have figured out the answer yourself. Hots are good, in fact, in this expansion they are excellent if you want to keep someone alive, but that doesn't mean that they don't have their limitations. Direct Heals are more accurate than Heals over Time, in the presence of uncontrollable factors, even considering the downranking nerf. These factors include other healers using alternate heal styles and random incoming damage to your heal target. Things such as crushing and critical blows make heals over time actually heal more effectively (since they can tick without necessarily being overhealed all the time) but sometimes they lack the magnitude to heal the target accurately, that is to say return the target's health to a reasonable amount in a timely fashion, before he gets the next damage spike that will kill him.

Obviously, Gruul is going to hit rather hard after a certain number of growths, so it seems to me that in your particular case it wasn't a setup problem but rather a healing strategy problem which you eventually sorted. Whether that particular setup could work in other encounters is entirely up to you to judge, however I would personally recommend a better mix of classes due to the reasons I mentioned in my original comments.