I am looking for information regarding the mechanics of Amplify/Dampen magic since I could not find anything close to the answer I am looking for by searching here and elsewhere.

Specifically, I know how Amplify and Dampen work versus players and regular mobs(same as positive or negative +damage/heal) and was wondering if the same mechanics apply to elites and raid bosses. The reason I am asking is because if the same mechanics do apply than Amplify and to a lesser extent Dampen Magic are somewhat under-utilized in the raid environment.

For example, against Prince, would Amplify Magic make Shadow Nova hit for 103 extra damage(3 second cast = ~86% of +damage), or are the mechanics modified? If they are unmodified which is what I would assume, then Amplify Magic is fairly powerful at least on tanks on many fights where there is none to a moderate amount of large damage magic attacks. Dampen would be good in the other case of many frequent low damage magic attacks.

The +heal from players is applied before the coefficient, so a +1000 +healing priest becomes a +760 +healing with 240 reduction (not sure about the actual value got it off the top of my head). However mobs get direct reduction which means a 700 spell will become a 680 spell with 120 reduction. If you look at it more deeply, you'll see that there's some breakpoint of damage taken per spell VS amount and type of healing done to you where dampen/amp become useful. On very low damage taken fights (such as curator, illhoof (works on chains too per tick), dampen magic is a no-brainer I think. On fights like grul and mag (after adds are dead) amp magic is a no-brainer.
It doesn't seem to affect bandages/VE but I didn't really verify.

To the best of my testing, Amplify and Dampen Magic work exactly like a +/- Spell Damage and Healing bonus to anyone attacking that target. So if your tank has +360 healing from amplify magic, a healer with +1500 healing would have +1860 healing against the tank.

Dampen Magic used to reduce damage as low as 1, but mages would dampen magic themselves and AoE a dozen or so Air Elementals in Silithus, so it was nerfed. I believe Dampen Magic cannot reduce the actual damage dealt below 50% of the base. Dampen applies before resistance checks.

Amp/Dampen have no effect on Vampiric Embrace and Bandages since they get no benefit from +healing.

When in doubt, your tank should have Amplify Magic.

Amp/Damp also work on certain abilities, and not on others. For example extensive testing (back before 2.1) showed that the nether burn from Netherspite was not affected by the +extra spell damage taken from Amplify Magic, making it only beneficial (untested on the breath)

Yeah, as far as I've ever been able to tell it works exactly like +/- dmg/heal. On the other hand, lots of mob abilities have a lot of weirdness going on in terms of what their coefficients are and how they react with this spell. I've never been able to find a systemic rule to explain it; it seems to largely be left up to the whimsy of whomever at Blizzard coded the spell/ability in question. I just test 'em out on each new encounter where they seem like they'd be useful.

By the buy, Dampen really shines on Illhoof. The full effect of the spell is applied to each tick of the demon chains, not to mention totally trivializing the imp damage until Kil'rek throws up a debuff. The trick is to convince the healers that it's not the end of the world if they lose 180 +heal on a few non-tank targets.

Most mob spell don't follow the mechanics for player spells, concerning how +spell power affects final spell damage.
For most mob spells, it's either 100% or 0% benefit. Also note that the calculations are made before damage multipliers and resist.

So, imagine an elemental hitting your bear tank for 2k nature damage (pre modifiers) at +25% damage debuff, with a 50% resist.
So 2k hit => 2500 hit => 1250 hit post resist.
With -100 dampen magic:
2k hit => 1900 hit =>2375 hit => 1187 hit post resist.

Non-pysical melee attacks and most spells get 100% benefit. Have a mage spellsteal a shaman mob lightning shield and it will do 1k damage when being hit. Same with the immolation Auras in Mechanar and Alterac Valley (both fixed).
On Hakkar's poison, dampen (-90) removed 9 damage from every DoT tick, it looked like the -90 was spread over the duration.

As to where it's useful and where not:
It was awesome vs. Patchwerk, it was a huge help against Maexxna (before HoTs stacked). I have mixed feelings about most BC fights, that mix both, huge physical and huge magic damage.
If magic damage comes in small steps, amplify will most likely hurt a lot. If the magic damage comes in large chunks, amplify will make healing easier - as long as it's not spikey enough to kill the tanks/people.

Amplify on hard hitting bosses (I use it on myself when tanking Maulgar adds actually, since the damage shield decreases the damage increase), dampen magic on Illhoof.
On most other fights, argueing with your tanks and healers will usually take longer than just killing the boss. Even I don't know what to prefer in most fights.

I think the basic rule of thumb for amp is: "Is your tank getting healed more than 3x between the monster's magical damage hits them?" Answer is yes for fights like Attumen, Moroes, Big Bad Wolf, Prince Malc, Gruul, Mag after adds are dead.

The basic rule of thumb for damp is "Is the target going to be taking quick small amounts of spell damage 2-3x times before they receive a heal?" Answer is yes for fights like Maiden (melees), Illhoof.

It's spread out over time.

For spells cast by other players, Amp. Magic and Dampen behave just like +spell dmg effects. So the effects of Amp or Dampen are spread over the duration of the HoT.

Enemy mob abilities, OTOH are inconsistently affected. Certain HoT/DoT effects such as the General's Aura during the Rajaxx event or Ayamiss' poison receive the full effect of Amp/Dampen per tick. Other effects such as most enemy casted spells treat Amp./Dampen like +/- spell damage. And still other spells and effects are completely unaffected by Amp./Dampen.

I am trying to explain the ROI(return on investment) to our raid squad in relation to Dampen Magic on VashJackson...the best way i have come up with is that with the raid receiving less damage all the time on the lightning and poison bolt damage, which gives the healers the opportunity to be more efficient healing the raid.

The payoff off for Dampen magic is that you receive less damage every magical attack, the healing penalty is paid only when you get a heal...which means that you should only be paying a penalty equal to or less than the initial gain of less damage taken over several magical attacks.

The additional gain is that healers will have more time to decide when and how to heal the 5 or so peeps getting hit for 2-3k lightning damage.

In my opinion, there should be an opportunity for Amplify Magic or Dampen Magic in 90%+ boss fights. They are definitely under utilized!!! And they are there for something......also the mana cost would lend to thinking they are worth more than players think.

I use Dampen essentially never. There just are no fights that are non-trivial where the extra damage reduction is not outweighed by the healing reduction.

Amplify on the other hand I will use pretty much anywhere. I don't put it up for Kael (although I still argue that it is a benefit) or Hydross but I'll hit the tanks and generally paint it around otherwise. I am lazy however and on iffy fights like Vashj or Rage I don't bother. Besides, I hate people arguing about how they died because they were amped and got no heals. I'd personally blame the lack of heals but that's never a recipe for good healer relations and yeah, I like to not be the last person chosen to live.

Damned if I wouldn't love group versions =/