Ok, I have a feeling it's Marked for Death that's interfering somehow. I'll remove all my talents and shoot a few shots with a fixed damage range weapon every time I add a talent point to see where it goes wrong with the predicted damage.
Sanity Checks
Settings:
Disable Averaging - Yes
Disable Target AP Buffs - Yes
Using the fine light crossbow (29-29 damage, 2.7 speed) I have 2037 AP, 22.05% crit. I have RED in my helm. No HM on target.
My predicted Auto-Shot is 317,7 non-crit, 654,5 crit.
Observed damage 317-318 and 654-655.
Predicted Steady Shot is 397 and 818 crit.
Observed: 396-397 and 817-818.
My predicted Serpent Sting ticks for 213,5. Observed ticks 213-214.
So far correct. Arcane Shot and Multi-Shot are also spot on.
Now I take 5/5 Lethal Shots and 5/5 Mortal Shots. Predicted Arcane Shot crit is 1376 and predicted Multi-Shot crit is 1082. Observed values still match.
So much for the sanity checks. Mortal Shots at least is not the issue.
Aimed Shot
Now for Aimed Shot; predicted values 458 and 1088. Observed values are 454-455 and 1081-1082
The observed values match a non-normalized weapon and ammo damage contribution. Predicted values without normalization are 455 and 1082. If noone comes with any counter evidence in this case, I'll adjust the calculations for the next version.
Intermediate Talents
Next I spec
this build to check whether the passive talents are working correctly.
New stats: 2537 AP, 27.7% crit. TSA is active. No HM on target.
Predicted Auto Shots: 396,7 and 817,7.
Observed: 397 and 817-818
Predicted Steady Shot: 482 and 1147
Observed: 481-482 and 1147-1148
Predicted Serpent Sting tick: 303,4
Observed: 303-304
Predicted Aimed Shot: 543 and 1292
Observed: 543-544 and 1292
Predicted Multi-Shot: 543 and 1292
Observed: 543 and 1291-1292
Predicted Arcane Shot: 686 and 1631
Observed: 686 and 1631-1632
So far so good.
Next tier talents
5/5 Master Marksman
2/2 Rapid Recuperation
3/3 Imp. Steady Shot
This build
No changes in damage (didn't expect any of course).
Marked for Death
With 5/5 in this talent, I run the tests again. 2537 AP, 32,7% crit. No HM on target.
Predicted Auto Shot: 416,5 and 858,0
Observed: 397 and 817
First puzzle, but not long. Bug in Marked for Death calculations not turning off if there's no HM on target. After fixing that I come to:
Predicted Auto Shot: 396,7 and 817,1
Which matches the observed values better.
Predicted Steady Shot: 482 and 1147
Observed: 481-482 and 1197.
Puzzle #2. Crit values are higher than expected. Reading the Marked for Death talent again I observe the following:
Effect 1: Increases your damage done by your shots and the damage done by your pet's special abilities by 1% on marked targets,
Effect 2: and increases the critical strike damage bonus of your Aimed Shot, Steady Shot, Kill Shot or Chimera Shot by 2%.
Effect #2 is not dependant on the target being marked (this was reported a little back in this thread as well).
Adding 10% crit strike damage bonus to Steady Shot leads to a predicted crit value of 1213, which is close but no cigar.
So I check the calculations for critical strike bonus and re-do them in the following way:
(100% + Mortal Shots + Marked for Death) * RelentlessEarthstormBonus
This leads to a crit damage bonus of 148,40%
482 * ( 1 + 148,4% ) = 1197,3
This seems to be the correct value.
I continue testing the other shots:
Serpent Sting predicted tick: 303,4
Observed: 303-304
Aimed Shot predicted damage: 543 and 1349
Observed: 543-544 and 1349-1350
Arcane Shot predicted: 686 and 1631
Observed: 685-686 and 1631-1632
Multi-Shot predicted: 543 and 1292
Observed: 543-544 and 1291-1292
With that, it seems at least Marked for Death is working without a HM on the target.
Next I put up HM. I have the 110 AP bonus and 22 from the Glyph of Hunter's mark. AP tooltip is 2537 and actual is 2667.
Predicted Auto Shot: 434 and 894,1
Observed: 434-435 and 894-895
Serpent Sting: 310 tick
Observed: 310-311
Steady Shot: 525 and 1303
Observed: 525-526 and 1306
A very slight overshoot there. I have been unable to explain this. Either it's rounding errors on multiple points or I am missing something.
Arcane Shot: 742 and 1765
Observed: 745 and 1770
Both non-crit and crit are slighly higher than predicted. Again I'm not sure what causes this.
Multi-Shot: 588 and 1399
Observed: 588-589 and 1399-1400
Aimed Shot: 588 and 1462
Observed: 588-589 and 1461-1462
Chimera Shot
Last but not least, Chimera Shot itself.
Without HM predicted: 817 and 2029
Observed: 817-818 and 2030-2031
With HM predicted: 895 and 2222
Observed: 895-896 and 2222-2223
Now the CS-Serpent effect:
CS-Serpent without HM: 727 and 1457
Observed: 728 and 1732
CS-Serpent with HM: 745 and 1490
Observed: 746 and 1775
Crits much higher than expected. After checking the calculations, the predictions used the wrong critical hit bonus (it was using a flat 100% damage bonus instead of factoring in Mortal Shots and RED). After fixing that they matched the observed values correctly.
Conclusion
RangedweaponDamage = Average( low-end; high-end )
AmmoDamage = RangedSpeed * AmmoDPS
Steady Shot: BaseDamage + Normalized Weapon/Ammo Damage + AP*20%
Arcane Shot: BaseDamage + AP*15%
Auto Shot: RangedWeaponDamage + AmmoDamage + AP/14*RangedSpeed
Multi-Shot: RangedWeaponDamage + AmmoDamage + AP/14*2,8 + BonusDamage
Aimed Shot: RangedWeaponDamage + AmmoDamage + AP/14*2,8 + BonusDamage
Chimera Shot: RangedWeaponDamage + AmmoDamage + AP/14*2,8
These 4 use the same calculations, except for the strange fact AP contribution is normalized for Multi/Aimed/Chimera but not for auto
EDIT: I had Chimera as AP*20% before but 20% is of course the same as AP/14*2,8. Thanks to the reader who pointed that out!
Serpent Sting: BaseDamage + AP*20%
Untested so far:
Kill Shot (for obvious reasons)
Explosive Shot (will need to respec again)