Autoshot gains 88% more from RAP increases than steady shot while Steady gains 10% more from ranged weapon upgrades than autoshot. Since the larger part of our damage comes from RAP, rather than weapon damage, Autoshot wins on this count. Not that this is very important though.
The question of which glyph is generally better (and which way they scale comparatively) requires us to see what the difference is between them
SrS glyph allows 2 more steadies in a timeframe of 105 seconds.
Hawk glyph adds additional 6% haste to our autoshots (as everything else is already capped - and we aren't looking at volley)
As a beastmaster hunter we have the following modifers:
I will work with a 2.9 weapon speed from envoy of mortality or giant's bane from ulduar. Siren's Call has 2.8 speed, but the difference is negligable.
I will also use my own hasterating which doesn't seem excessive either way; I have 336 hasterating, translating to roughly 10% haste at 32.79 rating per haste rating point.
Weapon speed: 2.9
15 % Haste from quiver / autoshot ability
20 % Haste from Serpents Swiftness
10 % Haste from gear
15 % Haste from Improved Aspect of the Hawk
60 % Haste total.
at weapon speed 2.9 with 60% additional haste (60% more shots fired in the same amount of time) I get 2.9 / 1.6 = 1.81 as the shot speed.
This is our basis for comparison as this is without the hawk glyph.
At 1.81 speed we fire 58 autoshots during the 105 seconds we use as a frame.
Adding an additional 6% haste results in a speed of 2.9 / 1.66 = 1.75 (1.7498..) for each autoshot.
With this speed increase we fire 60 autoshots in the same 105 seconds.
To check the gear haste I will do 3 additional calculations here: 0% gear haste, 5% gear haste and 15% gear haste to see the trend:
Thus; we have the choice of 2 additional steadies or 2 additional autoshots in 105 seconds between these glyphs.
(the 15% gear haste was almost 60/62 but cut due to rounding to 2 digits before calculating the amount of shots fired)
Unless I'm mistaken Autoshot wins out over steadi except at exremely low gear levels.
I have calculated the uptime of Quick Shots to 100% for this purpose as it is my intuitive understanding that QS will reproc off itself; it lasts 12 seconds and I have 1,24 shots per second = 14 shots per QS proc. QS has a 10% proc chance, so it should reproc during these 14 shots. (Caveat; this is for napkin math)
I assume the SrS glyph showing slightly better than the hawk is due to the actual chance of QS to proc and reproc (my spreadsheet shows a typical average uptime of 57.76%)
Something you forgot in your calculations. You not taking critical strikes into account.
Assuming that an average BM hunter will have around 40% crit, and autoshots only crit for 203% damage vs the 234%. This in my opinion would put greater weight on steady shot.