I'm a bit limited in gear selection for this sheet, as I needed to use a fairly old version for Rupture support to still be present, and it lacks in a lot of the more recent item discoveries, but I ran a comparison with the best gear that the sheet supports, which are listed in this comparison. Also, this was run from a Horde point of view, so no Draenei hit bonus.

The results are as follows:
Standard 51/13/7 build, Envenom cycle
Blue socket: Nightmare Tear
Red/prismatic sockets: Bright Cardinal Ruby
Yellow sockets: 4 Stark Ametrines, 7 Pristine Ametrines
DPS: 12589.5

Spec: Talent Calculator - World of Warcraft
Rupture Cycle
Same gems as the first case, Cat's Swiftness in place of Icewalker
DPS: 12697.5

Spec: Talent Calculator - World of Warcraft
Rupture Cycle
Blue socket: Nightmare Tear
Red/prismatic sockets: Bright Cardinal Ruby
Yellow sockets: Stark Ametrine
DPS: 12586.2

Spec: Talent Calculator - World of Warcraft
Rupture Cycle
Same gems as first case
DPS: 12581.5

At this gear level, it appears that the standard 4+ envenom rotation begins to catch up to the rupture rotation, though the rupture rotation still has an edge, though again not enough of an edge to account for Tricks increasing the other rogue's DPS by an average of 5% instead of 3% in the scenario where you have a pair of rogues to trade tricks, but enough to cover the lost DPS if you're using Tricks on a tank.

EDIT: Corrected typo in Envenom cycle DPS.
EDIT2: Wasn't a typo after all, but an error switching enchants on my part.

Were you able to generate EP values?

Given that this is a modified version of Aldriana's sheet, yes.
Envenom cycle:
Strength 1.1
Agility 2.02
Crit 1.86
White Hit 1.7
Spell Hit 2.96
Expertise 2.51
Haste 1.95
Armor Penetration 1.35

Rupture cycle:
Strength 1.1
Agility 2.06
Crit 1.92
White Hit 1.68
Spell Hit 2.92
Expertise 2.56
Haste 1.96
Armor Penetration 1.36

As a note, despite the fact that the EP for Agi is greater than two, gemming for it was a loss due to crit cap issues.

I have to wonder if you are far enough below the crit cap (Heavy Hit rating) if gemming for Agi might make a considerable impact. It might be irrelevant if crit cap is unavoidable by BiS.

To test the effect of getting more space below the crit cap, I reran it with an extra 200 hit rating present. That seems to lower the value of agility significantly, with the Stark Ametrines and Deft Ametrines having the exact same EP value at a combination of 9 Stark/2 Deft, with all the red slots still being used for Bright Cardinal Rubies. This suggests to me that Agility gemming isn't viable, even with the Rupture spec.

Also, just to cover the bases I missed in my earlier comparisons:
Standard 51/18/2 build, Envenom cycle, Icewalker
Blue socket: Nightmare Tear
Red sockets: Pristine Ametrine
Yellow/prismatic sockets: 6 Quick King's Amber, 6 Pristine Ametrine
DPS: 12606.9

Spec: Talent Calculator - World of Warcraft
Rupture Cycle, Cat's Swiftness
Blue socket: Nightmare Tear
Red sockets: Pristine Ametrine
Yellow/prismatic sockets: 5 Quick King's Amber, 7 Pristine Ametrine
DPS: 12623.6

I've also been wondering if a Subtlety template could pull ahead using backstab.

Rupture has a 16 seconds base time with 5*CP

- 20 seconds with glyph
- 26 with glyph of backstab

Considering Shadowstep has a 30 seconds CD I'm wondering if it would be a dps gain to wait on it to ruputre...

Is there a spreadsheet with TotT accounting for a 7-8% (7.5) dps increase? If with the 2-3% (2.5) damage increase subtlety pulls ahead then it's viable as long as another rogue in the raid has the same build (filthy tricks).

Subtletey has been completely locked out from high end raiding since the HAT fix - 3.3.3 might be it's revival.

Wouldn't you take the Hemo glyphinstead of the the Shadow Dance one?

Shadowstep

30 seconds CD.

Your template isn't good... taking 2/2 Improved ambush without initiative - and simply taking 2/2 ambush (in end game gear you'll be facing severe crit cap issues anyway)

Using backstab probably outdamages hemo by a large margin with a high rupture cycle - and wields more poison damage through daggers.

Taken from the PTR notes -

Filthy Tricks: Now Reduces the cooldown by 5/10 seconds and energy cost by 5/10 of Tricks of the Trade, Distract and Shadowstep abilities, and reduces the cooldown of Preparation by 1.5/3 minutes.

With 2/2 Filthy Tricks you'll be synching your Shadowsteps perfectly with your GoR Ruptures at 20 second intervals.

If you really wanted to take the Backstab glyph as well you'd either want to take 1/2 FT delaying it by a second or even 0/2 FT and delay it by 4 seconds, but frankly with the increase in TotT uptime that doesn't seem too smart. You could fill the points and delay it anyway but it seems a bit of a waste so at the end of the day you'd probably be better off filling that Glyph slot with something else. As 2 of your slots will almost surely be filled with Glyph of Rupture/Glyph of Tricks, you'd possibly be left with a choice of Glyph of Hemo or perhaps Glyph of Slice and Dice.

If we are strictly sticking to Daggers, poison damage will be the same. Also, I don't think Backstab will out perform Hemorrhage by a large margin.

With at least 51 Points in Sub Tree we have the following bonus damage,

Backstab - ((150% + 10%) + 465)*1.20 at 45 Energy (Sinister Calling and Opportunity)
vs
Hemorrhage - (160% + 10%) + 75 at 30 Energy (Sinister Calling only and Hemo debuff)

If you went the extra 3 points into Improved Backstab, it might be better due to combo point generation. But I think I would prefer the lower energy consumption vs 30% extra crit.

Also Hemo is another raid buff, no?

Would be cool to have one of 'em in raids, shouldn't it?

Hemo is not a raidbuff. Because it has a fixed number of charges and a fixed amount of damage per charge not pegged to the gear of those consuming the charges, it's basically a redistribution of the rogues personal DPS to other members of the raid. If threat mattered, it would be slightly advantageous, but as it is, it's really just a silly mechanic - it is not a raid buff.

Actually if we are strictly sticking to daggers poison damage would increase with hemo. Given that it costs 30 energy instead of 45 energy it will likely be used more often which leads to more poison procs.

Found a newer copy of Aldriana's mutilate sheet that still had rupture in it (1.2.1), made the same hacks to it (HfB nerf and forcing T8 4/5 in cell B12), and fixed another error I found in my earlier tests (forgot the poison nerf). Numbers don't look very good. With my gear, the best increase I could pull off was 0.93% (91 DPS), which would only cover the Tricks damage loss if you were using Tricks on a tank, and only barely then (assuming 4k DPS from the tank). With better gear (using Aldriana's BiS list, with the exception of switching in heroic WFS as I seem to remember TaiaJ getting nerfed since this sheet was made, so I didn't trust the numbers with it), the damage increase was far smaller, if any. Looks like the rupture change may have bought it a bit of scaling, especially at lower gear levels, but the fact that it only scales off of AP and crit, as opposed to envenom which scales off of AP, crit, and haste, seems to basically doom most uses of it for mutilate, at least from this test.

Correct. Thank you.

Including the higher frequency of mainhand attacks through Hemorrhage, we are also performing more finishers. Upping the poison procs. Yes?

could sub spec outperform the combat and mutilate in fights like putricide due shadowstep and high CP generation?

You would want to take Ghostly Strike because of Glyph of Ghostly Strike. Increasing it up to 252% I believe.

180*1.4 = 252. So yes you believe well.

That would place Gstrike at 252/40 = *1.063 damage per energy point.

Except that the glyph raises the ability cooldown from 20s to 30s, so the net benefit of using it over another CP-builder is halved due to less frequent usability. And as a sidenote, Ghostly Strike will consume the full energy cost even when the attack is dodged/parried/missed. Considering the damage difference from Hemorrhage isn't that big to begin with, it very likely doesn't fit in with Tricks, Rupture and Backstab glyphs (assuming Mavanas' initial analysis about GoBS holds true). And even with a glyph slot freed, Hemorrhage glyph should be a strong contender for the slot, especially if Backstab is dropped out of the equation. Additionally, [Glyph of Shadow Dance] is a very interesting option, as in a "Patchwerk" situation it should give you enough time (=energy) to weave in an additional Ambush every minute. Not to mention the utility aspect, which shouldn't be dismissed due to the number of short-lived/stunnable adds in current encounters.

Very true. PvP-wise, it's there for those bursty moments. I can't seem to find use for it in PvE.

It seems like tott, hemo and evisc are best glyphs for raid dps right now. I made an update of my analysis in that thread. I just want to comment on Shadow Dance glyph. If you pool energy before SD, as you should, you can have a maximum of 160 energy during the shadow dance, which is enough for 3 ambushes. Adding another 2 seconds gives you exactly enough energy for 4 ambushes, but in reality I am not sure if it will allow you to do the last ambush as you gain 45 last energy right when Shadow Dance expires.

P.S. just realized you can use tricks during the Shadow Dance, which will allow you to do one more ambush, and even an eviscerate to clear your combo points.

I estimated the effect of squeezing in 3 ambushes to be 473 dps, so an extra one will be around 158 extra dps. However, the raid dps benefit of hemorrhage glyph is around 290 dps. Shadow dance might replace eviscerate though because it's value was around 110 dps with agility gemming. I haven't actually recalculated its value with arpen gemming.

Hemo costs 30 energy talented so the difference gets even bigger

160/30 = *1.05333