I investigated this a bit myself, and after 15 minutes of stabbing rats and a respec, the answer seems to be that Murder applies twice on crits. On regular white attacks one does baseDamage*1.02 damage; on crits, one does baseDamage*1.02*2*1.02 damage, which is why it's showing up as 204% of base damage.
Also, apparently "critters" count as "beasts".
Are you sure it's base*1.02*2*1.02? That would yield ~208% base damage.
I was messing around hitting beasts and demons the other day (5% troll racial for beasts). I was seeing white crits with/without RED vs demons were 206%/200%, but vs beasts I was seeing ~221%/~215%.
This seems to indicate the slaying is more akin to base*1.05*(2+.05), and the RED is tacked on after for another 1.03 multiplier.
Those match my testing pretty well. I admit I didn't collect a lot of data. I just stabbed with a [Dirk] a few times and took the biggest hit and biggest crit. I figured if I saw 158 and 160 noncrits, 160 was the biggest I would see.
It's Base*1.02*2*1.02, which is 2.04 times a noncrit which does Base*1.02. It's 208% of what you *would* do without Murder on a noncrit, but it's 204% of what you actually do, with murder, on a non-crit.