You might find
this thread interesting.
Long story short, Relentless works for spells, and has weird mystic voodoo going with how it increases damage done. It basically ends up providing 4.5% extra crit damage base on spells, which is further multiplied by any crit damage multipliers you have. I.E. For an Elemental Shaman it turns into 9% extra crit damage.
As I understand it, it basically attempts to provide 3% extra damage on your base crit damage, before getting further multiplied. The base crit damage for spells is 150%, 3% of that is 4.5%, and twice that from the talent is 9%.
I'd expect it to be providing 6% extra crit damage for melee as well though, I'm unsure if someone has tested that.