So PVPing over the weekend and targeting some felguard or stupid cat named Mr.Stupidface every time I hit tab is really annoying (not to mention every battleground has somewhere around 10 hunters these days).

Is there a macro option ([]) to check if the unit is a pet? I tried

which is the extent of my macro knowledge, and it still targets pets. Thanks.

Yeah, I just found that out.

It doesn't really make sense to me, as I don't see [pet] being any different from [harm] or [combat].

Edit: I understand now that it's not [pet] that's in question, it's the /targetenemy command itself.

It's a bit complicated.
/targetenemy cycles through hostile targets, as if you were tabbing through them.
While you can use conditionals as [harm] with /targetenemy, you cannot check what you are targetting (and eventually change your target).
Also the [pet] conditional does something different than you might expect. According to WoW Wiki...
IIRC there was an addon that did something similar, e.g. retargeting if you target a civilian or pet, but alas I cannot recall its name

If you are into addon scripting you might want to try hooking the TargetNearestEnemy() method, but I am not sure how much is possible without tainting your UI.