It's a bit complicated.
/targetenemy cycles through hostile targets, as if you were tabbing through them.
While you can use conditionals as [harm] with /targetenemy, you cannot check what you are targetting (and eventually change your target).
Also the [pet] conditional does something different than you might expect. According to
WoW Wiki...
|
pet:<name>|<family> - Choose if your current active pet has the specified name or family (e.g. pet:bear/wolf)
|
IIRC there was an addon that did something similar, e.g. retargeting if you target a civilian or pet, but alas I cannot recall its name
If you are into addon scripting you might want to try hooking the
TargetNearestEnemy() method, but I am not sure how much is possible without tainting your UI.