Faerie Fire and Sunder Armor are applied independently of armor penetration, so they do not affect the armor penetration cap. You could think of them as stacking multiplicatively, but that's not quite correct because armor penetration itself does not act in a multiplicative fashion. The math is a little complicated, but suffice it to say that Sunder Armor and Faerie Fire provide exactly the same fractional damage increase (to armor-mitigated attacks) no matter how much armor penetration you have.
Precise mathematical details:
Let A be the target's armor value before armor penetration, but after debuffs. Let x denote the fraction of damage taken from an armor mitigated attack (1-damage reduction). C is the armor constant, 15232.5 for level 80 attacker. Let k be the nominal armor penetration fraction (as shown in the character pane).
One can work out (e.g.
in this post) that (assuming A > C/2)
(1-k/3)})
.
In particular, the damage is proportional to 1/(A+C) no matter the value of the armor penetration k. Again this only holds for A > C/2 = 7616.25, but this inequality is almost always true for raid bosses since they have 10643 base armor (8088.68 with Sunder Armor and Faerie Fire). The only situation in which this condition is violated is when
Shattering Throw is also active - so in fact one can work out (assuming the other debuffs are present) that Shattering Throw does become less effective at higher armor penetration levels.