Here's a question Kavan... do you model obscure talents, such as (for example) the mana return from resisted frost spells with
Frost Warding? Do you model the increased survivability from the same talent (since 30% of the frost attacks would not even take away from the damage shield)? It seems dubitable, since no matter how much frost damage I set myself taking, and no matter how high I set survivability, the rotation never asks me to cast Frost Ward.
If you're actually properly modeling all the effects of all the talents, it's conceivable to add constraints on the effects rather than on the specific talents.
But I'm afraid there are spells that are very difficult to properly model. Rather than try and model everything in the fight (which doesn't just make it harder for Kavan, it makes it harder for us... it's a lot easier to make a setting 'require Frost Wardx2' than it is to set survivability as high, input the expected frost damage incoming, and hope that the numbers add up to 'I want to ensure those points are important'), I believe it would be better to give more direct control over the talents we want in a build.
As far as the GUI... why not duplicate the talent pane. Mark or color code the talents as DPS, Utility, and Survivability. But rather than selecting specific points to put into a talent, have the player choose the priority of the talent. Priority would be modeled internally as a 0 to 1 float... visibly, it could be something like 'high, medium, low, ignore' (1, 0.7, 0.4, 0.1).
For talents that are properly modeled, they would default to 'high' priority (as in, the DPS increase from that talent is fully counted when weighting the talent). A talent that is fully modeled, but gives crap DPS, might still be skipped even at 'high' priority.
For talents that are incompletely modeled (or not at all), then the priority is normalized to the best DPS talents. For example Fire Throwing (partially modeled, since what spells it affects is known) would be scaled to be just as important as Living Bomb for a fire spec, assuming it was set to High priority (1). If it was set to Low priority (0.4), then it will only be taken if it doesn't interfere with a useful talent.
Due to the unbalanced value of one point talents (Living Bomb, Frost Elemental, pre-nerf Arcane Barrage, Icy Veins) it might be better to consider only multi-point talents when normalizing the value of non-modeled talents.
The end result is that players can look at the talents, adjust the balance of modeled talents if they want, and adjust the priority of unmodeled talents to their liking. An unmodeled talent would get scaled in value to the best (multi-point) modeled talent, multiplied by the user's priority, and possibly adjusted again based on whether the affected spells are part of the rotation. The user gets direct control over the talent tree (setting Flame Throwing to high pretty much ensure it will be taken in any fire build), Kavan has a much easier time with modeling (he only has to worry about what spells are affected by a talent, rather than exactly how much DPS/survival/utility it provides under contrived fight circumstances).
And, let's face it... players are irrational. Maybe Magic Absorbtion is a shitty talent, but they really want it so they can brag about having over 600 FR for Sapphiron. This method is easier to do, and it gives players more direct control of their talents... just like locking equipment when doing the optimizer.