if you switched out a 100 dmg/40 crit rating purple item for a +20 hit green item, you'd lose dps
That's effectively what you did. I would confidently say that your +hit gear is just lower ilvl items overall.
Get items that spread their item budget among all three, or at least two, of these stats when possible. Prince's cloak is best in game until BT, if you aren't hit capped already, for example. Ashyen's gift is a good ring for your current progression, +hit and good +dmg, etc etc..