Because 'the right people' often believe such power is 'wrong', they either refuse to use it or find some way to throw it away.

[Though there are some abilities that you cannot rid yourself of, as Dhaos knows.]