Any sort of sexual sin is just an easily identifiable transgression and much less a matter of opinion than e.g. greed, joylessness, or contempt for fellow believers; as a result churches tend to let their pastor off the hook for these other sins, and treat him as “other” or “special” until he commits a sin that is an easily identifiable event with an identifiable victim.