One of the standard ways to mitigate the effect of conflict of interest, if you can't make it go away, is to involve more people in the decision-making. If you're the only relevant expert in some field and you need to be part of this adjudicating committee, then, if we can't remove you, what we need to do is bring in additional people so that more people are party to the decision.
No one is going to think that's a perfect situation, but it might be the best we can do in a given situation.