It is not a matter of how fair or unfair an agency attempts to be. It’s a matter of fact that you have no legal recourse to ensure that they are fair to you. You do not have a presumption of innocence. You do not have the right to a trial by jury, or even an impartial judge.
You often grind this axe about how fair the Faa is, but that is irrelevant. All the things you mention are still relying on “fairness” that you have to hope for, by a bureaucrat that you can take no action against no matter the damage they do to you. If you don’t get that fairness, you have have to spend thousands to reverse even the silliest decision, and you are unable to exercise your privileges in the meanwhile.