The beauty of the staffing priority system is that the hiring managers don't know the reason you're on the list. There are many reasons you could be on the list, including prior military service or returning from disability, things don't have any stigma attached to them, so it's not necessarily that their reputation would follow them.
Also, I think there are many good places to work in the public service where doing the right thing is valued, not suppressed.