That's the risk and that's one of the black boxes of complaint procedures. You need the people receiving the complaint take it seriously, to listen to the person, and to not immediately dismiss it. If it's confidential, it doesn't necessarily mean that there's no record taken. If it's confidential, it doesn't necessarily mean that the person who is charged with taking these complaints doesn't write down on a particular day that something was said about this person.
What may happen is that after one, two, three, or five years, all of a sudden, the person taking the complaint realizes they're receiving a lot of comments about this one person. It puts them in a position where they may be more able to recommend either to their boss or to the complainants coming forward that maybe they should start thinking about formal complaints.