It depends on the purpose and how it's being used. It depends on who you're talking to. Consumers often see it the same way--not always, but often.
If you pay a toll, that can be seen as a fee. If it's going to pay for a bridge and is time-limited for a particular budget, it may not be, but if you're paying a fee for an ongoing service--for example, the air security charge--and it's dedicated to that in an envelope and it's transparent that it's for that envelope, it may not be seen that way. It's in the eye of the beholder.