It's a very complex issue to say that....
No, you shouldn't. We don't want the keys to your house, but we also want to stop the theft of our lands. If you feel that is a threat to your safety, think of how we feel when we look at all of the land that's been taken under the guise of the rule of law.
Should you be fearful? Only if you do not believe in human rights, only if you do not believe in peace and only if you believe that you are superior to indigenous peoples, just as the doctrine of discovery touted for all of the monarchs in Europe.