Yes, it is very fair.
In fact, the governments of those countries where they originally come from are aware of that. This has been a pattern since the end of the 1980s, after the Soviet war in Afghanistan when the mujahideen felt they'd defeated the Soviet Union—now what do they do? Then they were going to go back to their home countries. Those countries didn't want them there because they knew that they were going to come back and undertake activities designed to overthrow those governments.
Many of them did move to Europe and Canada and the United States from where they did continue to plot attacks on their home countries.