Thank you for that very valid and important question.
Truth be told, all Afghans are suffering now, yes. The Taliban are an enemy and a danger to all Afghan people.
When people leave the country to go to Iran or Pakistan, for example, just all that travelling and crossing the border are a danger in themselves. Once they arrive in Pakistan, in refugee camps like Quetta or elsewhere, they face many sad realities and dangers all around them. For instance, right now the U.N. has no official presence. They have representation under contract with agencies mandated by the Pakistani government and the U.N. They have to go to these offices to get some kind of registration documents, but the documents don't give them legal status. So they run the risk of being arrested at any time and deported to Afghanistan.
Even crossing the border is no easy task. First of all, there's a crowd. I don't know if you remember the crowds at the Kabul airport, but it's three or four times worse than that.
In addition, people sometimes have to pay Pakistani soldiers to let them cross.
On top of all these risks and perils, once they cross the border they are not out of the woods because they can be caught at any time by the Pakistani authorities and sent back to Afghanistan.