There are all kinds of reasons behind it. In the 1970s, when there weren't the same restrictions there are today, about 55,000 of that population were what today we would call non-English non-French allophones, and about another 30,000 were actually French-speaking families who chose to send their children to English schools. You had 175,000 or so students who today would still qualify under the current rules, and then you had the mix of the others. We just don't have access to the others any longer.
So yes, people left Quebec. Many English-speaking people did leave Quebec, but that population has fallen by more than 50%. If you take out the adult section of our school board, you're looking at about an 80,000 to 85,000 youth sector compared to 175,000 previously.