Defining who is an anglophone or francophone is always a bit tricky when it comes to determining what services we can offer. However, historically, let's be honest, it's the mother tongue that's used. If you look at the comparisons over time of English speakers, French speakers and mother-tongue speakers in Quebec, what happened was that there was significant immigration between 1970 and 1980. This happened following two external shocks, the October Crisis and the election of the Parti Québécois in 1970 and 1976 respectively.
This prompted a significant number of wealthy English speakers to leave Quebec. People a little older than me benefited greatly from the drop in house prices in the town of Mount Royal at that time. It also helped in the francization of the town a little.
Since then, we've observed that the average working income of anglophones is still higher than that of francophones, if we take mother tongue into account in both cases. However, when we do a slightly more sophisticated statistical analysis, where we say that the gap comes from the fact that anglophones are perhaps better educated, have more experience, and so on, then we note a significant difference. In 1970, a unilingual English-speaking man earned about 10% more than a bilingual French-speaking man. For women, the gap was smaller, at around 5% more for the unilingual anglophone. From 1980 to 1985, the opposite was true.
Francophones have therefore benefited from various measures, and I'm not talking about language laws, but rather the Caisse de dépôt et placement du Québec and francophone investments. A whole range of measures have significantly strengthened the French-speaking economy in Quebec, and English speakers, relatively speaking, have lost out.
Now, to get back to your more specific question, we usually use the mean. As you say, the median tends to overlook the fact that, by definition, there are people to the right or left of the median, dragging the average up or down. The trouble, I think, is that there might be some quibbling about this. Let's face it, academics sometimes like to split hairs, and split them some more. However, I think that the first official language spoken, as you mentioned in your question, is not as good an indicator of group membership as are the mother tongue and the language spoken at home. Those would be the two best indicators, in my opinion.