As I recall, the unrecognized parties that don't reach that threshold are not actually.... Obviously there are discussions that go on with their colleagues from recognized parties, but I don't think they figure in the actual divvying up of the number of slots in question period in formal negotiations. Those formal negotiations, as I understand it, take place between the recognized parties, and this collective would still be under the wing of the Speaker.
Again, going back in history, we were talking about the two Conservatives, when Mr. Charest and Mrs. Wayne were sitting. The circumstances might be different and negotiations might want to take different things into consideration. Ordinarily it's the recognized parties that are part of the formal negotiations, and those outside that kind of fend for themselves.