The Speaker could choose not to recognize a member in a particular party but choose to recognize the person next to him who has been behaving appropriately. Surely not all members of a particular caucus are misbehaving all at once in the same way.
On October 28th, 2010. See this statement in context.