I think there is merit in the notion that in setting an agenda, there be an ability.... There's certainly the ability of the third party to be there. I think I'd be prepared to try including them in the quorum, provided it didn't demonstrate a desire to shut down the work of the committee.
In terms of an opportunity to protect the rights of the minority, I don't think it's a bad idea.