I'll just point out one thing: this is not without precedent. The aboriginal affairs committee operates this way for reasons similar to those proposed—i.e., with the parliamentary secretary and an additional member from the Liberal Party—for what that's worth.
I'll also point out that as parliamentary secretary I can attend the meeting as an observer, a non-voting member, should this go the other way.