So in your case, each of the four of you could have one staff member, plus there could be someone from your House leader's office or your whip's office come in.
Let's say you need a technical expert to come in and advise you on procedure. You'd have the ability to do that and not have to kick someone out of the room.