You're entirely right.
The conference interpreting system you have in-house is excellent. It would be possible to ensure better quality than you have at the moment if you had a dedicated, tunnelled connection with fixed bandwidth.
Sound engineers would have to decide that, but to my knowledge, it's not possible to have a guaranteed quality for the connection from the many speakers who are joining from outlying locations to the centre hub where the interpreters are located.
They are sitting in excellent technical conditions with a professional system, but the sound is coming through a connection, like a bad telephone line, basically, that compromises sound to the level where you cannot really hear and speak at the same time unless you turn up the volume to a level where it damages your hearing.