This is absolutely true. One of the downsides of leaving it to the judiciary to supervise the relationship between the Governor in Council and an office-holder is that it is very time consuming. It is also very expensive, and often the results are unsatisfactory.
That is not to say that the procedures would be better if Parliament supervised the relationship between the executive branch with the Governor in Council and the office-holder. If the office-holder had to consult a committee or bring proposed action to the House of Commons for some sort of approval prior to removing an office-holder, that could also result in an expensive delay. So there are trade-offs.