First, as the legislation is drafted, I don't equate the parliamentary budget office with a full-fledged parliamentary officer. I see it much more as an internal support and resource--to parliamentarians, to committees, to individual MPs, to senators--than as an alternative accountability structure.
As for the other officers, I've read articles about the proliferation of these positions. It is true, I think, that over the years, in creating parliamentary agents, Parliament's intention was to delegate a certain accountability exercise, but in many cases it has diminished the authority of Parliament.
Take the Access to Information Act. Don't take this as criticism of the act and the value it adds to our political culture, but questions on the order paper and notices of motions for the production of papers had real value, real impact, before the Access to Information Act. A department, when it got a question on the order paper, took it quite seriously. Indirectly, what has happened now is that you're better off, as an MP, using ATI to get a document than using a notice of motion for the production of papers. I don't think it was Parliament's intention to diminish one of its own procedures by delegating this to an agent outside, but that's what has happened, in my view.
That's not to say that the system can't be strengthened and reined in, but along the lines of what you just outlined, that would be my comment.