It does worry me.
If I were a minister, the last power I would want would be a power to veto an access to information request. This would undermine the independence of the officials to whom the power has been delegated. I think there would even be a conflict of interest in a delegated power context. The idea was instituted in England and did not prove successful. I think it's inconsistent with our political culture.