The relation to the office of the Governor General might be in this context of section 41 contained within the Constitution Act of 1982, which, by the way, relates to the Constitution of Canada, which is defined in section 52. You can argue that section 41 relates only to those provisions in the Constitution Act 1867 and 1982 that relate to the office of the Governor General.
But I can't tell you where the line is between the office of the Governor General and the powers of the Governor General. My discussion, my paper regarding focusing on the Prime Minister's role, was as a possible strategy to avoid that argument. If all you're legislating is the Prime Minister's role, arguably, you're not touching the Governor General's powers or the office of the Governor General and section 41 wouldn't apply.