There is academic disagreement. The actual constitution talks about the “office of the Governor General”, which would definitely require unanimity under article 41. But I think the Clerk mentioned the possibility of distinguishing the “powers” of the Governor General as opposed to the “office” of the Governor General. That would require less than unanimity.
I don't think we should go down that road. As we've found out in the past, even constitutional amendments requiring less than unanimity are almost impossible.