Yes. An interesting historical note is that in 1926—everybody remembers the famous King-Byng crisis—when Arthur Meighen was sworn in as Prime Minister after the Governor General Byng dismissed Prime Minister King. He was faced with the awkward situation that he would lose his parliamentary one-seat majority if he had members resign to run for cabinet, because they were not cabinet members following the election. His solution was to retain all the cabinet posts for himself in order to get around that conundrum. He ultimately was defeated, but that legislation played a key role in the outcome of that event, which I think is why King subsequently got rid of it when he was Prime Minister.
On March 21st, 2017. See this statement in context.