You could also do that now. I don't see any governing party wanting to have their chair polled repeatedly. That just slows up their legislation in committee.
You could do that, but if you have a relatively quick byelection mechanism, you resign on Thursday, the byelection is held on the following Monday or Tuesday, and by a week afterwards you're back in business then—