If it required unanimous consent, you as the speaker could immediately say you don't give it because you want to finish that thought.
We always maintain—and we'll use you as an example—that when you have the floor, no matter how far afield we get with our little side deals, discussions, set-asides, and everything else, at the end of the day you always have the right to say to the chair, “I have have the floor and I want it back right now.”