I think this is a question that's better answered with a concrete example to illustrate the two different threats we have to contend with.
As you said, perhaps there are some great powers we might want to acquire. Also, in the report that Anthropic put out, on page 55 it describes an incident where Mythos was confined to a secure computer inside of a secure network. They asked Mythos to escape the secure computer and send a message. Mythos succeeded, so you see the first half—the great power that we want—but this is only the beginning of the story. Mythos also developed an exploit to gain broad Internet access on its own. Then, without being asked to, Mythos bragged about the exploit on multiple public-facing websites, once again, on its own.
Yes, we have to think about not only people tasking AI to do dangerous stuff but also AI doing dangerous stuff on its own. It's not really like a nuclear weapon. It's like a nuclear weapon that sometimes goes up on its own, makes a bigger nuclear weapon and tries to take over your government.
That being said, I will repeat what I said. We don't call for unilateral disarmament. Right now, the United States is in the lead, so we don't call for the west to unilaterally disarm and cede the race to China. Wherever it is developed first, it threatens all of our lives, so we have to enter a stable regime where costs are imposed on people who are pushing forward, and they also get the reassurance that the other side is not pushing. That's the only stable equilibrium to limit the race toward superintelligence.