Absolutely. It's a great question.
As AI development continues, AI is becoming more autonomous—able to independently take action and handle open work without human involvement. These are sometimes called AI agents. As I think everybody can appreciate, if we build systems that are much more competent than any group of humans in every domain, including hacking, persuasion, building new technologies, etc., and we don't control them, that's a bad thing.
How do you define and how do you create laws? First, obviously, human judgment would be involved. Just as with most laws, if a representative panel from signatory countries believes a system quite likely would be able to autonomously compromise national security, escape human oversight and upend international stability, we would call this superintelligent AI. If something could be trivially, covertly or quickly converted into superintelligent AI, that would count as well.
Third, there are two ways of developing superintelligent AI—on purpose and recklessly. I think here we should use the principle that we don't let people try to build a bomb, even if they don't know what they're doing. We also don't let people plot to overthrow the Canadian or U.S. government. That would be intent, but in the name of short-term competitiveness, one might still recklessly develop superintelligent AI by, for example, automating AI R and D.
The real thing that matters is this: What is the point of intervention? That's what you must define. We call those precursors. Two examples are large AI data centres and AI that can significantly automate AI research and development. There are others. We need to be flexible and be able to add precursors as they get discovered and as the landscape changes and so on. I think precursors are allowed, but they must be monitored within the regime and denied outside the regime.