It's very simple: critical thinking. It may sound a bit trite, but we're seeing that the more people use these systems.... If, let's say, you use it 100 times, and 99 times or 95% of the time it is getting a good answer, and it fails just once, the question is, how consequential is that failure?
We're seeing a lot of erosion of people's critical thinking and even capacity and capabilities using these systems. I study computer science. We see people not thinking as much about their code, and this can create security vulnerabilities. Do you see what I mean?
The thing is, as people push for more automation and essentially take the human out of the loop, the paper protocol and the safeguard is to insist upon the human in the loop, to insist that we still develop these critical skills as a form of resilience. This is a hard question.
How do we maintain our critical thinking and our autonomy when automation provides useful and reasonable capabilities? It's a challenging question that doesn't have one correct answer.