As Mr. Chhabra noted in the California instance, some of this is about what they're being used for and what they reveal.
In the case of Ontario, those of us who are citizens of this province know that our birthdate is included as the last six digits of our driver's licence number. That's not the case in a number of other provinces, so again, this is where context will matter.
In many of those instances, because of what you can potentially use a passport number for and the link back to the individual, it would probably be deemed personal information worth protecting. However, in and of itself, it doesn't necessarily suggest that the combination of numbers and letters is sensitive information, because, again, that's what it's for.
I think giving broad ambit to the commissioner to lay out and provide guidance around what can and cannot be done with this type of information and what protections need to be put around it—including, for instance, suggesting that express consent is the best approach in some of these contexts—is more important than just a broad categorization. Again, within that categorization are uses, contexts or, in fact, instances where it's not even necessarily personal information in and of itself.