Information & Ethics Committee on June 9th, 2022

[Nestor Maslej]

My apologies for that trouble with the microphone.

The AI Index doesn't comment directly on the amount of investment that exists in that space, but it alludes to the fact that this is becoming an area of increasing concern across a lot of different spaces. I'll highlight two points here.

The first is that with the data I provided from the NIST FRVT test, this test looks at 1:1 verification. In a brief that I submitted to the committee, in figure 1.1, I show the success of different models on a variety of different datasets. Now, in this figure, one of the things that is very clear is that the best-performing model is the one that performs on visa photos, and that's the one where you had a correct identification 999 times out of 1,000, whereas the worst model is the one that is for the wild photos dataset.

The wild photos dataset is a dataset of individuals whose faces might be obscured partially by shadows, or perhaps they weren't looking directly into the camera, and the top-performing models identified correctly 970 out of 1,000 faces. It's still very high, but there's a noticeable drop-off compared with the visa photos.

I think this is suggestive of the fact that if companies and agencies want to use these technologies and justify them on the grounds that, “Look, we tested them in the lab and they had a very high accuracy rate in the lab,” there has to be an attempt to qualify the difference between the settings in which these technologies are tested and the settings in which they are deployed. I think the committee is aware of this, but the index suggests that it is a pressing concern.

I will also add that we cite some research that was published a couple of years ago—and that I think has appeared in the committee as well—in the form of a 2018 paper by Timnit Gebru et al., entitled “Gender Shades”, which looks at the fact that a lot of facial analysis benchmarks tend to be overwhelmingly composed of light-skinned individuals, leading to subsequent bias, and that existing algorithmic systems tend to disproportionately misclassify darker-skinned females as the most misclassified group.

We allude to this, and I think there is a general sense in the research community that there should have been more work being done in this space, but I would be unable to comment as to the exact amount of investment that is being put into this particular field at the moment.

[Parm Bains, Liberal, Steveston—Richmond East, BC]

In November 2020, HAI published a report, “Evaluating Facial Recognition Technology”. I think you talked a bit about this in terms of the clarity of the images.

One of the concerns raised is that “FRT vendors may train their images with well-lit, clear images and with proper software usage from machine learning professionals”, but when deployed by law enforcement, FRTs rely on images produced by body cameras and other sources in “suboptimal” conditions.

Is this a problem that can be corrected? With respect to body cameras and the law enforcement technology that they're using, how can that be improved?

[Nestor Maslej]

It might be outside of the scope of my area of expertise to identify ways in which they could be improved.

I would, however, say that in the paper you're citing, the issue they talk about there is “domain shift”, which is the fact that very often the settings in which some of these algorithms are tested are radically different from the settings in which they are deployed.

At the minimum, there ought to be some kind of clarity and honesty in terms of the agencies that use these tools—whether it's companies or agencies—about the extent to which a difference exists between testing conditions and conditions in which these tools are actually deployed. I think it would be problematic if there were such a big discrepancy, that these tools were tested in one setting but then deployed in completely different settings. If there isn't a clear sense and a clear understanding as to whether this difference exists, then these technologies perhaps have a great likelihood of being misused and serving more nefarious purposes.

[Parm Bains, Liberal, Steveston—Richmond East, BC]

That takes me to my next question, which is about human error. It's also a concern with FRT. The same report indicates that while Amazon Rekognition “recommends a 99% confidence threshold on identity matching for use in law enforcement”, one of the sheriff's offices interviewed for the report stated, “We do not set nor do we utilize a confidence threshold.”

Do you think any use of FRT requires a trained professional who understands the technology's structure and design?

[Nestor Maslej]

Again, I didn't contribute to that report directly, so I wouldn't be best suited to answer the question, but I think perhaps the issue that the report is getting at there is one of institutional shift.

You might potentially have these technologies used by different individuals in different parameters. Certainly, being trained in the usage of these systems can be important, but I think there's also a recognition here that unless there is some kind of set regulatory standard about what is an acceptable benchmark or an acceptable framework, you might have different jurisdictions using these technologies in different ways. On the question of what this acceptable benchmark is, again, that is outside my area of expertise. I will leave it to you policy-makers to crack that one.

I think the point being made is that if a threshold doesn't exist, it's a lot likelier that individual agencies will make these assertions themselves. There are reasons certain agencies might favour lower or higher thresholds, and this can lead to potential misuse with some of the technology.

[The Chair, Conservative, Pat Kelly]

Thank you, Mr. Maslej.

Mr. Villemure, you now have the floor for six minutes.

[René Villemure, Bloc, Trois-Rivières, QC]

Thank you, Mr. Chair.

I thank the witnesses for joining us.

Ms. Polsky, if we define understanding as the ability to grasp everything that is at stake, I conclude from your statement that there is a great deal of misunderstanding among people, governments and users—in short, everyone who is involved to a greater or lesser degree in facial recognition.

I have three questions for you, Ms. Polsky.

In reality, the consent we give by clicking on “I accept” is not a choice. We have no choice but to consent. Is that right?

[Sharon Polsky]

That is, as I said in my remarks, the consent fantasy. Mr. Zuckerberg himself said to the U.S. Congress that even he doesn't read these things. The last time I counted—yes, I did count—the Google privacy policy, it was 38 pages long. Nobody is going to read it. As a result, they're clicking on...what? They don't know.

The problem or the catch there is that at least under Canadian legislation, a company or an organization is supposed to collect personal information only after they have informed consent. When even Mr. Zuckerberg acknowledges that nobody reads these things, they are collecting personal information without informed consent, contrary to the provisions of PIPEDA and, I think, all of the other privacy laws across Canada.

[René Villemure, Bloc, Trois-Rivières, QC]

I think that those who wrote the policies have not read them.

In Europe, users can continue without clicking on “accept”. Do you think that should also be implemented in Canada?

[Sharon Polsky]

If you're talking about getting rid of the cookie consents, that has become a farce, quite bluntly. You see them on so many websites. There are some websites where you can go in to adjust your cookie settings, but then you can't get past that. You must accept all cookie settings, which is contrary to the GDPR.

To say get rid of consent policies, well, the way to do that is to put the onus not on you and me and individuals to read through these library books, but on the organizations. Require them by law to stop collecting and distributing our information.

[René Villemure, Bloc, Trois-Rivières, QC]

Do you trust the industry to self regulate?

[Sharon Polsky]

We've already seen that happening in the United States, where the big technology companies have literally written the legislation that is being passed in several states. They call that privacy law. It's not. It doesn't protect individuals. It doesn't give them any greater right to protection or privacy. That's why in my remarks I said to craft these laws without the direct or indirect involvement of industry.

[René Villemure, Bloc, Trois-Rivières, QC]

You currently don't know anyone in the industry or any business that, in an effort to self regulate, would expand best practices, as we would like to see it done?

[Sharon Polsky]

There were many.... Not only am I the president of the Privacy and Access Council of Canada, but for roughly 30 years I've been doing privacy impact assessments and privacy consulting privately. Through that, I have been invited inside everything, from governments and public bodies to Fortune 100s.

I understand how they operate, the technologies they build and what they deploy. There are some that sincerely believe they're doing it right, but remember, without education, they're misled. They're maybe assessing their own understanding a little favourably.

You get into situations where it's a Canadian company and they store the information in Canada, but they use third parties in the United States to provide essential services for that app or that service. Companies don't recognize that as a problem. They don't notify users. They don't have any concern. They're ignorant, totally unaware that there is a privacy implication.

[René Villemure, Bloc, Trois-Rivières, QC]

To your knowledge, is the RCMP using facial recognition? If so, does it understand what it entails?

[Sharon Polsky]

I have had the opportunity to speak with a couple of very senior members of the RCMP, and they had, I think, a solid understanding. They are genuinely concerned. Their hands, I might say, are tied sometimes. Sometimes the technology is bought or trialled by somebody, and nobody else knows it.

That goes on in private sector organizations also. Instead of going through an approval process, somebody goes and buys something and plugs it in. If you don't know it's there, you can't monitor it and you can't sanction it.

Whether the RCMP is actually using facial recognition, I don't know for sure, but I wouldn't doubt it.

[René Villemure, Bloc, Trois-Rivières, QC]

Can you say a few more words about the required education? What kind of content is needed to raise awareness among people who are involved in that education?

[Sharon Polsky]

There was a professor at McGill a few years ago, with whom I was discussing developing some education to be rolled out to schools. There are media organizations and various privacy commissioners across the country, that have developed little courses, little programs. They're available; they're not mandatory, though.

I recognize that there's a problem, because it's federal, but education is provincial. However, to have, first of all—

[The Chair, Conservative, Pat Kelly]

I'm very sorry, but we're getting to be quite a bit over time. We're quite pressed today in our schedule.

I am going to have to switch and give the floor to Mr. Green, for up to six minutes.

[Matthew Green, NDP, Hamilton Centre, ON]

Thank you very much, Mr. Chair. My questions will be for Dr. Maslej.

The rapid expansion and development of AI technology comes with significant risks. Chapter 3 of your AI Index report outlines some of the harms of AI technologies, including “commercial facial recognition systems that discriminate based on race, résumé screening systems that discriminate on gender, and AI-powered clinical health tools that are biased along socio-economic and racial lines. These models have been found to reflect and amplify human social biases, discriminate based on protected attributes and generate false information about the world.”

Could you please elaborate on some of these harms and risks posed by the use of AI technologies, particularly at a time when investment in and development of technologies are so rapidly accelerating?

[Nestor Maslej]

Certainly. That's an excellent question. It speaks to the fact that although regulation of facial recognition technologies certainly matters, there are also other AI use cases that might merit regulatory attention.

One of the takeaways from the data we have on facial recognition is that facial recognition is in the middle of the pack in terms of total private investment. It's more invested than things like drones or legal tech, but it's behind NLP in medical and health care, which suggests, as well, that there are other AI use cases that might merit further attention.

As the data from the McKinsey survey suggests, facial recognition is not as embedded in certain business technologies and processes as are other AI systems. Again, this doesn't necessarily mean that we shouldn't care about facial recognition regulation; it's an issue of utmost importance. It's just that we should also be cognizant of the other problems AI might pose, especially at a time when AI is becoming increasingly ubiquitous.

You alluded to a couple of the different examples that we cited in the report. I'll speak to a couple of different ones.

We talked about the fact that there are résumé-screening systems that have been shown to discriminate based on gender. We cite evidence from a newspaper article that a couple of years ago, Amazon developed a machine-learning résumé-screening system that was shown to systematically downgrade the applications of women.

Again, it would be ideal for a lot of these companies, especially the very big ones, if someone gave them 100 résumés and they could just give them to a machine that says automatically that these are the best three candidates and just hire these three people.

The reason Amazon trained a biased system was that the system was ultimately trained on data from résumés that were submitted to Amazon previously. Overwhelmingly, the résumés that tended to be submitted to Amazon were submitted by men. This reflects the fact that the tech industry is mostly dominated by men. Given that men were mostly traditionally hired, the AI system learned to penalize the term “women”. That meant that if you included a resume that, for example, said, “I was captain of the women's swim team,” the algorithm saw that historically very few women have been hired at Amazon, this person has “women” in their résumé, so let's downgrade this résumé. Amazon claimed that this tool was not actually ever deployed to make hiring decisions, but the point stands that this bias remains.

We also talk about bias in multimodal linguistic models. We talk, as well, about bias in medical image segmentation. I could go on at length about that, but I'll perhaps give you the opportunity to pose additional questions.

[Matthew Green, NDP, Hamilton Centre, ON]

I do have additional questions on the types of regulatory frameworks that you think are needed. Very important to this committee is going to be our recommendation.

Doctor, if you could, talk about what frameworks are needed to protect Canadians when it comes to the use of these AI technologies, including the ones you just listed.

[Nestor Maslej]

I'll say a couple of things.

First, I'm not technically a doctor. Although I very much appreciate the title, I feel that I would be remiss not to correct that.

Second, it is perhaps outside my area of expertise to offer recommendations for the committee. I understand that they are very valuable and very essential, but I feel that I can comment most on the data and what impact—

[Matthew Green, NDP, Hamilton Centre, ON]

We can accept that.

Since we're on familiar terms, Nestor, if I could, given your subject matter expertise—and I would suggest, given your credentials, you have subject matter expertise—could you state for the record whether or not you support a moratorium on the use of facial recognition technologies and other forms of AI by law enforcement, until government is adequately able to catch up to its impacts?

[Nestor Maslej]

Again, I think answering that question is outside of my scope of expertise. I would defer more to the other witness on the panel, who I think has a bit more experience in this domain and can comment a bit more authoritatively.