Industry Committee on July 25th, 2022

[Viviane LaPointe, Liberal, Sudbury, ON]

For all Canadians, it's difficult to understand how a scheduled upgrade could cause such a massive failure of Rogers' network. Normally one would expect the upgrade to be tested extensively and deployed in a sandbox test environment. In fact, when Mr. McKenzie was responding to MP Kram, he indicated that this was done.

I want to dive a little deeper. Did you use both Cisco and Juniper in the sandbox test or only one of those manufacturers' products? How will you ensure future sandbox testing will have greater rigour and resilience?

[Tony Staffieri]

I'll start with two broad comments. One is that pretesting of certain changes, where practical, is always done. In certain cases it's impractical, and what you saw here was an outage that made its way throughout the entire network. It's difficult to have a test environment that replicates the entirety of a very complex network across the nation. That's why in this case we would have been unable to test this in what we would call a lab environment.

[Ron McKenzie]

It's very difficult to simulate the live full environment. It can be parts. Because this was the sixth time we had done this and in the previous five times there was never an indication of any issue, this was a very unique case in the way one particular manufacturer designed their system and how it handles flow control and traffic. The way it went into one mode flooded because the other mode handles traffic in a very different fashion. It was within minutes there was a complete loss.

[Viviane LaPointe, Liberal, Sudbury, ON]

Could the Canada-wide failure have been avoided with a staged regional deployment whereby you would have first deployed it in some smaller regional networks and then expanded gradually across the country, even over a few days if no issues were detected?

[Ron McKenzie]

All carriers use, and have used for many years, the concept of a common IP core. The common IP core is what's at the root here. A common IP core is obviously designed for security purposes. It lets you manage, and essentially I would describe it as the brain of the network. It's not something that traditionally is regionalized. All operators use a common core and then use what are called “access networks” to feed into that. The access networks can be managed. In this case—

[Viviane LaPointe, Liberal, Sudbury, ON]

Thank you. I have limited time.

I want to focus on what occurred during the 16-hour outage. The question I have to ask is, does Rogers have a robust and tested emergency preparedness and disaster recovery plan? I ask because part of a disaster recovery process is regular communication with affected stakeholders. That did not take place in this instance or, for that matter, in last year's outage. Even regular, hourly messages indicating that Rogers was aware of the issue and working to isolate and resolve the problem would have been better than the silence we saw.

As to the second part of my question, Mr. Staffieri, when did you learn of the outage and why didn't you inform the public and the government?

[Tony Staffieri]

In response to the first part of the question, we do have an emergency response team. Shortly after the incident occurred, that team, which is well prepared, pulled together the resources needed to go through a plan that is put in place well in advance on how to react to these types of situations. We were delayed in this instance because our own network had gone completely dark. Nonetheless, we quickly made arrangements for alternative connectivity and that caused some delay.

Notwithstanding that, in terms of communication, broadcast started as early as 5:30 in the morning that there was a network problem. We did not know fully until much later the full extent and that it was a national outage. Once we knew, we communicated that on social media, which was just before 9 a.m., and on radio stations.

When I spoke to Canadians—customers—afterwards, what they really wanted to know, which is what we wanted to know, was the root cause and when this was going to be fixed. We didn't know the answer to that. We always strive to be extremely transparent with customers and Canadians and share as much information as we know, but we didn't know the answer to those questions. As we got more information, we shared that publicly. We did not know until late afternoon that we had a fix that would allow the network to come back up. Once we knew that, we promptly communicated it through all the relevant media channels.

[The Chair, Liberal, Joël Lightbound]

Thank you Mr. Staffieri.

Thank you, Ms. Lapointe.

It's now over to Mr. Lemire for two and a half minutes.

[Sébastien Lemire, Bloc, Abitibi—Témiscamingue, QC]

Thank you, Mr. Chair.

Who was on your emergency response team to deal with the outage? Was Mr. Fernandes, your then chief technology and information officer, on it?

What did you learn from the event? Would you have the same people on your emergency response team if a widespread outage occurred today?

[Tony Staffieri]

In terms of our emergency response plan, we bring to bear, depending on the type of emergency, all the right resources, including me and whoever is responsible. You can imagine that every one of these has the full attention of the organization.

Certainly on this outage, Mr. Fernandes, who was our CTO, was immediately notified at just before 5 a.m., and was very much integral to the recovery plans and investigation of what went on that day, together with the rest of the network team. We have several command centres issued throughout the country. The outage happened just before 5 a.m. By 5:30 a.m., teams were already pulled together to start to work on the problem.

[Sébastien Lemire, Bloc, Abitibi—Témiscamingue, QC]

Thank you.

What do you make of the solution Mr. Champagne, the minister, somewhat forced on you? I'm talking about having to rely on your competitors for backup in an outage like the one that just happened.

[Tony Staffieri]

Minister Champagne's initiative to have the industry work collaboratively, in a very formal document, is going to be very constructive to a redundancy and resiliency program. As I've mentioned, we've made very good progress on moving that forward. We will have it completed within the 60-day time frame.

Importantly, it will provide for failover measures in the event that one network goes down; importantly, for 911, so that there is a robust plan to ensure that every single call transfers over; and finally, communication protocols, so that the minister and other important government agencies are alerted as quickly as possible about any future outages.

[Sébastien Lemire, Bloc, Abitibi—Témiscamingue, QC]

If your competitors ran into a similar problem, you would co‑operate with them.

Is that right?

[Tony Staffieri]

We would very much co-operate with our competitors. In these types of situations, there is always collaboration with our competitors in order to obtain a fix. Early on, I received calls from the CEOs of the other telecom operators asking how they could help. There's always been that type of collaboration and co-operation. It's in our interest as an industry for all Canadians to ensure that we recover quickly, but as you heard earlier, for technical reasons we were unable to transfer over our traffic volume to our competitors' networks.

[Sébastien Lemire, Bloc, Abitibi—Témiscamingue, QC]

Thank you.

[The Chair, Liberal, Joël Lightbound]

Thank you very much.

We'll now move to Mr. Masse for two and a half minutes.

[Brian Masse, NDP, Windsor West, ON]

Thank you, Mr. Chair.

Following up from Mr. Lemire's questioning, the interesting aspect of this is that without any type of essential service bill of rights or something of that nature, we have to rely upon private meetings amongst the CEOs and the boards in private meetings with the minister to find a solution. Is that really what's being presented here, that we rely on that and you guys come back to us later on, after you have those private meetings? What transparency is there for Canadians? I mean, this sounds like something that really is not going to build the trust that I think is necessary.

[Tony Staffieri]

Throughout this process, we have been as transparent as we can be. If you look at some of the filings we have made, including most recently the CRTC filing, we want to share all the information on this outage. It only makes our industry better and more prepared for this...to happen in the future.

As part of this agreement with our competitors on this memorandum of understanding, we look forward to being very transparent—because Canadians deserve to know—on how the industry is going to ensure that this does not happen again at this scale.

[Brian Masse, NDP, Windsor West, ON]

I guess I hesitate, because I've seen that show before, where we have a few industry players together and meeting in private and where there's no public access to any of those meetings. The minister can follow up, but the minister won't be able to provide documents, or doesn't have to, even to Parliament or this committee.

I fail to see how this process is going to build public trust. I think the other carriers that are going to be involved in this are probably going to get their reputations brought into this as well. Quite frankly, what's happened is that either through public negligence, through Parliament and the processes that we've had here, or through the industry itself, we have failed collectively on 911, something that was supposed to be guaranteed.

I'll finish with this, Mr. Chair. I don't think this process is going to satisfy the public. Those meetings will not be public. There will be heavily redacted materials. We'll have to go to access to information to find out what is going on even for scheduling and so forth.

I'm worried about it at this point in time, Mr. Chair.

[The Chair, Liberal, Joël Lightbound]

Thank you, Mr. Masse.

We now go to Mrs. Gray for five minutes.

[Tracy Gray, Conservative, Kelowna—Lake Country, BC]

Thank you, Mr. Chair. Thank you to the witnesses for being here.

We learned that with your core system going down, that also included 911 calls going down. There were also four emergency alerts that did not work, one dangerous person alert by the RCMP and three Environment Canada tornado warnings. Is that correct?

[Tony Staffieri]

I'm not aware of all the specific messages that didn't go out, but as I've said, it's unacceptable that even a single emergency message did not go out. That's what we're focused on correcting.

[Tracy Gray, Conservative, Kelowna—Lake Country, BC]

Okay. Did you have protocols and processes in place should there be an outage like what occurred, so that critical services like 911 and emergency alerts would still operate?

[Tony Staffieri]

We had procedures in place to ensure that 911 and emergency essential services would switch over to an alternate carrier's network. For very specific technical reasons, that automatic transfer did not happen.

[Tracy Gray, Conservative, Kelowna—Lake Country, BC]

In the submission that you made to the CRTC answering questions, it read, “While considered many times during the day, shutting down the RAN”—the radio access network—“was simply not a solution.”

Is that correct? Is that what you're referring to?