Industry Committee on July 25th, 2022

[The Chair, Liberal, Joël Lightbound]

Thank you.

Wrapping up the last round is Ms. Lapointe for five minutes.

[Viviane LaPointe, Liberal, Sudbury, ON]

Thank you, Mr. Chair.

Thank you, Minister.

Minister, when you met with the heads of Rogers and the other service providers, you asked them to work together on an action plan. You've said that a few times this morning. I get the sense, though, that the providers don't want to work with their competitors on a partnership initiative.

Is that your sense as well? What can we do to challenge that?

[François-Philippe Champagne, Liberal, Saint-Maurice—Champlain, QC]

I'd like to thank the honourable member for her question.

They actually don't have a choice. When I spoke to them, I made clear the frustration of more than 12 million Canadians. I directed them to enter into a formal agreement covering the three areas I mentioned, mutual assistance, emergency roaming and a communication protocol.

I also made clear to them that this was just the start, that more would be coming once the CRTC had completed its investigation. As committee members have mentioned, we will be looking at the investigation findings to see where we can take further action. I am quite ready to do that. Our focus is on finding solutions and improving network resiliency across the country.

[Viviane LaPointe, Liberal, Sudbury, ON]

What worries me is that the providers want market protection, and they want to play key roles in the country's infrastructure services. The Interac system is one that comes to mind. The system ground to a halt, costing the economy millions.

What is the best way to protect Canadians and Canadian businesses?

[François-Philippe Champagne, Liberal, Saint-Maurice—Champlain, QC]

The first thing we need to do is improve resiliency and competition.

I also spoke with the head of Interac, and I think there are lessons to be learned when it comes to building more redundancy into networks.

As I told Mr. Deltell, the head of Rogers has to answer some serious questions. I asked him those questions, but I think the committee has a role to play in getting to the bottom of what happened and identifying what further measures need to be taken.

We took some immediate actions after the outage, first to restore connectivity, but then to increase resiliency. I hope the committee can get to the bottom of things. The CRTC is doing that, and its recommendations will inform the additional measures we take. We all want the same thing, after all.

We want a more resilient network, and that is what I've demanded of the country's telecommunications companies. We want them to work together to build the enhanced resilience Canadians expect from the country's telecommunications system.

[Viviane LaPointe, Liberal, Sudbury, ON]

One of the questions I have for the Rogers executives this afternoon has to do with the communication plan.

I know you talked about it, but do you think they really have a communication plan to give Canadians the facts?

[François-Philippe Champagne, Liberal, Saint-Maurice—Champlain, QC]

They need to have one within the next 45 days, because that's one of the three requirements I set out for the country's telecommunications carriers.

I think you've touched on the frustration of the over 12 million Rogers customers, who did not get the information they needed during the outage. Seeing what transpired, we directed the carriers to provide what I call emergency roaming, like our neighbours to the south, but we went further by requiring the companies to develop a very clear communication protocol in the event of another outage of this magnitude.

That means a communication protocol needs to be in place to ensure not only Canadians, but also government authorities, are adequately informed. That way, the authorities can provide support to the companies affected should something like this happen again.

[Viviane LaPointe, Liberal, Sudbury, ON]

You told us about the first steps. What are the next steps for Rogers and the other carriers?

[François-Philippe Champagne, Liberal, Saint-Maurice—Champlain, QC]

Thank you for your question.

The next logical step is to see what the CRTC recommends once its investigation is complete. Obviously, I'm going to look carefully at the CRTC's recommendations to see what other requirements we can set out for Canada's telecommunications carriers. Keep in mind that the three measures I have already asked for need to be formally implemented in the next 45 days.

The next step, then, is the CRTC's investigation and recommendations, followed by the implementation of additional measures to improve resiliency.

Of course, the committee may have its own recommendations, which I will pay close attention to.

[The Chair, Liberal, Joël Lightbound]

Thank you, Ms. Lapointe.

Thank you, Minister, and thank you to the officials accompanying you today. We appreciate your making yourself available on such short notice.

[François-Philippe Champagne, Liberal, Saint-Maurice—Champlain, QC]

Thank you, Mr. Chair.

Thank you to the honourable members for the opportunity to speak with the committee today.

[The Chair, Liberal, Joël Lightbound]

That concludes the first half of the meeting.

We will now suspend for two minutes, so the Rogers executives can take their seats.

Thank you.

[The Chair, Liberal, Joël Lightbound]

Honourable members, please be seated. We are resuming the meeting of the Standing Committee on Industry and Technology.

We'll resume meeting number 31 of the House of Commons Standing Committee on Industry and Technology, pursuant to Standing Order 108(2) and the motion adopted by the committee on Friday, July 15, 2022, for this committee to shed some light on the Rogers Communications service outages in early July.

Without further ado, we have with us here today Mr. Tony Staffieri, president and chief executive officer of Rogers.

Also with us is Ron McKenzie, Rogers' chief information and technology officer.

Thank you for being here today.

Without further ado, I will turn the floor over to you for five minutes.

[Tony Staffieri, President and Chief Executive Officer, Rogers Communications Inc.]

Good afternoon, Chair and members of the committee. Thank you for the invitation to be with you.

My name is Tony Staffieri, and I am president and CEO of Rogers Communications. I am joined today by our new chief technology officer, Ron McKenzie.

I appear before you today, because as Rogers' CEO, I'm accountable for the outage that occurred on July 8. I'm also responsible for the specific actions we are taking as a company to make sure this does not happen again.

On that day, we failed to deliver on our promise to be Canada's most reliable network. More than a marketing slogan, we know just how critical the wireless phone and Internet services that Rogers provides are. Canadians need to be able to reach their families, businesses need to be able to accept payments and, most importantly, emergency calls to 911 simply have to work every time.

To those who were impacted by our outage, I am sorry.

Today I want to share with you how we are working to win back the trust and confidence of Canadians.

I will start with what happened and why there was a delay in restoring our service. I'll discuss the important steps we're taking to help prevent this from happening again, and I'll conclude with some of the steps we have begun to take to make things better for our customers.

Simply put, this outage was a result of a system failure following an update in our core network.

Given the enormous complexity of all modern networks, understanding what caused the outage took some time. Once the cause was identified, our technical experts needed more time to methodically bring traffic back up, as we had millions of customers trying to access their phones, home TV and Internet, all at about the same time.

To manage those returning traffic volumes, we had to physically disconnect the impacted equipment. Throughout this process, we had one singular and overriding focus: to get our customers up and running as quickly as we possibly could.

I understand the frustration our customers felt in not knowing when our networks would be back online. I wanted a timeline, but the fact is that we did not have one and didn't want to provide an estimate that might turn out to be wrong.

In the conversations I've had with customers and with small and large business owners, there is one thing everybody wants to know: What is Rogers doing, today, now, to learn from this outage and ensure it won't happen again?

I've said we will make every investment needed to do our best to make sure that won't happen. That investment begins with the work now under way through our enhanced reliability plan. Working with the government and our competitors, we are making significant progress on a formal agreement to ensure that 911 calls can always be made, even in the event of an outage on any carrier's network.

Making this a reality is the only responsible way forward, and I am personally committed to making it possible for all Canadians. I would like to thank Minister Champagne for his leadership on this.

When it comes to our network, we will do our part and then some. To guard against a system-wide outage, we will set a higher standard by physically separating our wireless and Internet networks and creating an “always on” network. To be frank, this added layer of protection will be expensive. We estimate that it will cost at least a quarter of a billion dollars, but we know it is the right thing to do.

We will also continue with our plan to invest heavily in reliability. We will spend over $10 billion over the next few years to build out and strengthen our network. This investment includes additional oversight, more testing and greater use of artificial intelligence to ensure that the upgrades we make to our network work as intended.

Finally, we have partnered with leading technology firms to do a full review of our network systems to learn from the outage and emerge stronger.

When this work is complete, we will share the key lessons with our competitors and other industry partners.

When it comes to making things better for those who were impacted by our outage, we have already extended five days of credit to every Rogers' customer. As well, we are working with our business customers to better understand the implications of the outage on their organizations.

Chair, I know that it is only through our actions and with time that we can restore Canadians' confidence in us. We can and we will do better.

I thank the committee for inviting us to speak with you today, and I look forward to your questions.

[The Chair, Liberal, Joël Lightbound]

Thank you very much.

We'll now move to the first round of questions with MP Michael Kram for six minutes.

[Michael Kram, Conservative, Regina—Wascana, SK]

Thank you very much, Mr. Chair.

Thank you to the representatives from Rogers for joining us today at the committee.

I was wondering if either of you could describe for the committee the quality assurance practices that Rogers has in place for major software or hardware updates.

[Tony Staffieri]

Through you, Mr. Chair, we have extensive policies and procedures, and I'll have my colleague walk through some of those. I can assure you that not only have they been extensive, but we also upgrade those continually.

Network resiliency and redundancy are extremely important to us and in our industry.

[Ron McKenzie, Chief Technology and Information Officer, Rogers Communications Inc.]

This was an unprecedented impact to what's called the “core” of the network. The core of the network you can think of as the brain of the network that controls all access of flow of information and flow of connectivity for all services to the Internet.

We have extensive...over two years in this particular program, with full rigour, full review and full analysis that was done. The implementation was part of a seven-stage program, and this was stage six. The previous five stages had no issues whatsoever.

This was truly an unprecedented incident. We have very thorough reviews, very thorough simulations and very thorough testing, but this was an unprecedented incident.

[Michael Kram, Conservative, Regina—Wascana, SK]

Does Rogers have a testing environment in place where you test updates before releasing them into the live production environment?

[Ron McKenzie]

That's correct. We do.

We do both extensive simulation and extensive testing. We also rely heavily, side by side, with the equipment vendors who work with us. As we design and as we test, the equipment vendors work right alongside us. They provide the same infrastructure to all operators around the world, so we also gain the benefit of the learnings of all other operators in terms of how they've executed and how they implement.

[Michael Kram, Conservative, Regina—Wascana, SK]

I guess the questions that have to be asked are, why was the root cause of this month's outage not caught in the testing environment, and also, why was the root cause of the April 2021 outage not caught in the testing environment as well?

[Ron McKenzie]

First of all, the events are very, very different.

Let me speak to the event of July 8. The way to characterize it is that there was no way for the engineer at the time.... When it had been performed five times prior with no incident whatsoever, there was no belief, no information at the time, that there was going to be any issue.

What happened was that when the code change was executed and the filter removed, the behaviour of the equipment, in the way it's designed between one vendor and a second vendor, was very different. That was the unknown at the time: the behaviour of one device, one manufacturer, who executes a standard one way, versus another manufacturer. It was that sequence that caused essentially the event to then flood the core network.

[Michael Kram, Conservative, Regina—Wascana, SK]

And the April 2021 outage?

[Ron McKenzie]

The April 21 outage was a software update performed by our partner. The software update in that instance was tested, but the nature of the way it was deployed caused the impact. That was done by our vendor. It was tested. It had been deployed in other networks around the world. It was an instance that the particular dynamic of the way it was deployed caused the incident in April 2021.

[Michael Kram, Conservative, Regina—Wascana, SK]

I have limited time in my questions, and I don't think we're going to get to the actual line of code in five minutes of committee work, but if Rogers could submit a document of a more technical nature to the committee, I would certainly find that helpful.