Public Accounts Committee on Oct. 30th, 2014

[The Chair, NDP, David Christopherson]

I now declare this 36th meeting of the Standing Committee on Public Accounts in order.

Colleagues, you will recall that we followed our choice of chapters following the tabling of the spring 2014 report, and we're working our way through that. Today we have the NDP choice for the public hearing, and it is chapter 8, “Meeting Needs for Key Statistical Data—Statistics Canada”, of the spring 2014 report of the Auditor General of Canada.

I am pleased to advise that of course we have our Auditor General here, Mr. Michael Ferguson.

Welcome, sir. It's good to have you here again.

We also have the appropriate government officials here to give their testimony and answer questions.

I'll ask you to introduce yourselves and your delegation when I call upon you to give your opening remarks.

If there are no interventions from colleagues, or questions, or any reason why we should delay—and I'm not hearing any—we will then proceed. We will begin with the opening remarks of the Auditor General.

Mr. Ferguson, you now have the floor, sir.

[Michael Ferguson, Auditor General of Canada, Office of the Auditor General of Canada]

Mr. Chair, thank you for this opportunity to discuss chapter 8, “Meeting Needs for Key Statistical Data”, from our 2014 spring report.

Joining me at the table are John Affleck, principal, and Colin Meredith, director, who were responsible for the audit.

As Canada's national statistical agency, Statistics Canada plays a fundamental role in generating information on Canada's socio-economic conditions. This information is used by governments, researchers, academics, businesses, and non-governmental organizations.

We examined whether Statistics Canada ensures the quality of key socio-economic information it produces and whether it generates this information efficiently and in response to priority user needs. Our examination of quality and the efficient use of resources was focused on four data products that provide important information on Canada's economic and social conditions: the consumer price index; the labour force survey; the national household survey; and the survey of employment, payrolls, and hours.

Overall, we found that Statistics Canada applied its quality assurance framework to ensure the quality of the statistical programs we examined. However, improvements are needed to better meet user needs.

We found that Statistics Canada consults mainly with federal, provincial, ans territorial users, with less attention paid to the private sector, municipalities and non-government organizations.

Without ongoing consultation and feedback mechanisms for the full range of users, the agency's data may become less relevant. For example, detailed information on job vacancies—by both occupation and location—could inform job seekers and policy-makers on which occupations are in demand and where. Policy-makers could also use this information to develop and manage policies and programs.

However, information on job vacancies is not broken down within provinces, so it is not possible to know if Alberta job vacancies are in Fort McMurray, Medicine Hat, or in any other community. Users told us that as a result of the shortcomings, this information is of limited value to them.

We found that the agency managed the implementation of the national household survey well. Nevertheless, introducing this voluntary survey to replace the mandatory long form questionnaire that was part of the previous census of population had an impact on users. The agency anticipated a drop in response rate as a result of the move to a voluntary survey. It took a number of steps to mitigate this risk, such as increasing the number of households in the initial sample from 3 million to 4.5 million, and targeting its efforts to encourage households to respond. Notwithstanding the steps taken, the actual response rate dropped from 94% in the 2006 census to 69%. While the number of responses to the 2006 census and the 2011 survey were similar, it is the response rate that determines how the data can be used.

Statistics Canada then took additional steps consistent with its quality assurance framework. It decided not to release data, especially for some sparsely populated, remote areas, or rural communities, because of low response rates. The agency estimated that reliable national data was unavailable from the national household survey for 3% of the Canadian population, up from 1% in the 2006 census. The agency also informed users that they should use caution when making comparisons between data from the national household survey and data from previous censuses.

We also examined whether the agency generated data for the four data products efficiently over the audit. We found that Statistics Canada had identified and implemented a number of approaches that resulted in more efficient use of resources. These included using common corporate services and identifying alternate data sources.

In order to maintain the relevance of its work, it is important for Statistics Canada to identify and respond to emerging needs. We examined how the agency assesses and addresses identified gaps, and how it ensures that its approved priorities are implemented.

We found that Statistics Canada had an adequate process to identify and implement its priorities, including emerging needs. However the agency did not systematically consider the need to consider cost-recovered work when external funding is withdrawn. For example, after work ended on a client-funded survey of older workers, the agency did not assess whether it should have continued to fund the survey out of its internal resources.

Statistics Canada has agreed with our five recommendations and has set deadlines for implementing them, ranging from April 2014 to January 2015.

Mr. Chair, this concludes my opening remarks. We would be pleased to answer any questions the committee may have.

Thank you.

[The Chair, NDP, David Christopherson]

That's very good.

Thank you, Mr. Ferguson.

Now we'll move along to the chief statistician of Canada, Mr. Smith.

You now have the floor, sir.

[Wayne Smith, Chief Statistician of Canada, Statistics Canada]

Thank you, Mr. Chair.

Good afternoon, committee members.

I have with me at the table today Stéphane Dufour, the assistant chief statistician for corporate services at Statistics Canada.

As chief statistician of Canada, it is my pleasure to speak to you about the Auditor General's report in which he assessed how well Statistics Canada meets the key statistical information needs of Canadians and their institutions. I was pleased that the chapter on Statistics Canada demonstrated that overall, the agency makes efficient use of its resources to produce relevant and timely data of good quality. Statistics Canada agrees with the Auditor General's recommendations, and we have already begun implementation of our action plan.

In my remarks today, I would like to focus on some of the key issues raised in the report. Essentially these issues centre on the challenge of meeting the diverse statistical needs of a multitude of users, while at the same time ensuring a robust level of quality. As a fundamental principle, since Statistics Canada defines quality as fitness for use, Statistics Canada would never sacrifice quality to deal with budgetary constraints. If need be, Statistics Canada limits the scope of its statistical program, rather than compromise the quality of the information it produces.

In addition to the Censuses of Population and Agriculture, which are conducted every five years, the agency also has 350 active surveys and makes use of approximately 500 administrative data sources to inform Canadians on the economic, social and environmental conditions of their country. Statistics Canada is held in high regard internationally for the quality of its data, its methodologies and its efficiency.

Since the agency cannot accommodate all information needs within its base budget, we undertake statistical work on behalf of external clients, primarily federal and provincial government departments, on a cost recovery basis. This allows us to respond to emerging needs while ensuring that all Canadians benefit from the information that is ultimately produced. Many surveys have been conducted this way over several decades. Normally, this work ends if the client no longer wishes to fund it. On occasion, Statistics Canada will assume responsibility for funding a program that it considers too strategic to allow to be discontinued. The agency's ability to self-fund these programs is, of course, extremely limited.

The Auditor General has noted that this process of considering whether to self-fund a former cost recovery program has not been formalized at Statistics Canada. The agency has now put in place a mechanism, in the course of its annual planning process, to formally review cost recovery programs that are losing funding, in order to determine whether it is necessary to provide, or to try to secure, permanent funding. In the absence of new external funding, maintaining a cost recovery program using existing funding would, of course, mean discontinuing some other programs.

In our rapidly changing world, there is an unquenchable thirst for information. Statistics Canada receives demands not only for new data on a wide variety of topics, but increasingly for data at the local level, and for very small populations. To fully respond to this thirst for data would require many times the budget we presently possess. Even with unlimited funds at our disposal, the issue of the burden that responding to surveys would impose on the Canadian population and businesses would remain intact. It's necessary for Statistics Canada to balance the burden that we impose on Canadians and their businesses in responding to our surveys against the value of the information that would be obtained.

To address the growing need for data for small areas and small populations, I am pleased to report that Statistics Canada is making great strides by developing and exploiting state-of-the-art techniques, such as micro-data simulation and smaller re-estimation, to extract the maximum value from existing datasets. Our growing use of administrative data is also increasing our ability to provide data for small areas and populations. These techniques produce, at affordable cost, estimates at a detail beyond that of a single survey, without imposing an additional response burden on Canadians and their businesses.

In the Auditor General's report, there is some focus on the National Household Survey and the agency's decision not to publish local estimates for more than 1,100 communities, representing 3% of the population, due to quality concerns. Lower response rates relative to those of the mandatory 2006 census long-form, due to the voluntary nature of this survey, were the principal driver behind this decision.

While there has been some erosion of data quality due to lower response rates, the National Household Survey nonetheless produced a massive data base of robust information at all levels of geography and for many small populations. To seek, as some have, to dissuade Canadians from using this rich and powerful data source does them a terrible disservice.

Relevance is one of the underpinnings of a successful national statistical office. Statistics Canada seeks to align its statistical and analytical programs with the highest priority information needs of the nation. To accomplish this, we have always engaged a wide range of partners, users, and stakeholders in order to understand the revolving information needs and to ground our statistics in a genuine understanding of the phenomena we are trying to measure.

As recommended by the Auditor General, Statistics Canada will in future systematically document this outreach process and demonstrate clearly how it is taken into account in the agency's program planning.

As part of the agency's move to expand its consultative base, the membership of the national statistics council has recently been modified to broaden the number of voices at the table. This council sits at the pinnacle of a system of advisory committees and brings together distinguished Canadians with strong knowledge of Canada's national statistical system and a strong interest in participating in its development.

In addition, we have begun to formalize the renewal process for the agency's numerous advisory committees, to ensure they are properly constituted given their mandates, thus helping the agency remain aware of and responsive to evolving needs and statistical methods.

Let me conclude by saying that I believe our national statistical system is a national asset of great value, which produces returns for Canadians well beyond their investment, by providing them with high quality information about Canada's economy, society, and environment, information that they require to function effectively as citizens and decision-makers in a rapidly evolving world.

The quality of Statistics Canada's programs, the morale of its employees, and its international reputation for excellence are very much intact.

Thank you.

[The Chair, NDP, David Christopherson]

That's very good, thank you.

That ends our opening comments.

We'll now begin the questions in the usual rotation, beginning with Mr. Hayes.

[Bryan Hayes, Conservative, Sault Ste. Marie, ON]

Thank you, Mr. Chair.

I'm going to speak to the management response and action plan, specifically recommendation number one. It's pointed out that many agency programs are primarily designed to meet the needs of federal and provincial policy departments, but the Auditor General has wanted you to have a look in terms of what has been done specific to the private sector, to advance the needs of the private sector.

I would ask for your response in terms of moving forward. How are you addressing contacting the private sector and making sure you're fulfilling its needs?

[Wayne Smith]

While we view Statistics Canada as a national statistical office and, therefore, try to meet the needs of the entire nation, all Canadians, the reality is that with the budget available our focus is primarily on government and primarily on the two most senior levels of government. A lot of our program certainly addresses the needs of federal departments, and the provincial and territorial governments as well. The census particularly addresses the needs of municipal governments.

Most of the rest of the use of our program is really residual use of the information that we've created in response to the needs of those governments, by other organizations. So non-government organizations use our information to, for example, develop positions on government policies, as do businesses. Businesses use our information—for example, the census data—extensively in their marketing efforts. Financial institutions use our information extensively in their analysis of financial markets and adjust their strategies in response to that.

We do meet regularly. I recently met with a group representing service industries to talk about our program and how it meets their needs and where they felt there were gaps that should be addressed.

I also participated recently in a business round table to talk about how our data is used in conjunction with, for example, big data sources in order to meet the needs of the private sector.

Every time we release the national accounts—the gross domestic product, for example—we meet with a table of senior economists from the private sector to talk about their views of the data we've just released, any concerns they have.

We've added three or four members to our national statistical council, who come from the business community, to hear their voices.

In our economic statistics programs and our advisory committees, we have representation from the business sector, and business associations as well. It's a very broad-based effort to make sure we're engaging with them.

[Bryan Hayes, Conservative, Sault Ste. Marie, ON]

The reason I ask is because on your behalf, but on my own and that of my staff, I'm familiar with a company in Sault Ste. Marie, Sault Ste. Marie Innovation Centre. They are a world renowned company. This is a company that over the past five years has completed 50 projects that use StatsCan data, and these are really important projects for our community. We asked them some questions in terms of their dealings with StatsCan. They made a comment that overall, the data provided by Stats Canada has brought to light the issues of Sault Ste. Marie and has led to a meaningful definition of what is going on in the community while attracting significant funding for projects. Stats Canada data is the foundation of all datasets they use in all the good they try to do for Sault Ste. Marie.

They were really impressed with you. I actually put the question to them in terms of what the process was for somebody who needs assistance. I'll give you their response.

I'm going to ask you, what is the process for an agency that needs assistance?

[Wayne Smith]

There are multiple roads into Statistics Canada. We put a lot of emphasis on self-service through our website. Most of our information is available or downloadable from our website if people want service directly. We also have a telephone inquiry service where people can call and where we will establish what their needs are and take appropriate action based on the identified needs. In each of our regions, we have essentially senior councillors whose job it is to make contact with major data users in their region. We have a number of people operating out of Toronto who, if they are approached by an organization like the one you're referring to, would make it their job to step in and say, okay, what is it you need, how can we help you, these are the things that we can do.

[Bryan Hayes, Conservative, Sault Ste. Marie, ON]

I want to get their response on record before my time runs out.

[The Chair, NDP, David Christopherson]

Time—

[Bryan Hayes, Conservative, Sault Ste. Marie, ON]

It's a 30-second response.

[The Chair, NDP, David Christopherson]

—is about to run out in less than 30 seconds.

[Bryan Hayes, Conservative, Sault Ste. Marie, ON]

He says that Stats Canada great to deal with and that any questions he asks are answered in a timely and appropriate fashion. You need to know that.

Thank you, Mr. Chair.

[The Chair, NDP, David Christopherson]

Moving along.

Mr. Allen, you have the floor, sir.

[Malcolm Allen, NDP, Welland, ON]

Thank you, Mr. Chair.

Thank you to our guests this afternoon.

If I could start with you, Mr. Smith, I'll take you to page 13 of the Auditor General's report, section 8.52, which basically is the response rate you got in 2006. It was 94% and was, basically, the last time or thereabouts that we had to do a mandatory long form, versus the 69% in 2011 where we were doing a voluntary form.

Are you concerned with that drop in respondents? Does that concern you at all as someone who works with data?

[Wayne Smith]

There's absolutely no doubt that I would have preferred to have a 94% response rate or any response rate higher than 69%, but the fact that the response rate was 69% is not in itself a condemnation of the data.

Response rates really have two impacts when you start talking about data. One is an impact on the statistical variability of the estimates—when you think about polls and when you talk about how this estimate is accurate plus or minus 2%, 95 times out of 100. That's where the 69% comes in. If we hadn't adjusted the size of the sample as we did, that would have resulted in the estimates being of a much poorer quality from that perspective. Because we adjusted the size of the sample and went from a 20% sampling rate to a 30% sampling rate, we actually got, as the Auditor General noted, the same number of responses from households and Canadians—it was actually a slightly higher number—and that took care of that issue. In terms of sampling variability, the estimates from the 2011 national household survey, as we demonstrated in some documentation that we released regarding coefficients of variation, were roughly as good as what we got from the 2006 census.

The second issue is non-response bias. There was a possibility that, because the proportion of people who answered is smaller and significantly different from 100%, those people might be significantly different in terms of their characteristics in the population as a whole. A lot of claims have been made. A lot of people raised concerns about that possibility. We spent a very large amount of time as were were publishing and prior to publishing the data, looking at that issue.

[Malcolm Allen, NDP, Welland, ON]

You could spend the next five days explaining to us how the dataset works, but I get only a few minutes.

I want to talk to Mr. Ferguson about the next paragraph. I have a specific question. I just need it to be clear in my head. You talked about a survey. There were 4,567 subdivisions of which 1,128 were not used due to quality concerns. So you said 25% were unusable due to quality concerns. Then it says, “data for an additional 686 census subdivisions, or 15 percent, was not released for confidentiality and other reasons.”

The simple question is, should I be adding those two numbers together, or is that number 686 in the 1,128?

[Michael Ferguson]

In that particular paragraph, we are referring to two different things, or two measures, I guess. We're referring to the percentage of census subdivisions, and then at the bottom of that paragraph, we are referring to the percentage of population.

In terms of the percentage of census subdivisions, the two are additive—the 25% and the 15%. But adding in the additional 15% really doesn't have any impact on the 3% of population. Whether you consider 25% or 15% of the census subdivisions, the impact was still the same in terms of the percentage of population.

[Malcolm Allen, NDP, Welland, ON]

I think that's attributable to the fact that we're looking at more rural components, so the number of folks doesn't really change significantly when you add the two together, which I get. But would that not also suggest that, when we're trying to take a picture of Canada through data, we don't get a very clear picture if we have a large percentage and, if you add the two together, 40% of subdivisions in rural Canada is pretty significant, not being usable?

I recognize, Mr. Ferguson, you may not want to answer that specifically, but is that not the case?

I see Mr. Smith shaking his head to say “no”, and I'll allow him to answer if he wishes.

But clearly if you don't have data, how do you make a decision? Data's meant to help folks make decisions. That's why you supply it for us. If we don't have data, how do we make informed decisions, regardless of who we are and what we're using it for?

[Wayne Smith]

To be clear, the data for those areas is included in the national picture and in the provincial picture. It's included in the census division picture. It's quite possible by combining several areas to generate an area for which we can provide data. That's been done.

The other consideration is that this data is in fact available. The response rates are in fact considerably higher than what you would obtain in a typical poll. A 50% non-response is still....

In our view, the data was not of sufficient quality. For our standards, it's not unusable. We simply feel that it's something we wouldn't put into people's hands without counselling them to be very cautious in the use of it.

Your point that those communities that did not receive data representing them have suffered a loss is absolutely true.

[The Chair, NDP, David Christopherson]

Sorry, Mr. Allen, time has well expired.

Moving along, Mr. Woodworth, you now have the floor, sir.

[Stephen Woodworth, Conservative, Kitchener Centre, ON]

Thank you very much, Mr. Chair.

My thanks to the witnesses for attending today. I have quite a number of questions, but I want to just clear up one or two points that arose a few moments ago.

I'll direct some questions to Mr. Smith.

You didn't have time to tell us the conclusion, if any, that you reached when you spent all that time looking at the issue of possible non-response bias. I wonder if you could just take a quick moment to tell us if you reached a conclusion as a result of that possibility regarding the national household survey, and what it was.

[Wayne Smith]

Essentially the conclusion was that, of the various claims that had been made about all the groups that would be under-represented, we found no real evidence that had occurred. We did find a slight discrepancy with regard to certain income data for low-income families. We identified that in information we published at the time the data was released.

Generally speaking, any serious bias in the data, either as people had suggested might occur or that we found that hadn't been previously suggested might occur, didn't happen really, which is why [Technical Difficulty—Editor] robust.

[Stephen Woodworth, Conservative, Kitchener Centre, ON]

Sure. On this side, at least, we're looking at evidence-based solutions, so it's good to know there's no evidence, or no significant evidence, in support of that.

On the question of the small population subgroups, for example, low-income earners, as I understand it from the Auditor General's report, your agency actually targeted its efforts following up with households that didn't initially complete their questionnaires based on geographic areas known to include relatively high proportions of certain of those subgroups, including low-income earners. You actually proactively went after that subgroup. Is that right?