Public Accounts Committee on May 4th, 2010

[Randy Hoback, Conservative, Prince Albert, SK]

Are you satisfied that the evaluators are arm's-length enough? They are part of a department, part of a ministry. Are they able to take a situation or problem and effectively relay it up the chain of command? Would it be easier if an external actor came in and looked at it and provided input?

[Sheila Fraser]

I really believe that the evaluators should be part of the department. This is a management tool. It is much like internal audit.

I know we've had discussions with the committee about where internal audit should report. We've always taken the position that internal audit evaluators should be reporting to the deputy head rather than to some external organization—say, the Treasury Board Secretariat.

This is really to be used by management to improve programs. It is not an external audit function or an external evaluation function. The responsibility really should come back to the deputy head to ensure that someone performing the function is well qualified and doing appropriate work and that the results are acted upon.

[Randy Hoback, Conservative, Prince Albert, SK]

You view this as more of an internal function. You don't think there would be any benefit in looking at it as more of an external function in the delivery of programs?

[Sheila Fraser]

I don't see a benefit to doing that, no.

[Randy Hoback, Conservative, Prince Albert, SK]

Are you satisfied with what you've seen, with the examples that you're given, that the deputy ministers are actually taking this function seriously enough and proceeding with the recommendations?

[Sheila Fraser]

I'm certainly pleased to hear of the progress today. As we always say, though, we are cautiously optimistic until we actually audit to see that things have improved. I would expect that at some point in the future the office will return to see how the policy is actually being implemented and whether some of these issues have been dealt with.

[The Chair, Liberal, Shawn Murphy]

I'm sorry to interrupt, but your time is up. Thank you very much.

Now we're going to go back to Mr. Christopherson for five minutes.

[David Christopherson, NDP, Hamilton Centre, ON]

Thank you very much, Mr. Chair.

There's one last area I wanted to follow up on. The auditor's report talks on page 22, starting at paragraph 1.52, about the shortage of experienced program evaluators in the federal government:

The shortage of experienced program evaluators is a long-standing concern. It has been noted in past Office of the Auditor General audits and in diagnostic studies by the Treasury Board of Canada Secretariat, and it was the subject of recent discussion within the federal evaluation community. A 2005 report by the Secretariat Centre of Excellence for Evaluation stated that “[t]he scarcity of evaluation personnel is probably the number one issue facing Heads of Evaluation.”

Two paragraphs later, it reads: “According to officials in the six departments, despite these”--Mr. Dion raised these earlier--“increases in both funding and staff, it remains a challenge to find experienced evaluators, particularly at the senior levels. In their view, the shortage of experienced evaluators has affected their ability to hire the people they need.”

This does get interesting: “For example, in one collective staffing process, the pool of experienced evaluators was depleted before the demand was met. They also indicated that the shortage of experienced evaluators has led to evaluators being hired away by other federal evaluation units.”

That may work well from a micro point of view, but from a macro perspective, it solves nothing. It's a legitimate concern, and we face it. I was thinking, Madame Fraser, that we went through something similar with the Canada Revenue Agency, and there it was analysts who had expertise in international investment income. As a result of not having the experts, there were likely untold amounts of money not coming in, for the simple reason that—no fault of anyone—there just weren't the experts. I'm seeing this as the same thing.

Can you collectively give me a sense of how we're tackling this? Are we speaking with educational institutions and provinces about trying to make sure we're developing them?

There's my very long-winded question.

[Alister Smith]

Thank you for those questions.

It is a challenge to find experienced evaluators. Clearly we're building up this community rapidly. We're building it up largely through intake from universities.

We have a consortium of universities working with us to offer evaluation courses and help with certification. They are from right across the country, from Carleton and Ottawa universities here through Quebec's Université Laval and l'Université de Montréal to the University of Saskatchewan, the University of Victoria, the University of Waterloo. It's right across the country. We're also working with the Canadian Evaluation Society in developing our standards.

We have a competency standard in circulation now that I think is quite good. We have a lot of courses being offered. We have a lot of support from the Canada School of Public Service. We're doing everything we can to build up this community.

It's true that when you start to get evaluators in one department, they could be poached by another department. We will keep trying to build up the community. In the end, as long as those evaluators remain within government, we're better off than we were without having them in government.

[David Christopherson, NDP, Hamilton Centre, ON]

Sure, but you're just squeezing the juice in the orange around in terms of the problem, aren't you? You solve yours, and your colleague now has a problem.

[Alister Smith]

We've ramped up and have probably doubled the number of people who are evaluating over this last decade. We're making headway. It's true that they're in real demand. I think this will also encourage universities to offer more courses to encourage more students to come to this line of work.

[Ian Shugart]

Mr. Chairman, if I may, I would add that one of the responsibilities I have as a deputy minister is to give full support to the evaluation group. Nothing motivates a team of public servants more than knowing their work matters and they have a platform for making a contribution.

I suspect that every deputy minister could say what I'd say about my own team. They are a very impressive group of young people and more mature people who enjoy their work.

As a manager, you have to get used to losing good people. I don't see my responsibility as being restricted to only Environment Canada, because it is a collective pool, but I certainly see my responsibility as meeting the obligations of the policy to hang on to and recruit as good a team as possible.

The data shows the numbers have increased. We use our people well, in conjunction with contractors. We've been known to recruit contractors when they have done a good job.

I would say that it is a terrific career. In an evaluation unit, there is probably no better way to learn what's going on in a department than to get right in there, from the vantage point of evaluation, work with the managers, and find things out.

These people become very good policy and program people and their careers often take a turn from evaluation into other areas. Frankly, we lose some people from evaluation because of that, but we recruit from other areas of the public service as a result. It's a very complicated situation.

[David Christopherson, NDP, Hamilton Centre, ON]

Thank you.

Thank you, Chair.

[The Chair, Liberal, Shawn Murphy]

Thank you, Mr. Christopherson.

Mr. Dreeshen, for five minutes.

[Earl Dreeshen, Conservative, Red Deer, AB]

Thank you very much, Mr. Chair.

Thanks to the witnesses.

I'd like to talk about what Environment Canada is doing.

Mr. Shugart, you pretty well went through the processes that were involved in your evaluation. You also spoke earlier about provincial and federal projects and finding out how those things work together. Could you explain some of the issues associated with that?

[Ian Shugart]

Mr. Chairman, I offered it as an example of the type of question that might come up in an evaluation that could be relevant to performance.

It should be the case that when a program is developed and adopted, at the outset, the appropriate federal and provincial analysis is done. In principle, the Government of Canada shouldn't engage in programs for which it does not have a jurisdictional responsibility, but there are of course many shared jurisdictions in Canada.

Certainly in my field, our work with the provinces is absolutely critical. An evaluation can expose factors of the relationship that can be as simple as how well things are working. A smaller province may not have the capacity that a larger province has and it may rely on us to carry a bigger load. It may not be the case for all programs, but it could be the case for some programs. That type of thing could come to the surface in an evaluation.

On the other hand, there may be deeper questions. Over time provinces may not have had the capacity or the interest or it could be that the federal government has not had the capacity in a particular area. Something that was true 20 years ago for an older program may not be quite the case today.

Those are the types of adjustments to programs that sometimes involve the federal-provincial relationship. An evaluation can be very useful in analyzing and bringing that to the attention of senior managers.

[Earl Dreeshen, Conservative, Red Deer, AB]

Your department is therefore able to give that type of advice to provinces as well.

[Ian Shugart]

In some cases, where we provide a service that provinces need, provinces might be consulted in an evaluation. They may tell us that they don't find this to be helpful. They may say this is absolutely essential to their responsibility and to please keep it up.

When we confront those types of results, it's then our responsibility as senior managers to make sure our minister is informed. When we give policy advice to the minister, that type of thing will be taken into account.

[Earl Dreeshen, Conservative, Red Deer, AB]

Thank you very much.

My next question is for Treasury Board. In the report, on page 45, recommendation 1.82 spoke about regularly identifying gaps that need to be acted upon. I was just wondering if perhaps you could speak about what some of those gaps might be and what actions have been taken or what actions are planned to be taken in order to address this.

[Alister Smith]

Some of the gaps that we ourselves have found over the years are pretty similar to what the Auditor General has mentioned--a lack of performance information, in particular. One of the focal points of the new policy is to improve performance measures and performance information, which will go a long way to help in assessing the effectiveness of programs. That has certainly been a gap we've looked at.

We've looked at, through our management accountability framework assessments, the quality of evaluations, the governance of the function within departments, because we do want to ensure that evaluators have access to deputy ministers and can tell them the news about a particular program, even when the program managers may not be all that keen to have that news going to the deputy. So we've looked at areas like that within our MAF assessments, and we've been fairly candid with departments where we have seen gaps. So we have a mechanism that works very well in that way. We also have a community of practice that has an evaluation, with others, with pretty much ongoing dialogue on all nature of problems, but also best practices that departments can borrow from each other, including the ones Mr. Shugart mentioned on surveying clients after a program has been delivered.

[Earl Dreeshen, Conservative, Red Deer, AB]

Thank you.

[The Chair, Liberal, Shawn Murphy]

Monsieur Dion, pour cinq minutes.

[Stéphane Dion, Liberal, Saint-Laurent—Cartierville, QC]

Thank you Mr. Chairman.

Because we're coming to the end of our discussion, perhaps I can sum up. All your departments, including Treasury Board, will have to cut their expenditure by 5%. Is that correct? Is Treasury Board not subject to that 5% cut?

[Michelle d'Auray]

Yes, Mr. Chairman. The strategic review for the Treasury Board Secretariat is taking place this year.

[Stéphane Dion, Liberal, Saint-Laurent—Cartierville, QC]

That is also true for Environment Canada and Citizenship and Immigration Canada. One would hope that during such a difficult and necessary exercise one could depend on quality evaluation functions. Yet the report tells us the opposite.

I'll sum up the main conclusions. Only 5% to 13% of programs have been evaluated, and more than two-thirds of those evaluations lack adequate data in order to be effective. Yet this morning we are being told that since the Auditor General's report came out one year ago significant progress has been made. The report states that Treasury Board has made little progress in "developing tools to help departments address the longstanding problem of insufficient data", and, later on in the report, that it did not issue "adequate guidance or tools".

Do you still say this is not true?