Public Accounts Committee on May 4th, 2010

[Michelle d'Auray]

We acted and we took note of the report's conclusions, but we also took note of the factors we identified. In November 2009, we provided departments with guides on performance measures. We will be finalizing that this year.

Therefore, yes, we took note of the recommendations and we took measures in order to ensure that data collection complied with the standards and met the requirements.

As the chairman requested, copies of that material will be provided to the committee.

[Stéphane Dion, Liberal, Saint-Laurent—Cartierville, QC]

Ms. d'Auray, you said that over the past year 15% of programs were evaluated. Therefore we're getting close to 20% of evaluations per year.

However, my concern is that that quantitative goal may be met at a cost in terms of quality, which is lacking according to the Auditor General's report. I would like to ask Mr. Shugart to react to paragraph 1.91, on page 32

page 32 in English, paragraph 1.91.

It states:

[...] Environment Canada estimated that it would have to double the complement of its evaluation unit over the next four years, or sacrifice evaluation depth in order to achieve full coverage.

Do you still agree with this paragraph?

[Ian Shugart]

That was an estimate. In the department there was a 100% increase in our capacity.

We did increase from about four evaluators in 2004-05 to where we have 12 or 12.5 FTEs in evaluations today. So in that period of time we more than doubled our evaluation capacity.

Going forward, the same level, generally speaking, to achieve 100% coverage over time probably is fair. With the increased supply of evaluators that we hope will be available we think that is achievable. But it is absolutely true that we will have to manage this carefully and we will have to focus our evaluation resources in those areas where the greatest value from evaluation can be achieved.

As I mentioned, we may be flexible in how we implement this policy. We may cover a broad area of programs in one evaluation rather than each individual program, which would be an efficient way of doing it. And if we make wise selections and have good evaluation plans, then we will be able to achieve 100% coverage in a very efficient way. I would look to the professionals in the evaluation branch to help me with evaluation plans that are efficient, as well as meeting the goal.

So it will be a challenge, but it is a necessity, and we will proceed as best we can.

[Stéphane Dion, Liberal, Saint-Laurent—Cartierville, QC]

May I ask Mr. Yeates to comment on the same topic?

[Neil Yeates]

Yes, certainly, Chair.

We are working to increase our FTEs, similar to Environment Canada. We had three back in 2004-05 and we're ramping up to 20. We think that is sufficient capacity for us.

Our current five-year evaluation plan provides 100% coverage for all our grants and contribution programs. So we have a little bit more to do beyond that, but we feel confident we will have the capacity to do the necessary work.

[The Chair, Liberal, Shawn Murphy]

Merci, Monsieur Dion.

Mr. Saxton, for five minutes.

[Andrew Saxton, Conservative, North Vancouver, BC]

Thank you, Mr. Chair.

My question is also for Mr. Yeates. In your opening remarks you stated:

With more and better studies, the results and conclusions of CIC evaluations are increasingly used by senior managers to inform program and policy decisions.

Can you elaborate on that, please?

[Neil Yeates]

Yes, I'd be pleased to, Chair.

The evaluation committee we have, which I chair and Elizabeth Ruddick supports, meets about monthly now. We review all the terms of reference for our forthcoming studies, and this is very important for us, because it makes sure that the right questions are being asked within an evaluation. Then we review the studies themselves and the management response we have from the particular program area. And we've had some very good discussions about what that means for us.

I can give you an example of that. We've just recently completed an evaluation of our language programming, which is provided throughout Canada. That's led us to rethink how we do some of our language assessment, both overseas for new immigrants but also the programming we have here in Canada. So it's an example where a specific program evaluation ended up having broader impacts in terms of our service delivery system.

[Andrew Saxton, Conservative, North Vancouver, BC]

Thank you.

My next question is for the Treasury Board secretary. Can you elaborate and expand on the importance of the management accountability framework?

[Michelle d'Auray]

The management accountability framework was established in 2003 in order to assess the department's management results overall, over a broad range of areas of management. Evaluation is one of currently 19 areas of management that we assess.

For example, we encourage departments to assess their capacity in terms of audit, in terms of financial management, and in terms of people management. We also look at the areas around performance management and we have an assessment also of the nature of the Treasury Board submissions that they send to us.

A lot of these areas, including evaluation, are what we consider core to the management functions and good performance in the area of management of departments and organizations. So we assess, we have performance indicators. Some of the assessment measures are what we would consider to be objective and some of the assessment measures are self-assessment by departments and organizations that they provide to us.

We report back to departments. In fact we have published on our website the results of the management assessments. We work with departments on establishing what would be the management priorities for the coming year.

That's where--in some instances, for example--the performance around evaluation has not necessarily been at the highest standard. We have occasionally identified where that is the case in a department where that is a priority for improvement for that department. That then becomes a management priority for that organization for a given year in order to focus their attention on improving the management capacity of a particular area, including, for example, evaluation.

[Andrew Saxton, Conservative, North Vancouver, BC]

Thank you very much.

Mr. Chair, I have no further questions.

[The Chair, Liberal, Shawn Murphy]

Thank you, Mr. Saxton.

I have one issue I want to clarify to you, Madame d'Auray, and that is the whole issue of departmental performance reports. It's long been a concern of this committee, and it's been a concern of the Auditor General, that the content and the way these reports are prepared and tabled in Parliament is of limited use and limited assistance to parliamentarians and others.

I guess when I read this report.... You look at performance and the first thing you would look at to determine performance is you would look at evaluations. You would think that the evaluations that have been done and the evaluations that will continue to be done would form a very large component of the departmental performance reports. They would either be produced in detail or they would certainly be summarized. I'm not sure if that's the case; I don't think it is. But can you tell the committee to what extent these evaluations that are being done now are used in the departmental performance reports that are tabled in Parliament now?

[Michelle d'Auray]

If I may, Mr. Chair, I would ask Mr. Smith to respond to that.

[Alister Smith]

Yes, Mr. Murphy.

The departmental performance reports do include performance measures, and they continue to improve. These performance measures or performance indicators are consistent with the types of measures used in evaluation. But the structure of these reports follows auditing standards. We're auditing performance against strategic outcomes for the programs at a fairly high level and at the levels of program activity.

In some respect here evaluation is dealing with programs at a lower level of granularity. So they're not inconsistent, it's just there's a continuum of information on performance from the program level up to the strategic outcomes. I understand you have a hearing on this.

[The Chair, Liberal, Shawn Murphy]

Yes, we're having a hearing.

I don't entirely agree with you.

Do you have any comment, Madam Auditor, on that point?

[Sheila Fraser]

Mr. Chair, that's not something we have looked at recently. We didn't include it in this audit.

[The Chair, Liberal, Shawn Murphy]

It is a timely issue.

As Mr. Smith pointed out, we are having a hearing. I believe it's a week from this Thursday. It will be an interesting hearing. We will be looking at the departmental performance reports of two departments, transport and industry, I believe.

You have one minute, Mr. Lee, sixty seconds.

[Derek Lee, Liberal, Scarborough—Rouge River, ON]

I have a very short question.

Has Treasury Board looked at the issue of transparency in this evolving field of evaluation--transparency in the sense of intra-departmental transparency of evaluation data as it's being collected, and when it's finally collected, the preparation of a report before it's made public? Because you've got the potential for sharing inter-departmentally as well as intra-departmentally with Parliament and under access to information.

I'm just wondering, as we develop this science of evaluation, if any thought has been given to the potential issues of transparency and access to various parties.

[The Chair, Liberal, Shawn Murphy]

Mr. Smith or Madame d'Auray.

[Michelle d'Auray]

I'll begin, and I'll turn to Mr. Smith for more information.

In terms of the transparency, departments are in fact encouraged to post the evaluation reports on their websites, and they do so fairly frequently. On the sharing of data, that normally happens in terms of horizontal evaluations, where that evaluation has to cover more than one program or a program covers more than one organization.

[The Chair, Liberal, Shawn Murphy]

It is getting close to eleven o'clock. On behalf of all committee members, I want to thank you very much for your appearance today. It has been a very valuable meeting.

Before we adjourn, I want to ask all the witnesses if they have any concluding remarks.

I'll start with you, Ms. Fraser.

[Sheila Fraser]

Thank you, Mr. Chair.

I would like to thank the committee members for their interest in this audit. As I mentioned, we believe, as do other witnesses today, that the evaluation function is a very important one, especially in a time of fiscal restraint.

I was pleased to hear about the actions that government has taken since the completion of our audit. I would certainly encourage the committee to perhaps follow up at some appropriate time in the future on progress that's being made.

[The Chair, Liberal, Shawn Murphy]

On that point, Madam Auditor, as you know, and for the purpose of everyone in the room, we do have action plans from everyone involved and we certainly will be following up very closely.

Madame d'Auray, any closing comments?

[Michelle d'Auray]

I would echo the Auditor General's comments about the importance of the function and note that we are on the right path to re-establishing and rebasing the function as a core element of the government's expenditure management system.

Thank you.

[The Chair, Liberal, Shawn Murphy]

Mr. Yeates, do you have any closing comments?