Public Accounts Committee on May 4th, 2010

[The Chair, Liberal, Shawn Murphy]

Thank you very much, Mr. Christopherson.

Thank you, Mr. Shugart.

Mr. Saxton, seven minutes.

[Andrew Saxton, Conservative, North Vancouver, BC]

Thank you, Mr. Chair.

Thank you to the witnesses for being here today.

My first question is for the Auditor General. Can you please elaborate on the purpose of the audit?

[Sheila Fraser]

The purpose was really to see the capacity of the evaluation function across government in terms of, obviously, human resource capacity, but also were there competency profiles, were there standards that were being set, and were departments in a position to be able to meet the requirements of the new policy, or was it expected the departments would be in a position to meet the requirements of the new policy that would be coming into effect in 2013?

[Andrew Saxton, Conservative, North Vancouver, BC]

Have you done an audit like this previously? How long has it been since the last audit was done?

[Sheila Fraser]

The last audit on the evaluation function was in 2000. We did do a similar audit on the internal audit function in 2004.

[Andrew Saxton, Conservative, North Vancouver, BC]

Thank you.

In your report you note that Environment Canada already had good practices in place. Can you expand on some of these good practices?

[Sheila Fraser]

We note in the report that they had good practices regarding continuous improvement, and I believe one of the examples we had was surveys they use to collect information on how the function should be improved. That was something we did not see in the other departments.

[Andrew Saxton, Conservative, North Vancouver, BC]

Thank you.

My next question is for the Secretary of the Treasury Board. With the new policy on evaluation, what gaps are you hoping to address that the previous policy did not?

[Michelle d'Auray]

One of the issues was coverage. Another was the nature. Another was the ability to tailor the type of evaluation to the type of program. Another was clarity around the collection of data, or performance measurement. We also addressed the issue of governance, the independence, if I could put it that way, of the evaluation function as well as the responsibilities of the deputy head in relation to the evaluation function.

We covered a range of issues. A lot of those are very similar to the issues raised by the audit. At the same time, they also address a number of issues we have found as part of the capacity to assess programs in light of expenditures.

The second component, the really important one, was the series of interviews that were held with deputy heads around the usefulness and importance—and this came directly from colleagues—of the use of evaluation as a management tool for the effectiveness of programs and the results. All those elements are now in the policy, the directive, and the standard.

[Andrew Saxton, Conservative, North Vancouver, BC]

Thank you.

Can you explain how evaluations that address program value for money will better serve government decision-makers?

[Michelle d'Auray]

Essentially, an effectiveness evaluation looks at the cost, the effectiveness, the results, and can occasionally look at the process around the delivery. It looks at the effectiveness and efficiencies around the program and the initiative. If you're setting up an initiative or program to achieve a certain set of results, it really is critical to set out what your performance measures will be—which is now clearly the responsibility of the program managers—for us to be able to assess whether the program and the initiative are achieving the intended results. Sometimes it can do so over a very long period, so we measure periodically. That's why we need to cover the entire program suite within a five-year period. On average, 20% of the program should be reviewed.

It also should give us some baseline information and means of addressing whether the programs are meeting the objectives and how they can be adjusted to do so. We found that particularly key in supporting the strategic reviews that departments are currently undertaking to identify the least-performing programs or initiatives, in order to reallocate expenditures to higher-performing programs. It's very difficult to do so if you don't have very good effectiveness evaluations.

Perhaps my colleagues would like to add something to that.

[Ian Shugart]

I could, Mr. Chairman.

I would say that there are benefits to policy-making if an evaluation shows that a program is not performing well in some of its objectives. That is relevant for ourselves and for central agencies in the process of determining whether a program is renewed. It may also tell us that part of the program is working very well but another part isn't.

Sometimes this may be due to capacity at the federal level vis-à-vis the provincial level. For example, a province may be able to do better than the federal government with respect to one aspect of a program.

Another area of benefit is in the management of programs. We posted an audit on environmental emergencies, where one of the details that came out in the audit had to do with the roles of our environmental emergency officers. On the ground, in the case of an emergency where we're providing support, it's very important to have a very clear delineation between the responsibility of environmental emergency officers and enforcement officers. This was an area that the evaluation showed was not entirely clear. That allowed us to issue policy guidance and an actual statement of responsibilities for our environmental emergency officers, as opposed to our enforcement personnel, so that on the ground there's no confusion and there are no legal problems about who is authorized to do what. It's a small point, but it allowed us to improve the delivery of the program on the ground.

These are a couple of examples where evaluations actually do make a difference, sometimes in the continuation of the whole program, at other times in the improvement of its delivery.

[The Chair, Liberal, Shawn Murphy]

Thank you, Mr. Shugart.

Just one question before we go to the second round.

Following up on your comment, Ms. Fraser, about the HR capacity, we're into a situation where there's going to be more work. If you look at exhibit 1.8 on page 25 of the English copy of the report, it seems that close to 90% of the person-years involved in these evaluations are done by outside contractors. Perhaps I'm wrong, but that to me would be some suggestion that if they're using outside contractors and consultants, perhaps there's not enough HR planning going into this whole process. We all know it's much more expensive and it does not create any capacity within government to do this type of work.

Do you have any comment as to that trend? It really hasn't improved at all in the five years of the audit between 2004-05 and 2008-09.

[Sheila Fraser]

Thank you, Chair.

That was one of the major issues raised by the various departmental functions. We note in the report that for the period we looked at, about 90% of the evaluations were wholly or partially conducted by contractors.

There will always be a need for contractors. I think we have to recognize that. But I was pleased to hear the secretary mention the work that has been done on recruiting and qualifying people, so that the capacity will remain within government and it won't be contractors coming in and then leaving with the information. It's important that there be sufficient capacity within departmental staff. I don't know the progress that has been made, but there certainly would appear to have been planning done, and efforts done, within the last year.

[The Chair, Liberal, Shawn Murphy]

To you, Madame d'Auray, I appreciate what the auditor has just said, that there will always be some need to hire outside expertise, but do you agree with me that when 90% of your work is done by outside contractors, you're really developing very little, if any, in-house expertise? The capacity will never be developed, especially in an environment where you're trying to increase the number of evaluations that you're doing, 100% coverage over a five-year cycle. Do you see real problems there? Do you see, in the past anyway, certainly a lack of HR planning?

[Michelle d'Auray]

Thank you, Mr. Chair.

We did recognize that gap, and we are working on many fronts to fill it. One is the work we are doing to recruit. We have also set guidelines on what the core competencies are. There has been some increased funding in departments. We've seen a growth in the number of evaluators available in the government. We're at just over 500 right now.

The Auditor General made an interesting comparison to the work that started with the audit community. We are following, I would say, a similar pattern in terms of identifying what the competencies are, identifying what the governance structure should be, and identifying the nature of the work of the evaluators and the evaluation function.

I think that in departments there is a growing recognition of the need to manage and oversee the evaluations themselves and to sometimes, as necessary, bring in contracting expertise, because not all the evaluators will necessarily have all the expertise necessary to cover the work.

I know that my colleague could speak from departmental experience about what that means, if that's possible.

[The Chair, Liberal, Shawn Murphy]

Perhaps we can catch that later on.

I'd like now to move on to the second round.

Mr. Lee, you have five minutes.

[Derek Lee, Liberal, Scarborough—Rouge River, ON]

Thank you.

I'd like to go to Treasury Board.

There's a reference to a five-year cycle, or a five-year program. When did the five years begin?

[Michelle d'Auray]

The new policy was put in place in April 2009. The five years will end in 2013-14, for the transition period. The formal period, when there will be the requirement to cover all of the evaluation in the five years, is from 2013-14 to 2017-18. That is when the obligation to cover 100% of direct program expenditures kicks in.

[Derek Lee, Liberal, Scarborough—Rouge River, ON]

It is 2017.

[Michelle d'Auray]

It is by the end of that period.

[Derek Lee, Liberal, Scarborough—Rouge River, ON]

We should come back then.

Who is responsible for the implementation of this five-year cycle? Is it Mr. Smith?

Hello, Mr. Smith. Mr. Smith was a wonderful witness last year at another committee. He was very helpful, very professional.

Over to you, Mr. Smith.

[Alister Smith, Assistant Secretary, Expenditure Management Sector, Treasury Board Secretariat]

Thank you, Mr. Lee.

Yes, we are responsible. Treasury Board Secretariat is responsible for providing oversight of the function.

[Derek Lee, Liberal, Scarborough—Rouge River, ON]

You're wonderful at that. Is it you--not we—is it you?

[Alister Smith]

Yes, it is.