Public Accounts Committee on May 4th, 2010

[Daryl Kramp, Conservative, Prince Edward—Hastings, ON]

Yes.

[Sheila Fraser]

Chair, I think everyone has agreed that the evaluation of effectiveness is absolutely critical to making good decisions about program spending so that we can know whether programs are actually getting the results that were intended. I would say this is even more critical in the current economic times we are going through, because government does have to make difficult choices, and it should be making those decisions based on good information. So when we look at the various areas we think are important in the management of government, we think that the evaluation function is a really important one. That is why we did this audit.

If I could just add, I think we have seen in the past when economic times have been difficult that it has been very easy to cut functions like evaluations and internal audits and these sorts of back-office functions. I think it is important, with the new policy that has come out, that government is ensuring that this function continues; otherwise decisions are going to be made without the kind of information managers really need.

So that's why we thought this audit was important, and I would hope the committee thinks it is important.

[Daryl Kramp, Conservative, Prince Edward—Hastings, ON]

Fine. Thank you very kindly.

A question now to Treasury Board. In your action plan you mentioned you established an interdepartmental advisory committee of deputy ministers. You go on to say you'll be working to develop a best practices handbook--in other words, to share the cost-benefit and to share your experiences.

Can you tell us the present status of that process?

[Alister Smith]

We've established a committee at the assistant deputy minister level. We also have working groups looking at specific issues across government in harnessing the expertise of our senior evaluators across government and heads of evaluation. We are pulling a lot of the information together, especially from the working groups, into a guide for evaluators. We have a best practice guide on performance reporting, but this would be different. It would be a guidebook for evaluators. That's the objective we're working toward.

[Daryl Kramp, Conservative, Prince Edward—Hastings, ON]

Thank you.

A question now to CIC. In your departmental response, you agreed you would have an external evaluation expert at the departmental evaluation committee and this item would be brought to the evaluation committee for a decision by April 2010. It's now May. What's the status?

[Neil Yeates]

The evaluation committee has agreed that we are going to proceed. We've actually been in touch with someone to invite them to join our committee, so we're pleased with that.

[Daryl Kramp, Conservative, Prince Edward—Hastings, ON]

Thank you.

A question now to the Auditor General. You've stated in your opening statement that the lessons learned from the government's recent strengthening of internal audit could be applied usefully to the program evaluation, and that is basically what you stated in your response to me earlier. But what, in particular, did you have in mind when you were offering that assessment, what lessons particularly?

[Sheila Fraser]

Chair, I think we've noticed a lot of improvement in the internal audit function. The secretary referred to many of the actions that were taken, evaluating what are the competencies that should be there, focusing on the human resource capacity, standards and practice in the community, and more recognition of the importance of the function. So I think there's much more attention to that function, the importance of the function, and the governance as well of the function. We saw that those issues were addressed in the internal audit function. I shouldn't pre-judge--we're doing a follow-up audit--but I think we can see that a lot of effort has been made there. And I think we see signs of the same processes being applied to the evaluation function.

[Daryl Kramp, Conservative, Prince Edward—Hastings, ON]

Great. Thank you.

[The Chair, Liberal, Shawn Murphy]

Madame Beaudin, five minutes.

[Josée Beaudin, Bloc, Saint-Lambert, QC]

Thank you Mr. Chairman.

Thank you and good morning to all of you.

I have one or two main questions. Environment Canada seems to be the only department that has an internal process for systematically identifying areas where improvement in evaluation methods is required.

Should we take from that that you do not have any problems with resources? You have enough evaluators to undertake these evaluations?

[Ian Shugart]

That may be somewhat too positive a conclusion. Each department makes its decisions differently. We acknowledge rather early in the process that quality improvement would be very useful. In order to guarantee manager participation in the programs, we felt it would be useful to have a commitment to

surveys that resulted from that, where after evaluations we would follow up with the program managers.

As well, we concluded that if we learned valuable lessons we should write them down and create a best-practices registry and discuss these within our community. Some rather simple improvements led to better practices. However, I'm certain that throughout government my colleagues have improvements that would be very useful. Therefore in the future I will also try to draw on their experience.

[Josée Beaudin, Bloc, Saint-Lambert, QC]

It occurs to me that your expertise could be useful to your colleagues. Do you work together? Do departments, for example, work with you, Ms. d'Auray, in order to share best practices?

Furthermore, if you do experience problems with respect to resources for necessary evaluations, how will Environment Canada proceed?

My question is for Mr. Shugart. How do you evaluate that lack of resources? How will you proceed within your department?

I therefore have two questions. Do you work together? How will you approach the situation?

Perhaps I have one last question. Ms. d'Auray, what is your contingency plan if departments end up experiencing problems in meeting the necessary requirements? What is the contingency plan? What measures will be used?

[Michelle d'Auray]

Thank you Mr. Chairman.

With respect to your first question on whether or not departments work together, the answer is yes. In June 2009, we established a practices community. We have regular meetings where we share information on best practices. We have also had several discussions on key skills. There are also four working groups that focus on the very specific questions that evaluators ask, as Alister Smith mentioned. The secretariat doesn't know everything but we do encourage sharing information and skills.

You asked a question about our contingency plan. We are quite encouraged by the progress and percentage of coverage achieved to date. If necessary, we will assess any measures that need to be taken. For now, given the amount of flexibility under the evaluation policy that we suggested, we are quite confident that departments will meet the requirements.

[Josée Beaudin, Bloc, Saint-Lambert, QC]

You didn't mention a contingency plan. If the departments experience short-term difficulties, how will you react in order to stay the course?

[Michelle d'Auray]

We have given them four years for this transition. Give us at least enough time to get halfway through that transition period. At that time we'll be able to assess whether or not adjustments need to be made. For now let's give them a chance to make some progress.

Perhaps Mr. Shugart wants to respond to your second question.

[Ian Shugart]

I will be brief Mr. Chairman. Perhaps it would be useful to describe the internal process of a department such as Environment Canada. There is an evaluation committee that I chair. Each year, within our planning process, an evaluation group drafts the plan for the coming year. This plan identifies the programs that will be evaluated. The budget is part of that action plan. The committee must decide whether to accept or reject the plan proposed by the group and make any changes. It must also decide whether the budget will be approved or not. Within a broader context, that budget for evaluation functions is one of several requests, several financial pressures. We have to make choices. We can modify the plan in order to adapt to the financial circumstances, but we do have that goal, that requirement, obligation, to cover 100% of our programs by the end of the period.

As Mr. Christopherson pointed out, it's a question of balance. One must make choices and changes for each evaluation.

[Josée Beaudin, Bloc, Saint-Lambert, QC]

Thank you.

[The Chair, Liberal, Shawn Murphy]

Thank you, Madame Beaudin.

Mr. Hoback, for five minutes.

[Randy Hoback, Conservative, Prince Albert, SK]

Thank you, Chair.

I'm curious about how the internal audit process works. Ms. Ruddick, perhaps you could explain it to me. When you're doing evaluations internally, how do you go about passing the information up through the department? If you come across a program that you feel doesn't meet the needs or the intention it was originally created for, how is that relayed through your department?

[Elizabeth Ruddick, Director General, Research and Evaluation, Department of Citizenship and Immigration]

The process in CIC is similar to that in other departments. We have an evaluation committee, chaired by my deputy minister, which meets almost monthly, in fact, because of the workload we have. We come each year with a plan and decide what evaluations will take place. We have various stages of providing information within the department before we finalize the report. For example, we will brief the program managers internally, we will check with them that there are no errors of fact in what we found, then we will finalize the report, and it gets tabled at the evaluation committee with the deputy minister for approval.

Along with the report, there is a list of findings. The responsible program managers—and often this may cut across a number of areas in the department, because it's not just the people designing the policy but also those who are implementing it, and it may involve our regional offices as well—provide a response, and at the same time there is an action plan in terms of how they are going to address the findings.

We also have in place a process to track progress against that action plan on a periodic basis to make sure that the changes proposed and agreed on at the evaluation committee are in fact being implemented.

[Randy Hoback, Conservative, Prince Albert, SK]

Would you look at the actual intention of the program and at whether it is relevant to today's needs? If it's a program that's been going on for four or five years, would you ask whether this is a program that's actually needed or not and report that also?

[Elizabeth Ruddick]

Yes, absolutely. There are a number of questions that we are mandated by Treasury Board to respond to. One of them is program relevance. We look at the objectives and the delivery. We may find, for example, that there are some problems in the delivery or, as was mentioned by Mr. Shugart, that because of some of the relationships across the various groups delivering the program there may be some misunderstandings. We can address that. We're looking at how effectively the program is being delivered, how the money is being spent, and we'll often look at other options for program delivery, if that's possible.

[Randy Hoback, Conservative, Prince Albert, SK]

Ms. Fraser, when you were going through your process of looking at the systems, did you review what external education these people were getting throughout their careers? For example, would Ms. Ruddick have the ability to look at other companies or businesses and say, this is how they go about doing their internal evaluations? Did you look at anything like that?

[Sheila Fraser]

We didn't look at that specifically. We did look, though, at whether there was a competency profile and whether there was an agreement on what evaluators should have as competency. That did not exist at the time we completed the audit. We've heard today that it has been completed since. We would look to whether a standard has been set and what kinds of minimum qualifications you would expect at different levels. That is critical to ensuring that the evaluators are well qualified and able to do their job.