Public Accounts Committee on May 4th, 2010

[The Chair, Liberal, Shawn Murphy]

I would like to call the meeting to order at this time.

I want to welcome everyone here. Bienvenue à tous.

Colleagues, this meeting is called pursuant to the Standing Orders. We're dealing today with chapter 1, “Evaluating the Effectiveness of Programs”, from the fall 2009 report of the Auditor General of Canada.

We have a number of witnesses before the committee this morning.

From the Office of the Auditor General of Canada, of course, we have the Auditor General herself, Sheila Fraser. She is accompanied this morning by Assistant Auditor General Neil Maxwell and Principal Tom Wileman.

From the Treasury Board Secretariat, we have the secretary, Michelle d'Auray. She is accompanied by the assistant secretary, Mr. Alister Smith.

From the Department of Citizenship and Immigration, we have the deputy minister and accounting officer, Neil Yeates. He is accompanied by Elizabeth Ruddick, director general of research and evaluation.

Finally, from the Department of the Environment, we have the deputy minister and accounting officer, Ian Shugart. He is accompanied by William Blois, the associate director of the audit and evaluation branch.

On behalf of all members of the committee, I want to extend a very warm welcome to everyone.

We will now have opening statements

Ms. Fraser, you're first. You have up to five minutes.

[Sheila Fraser, Auditor General of Canada, Office of the Auditor General of Canada]

Thank you, Mr. Chair.

We thank you for this opportunity to present the results of an audit included in our November 2009 report on evaluating the effectiveness of programs in the federal government.

As you mentioned, I'm accompanied today by Neil Maxwell, Assistant Auditor General, and Tom Wileman, principal, who were responsible for this audit.

I would like to point out to the committee that the work for this audit was substantially completed on May 31 of 2009.

Effectiveness evaluation is an established tool that uses systematic research methods to assess the extent to which a program is achieving its objectives. Over the past 40 years, the federal government has made repeated efforts to establish the evaluation of effectiveness as an essential part of program management.

One of the most important benefits of effectiveness evaluation is to help departments and agencies improve their programs. Departments also need to be able to demonstrate to Parliament and taxpayers that they are delivering results for Canadians with the money entrusted to them. Sound information on program effectiveness is particularly important in light of the recent budgetary measures to contain administrative costs and review government operations.

In this audit, we examined how examination units in six departments identified and responded to increasing needs for effectiveness evaluation. We also looked at whether they had built the required capacity to respond to those needs. In addition, we looked at the oversight and support role of the Treasury Board of Canada Secretariat in monitoring and improving the program evaluation function in the government, particularly with respect to the evaluation of program effectiveness. The period covered by our audit was 2004 to 2009.

Overall, we found that the six departments were not sufficiently meeting the needs for effectiveness evaluation. Annual coverage of departmental expenses by evaluation was low, ranging from 5% to 13% across the six departments.

In addition, the effective rate of coverage was even less because many of the evaluations we reviewed did not adequately evaluate effectiveness. Of the 23 evaluation reports we reviewed, 17 noted that the analysis was hampered by inadequate data, which limited the evaluation of program effectiveness. This lack of performance data is a longstanding problem as noted in my office's earlier audits of the evaluation function.

With respect to the six departments' capacity to meet the needs for effectiveness evaluation, department officials told us that they have not been able to hire enough experienced evaluation staff and have used contractors extensively to meet requirements.

Of the audited departments, Environment Canada had internal processes to systematically identify areas for improvement in effectiveness evaluation. For example, Environment Canada solicits client feedback through post-evaluation surveys. Such processes ensure that departments are following the management cycle for continuous improvement.

The situation with respect to program evaluation in the federal government is not unlike that of internal audit before the policy on internal audit was implemented. Lessons learned from the government's recent strengthening of internal audit could be applied usefully to program evaluation.

We believe that implementation of the new requirement to evaluate all direct programming spending faces serious challenges. Earlier requirements for full coverage were never met, and current legal requirements for effectiveness evaluation of all grant and contribution programs have been difficult to meet. Department officials told us that they have concerns about their capacity to implement the expanded coverage requirement in the new evaluation program.

In our view, it will be important for the secretariat and departments to carry out effectiveness evaluation of programs that are susceptible to significant change because of shifting priorities and circumstances. These are programs where evaluations of the relevance, impact, and achievement objectives can be put to best use. During the transition to full coverage, these programs may present the biggest opportunities for effectiveness evaluation. Taken together, the increasing needs for effectiveness evaluation and the department's limited capacity to meet those needs pose a serious challenge to the function. Concerted efforts by both the secretariat and departments will be needed to meet this challenge.

The secretariat and the audited departments have agreed with all of our recommendations. In several cases, they have made commitments for remedial action in their responses. In line with the committee's request for action plans and timelines, the committee may wish to explore the progress made to date in addressing the issues raised in the chapter.

Mr. Chair, this concludes my opening remarks and we would be pleased to answer the committee's questions.

[The Chair, Liberal, Shawn Murphy]

Thank you very much, Ms. Fraser.

We're now going to hear from the Treasury Board.

I'll turn it over to you, Madam d'Auray.

[Michelle d'Auray, Secretary of the Treasury Board of Canada, Treasury Board Secretariat]

Thank you, Mr. Chair.

Good morning.

Good morning. Thank you for this opportunity to speak about the evaluation function in the Government of Canada. As you mentioned, I'm here with my colleague Mr. Alister Smith, Assistant Secretary of the Expenditure Management Sector. Mr. Smith is responsible for my department's Centre of Excellence for Evaluation. This centre is responsible for evaluation policies and works very closely with the government evaluator community.

As Ms. Fraser mentioned, evaluation is a longstanding management tool that is vital to the sound management of public spending. It involves the systematic collection and analysis of evidence on the outcomes of programs. This invaluable information is used to make judgments about the relevance, performance and value for money of programs. It is also used to examine alternative ways to deliver programs or achieve the same results.

Finally, it supports policy and program improvement, expenditure management, cabinet decision-making, and public reporting to Parliament and Canadians.

Given the increasingly important role evaluation plays in support of program effectiveness and expenditure management, we are in full agreement with the recommendations contained in the Auditor General's report. They mirror in large part what the Treasury Board Secretariat has learned through extensive consultations and monitoring activities. And they are reflected in and addressed by the actions we have taken as part of the implementation of the new policy on evaluation that was issued in April 2009. Unfortunately, the scope and timing of the Auditor General's report did not allow for recognition of these improvements, as it focused on the period up to the introduction of the new policy.

Our action plan, in response to the Auditor General's report, which we provided to the committee, outlines what the secretariat has undertaken and delivered since the report's publication and what we will continue to do. Allow me to highlight some of our actions.

One of the Auditor General's concerns was that evaluations were not adequately assessing effectiveness, and that they lacked performance information. This concern has now been addressed under the new policy, which sets a clear standard for evaluation quality, as well as responsibilities for performance measurement. It also requires that all evaluations examine program effectiveness.

The new policy also requires each departmental head of evaluation to prepare an annual report to their deputy head on the state of performance measurement in their organization. This report will assist the deputy head in ensuring that the key data needs of program evaluations are met.

Finally, the policy has also expanded the evaluation coverage requirements to cover all direct program spending over a five-year cycle, after an initial transition period.

The Auditor General also recommended that the Treasury Board Secretariat should do more to monitor and support departments to help them identify priorities for improvement. This is addressed under the new policy that calls on the Treasury Board Secretariat to provide functional leadership for evaluation across the government. This includes regular monitoring and annual reporting to Treasury Board on the health of the evaluation function. Our first report will be issued before the end of 2010-11.

Much of our monitoring and support work is carried out through the annual management accountability framework assessment process, which assesses evaluation quality, neutrality, coverage, and use. It is also carried out through the advice and guidance we provide to departments on performance measurement frameworks, which are required under the management, resources, and results structure policy.

The secretariat has also allocated resources to our centre of excellence for evaluation to strengthen the evaluation expertise we provide to departments.

We appreciate that the new policy represents some important changes for departments—as the Auditor General noted by calling on the secretariat to help departments prepare to implement the new coverage requirements. This is why there will be a four-year phase-in period before departments are required to meet the comprehensive coverage requirement in their five-year evaluation plans, beginning with the 2013-2014 to 2017-2018 planning period.

I will turn now to the support we have provided to departments during the transition period, largely through the secretariat's centre of excellence for evaluation. For example, in November 2009 we issued a draft guide to developing departmental evaluation plans, which will be finalized and issued this summer. This provides guidance to departments with regard to evaluation timing, coverage, prioritization, and instruments.

We also issued, in November 2009, a draft guide to developing performance measurement strategies to support heads of evaluation in assessing the department's performance measures. This too will be finalized this fall, after integrating feedback and recommendations from departments.

We also set up, in June 2009, the evaluation community of practice with a website for exchanges of best practices. And we have held regular meetings to guide the capacity development of the evaluation community.

In addition, we provided preliminary guidance to departments on the possible merits of including external experts on departmental evaluation committees. The final document will be integrated this fall in a guide on the evaluation function, which will set out the expectations of the secretariat in relation to the evaluation policy and directive.

We recently led a post-secondary recruitment initiative for graduates with evaluation-related backgrounds. This led to the establishment of two pools of pre-qualified evaluators at the entry and intermediate levels. We also continue to work with universities and the Canada School of Public Service to promote and develop the types of evaluation skills and knowledge we need.

All these improvements have addressed the Auditor General's concerns over the quality, capacity and program coverage of the evaluation function in the government.

In sum, even though much remains to be done and even though we have attempted many times to improve the evaluation function within the federal government, I am of the opinion that with the new policy on evaluation, the guidelines and guides, and especially through our interactions with the deputy ministers and evaluators, we are laying the foundation for building a stronger, more competent and productive evaluation function in the Government of Canada in order to ensure better expenditure management.

Thank you.

[The Chair, Liberal, Shawn Murphy]

Thank you very much, Madame d'Auray.

Now we're going to hear from Neil Yeates, the Deputy Minister of Citizenship and Immigration.

[Neil Yeates, Deputy Minister of Citizenship and Immigration]

Good morning, Mr. Chairman, ladies and gentlemen.

I'm Neil Yeates, Deputy Minister of Citizenship and Immigration, as the chair has noted. I'm joined by Elizabeth Ruddick, who is the director general of research and evaluation at CIC.

I would like to thank the committee for inviting me back to speak today. Today I will focus my brief remarks on chapter 1 of the Auditor General's report, and afterwards, we will be happy to answer any questions you have.

With its focus on results and accountability, the government has emphasized the importance of the evaluation function in assessing the effectiveness of federal policies, programs and services.

Our Evaluation Division leads the evaluation function, and has developed an action plan, tabled here today, to respond to the Auditor General's recommendations.

I'd like to highlight progress we've made over the five-year period between 2004 and 2009. CIC initiated significant changes to the evaluation function with the creation of the research and evaluation branch and the establishment of an evaluation committee, as well as the implementation of a formal evaluation policy. Funding for the function increased from about $650,000 in 2004-05 to about $2 million in 2008-09. That's along with an increase in the number of professional staff, from three full-time equivalents in that initial year to 13 in 2008-09.

In the past the focus was on evaluating grants and contributions to meet the requirements of the Treasury Board and the Federal Accountability Act. This growth in resources has allowed us to increase the coverage of departmental programs and the rigour and quality of the evaluations themselves. With more and better studies, the results and conclusions of CIC evaluations are increasingly used by senior managers to inform program and policy decisions. As a result, under the management accountability framework, the evaluation function--initially assessed as unacceptable in 2006--reached a rating of acceptable in 2008, showing steady improvement over a relatively short period of time.

The department agrees with the OAG's findings, Mr. Chairman. We have developed an action plan that includes the renewal of the comprehensive departmental performance measurement framework and program activity architecture, and will further the integration of the framework into our business planning process. This will improve the availability of performance information for evaluations.

CIC is also adding an external evaluation expert to the department evaluation committee. We are currently identifying a list of potential candidates and developing terms of reference for such an expert.

A process is now also under way for soliciting client feedback at the end of evaluations. We are finalizing an internal client survey and we will carry out the survey this year. The survey will be administered to senior managers of programs that have recently been evaluated, as well as to members of the evaluation committee.

Mr. Chairman, the Auditor General's report observed that my department's coverage of spending was low, particularly for grants and contributions. This is due largely to the renewal cycle of CIC's grants and contributions and the fact that about 88% are concentrated in only two programs. I'm happy to report that between fiscal year 2009-10 and this fiscal year we will have evaluated these two large programs, accounting for this large proportion of our grants and contributions budget. The other programs are much smaller in comparison, but all will be evaluated over the five-year cycle.

Recognizing the need for more comprehensive evaluations and a broader coverage, CIC has increased the non-salary evaluation budget by $500,000 this fiscal year, 2010-11. We will add an additional $500,000 in 2011-12, for a total non-salary budget of $1.5 million in 2010-11 and $2 million in 2011-12 and ongoing. As well, by the end of this fiscal year the FTE complement will reach 20 persons devoted to this function.

Mr. Chairman, over the past several years some evaluations have had to be postponed or rescheduled for various reasons, including a lack of available performance data, which has created challenges for completing those and other evaluations in a timely manner.

To avoid similar problems in the future, the Evaluation Division is working closely with CIC staff to develop more robust data collection strategies and tools, to ensure that data collected through our administrative systems will be available in the format required at the time any program is being evaluated.

These are just some of the ways we are working to address the Auditor General's concerns in a timely fashion. We are ready for your questions now.

Thank you very much.

[The Chair, Liberal, Shawn Murphy]

Thank you very much, Mr. Yeates.

Finally, the committee will hear from Ian Shugart, the Deputy Minister of the Environment.

[Ian Shugart, Deputy Minister of the Environment]

Thank you, Mr. Chairman.

I want to add my comment right at the outset that we at Environment Canada concur with the Auditor General regarding the valuable contribution that effectiveness evaluations can bring to decision-making, and we support the recommendations she has made.

At Environment Canada the evaluation function is an important contributor to our decision-making. It provides an essential source of information on the relevance and performance of the department's programs and has an important role to play in managing for results.

We use evaluations to make decisions ranging from program improvement, such as how an existing program should modify its activities or processes to enable it to better meet objectives, to ensuring whether there is an ongoing need for intervention or an appropriate role for the federal government before renewing programs. It provides me with information I need to demonstrate accountability for the use of public funds.

Environment Canada's evaluation function also plays an active role in the review of memoranda to cabinet, Treasury Board submissions, the department's performance measurement framework as well as individual programs' performance measurement strategies and plans. This ensures that program managers are considering, planning for and collecting performance information that can be used in future evaluations.

Our valuation function has had continuous improvement processes in place for several years, and we continue to look for ways to improve the value of our evaluations to support the department. Some of this was noted, and we were pleased to see that in the report of the Auditor General. We've accepted the two recommendations that the report addressed to Environment Canada. In particular, we agree with the recommendation to develop and implement an action plan to ensure that ongoing program performance information is collected to support effectiveness evaluation.

Past evaluations have included recommendations for improvement in the area of performance measurement. We think we're now starting to see improvements in the department with respect to an increasing number of programs that are developing and implementing performance measurement strategies, and we will continue to work hard at this, because it is critical.

Environment Canada also accepts the Auditor General's recommendation to consider the merits of including external members on departmental valuation committees. We have recently received a preliminary guidance from the centre of excellence for evaluation on this issue, and we're pursuing that examination.

Finally, I'd like to speak briefly to the concerns identified in the report regarding the capacity of departments to evaluate all direct program spending every five years. This policy was put in place to increase coverage, and this is an important goal, one we agree with. It recognizes the benefit of maintaining a broader base of effectiveness information for departmental activity. It is important to acknowledge that there are challenges inherent in striving for greater accountability. Increased evaluation coverage, in order to have more information on program performance, has to be balanced against the need to focus on program delivery and the realization of results.

The four-year implementation period for the policy allows us, we believe, the time to adapt our approach, realistically to expand the scope of evaluation activity within the context of department priorities, resources, and program requirements. To increase coverage within current funding levels, Environment Canada will adopt a flexible risk-based approach to planning the study's scope, the approach, and the level of effort for each evaluation. In so doing we will ensure that our evaluation resources are focused on areas within the department where evaluation information is most needed.

Finally, where appropriate, we will conduct evaluations that take a broader perspective on some areas of the program activity architecture, as opposed to conducting individual evaluations of each program within the PAA element.

These are some of the changes that are taking place at Environment Canada that were motivated both by the Auditor General's report and by the new policy on evaluation. We're looking forward to and believe we will see positive impacts in our evaluation program resulting from both of these.

Thank you, Mr. Chairman.

[The Chair, Liberal, Shawn Murphy]

Thank you, Mr. Shugart.

We'll now going to go to the first round of questions, starting with the Liberals, seven minutes.

Monsieur Dion.

[Stéphane Dion, Liberal, Saint-Laurent—Cartierville, QC]

Thank you Mr. Chairman.

Good morning to you all.

Ms. Fraser, I expect that for the Office of the Auditor General, it's a dream come true to be able to evaluate evaluations. You must feel like a fish in water. I suppose what you found however is less of a dream come true. I'd like to begin by making sure I understood you. You stated that you examined 6 departments and that you reviewed 23 evaluation reports, is that correct? I'm assuming that's a sample. What percentage of the evaluations carried out by those six departments over those years would that represent?

[Tom Wileman, Principal, Office of the Auditor General of Canada]

Mr. Chairman, it wasn't that kind of a sample. We asked the departments to provide us with a list of all the evaluations they undertook in order to examine their effectiveness. We chose some of the evaluation reports on their lists, on effectiveness, according to a procedure that we have adopted. We then reviewed those reports.

[Stéphane Dion, Liberal, Saint-Laurent—Cartierville, QC]

How many evaluations did you receive? Out of how many did you review 23?

[Tom Wileman]

I do not have that information. Perhaps I could provide you with that information later.

[Stéphane Dion, Liberal, Saint-Laurent—Cartierville, QC]

Can you give me an idea? How many evaluations do you think those six departments would have undertaken?

[Sheila Fraser]

Mr. Chairman, on page 9 in exhibit 1.3, you can see how many evaluations were completed over that period. The number varies from 14 to 42 by department.

[Stéphane Dion, Liberal, Saint-Laurent—Cartierville, QC]

Out of the 23 evaluations, 17 did not have adequate data. Therefore one can conclude that they weren't particularly enlightening.

[Sheila Fraser]

Yes, the evaluation was not complete because of a lack of information.

[Stéphane Dion, Liberal, Saint-Laurent—Cartierville, QC]

That leaves six. Can one assume that the six other evaluations were of acceptable quality?

[Sheila Fraser]

Yes, I believe so.

[Tom Wileman]

There were problems in 17 out of the 23 evaluations. The departments pointed out in their report that there were shortcomings in performance data. The other departments did not mention that. Our review showed that 17 evaluations out of the 23 had in fact indicated, in the performance evaluation, a shortage of information on performance, performance data.

[Stéphane Dion, Liberal, Saint-Laurent—Cartierville, QC]

I know, but were you satisfied with the other six evaluations? Did this lead to program improvement, or to the elimination of bad programs? What was the outcome?

[Tom Wileman]

The purpose was to determine to what extent the 23 reports were complete, based on certain criteria. The other six did not mention any problems or issues with respect to data collection or methods. In those reports, it appeared to us that the procedure was more satisfactory with respect to program evaluation standards.

[Stéphane Dion, Liberal, Saint-Laurent—Cartierville, QC]

There were problems in more than two-thirds of the evaluations you reviewed. The other six perhaps gave results. What is even more worrisome is that this only represents 5% to 13% of the programs per year. The others aren't even evaluated. Therefore very few are evaluated. For those that are evaluated, in more than two-thirds of cases, the evaluations are not satisfactory. One would hope that at least the others would lead to results. That's what your report states.

However, six departments—we heard this again this morning—increased their resources in this area. I did the math. If one looks at exhibit 1.6 on page 22 in the English version and page 27 in the French version, from 2004-2005 to 2008-2009, there was an increase of 38% in the funding for program evaluations. With respect to staff increases, table 1.7 on page 23 in the English version and page 28 in the French version shows that there was a 54% increase in evaluation staff. We end up with this kind of result. Obviously one has to ask why we have such unsatisfactory results with an increase in resources.

How much more should resources be increased, given that only 5% to 13% of programs were evaluated, and badly evaluated, given that your goal is to evaluate them all over five years? Twenty per cent of programs should be evaluated over five years, satisfactorily, when you're having difficulty in recruiting competent staff, which is another aggravating factor. You have used contractors, but we don't know if the contractors will stay long enough to provide any memory or experience to the various departments and Treasury Board itself appears to be completely overwhelmed. According to your report, there is not enough staff and you have not provided sufficient leadership. How will we get there? The government is increasing its requirements and you are not able to meet the ones you had already. The gap is quite glaring: 5% to 13% of programs evaluated when the goal is adequate evaluations of at least 20%.

[Michelle d'Auray]

Thank you Mr. Chairman.

You have asked several questions in your question. I will try to respond as fully as possible.

The points that were raised by the audit were in fact identified by Treasury Board Secretariat. That is why, over the evaluation period we reviewed and redrafted the evaluation policy. The policy now deals with evaluation in terms of program performance. I should point out the importance of data collection. That is why we put considerable effort into establishing data for measuring performance, because that was one of the deficiencies noted in Ms. Fraser's report.

We also provided for greater flexibility in the policy in terms of the type of evaluation, scope of coverage, as well as risk management, in other words their nature and importance. As Ms. Fraser pointed out, if these are programs that are going to change, then departments must be encouraged to focus on those questions.

We also started recruiting staff, as I pointed out in my opening remarks. We have created two recruitment pools at the intermediate entry level. There are approximately 1,500 individuals in these pools. We represent, in terms of the evaluation function, more than 500 individuals throughout government. Last year evaluation coverage was on average 15% for programs. In terms of grants and contributions programs, we have achieved more than 65% coverage.

I would say therefore that although not everything is perfect, we have made progress. We are now working in a very concrete and practical fashion with the evaluation community and with departments.

[The Chair, Liberal, Shawn Murphy]

Thank you very much.

Madame Faille is next, for seven minutes.

[Meili Faille, Bloc, Vaudreuil—Soulanges, QC]

Thank you, Mr. Chairman.

I'd like to pick up where my colleague left off.

If one looks at exhibits 1.3 and 1.4 in the Auditor General's report, one can see that evaluations were planned and appear to have been completed within the time periods. Eighty-eight per cent of Canadian Heritage's evaluations were completed; 80% of Fisheries and Oceans' were completed, etc.

However, table 1.4 notes that a low proportion of total program expenses were evaluated. The estimated average annual percentage of program expenses evaluated is relatively low.

Could you please help us understand what you focus on and why, and who sets the priorities?

[Michelle d'Auray]

Priorities are in fact set by the departments because they are the ones who must establish their evaluation priorities. That being said, under the Financial Administration Act, 100% of grants and contributions programs must be reviewed over a five-year period. Therefore, as I explained earlier, to date more than 65% of grants and contributions programs have been evaluated, if you include 2009-2010.

As my colleague, Mr. Yeates, indicated, if one includes his evaluations or the evaluations he is about to complete, that coverage would be 88% of his expenditures. This is a process that takes place over five years and the same approach is used for all programs subject to evaluation under the evaluation policy.

We really have concentrated our efforts on the fundamentals, that is, data collection, recruitment, identification of skills for evaluators. We now have a guide to evaluation skills.

[Meili Faille, Bloc, Vaudreuil—Soulanges, QC]

Excuse me for interrupting. You started with data collection, but that is what appears to be lacking in the evaluations. Did you list that first because that is your main priority?

[Michelle d'Auray]

Mr. Chairman, it is one of the fundamental factors in our ability to undertake evaluations.

My colleagues have examined these issues, especially in the area of the environment, and they might like to expand on their approach.

[Neil Yeates]

Yes. I would add that it is true that this is a significant challenge for us.

The data is often very expensive for us to collect. At CIC we have an income tax file that we created with Statistics Canada a number of years ago to track immigrant outcomes over time. So that has proven to be very useful for us in looking at economic outcomes for immigrants. It's excellent data.

In some of our other programs, for example our language classes and so on, we have not had a good system for collecting language outcome data.

[Meili Faille, Bloc, Vaudreuil—Soulanges, QC]

Have you just done that this year?

[Neil Yeates]

No. We've had the income tax data file for 20 years--a long time.

[Meili Faille, Bloc, Vaudreuil—Soulanges, QC]

That is what I was wondering, because I remember meetings with Ms. Ruddick on data for the immigration system.

Here's my point. It seems to me that every time you come here we're talking about inadequate data, etc. I cannot understand why after 40 years, we're still evaluating programs. Information such as the criteria on the basis of which programs are evaluated have not been determined or assessed, and it is not a priority. Yet, today, in 2010, all of a sudden, this is a priority because the Auditor General has provided the background. One can see that this has not been resolved over the years.

Furthermore, in paragraph 1.96 to 1.100, the Auditor General points out that during her exchanges with the various departments, the departments expressed their concerns over their ability to evaluate programs. The report goes on to say that they were not able to undertake the required improvements on a regular basis.

I have a question. When you identify a weakness or a need to make an improvement in departments, why do you not act on those improvements? How does your decision-making process rank the importance of resolving problems as soon as they arise or as soon as they are identified?

[Michelle d'Auray]

Mr. Chairman, perhaps I'll begin by answering and then I'll ask my colleagues to expand on their own programs and activities.

First, with respect to data, we recommended that departments work with their managers. This was for the establishment or renewal of programs, in order to identify performance measurements unique to those programs. We also requested, through our policy, that the evaluation heads in each department draft an annual report on the quality and capture of data.

Those are the measures that we identified. The Auditor General had of course also identified them but we had identified them at about the same time as the audit did. This is what triggered our changes in the evaluation policy, precisely to address those deficiencies. We had also identified them. That is why we agree with the recommendations. Our reflections and the consultations that we undertook with the departments, which led to the new policy, reflected the same deficiencies and observations raised by Ms. Fraser in the audit.

That being said, we are increasingly using the evaluation measures for program reviews, especially for strategic reviews. The same applies to program renewal, under the transfer payments policy. We officially require performance evaluations, because we must focus on the effectiveness evaluation issue, performance evaluations. We are much more demanding with respect to evaluations than we were previously.

I'm not saying all will be resolved by tomorrow morning but I do think we are on the right path. The connections have been made between performance measures, program activity architecture, and expenditure reviews, and the full cycle is now integrated.

[The Chair, Liberal, Shawn Murphy]

Merci, Madame Faille.

Mr. Christopherson has seven minutes.

[David Christopherson, NDP, Hamilton Centre, ON]

Thank you, Chair.

Thank you all very much for your attendance today. It's good to see you all.

Right up front, the last thing in the world I am is an academic, so once we get into the world of evaluations and data you have your best chance for smoking me over, because it's not my area.

Madame d'Auray, on page 4 you said:

Much of our monitoring and support work is carried out through the annual management accountability framework assessment process, which assesses evaluation quality, neutrality, coverage, and use.

So the first thing that struck me when I heard you say that was why didn't this management accountability framework assessment process pick up these deficiencies long before the AG arrived on the scene?

[Michelle d'Auray]

I would say that it did. It led us to undertake a fairly comprehensive review of the evaluation policy that we renewed and redid. It caused us to look at the performance measurement information framework. It led us to have a fairly extensive series of conversations with deputy ministers about how they used evaluation. And it led to the fairly extensive work plan and very pragmatic approach we are taking today.

As I said earlier, it's not perfect, but the gaps were very similar. The approaches we're taking are to address the biggest gaps and make the linkages between expenditures, program effectiveness, improvements, and decisions about whether or not the programs should be maintained, changed, or improved.

So I don't want to say this is all a perfect situation, but in parallel, as we were looking at and assessing departments in their capacities--and the quality, scope, and coverage--we came to the same conclusion the Auditor General did.

[David Christopherson, NDP, Hamilton Centre, ON]

And this was prior to the Auditor General arriving?

[Michelle d'Auray]

We were doing this in the same timeframe, and we were encouraging departments to make improvements.

[David Christopherson, NDP, Hamilton Centre, ON]

I'm sorry, it's a small matter, but I just want to be clear that you started this work, identified this, and began something prior to the Auditor General arriving.

[Michelle d'Auray]

Yes.

[David Christopherson, NDP, Hamilton Centre, ON]

Auditor General, is that correct? Did you see all this beginning when you arrived?

[Sheila Fraser]

We do note in the report activities on monitoring and oversight in paragraphs 169 through to 174. We do note that over the years a number of issues were identified that were very similar to what we found.

[David Christopherson, NDP, Hamilton Centre, ON]

Good.

Just to drill down one more step, did the analysts in the line ministries pick this up too? Did they report to their supervisors that they were unable, from a professional perspective, to deliver the kinds of quality evaluations they would like to?

I'm just checking to see if the system worked all the way through, or did it take the AG to come in and trigger everything--because we have both. We get into all kinds of situations, so I'm just trying to get a sense of how well our systems were working underneath. Right at the beginning, were the line ministry analysts able to determine they had a problem, it went up, and then it got bumped up further to the Treasury Board, where things started to happen? Then did the AG come in, see some of this, and then do her further work?

[Neil Yeates]

I can start off, Mr. Chair, speaking to the experience in CIC.

We've been working on ramping up our evaluation capacity and function for several years. I noted in my opening remarks that in the management accountability framework we received an unacceptable rating in 2006. So that was well before the AG's work started on this. We knew we had some pretty significant challenges where we needed to improve.

We also had discussions within the department about the challenges we faced. So the Treasury Board assessment wasn't news to us. We've been working on those issues ever since. That's why I say we now have a five-year evaluation plan in place that will provide the 100% coverage that's expected in the policy.

So I think there is a fair history to this.

[Sheila Fraser]

I just want to add some context. I think the major impetus for all this was a change in policies and the requirement to do more evaluation of grant and contribution programs, and then the recent evaluation that sets all direct spending. Those requirements were not in place previously.

As I mentioned, this audit is about a year old now. We were doing this to see how the evaluation functioned across government. Was it prepared to be able to assume these new responsibilities under this new policy that came out? We can see, from what the secretary said, that a lot of work has been done in the last year to try to increase the capacity.

We also note in the report that most of the departments we looked at did not have formal quality management systems in place, or continuous improvement. Environment Canada did, and it can be a model for others in how they do that.

[Ian Shugart]

Mr. Chairman, if I could just add and maybe connect the committee's preoccupation on—

[David Christopherson, NDP, Hamilton Centre, ON]

My next question was directly to you. You can answer my question and throw your comments in, how's that? It's a simple question.

Again, in your opening remarks you stated that

It is important to acknowledge that there are challenges inherent in striving for greater accountability. Increased evaluation coverage in order to have more information on program performance has to be balanced against the need to focus on program delivery and the realization of results.

I was just trying to understand how that was unique to this situation, unique to your ministry. It sounds like the balancing act that as deputy you do every day. So help me understand what you meant by that. I just didn't quite get it. I understand that you have the challenge, but why is this challenge unique enough that you needed to underscore it in your remarks to us?

[The Chair, Liberal, Shawn Murphy]

Go ahead, Mr. Shugart.

[Ian Shugart]

Primarily, Chair, it's because the Auditor General in her report refers to the challenge that audit staff and officials within departments have recognized in completing the policy set out by the board. So I anticipated that this is an issue that is germane to this whole debate, and simply wanted to refer to it in that way.

You're absolutely right, it is characteristic of the kind of thing that we have to do all the time, and I think I would say that this is an excellent example of the kind of continuous improvement that we try to achieve, and it's relevant to the issue of data. As a deputy minister, even before the audit, before the policy, before the requirement of 100% coverage in grants and contributions spending, I would receive an evaluation report, and typically that evaluation report would indicate that we were able to answer these questions because we had data, but not these questions because there is no performance data. So even within the context of a particular evaluation, it's not a complete lack of data that we have; sometimes there is data but it is not developed by program managers to support evaluation per se.

For example, we might be able to know what the coverage of a particular program is, but we might not have data that relates to service standards, for example, and an evaluation--for which we decide in the department what program will be evaluated and what is the scope and the nature of the evaluation--would identify relevant questions. There might be data for some of the questions but not for others. And evaluation staff will, in their interaction with the AG and her team, have identified that one of the areas for improvement that we need is to have performance management.

In Environment Canada we recently redid our performance activity architecture. We already have a performance management framework, but it is by no means complete. So within the department we continue to work on our performance management framework, all of the data development that will in fact support that PAA, and it will then result in more data that's available for evaluations in the future.

The last comment I would make is that I as a deputy don't welcome and yet I do welcome the policy on 100% coverage. I don't welcome it because it's another thing I have to meet the requirement for; it's another pressure, another obligation. I do welcome it because it is the right thing to do, and because it will add a discipline to everything we do, both in program delivery and in evaluation. It will force us to develop the performance data, and so on. Will it be perfect in three or five years? No, it won't, because there will still be relevant questions that should be asked, and we may not have all of the data, but we will be improving, and I'm confident of that because we're already on a trajectory of improvement.

[The Chair, Liberal, Shawn Murphy]

Thank you very much, Mr. Christopherson.

Thank you, Mr. Shugart.

Mr. Saxton, seven minutes.

[Andrew Saxton, Conservative, North Vancouver, BC]

Thank you, Mr. Chair.

Thank you to the witnesses for being here today.

My first question is for the Auditor General. Can you please elaborate on the purpose of the audit?

[Sheila Fraser]

The purpose was really to see the capacity of the evaluation function across government in terms of, obviously, human resource capacity, but also were there competency profiles, were there standards that were being set, and were departments in a position to be able to meet the requirements of the new policy, or was it expected the departments would be in a position to meet the requirements of the new policy that would be coming into effect in 2013?

[Andrew Saxton, Conservative, North Vancouver, BC]

Have you done an audit like this previously? How long has it been since the last audit was done?

[Sheila Fraser]

The last audit on the evaluation function was in 2000. We did do a similar audit on the internal audit function in 2004.

[Andrew Saxton, Conservative, North Vancouver, BC]

Thank you.

In your report you note that Environment Canada already had good practices in place. Can you expand on some of these good practices?

[Sheila Fraser]

We note in the report that they had good practices regarding continuous improvement, and I believe one of the examples we had was surveys they use to collect information on how the function should be improved. That was something we did not see in the other departments.

[Andrew Saxton, Conservative, North Vancouver, BC]

Thank you.

My next question is for the Secretary of the Treasury Board. With the new policy on evaluation, what gaps are you hoping to address that the previous policy did not?

[Michelle d'Auray]

One of the issues was coverage. Another was the nature. Another was the ability to tailor the type of evaluation to the type of program. Another was clarity around the collection of data, or performance measurement. We also addressed the issue of governance, the independence, if I could put it that way, of the evaluation function as well as the responsibilities of the deputy head in relation to the evaluation function.

We covered a range of issues. A lot of those are very similar to the issues raised by the audit. At the same time, they also address a number of issues we have found as part of the capacity to assess programs in light of expenditures.

The second component, the really important one, was the series of interviews that were held with deputy heads around the usefulness and importance—and this came directly from colleagues—of the use of evaluation as a management tool for the effectiveness of programs and the results. All those elements are now in the policy, the directive, and the standard.

[Andrew Saxton, Conservative, North Vancouver, BC]

Thank you.

Can you explain how evaluations that address program value for money will better serve government decision-makers?

[Michelle d'Auray]

Essentially, an effectiveness evaluation looks at the cost, the effectiveness, the results, and can occasionally look at the process around the delivery. It looks at the effectiveness and efficiencies around the program and the initiative. If you're setting up an initiative or program to achieve a certain set of results, it really is critical to set out what your performance measures will be—which is now clearly the responsibility of the program managers—for us to be able to assess whether the program and the initiative are achieving the intended results. Sometimes it can do so over a very long period, so we measure periodically. That's why we need to cover the entire program suite within a five-year period. On average, 20% of the program should be reviewed.

It also should give us some baseline information and means of addressing whether the programs are meeting the objectives and how they can be adjusted to do so. We found that particularly key in supporting the strategic reviews that departments are currently undertaking to identify the least-performing programs or initiatives, in order to reallocate expenditures to higher-performing programs. It's very difficult to do so if you don't have very good effectiveness evaluations.

Perhaps my colleagues would like to add something to that.

[Ian Shugart]

I could, Mr. Chairman.

I would say that there are benefits to policy-making if an evaluation shows that a program is not performing well in some of its objectives. That is relevant for ourselves and for central agencies in the process of determining whether a program is renewed. It may also tell us that part of the program is working very well but another part isn't.

Sometimes this may be due to capacity at the federal level vis-à-vis the provincial level. For example, a province may be able to do better than the federal government with respect to one aspect of a program.

Another area of benefit is in the management of programs. We posted an audit on environmental emergencies, where one of the details that came out in the audit had to do with the roles of our environmental emergency officers. On the ground, in the case of an emergency where we're providing support, it's very important to have a very clear delineation between the responsibility of environmental emergency officers and enforcement officers. This was an area that the evaluation showed was not entirely clear. That allowed us to issue policy guidance and an actual statement of responsibilities for our environmental emergency officers, as opposed to our enforcement personnel, so that on the ground there's no confusion and there are no legal problems about who is authorized to do what. It's a small point, but it allowed us to improve the delivery of the program on the ground.

These are a couple of examples where evaluations actually do make a difference, sometimes in the continuation of the whole program, at other times in the improvement of its delivery.

[The Chair, Liberal, Shawn Murphy]

Thank you, Mr. Shugart.

Just one question before we go to the second round.

Following up on your comment, Ms. Fraser, about the HR capacity, we're into a situation where there's going to be more work. If you look at exhibit 1.8 on page 25 of the English copy of the report, it seems that close to 90% of the person-years involved in these evaluations are done by outside contractors. Perhaps I'm wrong, but that to me would be some suggestion that if they're using outside contractors and consultants, perhaps there's not enough HR planning going into this whole process. We all know it's much more expensive and it does not create any capacity within government to do this type of work.

Do you have any comment as to that trend? It really hasn't improved at all in the five years of the audit between 2004-05 and 2008-09.

[Sheila Fraser]

Thank you, Chair.

That was one of the major issues raised by the various departmental functions. We note in the report that for the period we looked at, about 90% of the evaluations were wholly or partially conducted by contractors.

There will always be a need for contractors. I think we have to recognize that. But I was pleased to hear the secretary mention the work that has been done on recruiting and qualifying people, so that the capacity will remain within government and it won't be contractors coming in and then leaving with the information. It's important that there be sufficient capacity within departmental staff. I don't know the progress that has been made, but there certainly would appear to have been planning done, and efforts done, within the last year.

[The Chair, Liberal, Shawn Murphy]

To you, Madame d'Auray, I appreciate what the auditor has just said, that there will always be some need to hire outside expertise, but do you agree with me that when 90% of your work is done by outside contractors, you're really developing very little, if any, in-house expertise? The capacity will never be developed, especially in an environment where you're trying to increase the number of evaluations that you're doing, 100% coverage over a five-year cycle. Do you see real problems there? Do you see, in the past anyway, certainly a lack of HR planning?

[Michelle d'Auray]

Thank you, Mr. Chair.

We did recognize that gap, and we are working on many fronts to fill it. One is the work we are doing to recruit. We have also set guidelines on what the core competencies are. There has been some increased funding in departments. We've seen a growth in the number of evaluators available in the government. We're at just over 500 right now.

The Auditor General made an interesting comparison to the work that started with the audit community. We are following, I would say, a similar pattern in terms of identifying what the competencies are, identifying what the governance structure should be, and identifying the nature of the work of the evaluators and the evaluation function.

I think that in departments there is a growing recognition of the need to manage and oversee the evaluations themselves and to sometimes, as necessary, bring in contracting expertise, because not all the evaluators will necessarily have all the expertise necessary to cover the work.

I know that my colleague could speak from departmental experience about what that means, if that's possible.

[The Chair, Liberal, Shawn Murphy]

Perhaps we can catch that later on.

I'd like now to move on to the second round.

Mr. Lee, you have five minutes.

[Derek Lee, Liberal, Scarborough—Rouge River, ON]

Thank you.

I'd like to go to Treasury Board.

There's a reference to a five-year cycle, or a five-year program. When did the five years begin?

[Michelle d'Auray]

The new policy was put in place in April 2009. The five years will end in 2013-14, for the transition period. The formal period, when there will be the requirement to cover all of the evaluation in the five years, is from 2013-14 to 2017-18. That is when the obligation to cover 100% of direct program expenditures kicks in.

[Derek Lee, Liberal, Scarborough—Rouge River, ON]

It is 2017.

[Michelle d'Auray]

It is by the end of that period.

[Derek Lee, Liberal, Scarborough—Rouge River, ON]

We should come back then.

Who is responsible for the implementation of this five-year cycle? Is it Mr. Smith?

Hello, Mr. Smith. Mr. Smith was a wonderful witness last year at another committee. He was very helpful, very professional.

Over to you, Mr. Smith.

[Alister Smith, Assistant Secretary, Expenditure Management Sector, Treasury Board Secretariat]

Thank you, Mr. Lee.

Yes, we are responsible. Treasury Board Secretariat is responsible for providing oversight of the function.

[Derek Lee, Liberal, Scarborough—Rouge River, ON]

You're wonderful at that. Is it you--not we—is it you?

[Alister Smith]

Yes, it is.

[Derek Lee, Liberal, Scarborough—Rouge River, ON]

If you're the quarterback, and you throw the ball, do you have people out there to catch it?

[Alister Smith]

We hope that we will have more and more people to catch it as we move along, yes.

[Derek Lee, Liberal, Scarborough—Rouge River, ON]

So it really is you. I don't want to put more weight on your shoulders than is fair. You are the quarterback for the implementation of this whole thing across government. It's you.

[Alister Smith]

Yes, that's correct.

[Derek Lee, Liberal, Scarborough—Rouge River, ON]

That's a big responsibility.

[Alister Smith]

It is.

[Derek Lee, Liberal, Scarborough—Rouge River, ON]

It reminds me of a vignette of evaluators who go out to a food bank. They have their clipboards and their video cameras, and the food bank manager comes out and says, “We're too busy to do this now. Come back when nobody's hungry.”

Have you established benchmarks for the implementation of this across government?

[Alister Smith]

We've established milestones. We've established a great number of particular objectives. This year, as you have heard before, our primary objective is to ensure that our guidance is firmly determined, that it is supported by departments, and that they see this as pragmatic and doable. We hope to finalize our guidance to establish good departmental evaluation plans and to report annually on the health of the evaluation function. We hope to then move over this four-year period to the five-year real implementation period, when we will try to ramp up spending coverage to 20% a year from the current level of 16%.

[Derek Lee, Liberal, Scarborough—Rouge River, ON]

In the Auditor General's report there was reference to a March 31, 2010 deadline for a document called a guidance document. Did Treasury Board succeed in getting that guidance document out there on time?

[Alister Smith]

All of our draft guidance documents—and there are at least three different parts of this—have been in circulation in departments over the last few months. We're finalizing them now, and we will be finalizing them at different stages through 2010-11. By the start of the next fiscal year, we hope to have all this guidance in a very solid form.

It's very important that we ensure that we've thought through all the issues, such as, for example, the calibration of effort, before we require departments to follow this.

[The Chair, Liberal, Shawn Murphy]

Was that a clear enough answer? I'm not sure.

[Derek Lee, Liberal, Scarborough—Rouge River, ON]

I was interested in the concept of the calibration of effort. I don't know what it is.

This five-year cycle, of course, is way out beyond the world that we around this table know. This is way into the future, so we're relying on you professionals to do this. I think we've made progress in focusing on some of the tangibles, but is there anything you wanted to add, Mr. Smith?

[Alister Smith]

No. We know that we have to continue to build the community, build up competencies, and build up our indeterminate personnel within government. We have some real challenges, clearly, but we're working on many, many fronts. We're working with 11 universities to build up the community. We have a community of practice well established. We have a lot of guidance, as we mentioned, in circulation. We have many expert evaluators within government helping us. So I'm confident that we can, over this four-year period, ramp up our capacity in order to undertake full coverage.

[Derek Lee, Liberal, Scarborough—Rouge River, ON]

This is a tough question for you, but do you have sufficient commitment and resources from the people who supervise you to do this job? Are there enough resources to accomplish the task?

[Alister Smith]

Within Treasury Board itself, or do you mean across government?

[Derek Lee, Liberal, Scarborough—Rouge River, ON]

Well, anywhere in the system. You're the quarterback.

[Alister Smith]

Frankly, I think anybody responsible for an initiative would want more resources rather than less. Certainly we will be trying to make the case. In departments it will be a matter for the deputies, in the first instance, to determine whether they require more resources. Mr. Yeates and Mr. Shugart, for example, mentioned they are very conscious of the need for resources in these areas. There are limits to the resources available for any of these initiatives, but we will continue to argue for more resources in this area.

[Derek Lee, Liberal, Scarborough—Rouge River, ON]

Thank you, Mr. Chair.

[The Chair, Liberal, Shawn Murphy]

Just to clarify one point, Mr. Smith, regarding the recommendation that was made—and Mr. Lee brought this up—Treasury Board Secretariat responded by saying that it was planning to issue, by March 31, “written guidance for making risk-based choices for evaluation coverage to support departments during the transition period”. Can you file a copy of that guidance with the committee?

[Alister Smith]

Absolutely.

[The Chair, Liberal, Shawn Murphy]

Can you do that within two weeks? Would that be fine?

[Alister Smith]

Yes. This is still the draft guidance, but we would be happy to provide it to you.

[The Chair, Liberal, Shawn Murphy]

Mr. Kramp.

[Daryl Kramp, Conservative, Prince Edward—Hastings, ON]

Thank you, Chair.

I have thought of a bit of a red herring in looking at a cost-benefit analysis of this whole process. I can recall being in business once, when an audit cost me $20,000 in professional fees. Goodness knows what it costs government. I know from my audit that there was an $11 discrepancy.

I'm asking for an opinion on the following. We have programs being run by the best people we have available in this country, by all of our program directors and ADMs. We also have an Auditor General who is doing a great job, and we're very, very pleased to see that. But we have auditors auditing the auditors of the department, who audit the internal auditors, who audit the evaluators. We're running down through....

I'm just wondering about the horrendous cost involved. Is the benefit of all of this comparable to the cost, or are we really just building another multi-levelled or layered bureaucracy? We have 350,000 civil servants in this country employed by the federal government alone. Is it necessary to audit the auditors, and to audit the other auditors who audit the auditors who audit the evaluators?

On my point, could you again just make a quick observation, Madam Fraser?

[Sheila Fraser]

I interpret that to mean why did we do this audit, right?

[Daryl Kramp, Conservative, Prince Edward—Hastings, ON]

Yes.

[Sheila Fraser]

Chair, I think everyone has agreed that the evaluation of effectiveness is absolutely critical to making good decisions about program spending so that we can know whether programs are actually getting the results that were intended. I would say this is even more critical in the current economic times we are going through, because government does have to make difficult choices, and it should be making those decisions based on good information. So when we look at the various areas we think are important in the management of government, we think that the evaluation function is a really important one. That is why we did this audit.

If I could just add, I think we have seen in the past when economic times have been difficult that it has been very easy to cut functions like evaluations and internal audits and these sorts of back-office functions. I think it is important, with the new policy that has come out, that government is ensuring that this function continues; otherwise decisions are going to be made without the kind of information managers really need.

So that's why we thought this audit was important, and I would hope the committee thinks it is important.

[Daryl Kramp, Conservative, Prince Edward—Hastings, ON]

Fine. Thank you very kindly.

A question now to Treasury Board. In your action plan you mentioned you established an interdepartmental advisory committee of deputy ministers. You go on to say you'll be working to develop a best practices handbook--in other words, to share the cost-benefit and to share your experiences.

Can you tell us the present status of that process?

[Alister Smith]

We've established a committee at the assistant deputy minister level. We also have working groups looking at specific issues across government in harnessing the expertise of our senior evaluators across government and heads of evaluation. We are pulling a lot of the information together, especially from the working groups, into a guide for evaluators. We have a best practice guide on performance reporting, but this would be different. It would be a guidebook for evaluators. That's the objective we're working toward.

[Daryl Kramp, Conservative, Prince Edward—Hastings, ON]

Thank you.

A question now to CIC. In your departmental response, you agreed you would have an external evaluation expert at the departmental evaluation committee and this item would be brought to the evaluation committee for a decision by April 2010. It's now May. What's the status?

[Neil Yeates]

The evaluation committee has agreed that we are going to proceed. We've actually been in touch with someone to invite them to join our committee, so we're pleased with that.

[Daryl Kramp, Conservative, Prince Edward—Hastings, ON]

Thank you.

A question now to the Auditor General. You've stated in your opening statement that the lessons learned from the government's recent strengthening of internal audit could be applied usefully to the program evaluation, and that is basically what you stated in your response to me earlier. But what, in particular, did you have in mind when you were offering that assessment, what lessons particularly?

[Sheila Fraser]

Chair, I think we've noticed a lot of improvement in the internal audit function. The secretary referred to many of the actions that were taken, evaluating what are the competencies that should be there, focusing on the human resource capacity, standards and practice in the community, and more recognition of the importance of the function. So I think there's much more attention to that function, the importance of the function, and the governance as well of the function. We saw that those issues were addressed in the internal audit function. I shouldn't pre-judge--we're doing a follow-up audit--but I think we can see that a lot of effort has been made there. And I think we see signs of the same processes being applied to the evaluation function.

[Daryl Kramp, Conservative, Prince Edward—Hastings, ON]

Great. Thank you.

[The Chair, Liberal, Shawn Murphy]

Madame Beaudin, five minutes.

[Josée Beaudin, Bloc, Saint-Lambert, QC]

Thank you Mr. Chairman.

Thank you and good morning to all of you.

I have one or two main questions. Environment Canada seems to be the only department that has an internal process for systematically identifying areas where improvement in evaluation methods is required.

Should we take from that that you do not have any problems with resources? You have enough evaluators to undertake these evaluations?

[Ian Shugart]

That may be somewhat too positive a conclusion. Each department makes its decisions differently. We acknowledge rather early in the process that quality improvement would be very useful. In order to guarantee manager participation in the programs, we felt it would be useful to have a commitment to

surveys that resulted from that, where after evaluations we would follow up with the program managers.

As well, we concluded that if we learned valuable lessons we should write them down and create a best-practices registry and discuss these within our community. Some rather simple improvements led to better practices. However, I'm certain that throughout government my colleagues have improvements that would be very useful. Therefore in the future I will also try to draw on their experience.

[Josée Beaudin, Bloc, Saint-Lambert, QC]

It occurs to me that your expertise could be useful to your colleagues. Do you work together? Do departments, for example, work with you, Ms. d'Auray, in order to share best practices?

Furthermore, if you do experience problems with respect to resources for necessary evaluations, how will Environment Canada proceed?

My question is for Mr. Shugart. How do you evaluate that lack of resources? How will you proceed within your department?

I therefore have two questions. Do you work together? How will you approach the situation?

Perhaps I have one last question. Ms. d'Auray, what is your contingency plan if departments end up experiencing problems in meeting the necessary requirements? What is the contingency plan? What measures will be used?

[Michelle d'Auray]

Thank you Mr. Chairman.

With respect to your first question on whether or not departments work together, the answer is yes. In June 2009, we established a practices community. We have regular meetings where we share information on best practices. We have also had several discussions on key skills. There are also four working groups that focus on the very specific questions that evaluators ask, as Alister Smith mentioned. The secretariat doesn't know everything but we do encourage sharing information and skills.

You asked a question about our contingency plan. We are quite encouraged by the progress and percentage of coverage achieved to date. If necessary, we will assess any measures that need to be taken. For now, given the amount of flexibility under the evaluation policy that we suggested, we are quite confident that departments will meet the requirements.

[Josée Beaudin, Bloc, Saint-Lambert, QC]

You didn't mention a contingency plan. If the departments experience short-term difficulties, how will you react in order to stay the course?

[Michelle d'Auray]

We have given them four years for this transition. Give us at least enough time to get halfway through that transition period. At that time we'll be able to assess whether or not adjustments need to be made. For now let's give them a chance to make some progress.

Perhaps Mr. Shugart wants to respond to your second question.

[Ian Shugart]

I will be brief Mr. Chairman. Perhaps it would be useful to describe the internal process of a department such as Environment Canada. There is an evaluation committee that I chair. Each year, within our planning process, an evaluation group drafts the plan for the coming year. This plan identifies the programs that will be evaluated. The budget is part of that action plan. The committee must decide whether to accept or reject the plan proposed by the group and make any changes. It must also decide whether the budget will be approved or not. Within a broader context, that budget for evaluation functions is one of several requests, several financial pressures. We have to make choices. We can modify the plan in order to adapt to the financial circumstances, but we do have that goal, that requirement, obligation, to cover 100% of our programs by the end of the period.

As Mr. Christopherson pointed out, it's a question of balance. One must make choices and changes for each evaluation.

[Josée Beaudin, Bloc, Saint-Lambert, QC]

Thank you.

[The Chair, Liberal, Shawn Murphy]

Thank you, Madame Beaudin.

Mr. Hoback, for five minutes.

[Randy Hoback, Conservative, Prince Albert, SK]

Thank you, Chair.

I'm curious about how the internal audit process works. Ms. Ruddick, perhaps you could explain it to me. When you're doing evaluations internally, how do you go about passing the information up through the department? If you come across a program that you feel doesn't meet the needs or the intention it was originally created for, how is that relayed through your department?

[Elizabeth Ruddick, Director General, Research and Evaluation, Department of Citizenship and Immigration]

The process in CIC is similar to that in other departments. We have an evaluation committee, chaired by my deputy minister, which meets almost monthly, in fact, because of the workload we have. We come each year with a plan and decide what evaluations will take place. We have various stages of providing information within the department before we finalize the report. For example, we will brief the program managers internally, we will check with them that there are no errors of fact in what we found, then we will finalize the report, and it gets tabled at the evaluation committee with the deputy minister for approval.

Along with the report, there is a list of findings. The responsible program managers—and often this may cut across a number of areas in the department, because it's not just the people designing the policy but also those who are implementing it, and it may involve our regional offices as well—provide a response, and at the same time there is an action plan in terms of how they are going to address the findings.

We also have in place a process to track progress against that action plan on a periodic basis to make sure that the changes proposed and agreed on at the evaluation committee are in fact being implemented.

[Randy Hoback, Conservative, Prince Albert, SK]

Would you look at the actual intention of the program and at whether it is relevant to today's needs? If it's a program that's been going on for four or five years, would you ask whether this is a program that's actually needed or not and report that also?

[Elizabeth Ruddick]

Yes, absolutely. There are a number of questions that we are mandated by Treasury Board to respond to. One of them is program relevance. We look at the objectives and the delivery. We may find, for example, that there are some problems in the delivery or, as was mentioned by Mr. Shugart, that because of some of the relationships across the various groups delivering the program there may be some misunderstandings. We can address that. We're looking at how effectively the program is being delivered, how the money is being spent, and we'll often look at other options for program delivery, if that's possible.

[Randy Hoback, Conservative, Prince Albert, SK]

Ms. Fraser, when you were going through your process of looking at the systems, did you review what external education these people were getting throughout their careers? For example, would Ms. Ruddick have the ability to look at other companies or businesses and say, this is how they go about doing their internal evaluations? Did you look at anything like that?

[Sheila Fraser]

We didn't look at that specifically. We did look, though, at whether there was a competency profile and whether there was an agreement on what evaluators should have as competency. That did not exist at the time we completed the audit. We've heard today that it has been completed since. We would look to whether a standard has been set and what kinds of minimum qualifications you would expect at different levels. That is critical to ensuring that the evaluators are well qualified and able to do their job.

[Randy Hoback, Conservative, Prince Albert, SK]

Are you satisfied that the evaluators are arm's-length enough? They are part of a department, part of a ministry. Are they able to take a situation or problem and effectively relay it up the chain of command? Would it be easier if an external actor came in and looked at it and provided input?

[Sheila Fraser]

I really believe that the evaluators should be part of the department. This is a management tool. It is much like internal audit.

I know we've had discussions with the committee about where internal audit should report. We've always taken the position that internal audit evaluators should be reporting to the deputy head rather than to some external organization—say, the Treasury Board Secretariat.

This is really to be used by management to improve programs. It is not an external audit function or an external evaluation function. The responsibility really should come back to the deputy head to ensure that someone performing the function is well qualified and doing appropriate work and that the results are acted upon.

[Randy Hoback, Conservative, Prince Albert, SK]

You view this as more of an internal function. You don't think there would be any benefit in looking at it as more of an external function in the delivery of programs?

[Sheila Fraser]

I don't see a benefit to doing that, no.

[Randy Hoback, Conservative, Prince Albert, SK]

Are you satisfied with what you've seen, with the examples that you're given, that the deputy ministers are actually taking this function seriously enough and proceeding with the recommendations?

[Sheila Fraser]

I'm certainly pleased to hear of the progress today. As we always say, though, we are cautiously optimistic until we actually audit to see that things have improved. I would expect that at some point in the future the office will return to see how the policy is actually being implemented and whether some of these issues have been dealt with.

[The Chair, Liberal, Shawn Murphy]

I'm sorry to interrupt, but your time is up. Thank you very much.

Now we're going to go back to Mr. Christopherson for five minutes.

[David Christopherson, NDP, Hamilton Centre, ON]

Thank you very much, Mr. Chair.

There's one last area I wanted to follow up on. The auditor's report talks on page 22, starting at paragraph 1.52, about the shortage of experienced program evaluators in the federal government:

The shortage of experienced program evaluators is a long-standing concern. It has been noted in past Office of the Auditor General audits and in diagnostic studies by the Treasury Board of Canada Secretariat, and it was the subject of recent discussion within the federal evaluation community. A 2005 report by the Secretariat Centre of Excellence for Evaluation stated that “[t]he scarcity of evaluation personnel is probably the number one issue facing Heads of Evaluation.”

Two paragraphs later, it reads: “According to officials in the six departments, despite these”--Mr. Dion raised these earlier--“increases in both funding and staff, it remains a challenge to find experienced evaluators, particularly at the senior levels. In their view, the shortage of experienced evaluators has affected their ability to hire the people they need.”

This does get interesting: “For example, in one collective staffing process, the pool of experienced evaluators was depleted before the demand was met. They also indicated that the shortage of experienced evaluators has led to evaluators being hired away by other federal evaluation units.”

That may work well from a micro point of view, but from a macro perspective, it solves nothing. It's a legitimate concern, and we face it. I was thinking, Madame Fraser, that we went through something similar with the Canada Revenue Agency, and there it was analysts who had expertise in international investment income. As a result of not having the experts, there were likely untold amounts of money not coming in, for the simple reason that—no fault of anyone—there just weren't the experts. I'm seeing this as the same thing.

Can you collectively give me a sense of how we're tackling this? Are we speaking with educational institutions and provinces about trying to make sure we're developing them?

There's my very long-winded question.

[Alister Smith]

Thank you for those questions.

It is a challenge to find experienced evaluators. Clearly we're building up this community rapidly. We're building it up largely through intake from universities.

We have a consortium of universities working with us to offer evaluation courses and help with certification. They are from right across the country, from Carleton and Ottawa universities here through Quebec's Université Laval and l'Université de Montréal to the University of Saskatchewan, the University of Victoria, the University of Waterloo. It's right across the country. We're also working with the Canadian Evaluation Society in developing our standards.

We have a competency standard in circulation now that I think is quite good. We have a lot of courses being offered. We have a lot of support from the Canada School of Public Service. We're doing everything we can to build up this community.

It's true that when you start to get evaluators in one department, they could be poached by another department. We will keep trying to build up the community. In the end, as long as those evaluators remain within government, we're better off than we were without having them in government.

[David Christopherson, NDP, Hamilton Centre, ON]

Sure, but you're just squeezing the juice in the orange around in terms of the problem, aren't you? You solve yours, and your colleague now has a problem.

[Alister Smith]

We've ramped up and have probably doubled the number of people who are evaluating over this last decade. We're making headway. It's true that they're in real demand. I think this will also encourage universities to offer more courses to encourage more students to come to this line of work.

[Ian Shugart]

Mr. Chairman, if I may, I would add that one of the responsibilities I have as a deputy minister is to give full support to the evaluation group. Nothing motivates a team of public servants more than knowing their work matters and they have a platform for making a contribution.

I suspect that every deputy minister could say what I'd say about my own team. They are a very impressive group of young people and more mature people who enjoy their work.

As a manager, you have to get used to losing good people. I don't see my responsibility as being restricted to only Environment Canada, because it is a collective pool, but I certainly see my responsibility as meeting the obligations of the policy to hang on to and recruit as good a team as possible.

The data shows the numbers have increased. We use our people well, in conjunction with contractors. We've been known to recruit contractors when they have done a good job.

I would say that it is a terrific career. In an evaluation unit, there is probably no better way to learn what's going on in a department than to get right in there, from the vantage point of evaluation, work with the managers, and find things out.

These people become very good policy and program people and their careers often take a turn from evaluation into other areas. Frankly, we lose some people from evaluation because of that, but we recruit from other areas of the public service as a result. It's a very complicated situation.

[David Christopherson, NDP, Hamilton Centre, ON]

Thank you.

Thank you, Chair.

[The Chair, Liberal, Shawn Murphy]

Thank you, Mr. Christopherson.

Mr. Dreeshen, for five minutes.

[Earl Dreeshen, Conservative, Red Deer, AB]

Thank you very much, Mr. Chair.

Thanks to the witnesses.

I'd like to talk about what Environment Canada is doing.

Mr. Shugart, you pretty well went through the processes that were involved in your evaluation. You also spoke earlier about provincial and federal projects and finding out how those things work together. Could you explain some of the issues associated with that?

[Ian Shugart]

Mr. Chairman, I offered it as an example of the type of question that might come up in an evaluation that could be relevant to performance.

It should be the case that when a program is developed and adopted, at the outset, the appropriate federal and provincial analysis is done. In principle, the Government of Canada shouldn't engage in programs for which it does not have a jurisdictional responsibility, but there are of course many shared jurisdictions in Canada.

Certainly in my field, our work with the provinces is absolutely critical. An evaluation can expose factors of the relationship that can be as simple as how well things are working. A smaller province may not have the capacity that a larger province has and it may rely on us to carry a bigger load. It may not be the case for all programs, but it could be the case for some programs. That type of thing could come to the surface in an evaluation.

On the other hand, there may be deeper questions. Over time provinces may not have had the capacity or the interest or it could be that the federal government has not had the capacity in a particular area. Something that was true 20 years ago for an older program may not be quite the case today.

Those are the types of adjustments to programs that sometimes involve the federal-provincial relationship. An evaluation can be very useful in analyzing and bringing that to the attention of senior managers.

[Earl Dreeshen, Conservative, Red Deer, AB]

Your department is therefore able to give that type of advice to provinces as well.

[Ian Shugart]

In some cases, where we provide a service that provinces need, provinces might be consulted in an evaluation. They may tell us that they don't find this to be helpful. They may say this is absolutely essential to their responsibility and to please keep it up.

When we confront those types of results, it's then our responsibility as senior managers to make sure our minister is informed. When we give policy advice to the minister, that type of thing will be taken into account.

[Earl Dreeshen, Conservative, Red Deer, AB]

Thank you very much.

My next question is for Treasury Board. In the report, on page 45, recommendation 1.82 spoke about regularly identifying gaps that need to be acted upon. I was just wondering if perhaps you could speak about what some of those gaps might be and what actions have been taken or what actions are planned to be taken in order to address this.

[Alister Smith]

Some of the gaps that we ourselves have found over the years are pretty similar to what the Auditor General has mentioned--a lack of performance information, in particular. One of the focal points of the new policy is to improve performance measures and performance information, which will go a long way to help in assessing the effectiveness of programs. That has certainly been a gap we've looked at.

We've looked at, through our management accountability framework assessments, the quality of evaluations, the governance of the function within departments, because we do want to ensure that evaluators have access to deputy ministers and can tell them the news about a particular program, even when the program managers may not be all that keen to have that news going to the deputy. So we've looked at areas like that within our MAF assessments, and we've been fairly candid with departments where we have seen gaps. So we have a mechanism that works very well in that way. We also have a community of practice that has an evaluation, with others, with pretty much ongoing dialogue on all nature of problems, but also best practices that departments can borrow from each other, including the ones Mr. Shugart mentioned on surveying clients after a program has been delivered.

[Earl Dreeshen, Conservative, Red Deer, AB]

Thank you.

[The Chair, Liberal, Shawn Murphy]

Monsieur Dion, pour cinq minutes.

[Stéphane Dion, Liberal, Saint-Laurent—Cartierville, QC]

Thank you Mr. Chairman.

Because we're coming to the end of our discussion, perhaps I can sum up. All your departments, including Treasury Board, will have to cut their expenditure by 5%. Is that correct? Is Treasury Board not subject to that 5% cut?

[Michelle d'Auray]

Yes, Mr. Chairman. The strategic review for the Treasury Board Secretariat is taking place this year.

[Stéphane Dion, Liberal, Saint-Laurent—Cartierville, QC]

That is also true for Environment Canada and Citizenship and Immigration Canada. One would hope that during such a difficult and necessary exercise one could depend on quality evaluation functions. Yet the report tells us the opposite.

I'll sum up the main conclusions. Only 5% to 13% of programs have been evaluated, and more than two-thirds of those evaluations lack adequate data in order to be effective. Yet this morning we are being told that since the Auditor General's report came out one year ago significant progress has been made. The report states that Treasury Board has made little progress in "developing tools to help departments address the longstanding problem of insufficient data", and, later on in the report, that it did not issue "adequate guidance or tools".

Do you still say this is not true?

[Michelle d'Auray]

We acted and we took note of the report's conclusions, but we also took note of the factors we identified. In November 2009, we provided departments with guides on performance measures. We will be finalizing that this year.

Therefore, yes, we took note of the recommendations and we took measures in order to ensure that data collection complied with the standards and met the requirements.

As the chairman requested, copies of that material will be provided to the committee.

[Stéphane Dion, Liberal, Saint-Laurent—Cartierville, QC]

Ms. d'Auray, you said that over the past year 15% of programs were evaluated. Therefore we're getting close to 20% of evaluations per year.

However, my concern is that that quantitative goal may be met at a cost in terms of quality, which is lacking according to the Auditor General's report. I would like to ask Mr. Shugart to react to paragraph 1.91, on page 32

page 32 in English, paragraph 1.91.

It states:

[...] Environment Canada estimated that it would have to double the complement of its evaluation unit over the next four years, or sacrifice evaluation depth in order to achieve full coverage.

Do you still agree with this paragraph?

[Ian Shugart]

That was an estimate. In the department there was a 100% increase in our capacity.

We did increase from about four evaluators in 2004-05 to where we have 12 or 12.5 FTEs in evaluations today. So in that period of time we more than doubled our evaluation capacity.

Going forward, the same level, generally speaking, to achieve 100% coverage over time probably is fair. With the increased supply of evaluators that we hope will be available we think that is achievable. But it is absolutely true that we will have to manage this carefully and we will have to focus our evaluation resources in those areas where the greatest value from evaluation can be achieved.

As I mentioned, we may be flexible in how we implement this policy. We may cover a broad area of programs in one evaluation rather than each individual program, which would be an efficient way of doing it. And if we make wise selections and have good evaluation plans, then we will be able to achieve 100% coverage in a very efficient way. I would look to the professionals in the evaluation branch to help me with evaluation plans that are efficient, as well as meeting the goal.

So it will be a challenge, but it is a necessity, and we will proceed as best we can.

[Stéphane Dion, Liberal, Saint-Laurent—Cartierville, QC]

May I ask Mr. Yeates to comment on the same topic?

[Neil Yeates]

Yes, certainly, Chair.

We are working to increase our FTEs, similar to Environment Canada. We had three back in 2004-05 and we're ramping up to 20. We think that is sufficient capacity for us.

Our current five-year evaluation plan provides 100% coverage for all our grants and contribution programs. So we have a little bit more to do beyond that, but we feel confident we will have the capacity to do the necessary work.

[The Chair, Liberal, Shawn Murphy]

Merci, Monsieur Dion.

Mr. Saxton, for five minutes.

[Andrew Saxton, Conservative, North Vancouver, BC]

Thank you, Mr. Chair.

My question is also for Mr. Yeates. In your opening remarks you stated:

With more and better studies, the results and conclusions of CIC evaluations are increasingly used by senior managers to inform program and policy decisions.

Can you elaborate on that, please?

[Neil Yeates]

Yes, I'd be pleased to, Chair.

The evaluation committee we have, which I chair and Elizabeth Ruddick supports, meets about monthly now. We review all the terms of reference for our forthcoming studies, and this is very important for us, because it makes sure that the right questions are being asked within an evaluation. Then we review the studies themselves and the management response we have from the particular program area. And we've had some very good discussions about what that means for us.

I can give you an example of that. We've just recently completed an evaluation of our language programming, which is provided throughout Canada. That's led us to rethink how we do some of our language assessment, both overseas for new immigrants but also the programming we have here in Canada. So it's an example where a specific program evaluation ended up having broader impacts in terms of our service delivery system.

[Andrew Saxton, Conservative, North Vancouver, BC]

Thank you.

My next question is for the Treasury Board secretary. Can you elaborate and expand on the importance of the management accountability framework?

[Michelle d'Auray]

The management accountability framework was established in 2003 in order to assess the department's management results overall, over a broad range of areas of management. Evaluation is one of currently 19 areas of management that we assess.

For example, we encourage departments to assess their capacity in terms of audit, in terms of financial management, and in terms of people management. We also look at the areas around performance management and we have an assessment also of the nature of the Treasury Board submissions that they send to us.

A lot of these areas, including evaluation, are what we consider core to the management functions and good performance in the area of management of departments and organizations. So we assess, we have performance indicators. Some of the assessment measures are what we would consider to be objective and some of the assessment measures are self-assessment by departments and organizations that they provide to us.

We report back to departments. In fact we have published on our website the results of the management assessments. We work with departments on establishing what would be the management priorities for the coming year.

That's where--in some instances, for example--the performance around evaluation has not necessarily been at the highest standard. We have occasionally identified where that is the case in a department where that is a priority for improvement for that department. That then becomes a management priority for that organization for a given year in order to focus their attention on improving the management capacity of a particular area, including, for example, evaluation.

[Andrew Saxton, Conservative, North Vancouver, BC]

Thank you very much.

Mr. Chair, I have no further questions.

[The Chair, Liberal, Shawn Murphy]

Thank you, Mr. Saxton.

I have one issue I want to clarify to you, Madame d'Auray, and that is the whole issue of departmental performance reports. It's long been a concern of this committee, and it's been a concern of the Auditor General, that the content and the way these reports are prepared and tabled in Parliament is of limited use and limited assistance to parliamentarians and others.

I guess when I read this report.... You look at performance and the first thing you would look at to determine performance is you would look at evaluations. You would think that the evaluations that have been done and the evaluations that will continue to be done would form a very large component of the departmental performance reports. They would either be produced in detail or they would certainly be summarized. I'm not sure if that's the case; I don't think it is. But can you tell the committee to what extent these evaluations that are being done now are used in the departmental performance reports that are tabled in Parliament now?

[Michelle d'Auray]

If I may, Mr. Chair, I would ask Mr. Smith to respond to that.

[Alister Smith]

Yes, Mr. Murphy.

The departmental performance reports do include performance measures, and they continue to improve. These performance measures or performance indicators are consistent with the types of measures used in evaluation. But the structure of these reports follows auditing standards. We're auditing performance against strategic outcomes for the programs at a fairly high level and at the levels of program activity.

In some respect here evaluation is dealing with programs at a lower level of granularity. So they're not inconsistent, it's just there's a continuum of information on performance from the program level up to the strategic outcomes. I understand you have a hearing on this.

[The Chair, Liberal, Shawn Murphy]

Yes, we're having a hearing.

I don't entirely agree with you.

Do you have any comment, Madam Auditor, on that point?

[Sheila Fraser]

Mr. Chair, that's not something we have looked at recently. We didn't include it in this audit.

[The Chair, Liberal, Shawn Murphy]

It is a timely issue.

As Mr. Smith pointed out, we are having a hearing. I believe it's a week from this Thursday. It will be an interesting hearing. We will be looking at the departmental performance reports of two departments, transport and industry, I believe.

You have one minute, Mr. Lee, sixty seconds.

[Derek Lee, Liberal, Scarborough—Rouge River, ON]

I have a very short question.

Has Treasury Board looked at the issue of transparency in this evolving field of evaluation--transparency in the sense of intra-departmental transparency of evaluation data as it's being collected, and when it's finally collected, the preparation of a report before it's made public? Because you've got the potential for sharing inter-departmentally as well as intra-departmentally with Parliament and under access to information.

I'm just wondering, as we develop this science of evaluation, if any thought has been given to the potential issues of transparency and access to various parties.

[The Chair, Liberal, Shawn Murphy]

Mr. Smith or Madame d'Auray.

[Michelle d'Auray]

I'll begin, and I'll turn to Mr. Smith for more information.

In terms of the transparency, departments are in fact encouraged to post the evaluation reports on their websites, and they do so fairly frequently. On the sharing of data, that normally happens in terms of horizontal evaluations, where that evaluation has to cover more than one program or a program covers more than one organization.

[The Chair, Liberal, Shawn Murphy]

It is getting close to eleven o'clock. On behalf of all committee members, I want to thank you very much for your appearance today. It has been a very valuable meeting.

Before we adjourn, I want to ask all the witnesses if they have any concluding remarks.

I'll start with you, Ms. Fraser.

[Sheila Fraser]

Thank you, Mr. Chair.

I would like to thank the committee members for their interest in this audit. As I mentioned, we believe, as do other witnesses today, that the evaluation function is a very important one, especially in a time of fiscal restraint.

I was pleased to hear about the actions that government has taken since the completion of our audit. I would certainly encourage the committee to perhaps follow up at some appropriate time in the future on progress that's being made.

[The Chair, Liberal, Shawn Murphy]

On that point, Madam Auditor, as you know, and for the purpose of everyone in the room, we do have action plans from everyone involved and we certainly will be following up very closely.

Madame d'Auray, any closing comments?

[Michelle d'Auray]

I would echo the Auditor General's comments about the importance of the function and note that we are on the right path to re-establishing and rebasing the function as a core element of the government's expenditure management system.

Thank you.

[The Chair, Liberal, Shawn Murphy]

Mr. Yeates, do you have any closing comments?

[Neil Yeates]

Mr. Chair, I would add that I think we've made a lot of progress in the past couple of years, and I'm very optimistic the committee will see significant progress in the years ahead.

[The Chair, Liberal, Shawn Murphy]

Mr. Shugart, do you have any comments?

[Ian Shugart]

Only to thank the committee, Mr. Chairman, for their courtesy and interest.

[The Chair, Liberal, Shawn Murphy]

Again, I want to thank you very much for all your hard work. We wish you all the best in your future endeavours.

Before I adjourn, I believe there's another committee meeting coming in immediately, and I want to urge members to vacate the room as soon as reasonably possible so the other committee can get started on time.

Thank you again.

The meeting is adjourned.