Thank you, Mr. Chair.
We thank you for this opportunity to present the results of an audit included in our November 2009 report on evaluating the effectiveness of programs in the federal government.
As you mentioned, I'm accompanied today by Neil Maxwell, Assistant Auditor General, and Tom Wileman, principal, who were responsible for this audit.
I would like to point out to the committee that the work for this audit was substantially completed on May 31 of 2009.
Effectiveness evaluation is an established tool that uses systematic research methods to assess the extent to which a program is achieving its objectives. Over the past 40 years, the federal government has made repeated efforts to establish the evaluation of effectiveness as an essential part of program management.
One of the most important benefits of effectiveness evaluation is to help departments and agencies improve their programs. Departments also need to be able to demonstrate to Parliament and taxpayers that they are delivering results for Canadians with the money entrusted to them. Sound information on program effectiveness is particularly important in light of the recent budgetary measures to contain administrative costs and review government operations.
In this audit, we examined how examination units in six departments identified and responded to increasing needs for effectiveness evaluation. We also looked at whether they had built the required capacity to respond to those needs. In addition, we looked at the oversight and support role of the Treasury Board of Canada Secretariat in monitoring and improving the program evaluation function in the government, particularly with respect to the evaluation of program effectiveness. The period covered by our audit was 2004 to 2009.
Overall, we found that the six departments were not sufficiently meeting the needs for effectiveness evaluation. Annual coverage of departmental expenses by evaluation was low, ranging from 5% to 13% across the six departments.
In addition, the effective rate of coverage was even less because many of the evaluations we reviewed did not adequately evaluate effectiveness. Of the 23 evaluation reports we reviewed, 17 noted that the analysis was hampered by inadequate data, which limited the evaluation of program effectiveness. This lack of performance data is a longstanding problem as noted in my office's earlier audits of the evaluation function.
With respect to the six departments' capacity to meet the needs for effectiveness evaluation, department officials told us that they have not been able to hire enough experienced evaluation staff and have used contractors extensively to meet requirements.
Of the audited departments, Environment Canada had internal processes to systematically identify areas for improvement in effectiveness evaluation. For example, Environment Canada solicits client feedback through post-evaluation surveys. Such processes ensure that departments are following the management cycle for continuous improvement.
The situation with respect to program evaluation in the federal government is not unlike that of internal audit before the policy on internal audit was implemented. Lessons learned from the government's recent strengthening of internal audit could be applied usefully to program evaluation.
We believe that implementation of the new requirement to evaluate all direct programming spending faces serious challenges. Earlier requirements for full coverage were never met, and current legal requirements for effectiveness evaluation of all grant and contribution programs have been difficult to meet. Department officials told us that they have concerns about their capacity to implement the expanded coverage requirement in the new evaluation program.
In our view, it will be important for the secretariat and departments to carry out effectiveness evaluation of programs that are susceptible to significant change because of shifting priorities and circumstances. These are programs where evaluations of the relevance, impact, and achievement objectives can be put to best use. During the transition to full coverage, these programs may present the biggest opportunities for effectiveness evaluation. Taken together, the increasing needs for effectiveness evaluation and the department's limited capacity to meet those needs pose a serious challenge to the function. Concerted efforts by both the secretariat and departments will be needed to meet this challenge.
The secretariat and the audited departments have agreed with all of our recommendations. In several cases, they have made commitments for remedial action in their responses. In line with the committee's request for action plans and timelines, the committee may wish to explore the progress made to date in addressing the issues raised in the chapter.
Mr. Chair, this concludes my opening remarks and we would be pleased to answer the committee's questions.