Date Published: September 27, 2017
Publisher: The American Society of Tropical Medicine and Hygiene
Author(s): Christine L. Hershey, Achuyt Bhattarai, Lia S. Florey, Peter D. McElroy, Carrie F. Nielsen, Yazoume Yé, Erin Eckert, Ana Cláudia Franca-Koh, Estifanos Shargie, Ryuichi Komatsu, Paul Smithson, Julie Thwing, Jules Mihigo, Samantha Herrera, Cameron Taylor, Jui Shah, Eric Mouzin, Steven S. Yoon, S. René Salgado.
As funding for malaria control increased considerably over the past 10 years resulting in the expanded coverage of malaria control interventions, so did the need to measure the impact of these investments on malaria morbidity and mortality. Members of the Roll Back Malaria (RBM) Partnership undertook impact evaluations of malaria control programs at a time when there was little guidance in terms of the process for conducting an impact evaluation of a national-level malaria control program. The President’s Malaria Initiative (PMI), as a member of the RBM Partnership, has provided financial and technical support for impact evaluations in 13 countries to date. On the basis of these experiences, PMI and its partners have developed a streamlined process for conducting the evaluations with a set of lessons learned and recommendations. Chief among these are: to ensure country ownership and involvement in the evaluations; to engage stakeholders throughout the process; to coordinate evaluations among interested partners to avoid duplication of efforts; to tailor the evaluation to the particular country context; to develop a standard methodology for the evaluations and a streamlined process for completion within a reasonable time; and to develop tailored dissemination products on the evaluation for a broad range of stakeholders. These key lessons learned and resulting recommendations will guide future impact evaluations of malaria control programs and other health programs.
Over the past decade there has been an intensified effort in malaria control in malaria endemic countries from Ministries of Health (MoH) with support from international partners and funding agencies. Given the level of investment and effort in malaria control, members of the Roll Back Malaria (RBM) Partnership, including MoHs and their national malaria control programs (NMCPs), were interested in assessing the impact of their malaria control efforts. Information on the effectiveness of malaria control measures could impact future funding for malaria control and prevention and assist in prioritizing the use of those funds. This led the RBM Monitoring and Evaluation Reference Group (RBM MERG) to publish an initial methodology framework for conducting impact evaluations of malaria control programs in 2007,1 and to discuss key principles at an international multiagency workshop on impact evaluation in Tanzania in 2010. The President’s Malaria Initiative (PMI) has been working with host country governments, research institutions, and other national and international partners to conduct a series of impact evaluations in malaria endemic countries. To date, evaluations have been conducted in Angola, Malawi, Mali, Mozambique, Rwanda, Senegal, Tanzania (both for mainland Tanzania and Zanzibar) and Uganda, with evaluations underway in Democratic Republic of the Congo, Ethiopia, Kenya, and Liberia. The methodology framework for conducting an impact evaluation of national malaria control efforts is described in detail in the RBM MERG Impact Evaluation Framework2 and Yé and others in this supplement.3 This article describes the operational details for organizing and conducting the evaluations, the lessons learned, and recommendations for future impact evaluations.
The malaria impact evaluations conducted to date followed the same general process (Figure 1) with common, but in some cases unique, challenges (Table 1). The process included three phases: initiation, execution, and finalization of the evaluations. The initiation phase involved agreement between the NMCP and funding agencies to conduct an evaluation, identification of evaluation stakeholders, hiring of a local (and external) implementing partner, and holding an initial stakeholder meeting to introduce and get buy-in for the evaluation. The initiation phase also included identifying data sources, developing an analysis plan, obtaining institutional review board (IRB) approval/ethical clearance from relevant authorities, and accessing data. The second phase (execution) consisted of the data analysis, interpretation, and writing of the evaluation report. Stakeholders also reviewed and edited multiple drafts of the impact evaluation reports. During the finalization phase, the evaluation report was sent for review by all partners before finalizing and submitting for approval to the NMCP and PMI. In many cases, a stakeholder consultative meeting was held in country to review evaluation findings before submitting for approval. The process for conducting these evaluations took more than a year in all countries.
Outlined below are additional considerations when planning a malaria impact evaluation. Although not unique to impact evaluations, they are important to successfully completing these evaluations.
Currently, nine national-level malaria control impact evaluations have been conducted following the RBM-MERG evaluation methodology framework, with four more underway. Valuable lessons have been learned about the methods used in the evaluations2,3,11 and the process for conducting the evaluations. An evaluation methodology framework has been developed that can be adapted to the individual country context and a corresponding streamlined report format and operational framework as described here for conducting the evaluations have also been developed. The lessons learned from these evaluations are not unique to malaria and are applicable to other public health programs. The impact evaluations conducted thus far have relied on existing data in country. Looking forward, countries should use these evaluations to identify gaps in their data collection systems and design approaches to improve the data for monitoring the country’s malaria control efforts, with the benefit of improving the data available for future impact evaluations, including designing the evaluations prospectively. Discussions are needed to decide when follow-up impact evaluations should be conducted as many countries evaluated impact through 2010 or 2011. MoHs/NMCPs and funding agencies will need to continue to report on the impact of malaria control efforts and investments and this article and others in this supplement provide a framework for conducting impact evaluations in high burden countries. As malaria burden declines, new methods (and potentially new processes) will be needed to assess impact.