In examining the role of evaluators in the face of crises such as the global COVID-19 pandemic, what themes emerge? What were the challenges, successes and failures? These topics are assessed in the new publication, Policy Evaluation in the Era of COVID-19 (Eliadis, Naidoo & Rist, eds., Routledge, 2023) and the corresponding webinar hosted by the Max Bell School of Public Policy. The webinar took place at the occasion of the open access publication, thanks to the UN’s Independent Office of Evaluation of the International Fund for Agricultural Development.
The webinar was moderated by Leslie Fierro, the Sydney Duder Professor of Evaluation at Max Bell, with presentations from academic experts and evaluation practitioners who offered diverse perspectives on recalibrating evaluation strategies for future global challenges.
Meta-Analysis of Evaluation Key Trends and Learnings during COVID-19 – Mariel Aramburu
Mariel Aramburu MPP’21, the Partnerships Officer of the United Nations World Food Programme, offered a comprehensive meta-analysis of key trends and lessons in policy evaluation during the COVID-19 pandemic, focusing mainly on grey literature that emerged in 2020-2021. Articles, blogs and analyses during this period highlighted the fact that top-of-mind issues for evaluators during this period focused on the evolution of evaluation methodologies, ethical considerations—especially in relation to remote data collection— leveraging technology for data collection, and the imperative of prioritizing marginalized voices. Aramburu concluded by posing reflective questions on integrating teachings into evaluation practices and shared ongoing evaluation efforts by the COVID-19 Global Evaluation Coalition on the international response to the pandemic, which is set to conclude in 2024.
Implications for Evaluation: What We Learn from the UN and Country COVID-19 Response Plans and Reflecting on Future Scenarios – Indran A. Naidoo
Indran A. Naidoo, Director of the Independent Office of Evaluation of the International Fund for Agricultural Development, emphasized the significant role of joint plans between governments and the UN system in mitigating risks and strategizing crisis recovery. "We need to assess the impact of the crisis and also to redesign and repurpose policies and strategies to build back better." He critiqued the evaluation sector's responsiveness during the crisis, questioning the absence of formal evaluation communities and dedicated units in informing social-economic response plans. Naidoo underscored the need for a transformative shift in evaluation practices, urging for agility, multi-disciplinary approaches, and a reorientation towards immediate, solution-oriented evaluations using diverse data sources. He urged evaluators to adapt to new realities and recognized that evaluators no longer have the exclusive space in making program judgments. He marked a crucial need for them to reassess their relevance in addressing pressing societal challenges.
COVID Crisis: Time to Recalibrate Evaluation – Steve Montague
Steve Montague, Partner at Performance Management Network Inc., underlined the need for a departure from conventional evaluation practices, saying, "There's no more ‘business as usual’ for evaluators and evaluation functions." Montague presented three cases from the Canadian context that highlight promising emerging evaluation practices and stressed the importance of recalibration in timing, collaborative linkages with audit functions, and the development of leading indicators for predicting failure. His insights from these cases underscored a shift toward real-time evaluations and the significance of leadership, collaboration, and adaptable approaches in redefining the evaluative process. Montague called for a revisioning of evaluation as a thinking approach, advocating for rigorous causation analysis and contextualized behavioral systems over traditional research designs.
The Unbearable Lightness of Rights: Evaluation, Ethics and COVID-19 Responses – Pearl Eliadis
Pearl Eliadis, human rights lawyer and Associate Professor (professional) at McGill University, spoke to the inadequate integration of human rights within the evaluation framework. She emphasized the pressing need for a reevaluation of the role of human rights, highlighting the inadequacy of evaluation standards that mention human rights during and after the crisis, and proposed prioritizing human rights as a fundamental aspect in the evaluation ethical hierarchy. Drawing from various ethical frameworks, including international standards and scholarly works, Human rights standards are not situated as legal obligations or give priority in ethical frameworks. Given the disproportionate impact of the pandemic on marginalized groups, and various forms of systemic discrimination, she pointed out that evaluators are uniquely well positioned to develop the empirical data to support or refute the proportionality and necessity of emergency measures that detract from basic rights or are discriminatory.
The Impact of the COVID-19 Pandemic on the Effective Use of Evaluation in Supporting the Sustainable Development Goals – Robert E. Lahey and Dorothy Lucks
Robert E. Lahey and Dorothy Lucks explored the pandemic's effect on evaluation practice in supporting the implementation, management, and reporting of the Sustainable Development Goals. Lahey highlighted the complexity of the 17 SDGs and their 169 targets. Robert Lahey emphasized the crucial role of evaluation in aiding decision-makers in assessing progress, but also noted that, "Progress against the SDG targets stalled, and, in some cases, it worsened," Lahey said. Lucks noted the emerging lessons pointing to the potential for evaluation to assume a stronger role post-pandemic. Lahey and Lucks stressed the importance adapting evaluation tools and training to effectively support the SDGs and the need for evaluators to be facilitators, analyzing outcomes for disadvantaged groups, and engaging stakeholders early in the evaluation process. Their insights highlighted the challenges and opportunities that emerged from the pandemic's impact on evaluation and SDG support.
Knowledge Production in a Pandemic: Supporting Accountability at Pace: United Kingdom and Canada – Maria Barrados and Jeremy Lonsdale
Jeremy Lonsdale, former Director of the UK National Audit Office, presented the authors’ chapter on how the National Audit Office and the Office of the Auditor General of Canada sought to generate valuable information amidst the COVID-19 crisis. Despite considerable hurdles in data collection, site inspections, and interviews that rendered conventional audit practices impossible, Lonsdale emphasized the especially critical role of audits in scrutinizing public expenditure, especially during a time of unprecedented spending and emergency response. He underscored the necessity of audits, even in challenging times, and highlighted innovations like the "COVID cost tracker" and varied report types that emerged during the crisis, showcasing the adaptability and resilience of these audit institutions.
Afterword – Ray C. Rist
Ray Rist wrapped up the event with five observations. First, the shocking global death toll of COVID-19 underscores the challenge posed by incomplete data. As well, the ongoing nature of the crisis militates against premature assumptions about its resolution. Second, Rist pointed out the pandemic's influence on government roles, signaling a shift toward increased government intervention compared to previous decades. Third, he called on evaluators to recall the societal flaws exposed by the pandemic, particularly in terms of inequality and the neglect of vulnerable groups. Fourth, he addressed the impact of misinformation's impact on vaccination rates, he stressed the dire consequences of false beliefs about vaccines. Finally, he emphasized the impossibility of fully reverting to pre-pandemic normalcy and urged the audience to remain vigilant and prepared for future crises.
These discussions are drawn from Policy Evaluation in the Era of COVID-19, which is now available open access. Watch these presentations in their entirety here.
