Home \ Methods \ The Basics of Impact Assessment

What is ex-post impact assessment?

Impact assessment is a type of evaluation that is intended to determine the consequences of an intervention, in terms of outcomes of interest. This analysis can either be ex-ante, conducted before the intervention has been initiated and/or outcomes have been produced, or ex-post, which measures outcomes that have actually resulted from the intervention to date. This website primarily concerns ex-post impact assessment (epIA) of research activities of the CGIAR.

Ex-post impact assessment is only one component of a comprehensive evaluation package, which also includes ex-ante assessment, programme reviews, performance monitoring and evaluation and process evaluations, among others. In the case of research, process evaluations can be further composed of several forms of early-acceptance and adoption studies, which seek to provide feedback to the research process as it proceeds. Many terms have been coined for these process evaluations, including impact monitoring, follow the technology analysis and monitoring and evaluation. In comparison, ex-post impact assessment is “intended to determine more broadly whether the program had the desired effects on individuals, households and institutions and whether those effects are attributable to the program intervention” (Baker, 2000).

Figure 1 . Evaluation timeline for research, with a number of possible processes of feedback provision represented by black arrows.

The relationship between these different forms of evaluation is indicated in figure 1. One factor that principally distinguishes ex-post impact assessment from other forms of evaluation is that it is a summative form of evaluation, whereas all the other types are formative. Formative evaluation is primarily concerned with providing information during the implementation phase on how to improve the research program. Summative evaluation is concerned with the programme’s effectiveness, value or impact and is conducted after programme completion for the benefit of an external audience (MacKay and Horton, 2003).

Who does ex-post impact assessment in the CGIAR?

Ex-post impact assessments are conducted by impact assessment specialists in the International Agricultural Research Centres (IARCs), often in collaboration with national and international experts from partner institutions. Each IARC has an impact assessment focal point who is often positioned within the Office of Director-General or within the Economics/Social Science unit.

Impact assessment is supported at the System-level by the Science Council’s Standing Panel on Impact Assessment (SPIA), which has a mandate to:

• provide CGIAR Members with timely, objective and credible information on the impacts at the System level of past CGIAR outputs in terms of the CGIAR goals;

• provide support to and complement the Centres in their ex post impact assessment activities; (this includes facilitating inter-Center impact assessment efforts and providing a forum for exchange of experience from impact studies); and,

• provide feedback to CGIAR priority setting, and create synergies by developing links to ex ante assessment and overall planning, monitoring and evaluation functions in the CGIAR.

SPIA’s predecessor, the Impact Assessment and Evaluation Group (IAEG), was initiated in 1995, and reorganised into SPIA in 2000. Thus, System-level integration of impact assessment is relatively new in the CGIAR, as no system-level IA entity existed prior to IAEG. As a result, there is significant diversity among the Centres, in terms of IA approaches and methods.

Why conduct ex-post impact assessment?

The CGIAR represents only a small share of the world’s agricultural research portfolio, and it is crucial that these scarce resources are invested efficiently and effectively. In the absence of market forces, empirical analysis is needed to ensure that funds invested in the CGIAR’s agricultural research activities have been been effective in creating progress towards CGIAR goals. This information is needed not only to show that CGIAR centres have the capacity to help alleviate poverty, but also is essential to directing limited resources to those research areas that can most effectively produce benefits for the poor.

Accordingly, ex-post impact assessment of agricultural research has two commonly cited uses (Alston and Pardey, 1996; Özgediz, 1995; Maredia et al., 2000):

• Accountability : to provide empirical evidence of the effectiveness of past investment for driving outcomes of interest and validate the relevance of overall strategies pursued.

• Feedback: to help inform allocation decisions by providing insights as to the comparative effectiveness of different investment options, and to generate lessons that can improve the implementation of ongoing and future research.

Of these, accountability has typically received greater emphasis, as many important stakeholders demand strong evidence of impact to justify continued investment. Indeed, in the CGIAR, ex-post impact assessment first became of high-priority at the behest of donors who requested the comprehensive Impact Study of the mid-1980s. This emphasis was renewed and invigorated during the funding scarcity of the mid-1990s.

The feedback function of impact assessment is important, but considerably more challenging to fulfil, as it is affected by a number of methodological limitations. For outcomes of interest to become evident at wide scale and ripe for ex-post impact assessment, long and unpredictable lag times often must pass. As these lags transpire, research agendas, priorities, and even goals may shift as new research techniques become available, and comparative advantage evolves. This may limit the relevance of findings to current research decisions. In addition, impact assessment is primarily performed for a small subset of research activities with the most easily documentable impacts, and this makes comparative analysis across research topics difficult. These challenges need to be addressed for the feedback role of impact assessment to be more effectively realised in the future.

Ex-post impact assessment methods and disciplines

Ex-post impact assessment is an applied social-science activity, and has been serviced by the field of economics, as economic approaches allow the use of broadly acceptable quantifiable measures and values that can be directly related to costs. For this reason, much of the content of this site is focused on economic approaches. However, this is by no means the only discipline that deals with impact assessment, as other social science and interdisciplinary approaches are common, especially outside of the agricultural research arena. In fact, impact assessment is characterised by a diverse array of methods and approaches, which differ considerably according to the scale, scope and type of analysis. However, many of the challenges and methodological considerations outlined in this site can be generalised across most of the various techniques applied.

References

Alston, J.M., G.W. Norton & P.G. Pardey. 1996. Science under scarcity: Principles and practice for agricultural research evaluation and priority setting. Ithaca, NY: Cornell University Press.

Baker, J. 2000. Evaluating the Impact of Development Projects on Poverty: A Handbook for Practitioners. Washington, DC: World Bank.

Mackay, R. and D. Horton. 2003. Expanding the use of impact assessment and evaluation in agricultural research and development. Agricultural Systems. 78(2): 143-165.

Maredia, M., D. Byerlee and J.R. Anderson. 2000. Ex Post Evaluation of Economic Impacts of Agricultural Research Programs: A Tour of Good Practice. In The Future of Impact Assessment in the CGIAR: Needs, Constraints and Options. Rome, Italy: CGIAR Technical Advisory Committee Secretariat, FAO.

Özgediz, S. 1995. Strengthening Evaluation in the CGIAR: Needs and Options. CGIAR Task Force on Impact Assessment. Washington, DC: CGIAR Secretariat.

Continue on to key considerations for practitioners