Home \ Methods \ Key Considerations for Practitioners

Impact assessment practitioners often struggle with three inter-related and difficult challenges – establishing a viable counterfactual, attributing the impact of a particular research output, and coping with long and unpredictable lag times. This section discusses these challenges and provides links to more detailed sources on these topics.

The Counterfactual

The counterfactual, or the forecasted course of events that would have taken place in the absence of the research output assessed (i.e. the “without” scenario), is the analytical core of an impact assessment. Every epIA is premised on this element, as the “impact” is derived from the difference between observed events and the counterfactual. If the counterfactual is unrealistic, the results of an impact assessment are of little credibility.

Constructing a realistic and accurate counterfactual is a far from simple task. Agriculture is a dynamic sector that is influenced by a multitude of exogenous factors, including government policies, conflicts, resource changes, social changes, and climate dynamics, in addition to the effects of technical change. Technical change itself is the product of many innovations, and the contribution of any single one of these is difficult to isolate. Each innovation is the product of collaborative efforts among scientists and institutions, which are also difficult to attribute. Among these many drivers of change, it is a considerable challenge to determine what the course of events would be if a single research contribution were removed.

In the case of international-public-goods research outputs, there is no way to isolate a true control sample for comparison with a “treatment” group, as the product of research, information, can freely travel across borders and populations. Thus, those who adopt or benefit from an innovation are likely to differ with respect to key factors even before adoption takes place. As a result, “experimental controls” are virtually impossible and “quasi-experimental controls” must suffice.

The “before” scenario cannot necessarily be assumed as an accurate counterfactual to the “after” scenario, due to the fact that the context for agricultural production and resource management is constantly in flux. Thus, in the absence of research interventions, measures of welfare may be decreasing or increasing, and research benefits may even accrue through reductions in welfare losses over time. As a result, care must be taken to ensure that before-after (“reflexive”) comparisons accurately represent the course of events without the assessed research.

A more effective approach for the establishment of “quasi–experimental controls” may be the “double difference” approach, if there is no significant interaction between the adopter/beneficiary group and the non-adopter/non-beneficiary control population, and the groups are under reasonably similar conditions. This approach compares the relative changes in metrics of interest over time between the two populations to establish how trends have been influenced by an intervention.

Alternatively, “statistical controls” may be established via econometric methods. Using these methods, the influence of alternative drivers of change can be tested, quantified, and compared with the effects of research outputs. However, care needs to be taken in the establishment of relationships among the factors, so as to ensure that these methods are not susceptible to spurious correlations.

For a more detailed discussion of counterfactual development, please read:

Ezemenari, K., K. Subbarao, and A. Rudquist. Impact evaluation : a note on concepts and methods. Washington, DC, USA: Poverty Reduction and Economic Management Network, World Bank. 33 p.

For a more detailed discussion of the plausibility of counterfactual assumptions embedded in past economic studies of research impacts, please read:

Salter, A.J. and B.R. Martin. 2001. The economic benefits of publicly funded basic research: a critical review. Research Policy. 30: 509–532

Attribution

Establishment of the counterfactual relative to a specific programme is the same as “attribution,” or the identification of the specific causal pathway from the specific actions of a particular institution, relative to other drivers of change. The counterfactual at this level should identify the “next best” technologies or policies that would have been developed and adopted without the assessed research programme, and should analyse how farmers would adjust their practices so as to make best use of the tools consequently available. The timeframe of the development of these alternative technologies should also be established, as impact may be derived from faster delivery of similar outputs to those that would happen in the counterfactual conditions.

During attribution, the issue of “fungibility,” or whether an alternative source, such as the private sector, would have produced a similar research product to that of the assessed public programme, becomes important. If the private sector has been displaced by a public-sector programme with similar research products, the impact at the level of the specific organization may be little, even though the impact relative to a no-research counterfactual is considerable. In this case, the latter scenario is clearly not realistic, and hence is not a plausible basis for claiming impact. For those cases where public sector research is providing outputs that compete with those of the private sector, displacement effects need to be assessed in the counterfactual. Similarly, if the CGIAR has merely substituted for something that a NARS organization would have done in the absence of the CGIAR, then that needs to be brought out in the counterfactual assessment.

At the same time, it is clearly not always feasible or desirable to attribute results to the actions of partners in collaborative research efforts with complementary products. In many cases, the actions of each partner in isolation would not produce an adoptable output without the contributions of the other. Any attempt to “partition” the credit among these actors would hence be ad hoc and would risk offending the partners involved. In these cases, the only viable solution is to consider the collaborative efforts as a single programme and adopt joint attribution.

Identifying the use and application of agricultural and related research outputs may often be less than simple, especially in the case of research programmes that do not directly produce finalised tools or improved physical inputs. Much, if not most, agricultural research results in information embedded in documents, management recommendations, or policies. In such cases uptake is not a binary decision, and may not necessarily lead to full adoption or implementation. Furthermore, there is no distinct empirical marker of use, and other drivers of change may produce influence that is difficult to distinguish from that of the research output. For these forms of research, interview and case study approaches will often be necessitated to identify influence and trace impact from the new information, as opposed to other factors causing change.

For an in-depth discussion of various means of attributing research, please read:

Alston, J.M. and P.G. Pardey. 2001. Attribution and other problems in assessing the returns to agricultural R&D. Agricultural Economics. 25: 141-152.

For empirical examples of how impact can vary according to assumptions regarding attribution, please read :

Pardey, P.G., J.M. Alston, C. Chan-Kang, E.C. Magalhães, and S.A. Vosti. 2004. International and Institutional R&D Spillovers: Attribution of Benefits Among Sources for Brazil’s New Crop Varieties. Staff Paper P04-3. Minneapolis, MN, USA: University of Minnesota, Department of Applied Economics. 40 p.

Lag times

Research is a cumulative, evolutionary process, in which each new finding is partially a product of all previous findings that laid the foundation for the new discovery. Furthermore, each new discovery, if it leads to a “successful” innovation will take a long and uncertain number of years to be applied broadly. Consequently, care must be taken in the temporal attribution of research efforts, as research completed long ago could be partially credited for today’s discoveries. Even so, in most studies, previous research investments are taken as sunk costs, and thus rates of return are calculated for the marginal investment of the new research. This may be reasonable if the implicit counterfactual assumption of no alternate provision of the output is valid.

This issue of selecting a proper “lag time” and structure has been debated extensively in the impact assessment literature. Metrics of the economic efficiency of research, such as internal rates of return, can very widely according to assumptions regarding lags.

Lag times also present challenges for the timing of ex-post impact assessments, as it may take decades or more before research products are widely adopted and produce widespread benefits. As a result, it may be difficult to relate epIA results back to current activities, as research approaches, topics, and methods may shift as these lags transpire. While many epIA studies do make some attempt to project benefits into the future, lags may present difficulties for these projections

For an in-depth discussion of lag times and structures, please read:

Alston, J. M., B. J. Craig and P. G. Pardey. 1998. Dynamics in the creation and depreciation of knowledge, and the returns to agricultural research. EPTD Discussion Paper No. 35. Washington, DC: International Food Policy Research Institute.

Common measurement errors

While many ex-post impact assessments conducted within the CGIAR and in the development community at large have shown a great deal of rigour, certain problems tend to recur in a number of studies. This section discusses a few of these.

Poor quality data

Agricultural statistics are often difficult to reliably obtain, especially at wide scale, if the agricultural product of concern is not marketed through formal channels. Existing sources may often be of questionable veracity, and often must be triangulated, or assessed against other sources of information regarding production. Certain agricultural statistics may not even be based on field data, as population is occasionally used as a proxy basis for the calculation of certain production statistics. If such is the case, the impact of technological change cannot be distilled from the dataset, and an alternate data source must be obtained.

It is also imperative that the data used for calculating productivity shifts or benefits can be reasonably extrapolated over the scale assessed. Each assessment should explicitly attempt to identify the constraints to inferring broad replication of results, especially if these data come only from a small number of trials or villages.

For more detailed information on data collection for ex-post impact assessment, please read:

CIMMYT. 1993. The adoption of agricultural technology: A guide for survey design. Mexico , D.F.: CIMMYT. (obtain print copy from CIMMYT here)

Insufficient consideration of externalities

While many technologies have shown significant overall economic benefits via consumer price effects, perceived associated negative externalities have often limited confidence in these results among certain audiences. Rarely do ex-post impact assessments attempt to capture external effects in quantitative terms. As a result, certain estimates may be biased by the fact that associated environmental costs (or benefits) are ignored, or that government expenditures associated with agricultural production levels (such as subsidies) are excluded.

For an overview of methods by which certain externalities can be incorporated in epIA, please read:

Lubulwa, G. & J Davis. 1996. Inclusion of environmental and human health impacts in agricultural research evaluations: review and some recent evaluations. Working Papers IAP-WP13. Canberra, Australia: Australian Centre for International Agricultural Research. 13 p.

Unrealistic or unspecified counterfactuals

Ex-post impact assessment studies do not always sufficiently account for the dynamic nature of events in the absence of the assessed output or programme. If one technical solution is not offered, it is unrealistic to expect that farmers will statically continue to pursue production methods that do not embed alternative sources of available technology. It has been long recognised that farmers dynamically respond to available production possibilities, and adapt so as to remain technically efficient given their production frontier. Counterfactuals must take account of this and attempt to capture true “next best” options for farmers, including the adoption of alternative innovations that would still produced by other institutions in the absence of the assessed research. To do so, counterfactuals should always be explicit and empirically-derived.

For a critical review of the counterfactual methods employed in impact evaluations conducted by the World Bank Operations Evaluation Department please see:

Kapoor, A.G. 2002. Review of Impact Evaluation Methodologies Used By The Operations Evaluation Department Over Past 25 Years. Operations Evaluation Department Working Paper. Washington, DC, USA: World Bank. 20 p.

Nonparametric calculation of the Malmquist TFP index without accounting for contemporaneous technical change

In recent years a large number of studies analysing agricultural total factor productivity (TFP) with nonparametric variants of the Malmquist TFP index began to show rapid agricultural productivity declines across the developing world, indicating technological regress in the agricultural sector. Although these studies did not intend to primarily analyse the impact of agricultural research, these findings cast doubt on epIAs that demonstrated productivity growth as a result of agricultural research investments. However, these Malmquist TFP index results were later shown to be a product of improperly selected points of reference for technological change (“the technological frontier”). These prior studies assumed that all technological progress essentially affected all production possibilities equally, whereas the reality of contemporaneous technologies means than a new production technique is likely to move the production possibilities frontier in one direction. With this error corrected, most developing countries display positive TFP growth.

For further discussion of the calculation errors in previous Malmquist TFP index studies, please read :

Nin, A., C. Arndt, and P.V. Preckel. 2003. Is agricultural productivity in developing countries really shrinking? New evidence using a modified nonparametric approach. Journal of Development Economics. 71(2): 395-415.

Continue on to key methodology references