About Impact Evaluation
What is “Impact Evaluation”
An impact evaluation provides quantitative evidence about the magnitude and direction of effects produced by an intervention or program. Impact evaluations also seek to identify whether the intervention is the cause of the effects observed by comparing program recipients to like individuals who did not receive the program (a so called "business-as-usual" comparison group). If program recipients and business-as-usual individuals are reasonably similar in terms of demographics, opportunity, and risk, it is possible to conclude from the impact evaluation that the only permissible cause of observed differences between these groups is the program in question.
A Helpful Example
Suppose we are interested in the effect of a hypothetical opioid treatment program that we will call HealthU. If we evaluated the program by simply comparing the percentage of sober participants before the program started to the percentage of sober participants after the program finished, it may look like the program had a large impact (as shown in Figure 1 to the right).
However, this 20% change may be misleading. The participants of the program may have been impacted by other factors in the community. Many of them might have gotten sober with family or community support, even without HealthU. We call this the “business-as-usual” or comparison case.
Figures 2 shows two situations with the same blue line as Figure 1, but also with a green line denoting a hypothetical comparison of what would have happened had HealthU not been available (the business-as-usual case). This net impact of the treatment and business-as-usual case changes how we think about the program’s success. The left graph of Figure 2 indicates that the net impact on sobriety is smaller than originally assumed, though still positive. The graph on the right, however, shows participants would have been better off without the program.
Timing (when to do an evaluation)
The best time to begin an evaluation is before a program or intervention has been enacted. In certain cases, program rollout or enrollment can be equitably randomized such that the same number and demographics of individuals are served. When program or intervention receipt is random, a well-defined business-as-usual group is implicitly created allowing for the most reliable measure of program impacts. For this reason, these randomized control trials (RCTs) are the gold standard in terms of causal impact evaluation. When impact evaluation is made a part of the program administration process, specific implementation plans can be used to generate high-quality evidence without significantly impacting the population served.
While RCTs are an ideal, in many cases it is not feasible, equitable, or practical to design an impact evaluation prospectively. In these cases, identification strategies are employed to use information about individuals who participated in a program or received an intervention to identify appropriate business-as-usual groups that are reasonably similar to those program recipients. There are many different ways to identify business-as-usual groups, and the choice of identification strategy depends critically on the program model, intended population, and intervention rollout.