Policymakers and practitioners can produce better results for Minnesotans when they prioritize evidence-based services and rigorously evaluate services that do not yet have evidence of effectiveness. Evidence-based services have been found to be effective by at least one study that analyzes how participants did in achieving desired outcomes relative to comparable nonparticipants. We call these high-quality studies “impact evaluations.” Practices that have not yet been documented to be evidence-based may nevertheless be effective and can become evidence-based after having an impact evaluation conducted. Rigorous research, combined with context from local practitioners, gives policymakers access to valuable information about effectiveness.
While there are many types of evaluations, the Results Team rates programs and services in Minnesota using impact evaluations only. This type of evaluation is designed to identify cause and effect relationships between the program/service and desired outcomes.
Qualifying evaluations are impact evaluations that use either a randomized controlled trial (RCT) design or quasi-experimental design (QED) to rigorously assess effectiveness of a program or service on desired outcomes. Both RCTs and QEDs use an evaluation design that includes a treatment and a treatment as usual group.
The Results Team leverages existing impact evaluations from national clearinghouses. We match programs and services delivered in Minnesota to ones that closely resemble programs and services previously evaluated and featured in a national clearinghouse or meta-analysis (with respect to the nature, length, frequency, and target population). In some instances, programs or services in Minnesota have been rigorously evaluated with an impact evaluation, in which case, we use that study to assign a rating. “Evidence-based” practices are those that are rated “Proven Effective” or “Promising”.
Depending on the number of studies, research methods, and findings, we assign evidence ratings to Minnesota programs. The guidelines we use are described below. The Minnesota Inventory lists all the programs we have rated.
| Evidence Ratings and definitions | |
| Proven Effective | A Proven Effective service or practice offers a high level of research on effectiveness for at least one outcome of interest. This is determined through multiple qualifying evaluations outside of Minnesota or one or more qualifying local evaluation. Qualifying evaluations use rigorously implemented experimental or quasi-experimental designs. |
| Promising | A Promising service or practice has some research demonstrating effectiveness for at least one outcome of interest. This may be a single qualifying evaluation that is not contradicted by other such studies but does not meet the full criteria for the Proven Effective designation. Qualifying evaluations use rigorously implemented experimental or quasi-experimental designs. |
| Theory Based | A Theory Based service or practice has either no research on effectiveness or research designs that do not meet the above standards. These services and practices may have a well-constructed logic model or theory of change. This ranking is neutral. Services may move up to Promising or Proven Effective after research reveals their causal impact on measured outcomes. |
| Mixed Effects | A Mixed Effects service or practice offers a high level of research on the effectiveness of multiple outcomes. However, the outcomes have contradictory effects. This is determined through multiple qualifying studies outside of Minnesota or one or more qualifying local evaluation. Qualifying evaluations use rigorously implemented experimental or quasi-experimental designs. |
| No Effect | A service or practice rated No Effect has no impact on the measured outcome or outcomes of interest. Qualifying evaluations use rigorously implemented experimental or quasi-experimental designs. |
| Proven Harmful | A Proven Harmful service or practice offers a high level of research that shows program participation adversely affects outcomes of interest. This is determined through multiple qualifying evaluations outside of Minnesota or one or more qualifying local evaluation. Qualifying evaluations use rigorously implemented experimental or quasi-experimental designs. |
Visit our Finding and Using Evidence page to learn more about where you can find reputable information on programs from a variety of policy areas.
