The Colorado Steps to Building Evidence model, depicted in the image below, has been adopted by the Governor’s Office of State Planning and Budgeting and the Legislature’s Joint Budget Committee for use when considering budget requests. This model is essential in describing how the Colorado Lab does business. We meet government partners where they are on this continuum and support them in moving to the next level.
There’s a lot of interest in randomized control trials (RCTs), which are the gold standard for determining the causal impact of a program. While RCTs may be appropriate for established programs, new or innovative programs in particular need to establish themselves starting with Step 1 on the Colorado Steps to Building Evidence model.
Note. SB21-284 includes four levels of evidence: (1) Opinion-based, not included in the steps to building evidence; (2) Theory-informed work, a critical first step in moving a practice along the continuum, aligns with Steps 1, 2, and 3; (3) Evidence-informed, aligns with Step 4; and (4) Proven-practice, aligns with Step 5.
The first step in building evidence for a program is clearly and explicitly defining the essential program elements, understanding the logic behind why those elements are expected to lead to change in the desired outcomes, and formalizing program expectations and requirements via a manual. This is particularly important when providers are decentralized and diverse. Without clarity around the program itself, along with processes to determine adherence to the essential program elements, there is no coherent program to build evidence about.
The second step is to determine that providers are consistently and effectively delivering the program. A fidelity of implementation (FOI) rubric is an important tool for developing and tracking measures that are consistent with the essential elements of the program identified in the logic model. Implementation science provides best practices around how to engage in continuous quality improvement while collecting quality data.
The third step in building evidence is assessing program outcomes, but without a control group. Most programs take this step by monitoring trends in desired outcomes over time as part of a performance management process or by conducting a formal pre-post assessment in desired outcomes.
Steps 4 and 5
The fourth and fifth steps in building evidence establish causal evidence of a program’s impact by comparing the outcomes of those who were treated with the outcomes of a control group. The most reliable way to do this is to conduct an RCT like those used in vaccine and drug trials. Formal randomization is required to reach Step 5. There are other “quasi-experimental” methods, however, that mimic randomization by creating a control group using statistical approaches. These can often be accomplished using administrative data and take a program to Step 4.
To generate actionable information, Steps 1 through 3 must all be addressed in order. Leaping to Steps 4 or 5 without laying this foundation can lead to wasted money and no results, or even worse, misleading results. As such, rather than focusing only on those questions that can be answered using a preferred method, like an RCT, the Colorado Lab identifies the most appropriate methods to answer questions posed by government leaders. We meet decision-makers where they are to move programs along the evidence continuum, regardless of where they start. It is through this unique approach that we generate insights that can be used to improve policies, practices, and ultimately the lives of Coloradans.