Essential Elements: Apply Rigor

At the Colorado Lab, we use data to support learning and sustained action among our partners. We view “rigor” as one part of what helps a study meet that goal, striving to do work that is simultaneously rigorous, relevant, and resonant.

While the term “rigor” is widely used in research and evaluation, it is rarely and inconsistently defined, leading to misunderstandings and differing viewpoints. Some associate rigor with randomized controlled trials (RCTs) as the “gold standard for research,” while others may view rigor as a construct that limits creativity. The Colorado Lab believes that rigor is essential and encompasses three elements:

Employs an appropriate design for the goals and questions. We align design decisions with the study goals, ensure that interpretations about cause and effect are well founded, and articulate the populations, settings, and circumstances to which results can be applied.

Is systematic and transparent. We take steps to ensure the accuracy and trustworthiness of the results. We document our decisions and provide enough detail so that others can understand why design or analytic choices were made.

Asks challenging questions. We ask questions for which the answers are not known in advance, question our expectations and biases, and look for system/root causes and alternative explanations.

Consistent with this definition, we recognize there is no such thing as objectivity. Each evaluator comes to a study with their own history, values, and experiences. As such, it is important that each of us recognize the ways in which our identities and perspectives inform our focus and approach. At each stage of a study, we strive to name our decisions and assumptions, and build in processes that hold us accountable and are transparent. We also acknowledge that data are not themselves objective or free of bias. This is because people are involved in all aspects of data generation, including deciding what data gets collected and from whom, how data are analyzed and interpreted, and where and how data are presented or shared.

The Colorado Lab applies rigor at every step of building evidence.

A good example of applying rigor during the early steps to building evidence is our work with Colorado’s Early Childhood Mental Health (ECMH) Consultation program. ECMH consultants partner with early childhood professionals, providing them with the support and training to care for their own well-being and to promote the healthy social-emotional development of the children they care for. The Colorado Lab is working with the ECMH Consultation program to surface assumptions about the relationship between program components and intended outcomes through a theory of change. We are clearly defining the program’s essential elements and creating strategies to understand the extent to which these elements are being implemented across diverse communities. Building this strong foundation supports program improvement as well as subsequent evaluation work to assess the program’s impact, allowing us to distinguish between a failure of implementation (the program not being fully implemented) and a failure of theory (the program being unlikely to produce the desired impact).

Likewise, rigor is critical to the later steps of building evidence. As noted earlier, certain kinds of study designs, such as RCTs, have come to be viewed as having the “most” value and legitimacy and being “rigorous by design.” However, quantitative and qualitative methods each have unique strengths and limitations, and one, the other, or both in combination should be employed only when appropriate based on the goals and questions at hand. It is both possible—and necessary—to ensure rigor in quantitative, qualitative, and mixed methods work. All designs require thoughtful consideration to ensure rigor.

Learn more about rigor and see examples of how we consider rigor across the research/evaluation process in this brief.

Next up in our blog series we move from Systems to Action, looking first at why we prioritize actionability and how we identify data-informed pathways to reach desired outcomes and set wheels in motion to achieve targeted goals.