Our Methods

Evidence Rating: Search, Selection, and Assessment

We begin with a broad orientation search to define each strategy and identify appropriate search terms. We then conduct targeted literature searches, focusing first on systematic reviews and peer reviewed studies, then on selected sources of grey literature and the findings of relevant, reputable organizations that assess policy and program effectiveness (rating organizations). All searches are conducted electronically. The sources we search vary by health factor and strategy.

Retrieved articles are screened by date, relevance to topic of interest, applicability of findings, study type, and impartiality of author(s). We retain the most relevant, recent, rigorous reviews and studies for consideration in evidence rating. Ratings are assigned based on two analysts’ assessments of the strength of the overall body of evidence (e.g., type, quality, number of studies, consistency of findings, etc.) as it pertains to specified outcomes. We place the most weight on the findings of studies with designs that demonstrate causality; we consider study quality in conjunction with design. External content experts also review ratings.

Evidence Rating: Guidelines

Rating Evidence Criteria: Amount & Type Evidence Criteria: Quality of Evidence
Scientifically Supported
  • 1 or more systematic review(s), or at least:
  • 3 experimental studies, or
  • 3 quasi-experimental studies with matched concurrent comparisons

Studies have:

  • Strong designs
  • Statistically significant favorable findings
Some Evidence
  • 1 or more systematic review(s), or at least:
  • 2 experimental studies, or
  • 2 quasi-experimental studies with matched concurrent comparisons, or
  • 3 studies with unmatched comparisons or pre-post measures

Studies have statistically significant favorable findings

Compared to 'Scientifically Supported', studies have:

  • Less rigorous designs
  • Limited effect(s)
Expert Opinion
  • Generally no more than 1 experimental or quasi-experimental study with a matched concurrent comparison, or
  • 2 or fewer studies with unmatched comparisons or pre-post measures
  • Expert recommendation supported by theory, but study limited
  • Study quality varies, but is often low
  • Study findings vary, but are often inconclusive
Insufficient Evidence
  • Generally no more than 1 experimental or quasi-experimental study with a matched concurrent comparison, or
  • 2 or fewer studies with unmatched comparisons or pre-post measures
  • Study quality varies, but is often low
  • Study findings vary, but are often inconclusive
Mixed Evidence
  • 1 or more systematic review(s), or at least:
  • 2 experimental studies, or
  • 2 quasi-experimental studies with matched concurrent comparisons, or
  • 3 studies with unmatched comparisons or pre-post measures
  • Studies have statistically significant findings
  • Body of evidence inconclusive
Evidence of Ineffectiveness
  • 1 or more systematic review(s), or at least:
  • 2 experimental studies, or
  • 2 quasi-experimental studies with matched concurrent comparisons, or
  • 3 studies with unmatched comparisons or pre-post measures

Studies have:

  • Strong designs
  • Significant unfavorable or ineffective findings, or
  • Evidence of harm

 

Equity Analysis (April 2022-current)

Each strategy with an equity analysis identifies evidence-informed solutions to advance equity by assigning a disparity rating, summarizing impacts on disparities, acknowledging historical drivers of inequity, and asking guiding questions to center equity during implementation. The equity analysis is informed by the evidence identified in the targeted literature search.

Assigning Decision Makers

Each strategy is assigned a decision maker. We define decision makers as those who typically have authority or lead in initiating and implementing a strategy. Assigning decision makers is informed by searches of the available evidence and implementation research and can evolve over time.

Implementation Examples and Resources

Examples of strategies in action and tools to support implementation efforts are provided whenever possible. Gleaned from reviews and studies that assess effectiveness and a targeted internet search, these resources can help prepare communities to move forward with their chosen strategies.

If your community has implemented a strategy and would like to work with us to share your story, let us know.