Evidence Brief methodology

The purpose of Evidence Products

Evidence Products include Evidence Briefs, Research and Development Briefs, Evidence Summaries and potentially other forms as well.  They provide systematic reviews of justice related interventions. These products will support decision-making for investing in evidence-based policy and practice to improve the lives of the New Zealand public. 

The Ministry of Justice Evidence Briefs summarise New Zealand and international research, the provision of current services, and tell us how well an intervention reduces crime.  Evidence Briefs can put a numerical value on effectiveness and therefore demonstrate if the intervention worked, and for whom.

How Evidence Products are used

Evidence Products are designed to support investment and decision making in the justice sector. The suite of Evidence Briefs, developed by the Ministry of Justice, is a ‘first-stop-shop’ for decision makers and policy developers looking to influence justice outcomes.

To make good decisions on where the government invests, we need good evidence to understand: what is effective; for whom; when; and why.

The evidence to guide decision-makers can be mixed, depending on the robustness of research into an intervention. Evidence Products summarise this evidence in a way that decision makers and policy developers can use. This includes interventions that range from strong, high quality evidence bases to evidence that is inconclusive, or evidence that supports ending an intervention.

The suite can then support

  1. The design of policy and services to develop interventions with a high confidence of success and avoid interventions that should not be undertaken
  2. Research and evaluation to identify and address gaps and uncertainties in current evidence
  3. Build confidence with Ministers, the government and the public, that policy decisions are made on the best evidence.

Evidence products can also assess some socio-economic outcomes beyond reoffending.

Assessing evidence rating

Each Evidence Brief provides a rating based on the same criteria. The final rating is built around two separate assessments, one reflecting international evidence, and another – New Zealand evidence.

Effective programmes in one country often, but not always, work in other countries. Even if international research shows that an investment type can reduce offending, it may not be worth adopting unless we are able to replicate the results in New Zealand’s crime environment.

Both international and New Zealand evidence is assessed using the Maryland Scale[1] of Scientific Methods. This is, a 5-point scale with randomised controlled trials at the top (level 5) of the scale. At lower levels, there is an increasing risk that findings are subject to selection bias with a range of challenges to their validity.  Level 3 is considered to be the minimum to conclude that the intervention reduces crime.

[1] Sherman, L., Farrington, D., Welsh, B., & Mackenzie, D. (Eds). (2002). Evidence-Based Crime Prevention. New York: Routledge

Evidence rating for Evidence Briefs follow the Maryland Scale:

Level 5 Randomised, controlled trials
Level 4 Quasi-experimental design
Level 3 Comparison of outcomes in treated group after an intervention, with outcomes in the treated group before the intervention, and a comparison group used to provide a counterfactual (e.g. difference in difference).
Level 2 Use of adequate control variables and either (a) a cross-sectional comparison of treated groups with untreated groups, or (b) a before-and-after comparison of treated group, without an untreated comparison group.
Level 1 Before and after studies with no comparison group

The final rating is measured in a six-grade scale varying from one for a “harmful” outcome category to six for a “strong” outcome category.

Outcome categories

  New Zealand studies
    At least one level 4 or 5 study finds a statistically significant negative impact, no conflicting L4+ studies Studies show conflicting results, or no impact At least one level 3 study finds a statistically significant positive impact, no conflicting L3+ studies At least one level 4 study finds a statistically significant positive impact, no conflicting L4+ studies At least one level 5 study find a statistically significant positive impact, no conflicting L5 studies
International studies Meta-analysis/systematic review of 5+ studies finds a statistically significant positive impact, no conflicts Fair Promising Strong Strong Strong
MA/SR with fewer than 5 studies finds positive impact Inconclusive Fair Fair Promising Strong
MA/SR find conflicting results Inconclusive Inconclusive Fair Promising Strong
MA/SR shows no effect Harmful Poor Promising Promising Strong
MA/SR shows negative impact, no conflicting results Harmful Poor Inconclusive Fair Promising

Interpretation of outcome categories

There is also a standard interpretation for each evidence rating, as summarised in the following table.

Outcome category

Interpretation

Strong
  • Robust international and local evidence that intervention tends to reduce crime
  • Intervention likely to deliver positive outcomes if implemented well
  • Could benefit from additional evaluation to confirm intervention is delivering positive outcomes and to support fine-tuning of the intervention design.

Promising
  • Robust international or local evidence that intervention tends to reduce crime
  • Intervention may well deliver positive outcomes if implemented well
  • Further evaluation desirable to confirm intervention is delivering positive outcomes and to support fine-tuning of the intervention design.

Fair
  • Some evidence that intervention can reduce crime.
  • Uncertain whether intervention will deliver positive outcomes even if implemented well.
  • May be unproven in New Zealand or be subject to some conflicting research which show no effect
  • May benefit from trial approaches with a research and development focus.
  • Robust evaluation needed to confirm intervention is delivering positive outcomes and to aid in detailed service design.

Inconclusive
  • Conflicting evidence that intervention can reduce crime.
  • Highly uncertain whether intervention will deliver positive outcomes even if implemented well

 

Poor
  • Robust evidence that intervention tends to have no effect.

Harmful
  • Robust evidence that intervention increases crime.
  • Should be priority for divestment.

This page was last updated:

 
Share