Treating Youth in Conflict with the Law: A New Meta-Analysis

2.0 Method

2. Method

2.1 Design: Meta-analysis

Meta-analytic techniques, as a method of aggregating knowledge, have been used in several fields of study including education and medicine, and have more recently been adopted within the social sciences (Lipsey & Wilson, 1993) to investigate both the prediction and treatment of criminal behaviour.   Similar to the standard quantitative research method, the meta-analytic process contains three basic steps:

  • literature review - identify and gather relevant research studies;
  • data collection - extract data through pre-determined coding procedures; and,
  • data analysis - analyse the aggregated data using statistical techniques.

Rosenthal (1991) claimed that "meta-analytic reviews go beyond the traditional reviews [of the literature] in the degree to which they are more systematic, more explicit, more exhaustive, and more quantitative.   Because of these features, meta-analytic reviews are more likely to lead to summary statements of greater thoroughness, greater precision, and greater intersubjectivity or objectivity" (p.17).

In general terms, a meta-analysis is a statistical examination of a collection of studies that aggregates the magnitude of a relationship between two or more variables (Glass, McGaw & Smith, 1981).   These studies typically differ, however, on several important characteristics such as operationalisation of independent and dependent variables, sample size, sample selection techniques, and design quality.   A meta-analysis can describe the typical strength of the relationship under investigation, the degree of statistical significance, the variability, as well as provide researchers the opportunity to explore and identify potential moderating variables.   The outcome of a meta-analysis, an effect size (ES), can be interpreted as the estimated effect of the independent variable on the dependent variable.   In other words, an average effect size estimate of + 0.05 translates into the independent variable (e.g., treatment) effecting a 5% change in the dependent variable (e.g., recidivism).  

2.2 Sample: Study identification criteria

To gather eligible studies for the meta-analysis, a comprehensive search was conducted on the young offender treatment literature over the last 50 years including unpublished doctoral theses and governmental reports.   A secondary search was conducted using the bibliographies of the relevant literature, prior meta-analyses and the Internet.   An explicit set of criteria was established in order for a study to be included in the analysis:

  1. the study examined the effectiveness of a non-traditional response to youth delinquency (i.e., an intervention that is not a standard court ordered response to youth crime such as traditional probation or custody);
  2. the study consisted primarily of youth who were under 18 years of age and had committed an offence using current adult standards;
  3. the study used a control group or comparison group that did not experience the treatment under examination (or provided sufficient pre/post data);
  4. sufficient statistical information was reported in order to extract an effect size; and,
  5. the study measured the impact of treatment on at least ONE of the following outcomes of interest:
    • recidivism;
    • academic performance/attendance;
    • psychological well-being;
    • family functioning;
    • employment gains;
    • social skills;
    • anti-social attitudes;
    • substance abuse;
    • anger management;
    • anti-social peer pressure; and,
    • cognitive skills.

2.3 Data extraction: Coding procedures

Standardized information was drawn from each research study using a pre-designed coding manual.   In accordance with standard meta-analytic techniques, multiple definitions of each of the outcomes of interest were accepted .   For example, recidivism was defined as a new conviction or a new charge.   If statistical information was not contained in an individual study, but a non-significant relationship between treatment and the outcome was reported, the effect size was recorded as zero.   I n order to generate sufficient data for analysis, several coding techniques were used.   For example, if 70% or more of the study sample were male, we coded it as a "primarily male program" and if 70% or more of the study sample were first-time offenders, we coded it as a "primarily first-time offender program".   In addition, several variables were coded only if the authors made an explicit positive statement.   For example, the existence of program manuals or staff training was only coded as "yes" if the authors directly stated this to be true.   Therefore, the comparisons made in this report are subject to this limitation.   It should be noted, however, that this is a general issue within all meta-analyses.  

2.4 Data analysis: Effect size calculations

In accordance with the meta-analytic techniques of Rosenthal (1991), the phi coefficient (Pearson's r product moment correlation applied to dichotomous data) was used as the effect size estimate.   I n cases where multiple control groups were used in a single study, the results were combined in order to generate a single effect size for each program.   In addition, where multiple follow-up periods were reported in a single study, the longer period was selected.

Once the effect sizes from each study were calculated, a series of analyses across each of the outcome measures of interest were conducted.   First, the overall mean effect size, along with the corresponding confidence intervals, were calculated.   Additional analyses were conducted to explore whether certain variables had a moderating impact on effect size magnitude.   For example, if adequate information was available, the treatment targets of the program or the treatment dosage (i.e., number of hours exposed to treatment) were examined to determine possible effects on program success.   This provided a mechanism whereby specific program characteristics could be isolated for further study.

Date modified: