Evaluation of the Youth Justice Initiative

3 Evaluation Methodology

The implementation of the evaluation was guided by an evaluation matrix (evaluation questions, indicators and data sources) which was developed through the evaluation scoping process and further refined with the Evaluation Working Group.

3.1 Lines of Evidence

The methodology for this evaluation included multiple lines of evidence and employed the following data collection methods. A more detailed description of the evaluation methodology is included in Appendix B.

  • Literature, document and data review: A review was conducted of secondary data sources relevant to the YJI. These materials included:
    • Literature – peer-reviewed and grey literature related to youth justice, including trends and emerging issues and research on best practices;
    • Documents – program foundational and implementation documents, performance information, other special studies and reports, and financial budget and expenditure information; and
    • Data – Statistics related to youth justice produced by the Statistics Canada Canadian Centre for Justice Statistics (CCJS).
  • File review: A systematic review was conducted of YJSFP annual reports and financial claims submitted by provinces and territories to Justice Canada. In addition, 105 IRCS Face Sheets were reviewed of the 153 cases that were concluded or terminated during the period under study. The IRCS Face Sheets provide a record and chronology of Part B (court orders) and Part C (exceptional) cases, including tombstone information, interventions and risk assessments.
  • Key informant interviews: In total 35 in-depth key informant interviews were conducted with 39 individuals. Respondent groups included: 1) Internal Justice Canada (senior officials, YJI program managers and staff); 2) provincial/territorial representatives; and 3) other stakeholders (YJF project leads and other external stakeholders).Footnote 10
  • Case studies: Five case studies of YJSFP-funded programs and services were conducted. The case studies included document reviews and primary data collection (interviews, focus groups and/or surveys) with program delivery management, staff and stakeholders. Case studies were selected in collaboration with provincial/territorial representatives and included diverse programs from across the country:
    • British Columbia (Youth Forensic Psychiatric Services);
    • Yukon (Youth Achievement Centre);
    • Ontario (Single Case Management Model);
    • Nova Scotia (Restorative Practices – Residential Setting); and
    • Prince-Edward-Island (Community Youth Worker Program / Youth Intervention Outreach Program).

3.2 Limitations and Mitigation Strategies

The evaluation has several methodological limitations that should be noted. First, while there are strong data at the national and provincial/territorial levels that suggest youth justice trends are consistent with the objectives of the YCJA and the YJI, it is difficult to conclusively attribute or quantify the contribution to these shifts of the YJI. Provinces and territories assume the responsibility and the majority of the costs of the implementation of the youth justice system. While the case studies provide some insight into the implementation and effectiveness of YJI-funded programs and services, it is difficult to generalize these outcomes more broadly to the full efforts of YJSFP and IRCS funded programs. The key mitigation strategy for this limitation was to triangulate the case study evidence with more systematic data collection across jurisdictions (file review, key informant interviews) to assess the general contribution of the YJI to the implementation of the YCJA.

Second, program files and data review was hampered by delayed and/or inconsistent reporting. For instance, the IRCS Face Sheet tool to monitor IRCS cases is not used consistently by provinces and territories and, therefore, a significant number of these cases were not included in the analysis. Consequently this data should be interpreted with caution. As well, national statistical data have been presented in the report where relevant; however, due to Statistics Canada publication lags, these data are only available to 2016-17 in many instances.

Finally, a common measure of the impact of programs and services such as those funded by the YJSFP and IRCS is reduced recidivism. While there is anecdotal evidence and some regional studies of youth re-contact with police, there was limited empirical data on recidivism available to the evaluation. It should be noted also that increasingly, re-offending is recognized as an important, but limited measure of the impact of youth justice interventions. Complementary measures such as achieving reduced severity of crime, education/vocational goals or family or cultural reconnection were identified as important if future research were to be conducted in this area.

Date modified: