Youth Justice Initiative Evaluation

3. Methodology

This section provides details of the methods employed to collect and analyze information in answering the evaluation questions.

3.1. Key informant Interviews

Interviews were conducted either by telephone or in-person, depending on the location and preference of respondents, in the official language preferred by the respondent. In total, 68 respondents were interviewed (some respondents were interviewed in group interviews). Table 2 describes the types of interview respondents and the number of each type.

Table 2: Key Informant Interviews
Key Informant Group Number of Respondents
Justice Canada 15
Provinces/Territories 33
Youth Justice service providers 31
Total 68

Most interviews inquired broadly about the YJI and the federal contribution to youth justice. However, the interviews included nine targeted interviews in New Brunswick, Saskatchewan and Alberta with provincial government justice officials and managers and staff of specific youth justice programs, to investigate how a small sample of provincial programs were designed to achieve YJI objectives.

3.2. Document Review

The purpose of the document review was to understand the rationale for the YJI, the nature of the Initiative, and how it has been implemented. Most documents were provided by YJI program officials, but an on-line search was also conducted for government documents directly relating to the YJI during the five-year evaluation period. Overall, 25 documents were reviewed, including YJI-related legislation, policy and program documents, contribution agreements and YJI funding terms and conditions, briefing notes and minutes of meetings, records of decision from federal, provincial and territorial committee meetings, departmental planning and reporting documents, and other YJI-related reports and presentations.

3.3. Literature review and Statistical Analysis

The evaluation included a review of Canadian Centre for Justice Statistics (CCJS) statistics relating to youth justice, and relevant data from individual provinces and territories. Statistical material was accessed in two main formats. First, data was retrieved in the form of statistical compilations, for example, Statistics Canada’s key socioeconomic database (CANSIM) tables, such as the  2011-13 Youth Custody and Community Services Survey. Second, statistics were accessed in the form of existing analyses undertaken by government departments and agencies, primarily CCJS Juristat publications.

3.4. File Review

The review included program files and available financial and project data under the three funding streams to collect information related to program effectiveness and cost efficiency. Data was collected on the types of projects, programs and services funded. For the YJF, data was collected on the types of projects funded under each component of the Fund (Drug Treatment, Guns, Gangs and Drugs, and Main Fund). Data on contributions under the IRCS Program was also collected and analyzed, focusing in particular on Parts B, C and D funding. A review of the financial information was conducted to summarize and compare the planned and actual financial resources of the YJI, and to examine administrative costs in relation to grants and contribution expenditures. The review examined and worked extensively with grants and contributions data from YJI, and YJSFP data on provincial and territorial youth justice expenditures in order to characterize the nature and extent of YJI and national investments in youth justice, particularly in high priority areas. Finally, the review examined the terms and conditions of contribution agreements with the provinces, territories and community-based funding recipients, and descriptions of projects funded under the YJF.

In addition to this file review, the evaluation undertook an analysis of a sample of 30 evaluation reports from YJF projects designed to extract information regarding: where the project was located; the start-up year and project duration; total funding, type of project (pilot, research, information sharing, training); the target group (e.g. Aboriginal, FASD, gang member/at risk, youth with addictions); the approximate number of youth receiving support/services; types of project partnerships; and whether they addressed specific YJI outcomes, including:

3.5. Case Studies

Nine case studies were conducted to exemplify the kinds of projects funded under the YJF, and to demonstrate the extent to which funded projects have targeted priority youth justice issues in keeping with YJI objectives. All but two of the case studies involved on-site observation of activities funded by the YJF. All case studies included interviews with project managers and staff, and a review of available documents that described the projects and evaluated project outcomes.

Six of the case studies involved projects undertaken by NGOs active in youth justice. During the evaluation period approximately 325 projects were funded. The selection of the YJF case studies took into consideration the three funding components of the YJF (Drug Treatment, Guns, Gangs and Drugs, and Main Fund) and the types of activities represented by the funded projects (e.g., research, pilot projects, information sharing, training) as they are stated in the terms and conditions of the YJF. In addition, geographical areas (rural and urban), Aboriginal issues, amount of funding for projects, and year of the projects were considered. Twenty-eight informants were interviewed for the YJF case studies.

3.6. Thematic Studies

In addition to the YJF case studies, three studies were conducted that focused on specific subject areas. The purpose of these studies was to examine how the policy and program components of the Initiative worked together to respond to these issues. These issue studies included work done on FASD and the youth justice system, responses to the emerging issue of cyberbullying, and supporting the implementation of amendments to the YCJA. These studies involved interviews with Justice Canada YJI officials, provincial government youth justice officials and service delivery agencies in the related areas, and a review of available documents. Fourteen key informants were interviewed for these case studies. 

3.7. Methodological Limitations

There are two limitations to the methods and available data used to conduct this YJI evaluation.

3.7.1. Limited data available to measure the impacts of the YJI

The main limitation concerns the limited data available linking on-the-ground developments in the administration of youth justice to the programs and services that the YJI funds in part. Each province and territory has its own set of programs and services designed to address youth justice issues, and many of these are, on the face of it and according to recent academic literature, successfully moving youth justice in the direction sought by the YCJA and the YJI. However, the logic model for the Initiative and the outcomes identified for the YJI are phrased such that success is measured largely by whether or not YJI funds are being spent on programs, services and community-based projects that are intended to address YCJA and YJI objectives. The effectiveness of the programs and services funded under the YJSFP and the IRCS Program are seen as to be the responsibility of each province and territory. This is reasonable given that the provinces and territories are responsible for the administration of youth justice, but it means that evaluators are limited in their ability to establish a causal link between YJI contributions and improvements in the justice system response to youth justice issues. It is possible to describe the extent to which the Justice Canada contribution to provincial and territorial programs and services is being used in the priority areas established to achieve YJI objectives.Footnote 3 Also, it is possible to identify the progress of the youth justice system and associated programs and services, in aggregate, have made towards meeting the YJI objectives. It would require extensive research at the federal, provincial and territorial levels to evaluate the effectiveness of specific types of programs and services to which the Department contributes.

To deal with this limitation, this evaluation has been able to demonstrate through the triangulation of results, how YJI funds are spent and to assess results in general terms, by using evidence from the academic literature and research papers, file reviews and case study findings, and the views expressed by key informants.

3.7.2. Lack of full participation in key informant interviews

A second limitation is that one jurisdiction chose not to participate in the key informant interviews of provincial and territorial officials. This may be a limitation to the extent that the state of youth justice in this jurisdiction and/or the views of its officials may differ from those in other jurisdictions.