Evaluation methodology

• Previous Page
• Table of Contents
• Next Page

The evaluation was guided by an evaluation matrix developed as part of the planning phase, which outlined evaluation questions and corresponding indicators and data sources.

3.1 Lines of Evidence

The evaluation included multiple lines of evidence and employed the following data collection methods.

• Document and data review. The purpose of the document review was to systematically review relevant secondary data that provided descriptive information on the activities of the Program and evidence responding to most evaluation questions. Types of documents reviewed included:
  • IJP Program data (2017-18 to 2019-20)
  • Success stories reported by IJP programs in annual reports
  • Capacity building and training materials
  • Partnership Initiatives (e.g., RCMP training, mapping of community programs, development of information pamphlet and resources, online database development)
  • State of the Criminal Justice System Report and online Dashboard
  • FPT WG documents.
• Literature review: The literature reviewed included research reports and statistics related to Indigenous people, law and policy.
• Surveys of community-based justice programs (2019) and workers (2021) and CJS representatives (2021): Three surveys were completed and reviewed as part of this evaluation.
  1. IJD administered the survey of community-based justice programs in 2019 to better understand the type of programs and services provided.
  2. A separate survey of the front-line service providers, the Community Justice Workers (CJWs) was conducted by the Evaluation Branch in 2021 to collect data relevant to the evaluation questions. Given that there are about 200 CJWs employed (vacancies may impact number of CJWs employed at any given time), the response rate among CJWs was about 50%.
  3. A survey of CJS representatives was also completed by the Evaluation Branch. Of 54 CJS representatives, half were RCMP officers and half were provincial prosecutors (a few defence counsel and a judge also participated). Respondents were from 10 jurisdictions: British Columbia, Saskatchewan, Manitoba, Ontario, Quebec, New Brunswick, Nova Scotia, Yukon, Northwest Territories, and Nunavut.

• Interviews with Key Informants: Telephone interviews were conducted with 39 individuals to collect perceptions of federal representatives from Justice Canada and Indigenous Services Canada (n=14), provincial/territorial representatives (n=22) and IJP community-based justice program representatives (n=3).
• Case Studies: Three case studies were conducted with 12 communities focusing on prevention, civil/family mediation and family violence programming, and experiences of victims and survivors of crime.

The community programs selected for the case studies demonstrate a wide range of programming offered, serving both offenders and victims. Since they are not intended to be representative of the many diverse and unique programs supported by IJP, efforts were made to select programs with particular expertise in the area of study so that best practices and lessons learned could be identified and shared. The communities were selected to ensure regional representation as well as a mix of urban, rural and remote communities.

Each case study included a document and data review, interviews, and where possible, site visits. The focus of each case study is described below followed by a summary of the participants in Table 3:

• Four communities participated in the case study exploring crime prevention activities and measures. The focus was to explore the extent and type of prevention activities undertaken by the community program, characteristics of clients participating in the prevention measures (e.g., youth, former offenders, high-risk individuals, etc.), other partners and stakeholders involved, barriers to and impact and effectiveness of these activities. The four communities that agreed to participate were from Northwest Territories, Saskatchewan, Quebec, and British Columbia. A total of 29 interviews were completed mostly with program partners and stakeholders.
• Three communities participated in the case study designed to provide observation of selected community-based initiatives related to civil/family mediation and family violence. The participating communities included two from Alberta, and one from Saskatchewan. The methodology consisted of interviews with 38 individuals across three communities (all but three interviews were conducted in-person during community visits), and observation of two client circle groups.
• Five communities participated in the case study examining the experiences of victims and survivors of crime who had participated in restorative justice processes delivered by IJP community-based justice programs. The five communities that participated included: two from Quebec; and one each from Nunavut, Saskatchewan and British Columbia. The site-visits resulted in a total of 17 interviews with survivors and victims of crime, 7 interviews with self-referred participants in non-criminal matters, 19 interviews with program managers and staff, and 27 community members (Elders and community participants in the restorative justice process).

3.2 Evaluation Challenges and Limitations

The evaluation encountered a few methodological limitations or challenges, including:

• Reliability of the program data. There are inconsistencies in how data is reported across community programs resulting from different data collection systems (some jurisdictions have their own systems) and capacity for reporting in a timely manner. Program data was not available for all provinces and territories across the three years reviewed.
• Limited representativeness of CJS respondents. Though the CJS respondents were from 10 jurisdictions, half of the RCMP officers participating in the survey were from Nunavut, and most of the provincial prosecutors were from three provinces (Ontario, Manitoba and Saskatchewan).
• Respondent bias. Most of the findings are based on opinions (surveys, interviews) which introduce a potential for respondent bias from the sampling approach (selective, non-random), the voluntary nature of participation, and self-reporting (reporting on own activities).
• Evidence presented here does not apply to all programs. As noted in section 2.1, the community-based justice programs are based on their traditions and culture, and so they vary in their design, structure, operations and capacities. Some funded community organizations are well established, long-operating, and well known in the community while others may be more recently established or have limited resources and a lower profile in the community. Thus, any findings presented in this evaluation relating to community-based justice programs will apply to some but not all communities. Case studies were not intended to be representative of a wide a range of programming, therefore, any data from the case study interviews should be interpreted with caution.
• Most of the data was collected during the COVID-19 pandemic. The current circumstances related to community programs’ ability to deliver services, such as limited in-person support, lack of travel, and limited engagement with community, may have influenced their perceptions of the Program. The pandemic and remote working conditions impacted the evaluation data collection resulting in delays and limited access to community organizations during the prevention case studies (e.g., all case study interviews were conducted over the phone. It was hard to reach some community organization representatives because they were not in the office).
• Inability to complete the recidivism study and comparative costing analysis within the evaluation timeframe. The planned recidivism study was delayed due to the COVID-19 pandemic restrictions that prevented participating community-based justice program staff from accessing their offices to complete the data entry. Similarly, the RCMP division that completes the criminal records checks experienced pandemic restrictions resulting in further delays to the project. As the comparative costing analysis incorporates the findings of the recidivism study, it has also been delayed. The work on both studies will be completed at a later date.

The evaluation methodology involved multiple lines of evidence, primary and secondary sources of quantitative data (program data and surveys), and qualitative data (interviews of key informants and case studies). Limitations were mitigated by triangulating the data from these lines of evidence to arrive at the evaluation findings presented in this report.

• Previous Page
• Table of Contents
• Next Page