Evaluation methodology

• Previous Page
• Table of Contents
• Next Page

The evaluation was guided by an evaluation matrix (evaluation questions, indicators and data sources) which was developed through the evaluation scoping and design process. The methodology for this evaluation included multiple lines of evidence described below. Appendix B contains the list of evaluation questions, and a more detailed description of the evaluation methodology is included in Appendix C.

3.1 Literature, Document and Data Review

A review was conducted of secondary data sources relevant to the DTCFP. These materials included:

• Literature – peer-reviewed and grey literature related to DTCs, including trends and emerging issues, and effectiveness and cost effectiveness of DTCs;
• Documents – Justice and federal documents related to departmental mandate and priorities, program foundational and implementation documents, performance information, other special studies and reports, and financial budget and expenditure information; and,
• Data – Statistics related to drug-related offences produced by the Statistics Canada’s Canadian Centre for Justice Statistics.

3.2 Analysis of Drug Treatment Court Information System Data

Most DTCs report information on their participants through the federal Drug Treatment Court Information System (DTCIS). The analysis included all participants who were active in the DTCs (where information was accurately captured in DTCIS) at some point between April 1, 2015 and June 17, 2021 (N = 1,053).6 The DTCIS includes information on the profile of DTC participants, as well as completion status.

3.3 Survey of stakeholders

An online survey was conducted of DTC stakeholders. All DTCs funded under the program provided lists of stakeholders or agreed to distribute the link of the survey to their stakeholders. In total, 139 individuals completed the survey. Among DTCs that provided a list of stakeholders for the survey, a 49% response rate was achieved. The profile of surveyed stakeholders included: members of the dedicated DTC management, court (judges, prosecutors) and treatment team (56%), governance or advisory committee members (21%), external treatment or service providers (19%), defence counsel (9%), and other (e.g., policy, management).

3.4 Key informant interviews

A total of 63 in-depth interviews were conducted for the evaluation, including:

• 4 interviews with federal officials, and;
• 59 interviews for four case studies of DTCs consisting of Whitehorse (13 interviews), Edmonton (14), Regina (13), and Toronto (19). Case study respondent groups included:
  • DTC personnel (e.g., director) (n = 6);
  • external treatment/service providers/probation (n = 15);
  • court personnel/legal stakeholders (judges, Crown counsel, defence counsel) (n = 16);
  • current/past participants (n = 21); and,
  • other (n = 1)

3.5 Limitations and Mitigation Strategies

The evaluation has several methodological limitations that should be noted.

• Inability to complete recidivism study within evaluation timeframe: The evaluation could not develop a robust measure of the impact of the DTCs on recidivism or the cost effectiveness of DTCs compared to alternatives within the timeframe of the evaluation report. A planned recidivism study was delayed due to the COVID-19 pandemic restrictions that prohibited the review of court files on-site to develop a comparable sample of non-DTC participants. DTCs also do not track or do not release information on referrals to the program which prevented using this group as a potential comparator for a recidivism study. As a mitigation strategy, Justice contracted the Canadian Centre for Justice and Community Safety Statistics (within Statistics Canada) to conduct a recidivism study using data from DTCIS and Statistics Canada. The results of this study will also be triangulated with literature, including systematic reviews, as well as anecdotal evidence from the evaluation to assess the impacts of DTCs.
• Determining the demand for DTCs: The target for DTCs include those with CDSA and Criminal Code offences where substance use disorder was a primary factor. There is little data to understand the potential demand for DTCs, nor the profile of those who would potentially be eligible for the DTC. This was due to the fact that referrals to the DTCs are either not traced or unable to be shared for privacy reasons. This data gap hampered the assessment of the representativeness of DTC participants and barriers to access. The evaluation relied on the views and perceptions of stakeholders closely involved in DTC implementation to examine under representation.
• DTCIS data quality issues: The analysis of program data held in the DTCIS was hampered by a number of data quality problems. Data was not available for some DTCs, and for others was only partially available for the period under study. Further, there were inaccuracies found in how DTCs populate the DTCIS template. Specifically, with reference to the education and marital status variables, valid categories were being used as defaults if the information was unknown (e.g., some high school, single). There were also lags and gaps in data capture, especially in 2019-20 and 2020-21 due to capacity issues created by the COVID-19 pandemic; and, technical issues also restricted availability of participant risk data. The end result is that the DTCIS data included in this report represents a sub-set of all DTC data (e.g., not all DTCs are fully represented; and some variables such as those mentioned above were considered to be unreliable and were excluded from the analysis).
  In order to mitigate these data limitations, the program area and the Evaluation Branch devoted considerable time to cleaning, reviewing and validating the data that was available. As such, where the report relies on DTCIS data, for the variables in question, the limitations are minimal. As a result of the review and validation process, data with questionable reliability was excluded from subsequent analysis, thereby resulting in increased confidence in the reliability of the remaining data included in the analysis for the current evaluation report. Furthermore, where available, stakeholder experience gathered through the case studies, interviews and surveys was used to better understand trends.
• Variable survey response rate by DTC: While almost 50% of stakeholders completed the survey, the response rate by DTC was uneven (ranging between 31%-85% by DTC). Stakeholders from the Toronto DTC make up 30% of responses. To mitigate this limitation, the survey data are triangulated with the experience of DTCs that were included in the case studies.

---

• Previous Page
• Table of Contents
• Next Page