Drug Treatment Court Funding Program Evaluation
3. Methodology
The evaluation was guided by the evaluation matrix in Appendix B and included three main lines of evidence.
3.1. Document and Data Review
The document and data review included the following documents:
- DTC funding applications
- financial information for 2009–10 to 2013–14
- funding agreements for 2009–10 to 2013–14
- minutes of DTC directors’ meetings for 2009–10 to 2013–14
- research documents (e.g., participant outcomes studies conducted by the Research and Statistics Division with accompanying briefing notes, independent research conducted for the Vancouver DTC, and the evaluation of the housing pilot for the Winnipeg DTC)
- DTC evaluations (evaluations were not a requirement, but some DTCs have conducted evaluations since 2009)
- DTC websites
- relevant Justice Canada sub-studies and research studies, including a recidivism study and a study comparing the results of UDTs of graduates and non-completers over time in the program
- Drug Treatment Court Information System (DTCIS) Footnote 3 data results.
Profiles of each DTC were developed using the available documentation. To verify the content, each DTC was provided with its profile and changes were made, as requested.
In addition to these documents, the evaluation gathered publicly available information from DTC websites, Departmental Performance Reports, Reports on Plans and Priorities, Budget Speeches, and Speeches from the Throne.
A targeted literature review focused on the most recent research studies related to DTCs with an emphasis on meta-analyses.
3.2. Survey of DTC Stakeholders and Staff
To assess the opinions of DTC stakeholders and staff, the evaluation used a bilingual web-based survey. The survey questionnaire is in Appendix C.
The survey sample was developed by Justice Canada in collaboration with the DTCs. The total sample included 138 contacts representing the six DTCs. The survey was offered online between August 8 and September 2, 2014. Reminder emails were sent to encourage participation. A total of 65 individuals completed the survey, for a response rate of 47%.
DTC | Number of respondents | Number in sample | Percentage of response |
---|---|---|---|
Toronto | 21 | 41 | 51% |
Regina | 11 | 17 | 65% |
Edmonton | 10 | 29 | 35% |
Ottawa | 10 | 21 | 48% |
Winnipeg | 8 | 15 | 53% |
Vancouver | 5 | 15 | 33% |
Total | 65 | 138 | 47% |
Respondents were primarily members of the DTC team and/or members of the DTC governance or advisory committees. See Table 4 for further details.
Number | Percentage | |
---|---|---|
Members of dedicated DTC team | ||
Treatment provider | 11 | 17% |
Judge | 7 | 11% |
DTC director | 5 | 8% |
Federal Crown/paralegal | 5 | 8% |
Provincial Crown | 3 | 5% |
Probation or police service | 2 | 3% |
Administrative | 2 | 3% |
Case manager | 1 | 2% |
Housing assistance/manager | 1 | 2% |
Duty counsel | 1 | 2% |
No response | 4 | 6% |
External service providers | ||
Housing services | 4 | 6% |
Addictions treatment | 2 | 3% |
Other health services | 2 | 3% |
Provincial social assistance | 2 | 3% |
Employment services | 1 | 2% |
Intake | 1 | 2% |
Members of DTC governance or advisory committee | 22 | 34% |
Defence counsel | 3 | 5% |
Other | 1 | 2% |
Note: Respondents could provide more than one answer. Totals do not sum to n=65 or to 100%.
3.3. Case Studies and DTC Observations
The case studies focused on the experiences of individual DTC participants at four sites: Ottawa, Regina, Toronto and Winnipeg. The DTCs assisted with recruiting individuals for the case study interviews, which were conducted in person at the DTC offices. A copy of the case study interview guide appears in Appendix C. Each interview took approximately 30 to 45 minutes to complete. Participants had the option of ending the interview at any time.
The evaluation planned to conduct at least 12 case studies with participants at each DTC site for a total of at least 48 interviews. The intention was to divide the interviews evenly between program graduates, current participants who have substantial experience with the program (i.e., not recent entrants), and participants who did not complete the program. Current participants were included because they have more immediate reactions to the programs and can address any challenges or barriers they are presently experiencing.
Overall, 48 individuals were interviewed across the four case study sites, including 22 graduates, 18 current participants, and eight who did not complete the program (“non-completers”). The evaluation targets were generally met, although the non-completers proved to be a challenging group to reach, since they were no longer connected to the DTC and did not have a positive outcome (graduation). Demographic information of case study participants is presented in Table 5.
Total | |
---|---|
Ethnicity | |
Caucasian | 36 |
Aboriginal/Métis/Inuit | 10 |
Other visible minorities | 2 |
Gender | |
Male | 32 |
Female | 16 |
Age | |
18–24 | 2 |
25–34 | 14 |
35–44 | 9 |
45–54 | 18 |
55 and over | 5 |
Age when began using drugsTable note * | |
10 and under | 2 |
11–19 | 36 |
20–29 | 4 |
30–39 | 2 |
40–49 | 0 |
50 and over | 1 |
Drug of choiceTable note ** | |
Cocaine/crack cocaine | 30 |
Opiates | 12 |
Methamphetamines | 10 |
Talwin & Ritalin (Ts & Rs) | 1 |
- Table note 1
-
Note: Three case study participants did not provide the age when they began using drugs.
- Table note 2
-
Note: Some case study participants provided more than one drug of choice.
Most case study participants were charged with multiple crimes before entering the program. Criminal activity self-reported by participants involved, for the most part, property-related crimes such as shoplifting, breaking and entering, theft and/or drug-related offences, mainly trafficking and possession.
The following scale has been applied to report on case study interviews.
- A few: less than 15%
- Some: approximately 25%
- Many: approximately 50%
- Most: approximately 75%
- Almost all: approximately 100%
The site visits at the four DTCs for the case study interviews also provided an opportunity to observe a pre-court and court session of the DTCs, which showed an understanding of how the DTC team worked together during court sessions. In addition, the DTC directors presented an overview of the DTC services to the evaluation team.
3.4. Recidivism Study
A study was conducted to compare the rates of re-offending of individuals who participated in a DTC program from 2006 to 2010 with two separate comparison groups: a) individuals (N=151) who met the DTC eligibility criteria but who were arrested in a jurisdiction that does not have a DTC; and b) individuals who were eligible and referred to a DTC but did not participate in the program (N=45). The DTC group was comprised of those individuals who successfully completed the program (graduates N=104) and those who were terminated from the program (released N=290).
The four recently established DTC sites (Ottawa, Winnipeg, Regina and Edmonton) provided information (DTCIS) on the adult offenders referred to the DTC, including offenders’ names, date of birth, program status and dates, gender and charges at arrest.
Evaluators reviewed court files (Halifax and Moncton) to identify the individuals for the comparison group. The criteria to select adult offenders who were charged under the CDSA were similar to the DTC eligibility criteria (i.e., offenders had committed a non-violent offense; did not use a weapon in the commission of their offense; there was no indication the individual’s offence was gang or organized crime related and there was a clear reference to the individuals drug and/or alcohol abuse/addiction/dependence).
Individuals who participated in the DTC program between 2006 and 2010, and those who went to the court between 2004 and 2010 (comparison) were included in the analysis. This timeframe allowed for a post-program follow-up time of at least three years after program participation or completion. The names of all individuals (DTC and comparison group) were submitted to the Royal Canadian Mounted Police’s Canadian Police Information Centre (CPIC) to obtain criminal history information. The CPIC records were analyzed up to September 2013. For the analysis, the time elapsed after participation in the program until receiving a criminal conviction or, in the absence of a conviction, the time to the end of the observation period, was statistically modeled. Recidivism is defined, for the purpose of this study, as a new criminal conviction after completion or participation in the DTC program.
Cox regression model was performed to analyze the data. The analysis controlled for gender, age at arrest, total number and types (i.e., offences against a person, administration of justice and drug) of prior convictions, participant’s age of first conviction, years of criminal record and whether or not individuals participated in a DTC program. Even though, the study controlled for some measures, there are however other unmeasured differences in the underlying characteristics of the DTC and comparison group members (i.e., race, employment, health issues, education, drug of choice, and housing) that the study did not account for. The major limitation of the analysis was the lack of true experimental design, as practical and ethical constraints precluded the random assignment of persons to participate.
3.5. Drug Urine Tests Analysis
Part of the requirement of the DTC program is random testing of the participants for illicit drugs. Participants must willingly submit to random and frequent urine drug testing by program staff during participation in the DTC program. The dates and the results of these urine drug tests (UDTs) were recorded in the DTCIS database regularly.
For the present study, UDTs results were extracted from the DTCIS for five sites (Edmonton, Ottawa, Toronto, Winnipeg and Regina) for the period between April 1, 2009 and March 31. 2014. The results of the UDTs were analyzed over a period of 15 months at three-month intervals. Individuals who were not accepted or admitted in the DTC, those with less than 30 days in the program and those who had no UDTs were not included in the study. The total number of the participants included in this study was 672. Only participants who had remained in the program for 15 months were ultimately included in the study (n=98) for the between-subject average effect and interactions. Although this approach introduces selection bias as earlier non-completers are not part of the study, the results align with findings in the literature that indicate that retention and length of stay may lead to better outcomes, as discussed above.
A mixed repeated measures design with one between-subject factor and two within-subject factors was used to analyze the data. The between-subject factor referred to the groups (those who successfully completed the program and those who did not complete the program) and the within-subjects factors were time in months (3, 6, 9, 12 and 15) and UDTs (dirty and clean).
3.6. Limitations and Mitigation Strategies
The evaluation encountered several methodological limitations and undertook steps to mitigate their effects.
Methodological limitations and mitigation strategies
- An online survey was used to collect information from stakeholders and key informants, as well as from case studies with participants at each DTC site. The survey and the case studies were administered during summer months. This may have affected the response rate to the survey as well as the ability to recruit case study participants.
- Mitigation strategies:
- Multiple reminder emails were used to boost the response to the survey. In addition, the survey timelines were extended by one week.
- For case studies, the site visits occurred as late in the evaluation as possible given reporting timelines.
- The survey included many open-ended questions to provide more in-depth, detailed information than a traditional online survey.
- Mitigation strategies:
- The case study interviews were intended to include equal numbers of current participants, graduates and non-completers. However, recruitment efforts were less successful for non-completers, which limit the inclusion of their perspective in the evaluation findings.
- No mitigation strategies available.
- Drug Treatment Court Information System data:
- Given that the DTCIS was developed in the first instance as a case management tool, and in the second instance, as a data collection tool, many of the DTCIS fields are site-specific with drop-down menus that do not allow for each roll-up in a national basis.
- The extracted fields from the DTCIS database are not the same as the fields in the DTCIS input tool. The DTCs have a manual to guide data entry, but there is no manual for the DTCIS database extraction.
- It was difficult to determine the number of active participants, regardless of their status, by FY based on the current fields available in the DTCIS.
- DTCFP made efforts to streamline the DTCIS as it proved onerous to DTCs. As a result, some information on outcomes is no longer consistently tracked, such as the number, length and type of treatment sessions.
- The DTCIS does not capture information on recidivism during the program unless it is a reason for terminating a participant from the program.
- The DTCIS contains data entry errors, inconsistencies and other data quality issues, which make its use to support analysis of program results challenging.
- The DTCIS database is to be updated regularly, but sometimes there are gaps and interruptions, and thus the most recent information might not be available.
- Mitigation strategies:
- The DTCFP has a dedicated IT person to support the database and extract the necessary information as required. The IT expert downloads the information from each DTC monthly report and merges the data from the sites into the national DTCIS.
- The Evaluation Division undertook substantial efforts to understand the DTC and DTCIS fields, and to clean and analyze the data for the evaluation, which enabled this evaluation to include DTCIS data.
- Although this evaluation benefits from the availability of the completed recidivism study, global costs are not tracked by each DTC. This is particularly the case with DTCs that have NGO funding recipients. These recipients do not have access to the provincial costs (e.g., court time, Crown time) for the operation of the DTC.
- No mitigation strategies available
- Date modified: