Drug Treatment Court Funding Program, Summative Evaluation
3. Methodology
The DTCFP evaluation draws on five lines of evidence including a document review, data review, key informant interviews, case studies with participants, and a stakeholder survey.
3.1. Document Review
The document review provided information on the design, implementation and outcomes of the DTCs. Documents reviewed include:
- Memorandum of Understanding (MOU) between the Department and Health Canada
- Overview of the National Performance and Evaluation Reporting Requirements
- DTCFP logic model and performance measurement strategy
- Results-based Management and Accountability Framework for the National Anti-Drug Strategy
- DTC funding agreements
- DTC site evaluation frameworks
- DTC progress reports, process evaluations, and outcome evaluations3
- Publicly available material on the DTCs from the Internet.
Additionally, the Department is undertaking a recidivism study. However, it was unable to obtain the information required to complete this study prior to the conclusion of the evaluation.
3.2. Data Review
The data review was intended to provide information on program activity and outcomes. Several data sources supported this review including:
- DTCIS (Drug Treatment Court Information System) raw data. The Department created a DTCIS for the DTCs to use for case management and reporting purposes. The DTCIS is a Web-based system that includes numerous data entry screens. The system is designed to collect information about participants' demographics, treatment history, criminal record, program participation and referrals. While the intention is that both the court and treatment team will enter data, at this point only the treatment team is able to do so. To protect participant confidentiality, the Department does not have access to the DTCIS case management component. However, specific fields, stripped of personally identifiable information, are uploaded to the Department for performance management and reporting purposes. The evaluation included a review of an Excel-based extract of this data for all sites except for Vancouver.4 The time period that the data covers varies by site.
- DTCIS pre-set tables. The Department has developed several pre-set tables to monitor the progress of the program. At the time of the evaluation, individual sites were in the process of implementing the system and, therefore, the degree of information entered and available for each site varies.
3.3. Key Informant Interviews
Key informant interviews were used to collect information on the DTCFP's relevance, design and delivery, success and cost effectiveness. They also collected site-specific information for each DTC.
A list of potential key informants was reviewed and updated in consultation with the Director of each DTC. All key informants received the interview guide prior to the interview itself. The interviews were conducted over the phone or in person, in the respondent's preferred official language. Each interview took between 60 and 90 minutes to complete.
A total of 50 key informants participated in an interview. Their distribution is shown in Table 3.
| Court team | Treatment team | Other | Total | |
|---|---|---|---|---|
| Toronto | 5 | 3 | 8 | |
| Vancouver | 3 | 5 | 8 | |
| Edmonton | 4 | 3 | 7 | |
| Winnipeg | 4 | 3 | 1 | 8 |
| Ottawa | 5 | 4 | 1 | 10 |
| Regina | 3 | 5 | 8 | |
| Federal representative | 1 | 1 | ||
| Total | 24 | 23 | 3 | 50 |
3.4. Case Studies
Case studies provided information on participants' personal experiences with the program. In consultation with the Evaluation Advisory Committee, Winnipeg, Ottawa and Toronto were chosen as the case study sites.
The case study interviews were conducted in person at the DTC offices. Each interview took 30 to 45 minutes to complete. Participants had the option of ending the interview at any time. A total of 22 DTC participants took part in a case study as shown in Table 4.
| DTC | Active participants | Graduates | Did not complete | Total |
|---|---|---|---|---|
| Toronto | 3 | 3 | 2 | 8 |
| Winnipeg | 3 | 2 | 1 | 6 |
| Ottawa | 4 | 3 | 1 | 8 |
| Total | 10 | 8 | 4 | 22 |
Although case study participants were offered the opportunity to bring a trusted confidante, such as a family member, partner or close friend to the case study interview, none of the participants took advantage of this option.
During the case study site visits, the research team also observed a pre-court meeting and one weekly court session.
3.5. Stakeholder Survey
A survey with DTC stakeholders provided information on the relevance and success of the DTCs.
The survey targeted those who are directly involved in the DTCs. The sample included members of the Canadian Association of Drug Treatment Courts (CADTC), subscribers to the DTC Electronic Bulletin Board, partners of the funded DTCs, and stakeholders from the DTCs in Durham and Calgary. The list was reviewed and updated in consultation with the Directors of each DTC.
The survey was pretested with five respondents. Based on the results of the pretest, the wording of a couple of questions was clarified.
Following the pretest, survey packages were mailed to 238 respondents. They had the option of returning the survey either by mail, in a pre-addressed, postage-paid envelope included with the mailed questionnaire, or by toll-free fax. Telephone calls were made to non-responders to remind them to complete the survey and offer them the opportunity to complete the questionnaire over the phone. A total of 88 completed questionnaires were received, for a response rate of 37 percent. Table 5 shows the response rate by site.
| DTC | Number sent out | Number returned | Response rate |
|---|---|---|---|
| Funded DTCs | |||
| Edmonton | 30 | 19 | 63% |
| Winnipeg | 44 | 17 | 39% |
| Regina | 42 | 15 | 36% |
| Toronto | 48 | 16 | 33% |
| Vancouver | 29 | 9 | 31% |
| Ottawa* | 32 | 5 | 16% |
| Non-funded DTCs | |||
| Calgary | 5 | 3 | 60% |
| Durham | 8 | 4 | 50% |
| Total | 238 | 88 | 37% |
* The survey for Ottawa was in the field for a shorter period than the other sites due to delays encountered in obtaining the sample.
3.6. Limitations
A major limitation for the ability to report on outcomes is the relatively short time frame that most of the DTCs have been operational (approximately 2.5 years to 3.2 years). Although Toronto has operated since 1998, ongoing performance measurement began with the DTCFP and the DTCIS; therefore, long-term tracking of outcomes is not available for Toronto. Given that program duration varies in length between eight and eighteen months, sites that began accepting clients approximately three years ago have, so far, had little opportunity to produce graduates. Based on the site outcome evaluations, the number of graduates ranged from 21 in Winnipeg to 8 in Regina, making comparison of results between graduates and non-graduates or other reference groups premature.5 Moreover, graduates' time post-program may in many instances be quite short. As a result, it is premature to draw anything other than preliminary conclusions on outcomes such as reduced drug use and criminal activity.
The evaluation also experienced limitations with the available data. The evaluation had four main sources of information on outcomes: interviews, survey results, DTCIS data, and the individual DTC outcome evaluations. Ideally, these different sources would strengthen interpretations by corroborating findings across qualitative and quantitative data sources. However, the evaluation encountered several issues that limited the ability to use the data to determine outcomes.
The evaluation has not used DTCIS data due to issues with completeness and accuracy of the data.
- DTCIS pre-set tables. While the pre-set tables may provide the Department with a snapshot of the individual DTCs, their use for the evaluation is limited. The information does not allow for analysis by admitted applicants, participants who did not complete the program, and/or graduates. This type of analysis would enable the Department to determine similarities and differences between successful and unsuccessful applicants.
- Alignment of DTCIS pre-set tables and case management data. The evaluation found a number of inconsistencies between the case management data that was uploaded to the Department and the pre-set tables.
- Inconsistent DTCIS data across sites. Due to delays, DTCIS did not become operational in the DTC sites until the fall of 2008. This limited the time available for inputting data for the evaluation, and, as a result, the level of client information varies by site as does the time frame covered by the data.
The evaluation includes the results of the outcome evaluations that each individual DTC completed. The outcome evaluations reported on each of the DTCFP's intended outcomes including compliance with DTC program requirements, reduced drug use, reduced recidivism and improved social stability of participants. However, they differed greatly in the way in which these outcomes were defined and measured. As a result, cross-site comparisons and generalizations about the impact of the DTCFP are difficult to make.
The cost-effectiveness analysis of the DTCs is limited by the incomplete information available on DTC costs. The DTCIS data does not include cost information and the DTC outcome evaluations contain little information on cost effectiveness. Additionally, the DTCFP did not have financial statements for provincial costs associated with the DTCs as most of the funding recipients are NGOs. Recidivism data, which are important to show whether costs are offset by benefits, are also not available for all DTCs, and what is available is preliminary (see discussion in 4.3.1).
To respond to these limitations, the evaluation has gathered qualitative data on outcomes based on the experiences of DTC participants and the opinions of experienced criminal justice and treatment professionals who work in the DTCs. The evaluation has also included information from the literature on DTCs where relevant. In addition, although the site outcome evaluations do not provide comparable data, the evaluation draws out similarities and differences in their findings in order to ascertain whether there is a general direction of DTC effects across the sites, even if the measurement used is not consistent.
- Date modified: