3 Evaluation Methodology

The evaluation was guided by an evaluation matrix (evaluation questions, indicators, and data sources) which was developed through the evaluation scoping and design process. Appendix A contains the list of evaluation questions.

An EWG composed of ARADS representatives provided advice during the design and implementation of the evaluation. The Group coordinated access to documents and data, and provided feedback on evaluation products (i.e., the evaluation questions, preliminary findings and the draft evaluation report).

The evaluation included multiple lines of evidence and employed the data collection methods described in the table below. A detailed description of the methodology can be found in Appendix B.

Table 2: Lines of Evidence

Table 2: Lines of Evidence
Line of Evidence Description
Document Review The document review included a review of internal documents (e.g., ARADS administrative documents, progress reports, planning and policy documents, training documents, and advisory committee documents), as well as publicly available departmental and other government documents.
Administrative Data Review The administrative data review included information obtained through Justice Canada’s Integrated Financial and Material System.
Environmental Scan An environmental scan was undertaken with six comparable governmental initiatives operating in a similar context to ARADS. The objective was to investigate the activities, approaches, and organizational structures of similar initiatives in Canada. A total of nine individuals from the comparable initiatives were interviewed and documents were reviewed.
Key Informant Interviews A total of 41 Key Informants were interviewed for the evaluation. Including 35 Justice Canada representatives and six representatives from other government departments.

3.1 Limitations and Mitigation Strategies

Table 3 describes the main limitations related to the methodology, along with the mitigation strategies that were implemented. Overall, the evaluation process did not encounter constraints or limitations that prevented its ability to adequately address all evaluation issues and questions.

Table 3: Summary of Limitations and Mitigations Strategies

Table 3: Summary of Limitations and Mitigations Strategies
Line of Evidence Limitation Mitigation Strategy
Key Informant Interviews The key informant interviews had the possibility of introducing self-reported response bias and strategic response bias.Footnote 2 This risk was mitigated by using multiple lines of evidence, including objective sources of data, such as documents and administrative data, to arrive at the overall evaluation conclusions.
Key Informant Interviews Lack of knowledge of ARADS among some external interviewees (e.g., due to not being directly involved with ARADS or limited knowledge about ARADS activities). Interviewers focused on obtaining information around best practices, emerging trends and issues, and other relevant topics that these interviewees were able to speak to.
Environmental Scan Lack of publicly available environmental scan documentation. The evaluation team closely coordinated with the EWG to determine which organizations to include in the environmental scan. Additionally, during the interview process, the evaluators asked interviewees from the comparator organizations to share relevant documents.
All lines of evidence The role of ARADS has been evolving since its inception. As a result, it is possible that some interviewees outside ARADS could have been commenting on an outdated view of ARADS.
In addition, some changes were not well documented and the evaluators’ understanding depended on insights
from interviewees.		 The evaluation team integrated changes to ARADS in the data collection approach. When interviewees from outside ARADS commented on a feature, role or gap of ARADS, the evaluation team ensured those views were relevant given ARADS’ current roles, responsibilities and activities. The evaluators ensured that they were well aware of those changes so that the analysis and reporting would be accurate.