3. Evaluation methodology

The evaluation was guided by an evaluation matrix (evaluation questions, indicators, and data sources) which was developed through the evaluation scoping and design process. The methodology for this evaluation included multiple lines of evidence described below. Appendix A contains the list of evaluation questions, and a more detailed description of the evaluation methodology is included in Appendix B.

3.1 Key Informant Interviews

Interviews were conducted with 38 key informants between April and July 2022 including 20 PT representatives, 11 service providers representing NGOs across the country, as well as 7 Justice Canada Departmental representatives. Interviews were conducted via videoconference in a semi-structured format following an interview guide tailored to each respondent group. Key informants were provided with the interview guide in advance and given the option to participate in the interview in the Official Language of their choice.

3.2 Document and Data Review

A variety of documents and data were reviewed as part of the CFJF evaluation, including CFJF planning and reporting documents (e.g., PT and project funding contribution agreements and final reports), family justice related survey summaries and databases, as well as external research reports and statistics related to family justice needs and emerging trends. Survey data from two Justice Canada surveys were analyzed and summarized:

  1. the Parent Education Program Survey; and
  2. the Mediation Services Program Survey.

3.3 Case Studies

Three case studies were conducted on CFJF projects. Each case study addressed a different theme:

  1. Diverse and underserved populations;
  2. Implementation of Divorce Act amendments; and
  3. Supporting the well-being of family members engaging in the family justice system. Data collection for each case study included interviews with project stakeholders and a document and file review. Case studies were summarized in individual case study reports.

The projects selected for case studies are as follows:

3.4 Limitations, Challenges and Mitigation Strategies

The evaluation encountered a few methodological limitations or challenges.

Table 3: Summary of Limitations, Challenges and Mitigations Strategies
Line of Evidence Limitation or Challenge Mitigation Strategy
Key informant interviews and case studies Challenges included potential response biases from the sampling approach (selective, non-random), the voluntary nature of participation, and self-reporting (reporting on own activities). The evaluation used multiple lines of evidence and triangulation to confirm results.
Document and data review There is limited primary quantitative and varying levels of qualitative information available on program activities. Further, the timing and format in which final reports are received from PTs and NGOs varies. In addition, results were not always reported in a consistent manner across different jurisdictions. The collaboration of PTs and Justice Canada representatives to share information and data was crucial to the successful completion of this evaluation. Outcome information was based heavily, though not exclusively, on qualitative data from key informant interviews and file and document reviews. Qualitative data was complemented with quantitative data, where available, on how projects increased access to services, participant perspectives on the effectiveness of services (i.e., through exit surveys), and other statistical data and research describing the need for activities/projects funded through the CFJF.
Document and data review Due to the timing of the evaluation and lag in how data is collected and published (i.e., by Statistics Canada), limited data was available on the full impact of the COVID-19 pandemic on various indicators such as divorce rates, rates of family violence, and other indicators. The evaluation relied on interview perspectives from service providers and existing available research on COVID-19 impacts on separation and divorce to infer possible impacts on how the pandemic has changed the needs for and delivery of family justice services.