3. Evaluation Methodology

• Previous Page
• Table of Contents
• Next Page

The evaluation was guided by an evaluation matrix (evaluation questions, indicators, and data sources) which was developed during the evaluation scoping and design phase. It included multiple lines of evidence and employed the data collection methods described below. Appendix A contains a list of the evaluation questions.

3.1 Document, Data and Literature Review

The document and data review provided descriptive information on OIRSO’s activities; information regarding the need for OIRSO, including whether it represents a legitimate and necessary role for government; best practices which may represent potential improvements to design and delivery; and benefits of informal conflict resolution services. The review was ongoing throughout the evaluation conduct phase and included the following types of documents:

• Literature – peer-reviewed and grey literature related to informal conflict resolution, including trends and emerging issues and research on best practices.
• Documents and data – OIRSO foundational and implementation documents, performance information, OIRSO annual reports, and publicly available policies and reports.

3.2 Key Informant Interviews

A total of 17 interviews were conducted with OIRSO staff, partners as well as internal and external stakeholders. Internal stakeholders and partners included representatives from senior management, the Human Resources Branch, Occupational Health and Safety, Workplace Harassment and Violence Prevention and the Anti-Racism and Anti-Discrimination Secretariat. External stakeholders included union representatives and other government departments.

The following is a breakdown of the interviews by group:

• OIRSO staff (n=3);
• Internal partners and stakeholders (n=9); and
• External stakeholders (n=5).

3.3 OIRSO Evaluation Survey of Departmental Employees

A department-wide survey of Justice Canada employees (OIRSO Evaluation Survey) was conducted as part of the evaluation to obtain information on awareness and use of OIRSO services.

A multi-media approach was used to communicate the launch of the survey. An e-mail letter, in both official languages, was distributed to approximately 5,365 Justice Canada employees across Canada inviting them to participate in an anonymous online survey. Notifications and reminders were also posted on info screens in Justice Canada buildings, JUSnet, the departmental intranet site, and JUS Your Weekly Update encouraging employees to participate. A total of 327 employees responded to the survey (6% response rate).

3.4 Case Studies

Five case studies were conducted to understand an OIRSO client’s journey through the ICMS and to document challenges, successes, and improvements that could be made to specific processes. The case studies were selected in collaboration with the Evaluation Working Group and were validated by the case study participants. The experience with various OIRSO activities and case complexity were criteria considered in the selection process.

3.5 Process Maps

Process maps were completed to identify the complexity and level of effort associated with the formal grievance process to compare the efficiency of the formal grievance process with the informal process.

3.6 Limitations and Mitigation Strategies

Limited data available to conduct a cost savings analysis

Assessing the cost efficiency of informal conflict resolution services is complex and the approach undertaken in this evaluation evolved based on the availability of data. The original intent of the efficiency analysis was to assess the cost efficiency of the informal conflict resolution services provided by OIRSO, while demonstrating any benefits of the informal system and value for money associated with OIRSO. The approach was to determine an estimate of costs associated with OIRSO services, an estimate of those associated with the formal conflict resolution process, comparing the two and developing a modelling scenario that would project cost-saving estimates over time. It is recognized that both processes serve a purpose and that they are not always interchangeable.

However, activity data (such as volume of cases, time spent and level of effort) across the various parties involved in formal conflict resolution processes is not consistently tracked within the Department during the years covered in this evaluation. Therefore, the approach to the costing analysis shifted to developing scenarios of possible cases in the formal process and creating estimated proxy cost information for the level of effort required using examples of possible scenarios. Similarly, for the informal process managed by OIRSO, costing was to be completed on the case studies of clients.

Process maps were created for each of the scenarios in the formal process assisted by consultations with the Labour Relations and Compensation Division and Corporate Human Resource Planning, Programs and Systems, but information on costing was not available to facilitate the cost savings analysis. As a result, assessing efficiency relied on calculations made to assess the costs per OIRSO activity over time (i.e., since its inception).

Limited data available on OIRSO clients

While the evaluation considered anonymized client satisfaction data that OIRSO collects, performance data was unavailable for analytical purposes due to confidentiality concerns. As a result, the Evaluation Branch was limited to information that is shared publicly in OIRSO’s annual reports to protect confidentiality, and the OIRSO Evaluation survey was used to collect anonymous information from those employees who had used OIRSO services.

Potential Respondent Bias

Key informant interviews may introduce response bias due to their vested interest in the program. Also, since the list of interviewees was provided by OIRSO, response bias may be introduced by providing partners and stakeholders who might have more positive experiences and interactions with OIRSO. To mitigate this issue, the Evaluation Branch implemented several data collection tools and relied on various lines of evidence to verify findings against other sources and perspectives.

Potential non-response error in the OIRSO Evaluation Survey

Given the self-selected nature of the survey and a response rate of 6%, there is concern that the characteristics of the respondents may be different from those who did not respond to the survey. There is a possibility that individuals with less to say, who had a lack of awareness of OIRSO or have primarily negative feedback might have opted not to participate. Moreover, the survey was only available to current Justice Canada employees, and there could have been some employees who had experience with OIRSO between its inception and the time of the survey but who were no longer with the Department.

The table on the following page (Table 2) compares the characteristics of those who responded to the survey to the total population of departmental employees in 2023 as reported by the Treasury Board of Canada Secretariat.^{Footnote 1} To assess the nature of the potential non-response error, the characteristics of the respondents were compared to the overall profile of the Department’s employees. As indicated below, males were slightly underrepresented while females were slightly overrepresented. Visible minorities were slightly underrepresented while persons with disabilities appear to be over-represented; however, this may be due to differences in how disabilities are categorized. Law practitioners were also underrepresented while most other occupational groups were slightly overrepresented with a few exceptions.

• Previous Page
• Table of Contents
• Next Page