Evaluation of Litigation Services

3. Evaluation Methodology

To guide the evaluation, a methodology and evaluation matrix were developed that incorporated the flexibility of the TB Policy on Results (2016). The scope of the evaluation focussed on performance (effectiveness and efficiency) and key questions of interest to senior management, such as the impact of the changes made in 2016 to the litigation function. The evaluation included seven main performance questions, as summarized below.

Effectiveness

  1. To what extent has the Department provided high-quality litigation services?
  2. To what extent have the changes made to the litigation function since April 1, 2016 improved the effectiveness of litigation services?
  3. How effectively does the NLS (including CLS, regional offices, and National eDiscovery and Litigation Services) engage or collaborate with Justice staff (Portfolios, LSUs, other areas) in delivering litigation services? What, if any, changes are needed?
  4. Is the legal risk framework applied in an effective way to the litigation function?
  5. To what extent do litigation legal professionals have the expertise, tools, structures, and resources to support the delivery of litigation services?

Efficiency

  1. Are litigation services being managed and delivered efficiently?
  2. Are litigation services achieving appropriate resolution of litigation cases in a timely, cost-effective manner?

Four lines of evidence were used to address the evaluation questions: a review of program and administrative documents; key informant interviews; a survey of litigation counsel; and a review of administrative data.

A brief description follows and additional details can be found in Appendix C.

3.1 Document Review

The document review provided descriptive information on litigation-related activities, as well as information responding to most evaluation questions. The review was ongoing throughout the duration of the project and included the following types of documents:

  • administrative and internal program documents
  • publicly available departmental and other government documents
  • results for relevant questions from the Department of Justice Canada Client Feedback SurveyFootnote 11
  • results for relevant questions from the 2017 Public Service Employee Survey (PSES)Footnote 12

3.2 Key Informant Interviews

A total of 49 in-depth interviews were conducted with 122 individuals, representing the following groups:

  • NLS staff (23 interviews with 52 individuals; these key informants mainly represented management or senior counsel who serve on regional litigation committees)
  • Other areas of Justice that provide direct litigation services (5 interviews with 16 individuals)
  • LSUs (9 interviews with 27 individuals)
  • Justice Portfolios (6 interviews with 13 individuals)
  • Other Sectors (6 interviews with 14 individuals)

3.3 Administrative Data Review

Administrative data was obtained from Justice Canada’s Departmental Business Analytics System (i.e., Explore). Data was extracted from Explore’s Data Warehouse via Tableau, which includes data from iCase, Integrated Financial and Material System (IFMS), and Human Resources Management System (HRMS). Data was extracted between March and June of 2019. The data were analyzed using R statistical software and Excel.

The data review focussed on litigation files led by the NLS or by one of the other litigation units between FYs 2014-2015 and 2017-2018. The data review for the NLS considered volume by type of client, outcome data, legal risk and complexity ratings, and level of effort (hours).

The administrative data for the other litigation units was more focussed and was limited to the number of litigation files opened, closed, and actively managed. The TC LSU, due to the small number of litigation files handled, and the NSLAG LSU, due to the sensitive nature of their litigation files, were not included in the administrative data review.

3.4 Survey of Litigation Staff

The evaluation included an anonymous and confidential bilingual web-based survey of litigation staff to obtain their opinions on the effectiveness and efficiency of litigation services and the impacts of the NLS. The survey population included all litigation staff (counsel, notaries, paralegals, and legal assistants) within the NLS, as well as staff from other litigation units outside the NLS that were included in the evaluation.Footnote 13 In total, out of 1,615 staff who received the survey, 1,516 qualified to participate according to the selection criteria and 530 litigation staff completed the survey, for a response rate of 35%.Footnote 14 Survey percentages are reported based on the overall number of respondents who were able to respond to the question (i.e., excluding “don’t know,” “not applicable” responses).

3.5 Limitations, Challenges, and Mitigation Strategies

The evaluation encountered a few methodological limitations or challenges. They are briefly mentioned below by line of evidence and are discussed in more detail in Appendix C.

Table 1: Summary of Limitations, Challenges, and Mitigation Strategies
Line of Evidence Limitation or Challenge Mitigation Strategy
Key informant interviews Potential response biases from the sampling approach, the voluntary nature of participation, self-reporting, and the possible desire to affect outcomes Used multiple lines of evidence and triangulation to confirm results
Survey of litigation staff Potential response biases from the sampling approach, the voluntary nature of participation, self-reporting, and the possible desire to affect outcomes Census approach (all litigation staff were invited)
Used multiple lines of evidence and triangulation to confirm results
Respondents included several categories of staff (counsel, paralegals, legal assistants) Used complex skip logic
Relied on working group members to assist in identifying appropriate questions for each group
Administrative data review Ability to conduct a pre/post analysis with only two years before and after launch of NLS, due to the fact that many litigation files may take several years to conclude Limited use of this type of analysis for quantitative data
Also used qualitative methods for pre/post analysis
Changes in data fields during time period (FY 2014-2015 to 2017-2018) Relied on mapping of previous data to new fields Worked with NLS to ensure data were appropriately interpreted
Financial data is maintained for units within Justice Canada rather than by business line until FY 2018-2019 (e.g., litigation) Worked with Financial Management Services to obtain financial data for units that are now part of NLS over the full evaluation period, as well as construct financial data for litigation services (NLS and other units) over time
Date modified: