Evaluation of the Department of Justice Professional Development Function

3. Evaluation Methodology

The evaluation was conducted using a combination of qualitative and quantitative methods.

3.1. Document and Data Review

A systematic review of relevant information related to PD was conducted. The document and data review provided descriptive information, informed the development of data collection tools and addressed the evaluation questions in the Evaluation Matrix (see Appendix A). As part of the evaluation, data and reports from iCase (e.g., number of hours legal professionals spend on training) and PeopleSoft Training Self-Service (e.g., number and types of courses registered) were also reviewed. Only data for indeterminate employees or those who were with the Department for more than six months were examined in the evaluation.

3.2. Key informant Interviews

Key informant interviews were conducted to provide descriptive information on the following: the governance structure for the PD function; perceptions of the delivery model for PD in the Department more generally; and whether training needs were being met. A total of 34 interviews were conducted with 37 individuals, including individuals responsible for developing, delivering and/or coordinating PD within the Department, and regional and portfolio/sector representatives dedicated to the PD function. These stakeholder groups included the Management and Chief Financial Officer Sector, Legal Practices Sector, National Litigation Sector (NLS) (including regional representation), and various Justice Canada Legal Portfolios. Interviews were conducted by telephone and in person.

3.3. Online Survey

An online survey was administered to all departmental employees with indeterminate or term status over six months to gauge the extent to which PD needs were being met and whether any gaps existed. The survey also examined whether there were barriers to accessing PD activities and applying learning on the job. A total of 940 respondents completed the survey questionnaire with representation from all six regions. The completed surveys represent about 22% of 4,341 Justice Canada employees. Those working in the Law Occupational Group (LP and LCFootnote 6) represented 45.5% of respondents to the survey. These survey demographics slightly underrepresent the actual total of LP and LC employees (69%) who work in the Department.Footnote 7

3.4. Limitations

Challenges were noted with regards to the availability, consistency and accuracy of existing data, such as the nature and extent of training undertaken by individual employees, expenditures on training by type/subject matter, and about Department-wide training needs. As a result, examining and comparing data across these different sources (i.e., PeopleSoft, iCase, survey data and financial data) was challenging.

Data drawn from PeopleSoft may be unreliable. First, it is uncertain whether all portfolios, sectors and regional offices track actual course participation in PeopleSoft, or simply registrations. Key informants suggested that the registration figures may overstate participation. For example, some may register in a course and not attend, thereby compromising assessments of numbers and types of courses taken. There are also several training administrators across various sectors with the authority to create course codes in PeopleSoft, leading to challenges in recording and reporting information. Conversely, there may also be an issue of understating registration and participation numbers. Some PD activities are less formal in nature and may not require registration, be developed independently by portfolio, sector and regional managers, or not necessarily be recorded formally in PeopleSoft.Footnote 8 The system relies on individual employees registering their activities in the system, but it is not clear whether they do so in an accurate fashion.

There may also be confusion in the process for registering for CSPS courses on PeopleSoft, leading to underrepresentation of numbers. Comparison of the numbers and percentages of CSPS registrations in PeopleSoft, the evaluation survey, and a summary report from CSPS of departmental course registrations suggest that the School courses are not always recorded in PeopleSoft. Similarly, iCase data documenting time spent attending PD activities on the part of (mainly) legal professionals (counsel, notaries and paralegals) is also subject to inaccuracies because of its reliance on individual recording of time and different ways of capturing time. Some legal professionals, including most legal managers at the LC level and some at the LP level, have no obligation to time keep as they are excluded. This data is limited by virtue of not including the majority of other professionals.

In order to supplement existing data, an online survey was conducted and 940 employees responded. Although this represents only approximately 22% of all Justice Canada employees, this level of response keeps with modern norms for internal surveys, given employee time demands and “survey burnout”. A breakdown of demographic information from respondents indicates participation from employees in both legal and other positions, and displays frequencies roughly in line with employee distributions in areas such as gender and location.

Examining and comparing financial expenditures for different types of PD at the Department are challenging. The coding of LPD, CLEP and CSPS training in PeopleSoft and iCase may be inaccurately or inconsistently reported by employees, and there are a variety of ways to record the time. The ability to conduct reliable financial comparisons is limited because PD costs are often grouped in an “other” category in budgets. As a result, it is difficult to obtain detailed breakdowns of costs for various types of PD across different areas of the Department. Furthermore, for the purposes of this evaluation, we did not have complete access to the costs of PD and its management initiated across portfolios, sectors and regional offices.