Evaluation of the Department of Justice Professional Development Function

4. Key Findings

This section presents a summary of important findings from the evaluation of the Department’s PD function. Because this is the first evaluation of the function, some of the findings are in the form of basic details about the nature and extent of PD availability and usage, mainly in the 2016-17 fiscal year. Despite the limitations regarding the data, the triangulated approach of using multiple lines of evidence has helped mitigate the concerns. The Effectiveness section begins by examining the extent to which the PD function at the Department is achieving its objectives in relation to having meaningful impact, meeting employees’ PD needs, addressing barriers to accessing training, and aligning with departmental and governmental priorities. The Efficiency section examines governance, coordination, and whether resources are being used efficiently.

4.1. Effectiveness

4.1.1. Impact of Professional Development

Justice Canada employees report generally positive impacts of PD.

There is no systematic process in place to track impacts of all PD, particularly in the long term. For example, methods to draw findings on effectiveness should include assessments from participants and their managers over time, or finding correlations between participation in training and annual performance assessments, and job advancement. CLEP and LPD request that employees complete a survey following completion of training, but the information is not tracked or analyzed in a systematic or coordinated process with the main delivery providers, portfolios, sectors and regional offices.

That being said, the evidence appears to suggest that PD training at the Department is having positive effects. The online employee survey examined the perceptions of departmental employees about the impacts of the PD they received in 2016-17. Given the training they had received, respondents were asked to what extent they agreed with the following statements. These statements are indicators for measuring impacts.

  1. My knowledge and understanding in this area have increased.
  2. I have been able to apply the knowledge and/or skills in my work.
  3. This training had led to positive changes in my work (e.g., job responsibilities, confidence, career advancement, job effectiveness, etc.).

Employees were asked about the impacts of PD in these three areas depending on the training provider or the category of training. The following key results can be observed from employees’ responses (detailed charts with percentage by internal, external and CSPS training can be found in Appendix B):

4.1.2. Meeting the Needs of Employees

The majority of Justice Canada employees report that they get the training they need, although some barriers to accessibility were noted, including budget and operational constraints. Challenges in delivering training tailored to regional realities and needs were also noted, including limits to virtual participation related to technological issues and time zone differences.

The positive impact of PD noted above is corroborated by the most recent Public Service Employee Survey (2017) data in which 73% of Justice Canada employees indicated they “get the training they need to do their job” (strongly agree and somewhat agree). The results for the Department are also higher than the Public Service generally (66%). However, the percentage of Justice Canada employees reporting that they get the training they need to do their job has fluctuated in past years from 76% in 2011 to 68% in 2014.

Respondents to the evaluation online survey reported a moderate level of satisfaction with the internal training offered in 2016-17 (see Figure 2 below). Responses suggest that potential improvements could be made to satisfaction with internal training, particularly in terms of the relevance of training topics, and the extent of non-legal training activities offered. It was also reported that internal departmental courses may be cancelled due to low enrollment, which may occur more frequently in the case of French language courses. Finally, 53% of respondents to the evaluation online survey reported that they needed more PD than they received.

Figure 2 : Professional Development Survey - Level of Satisfaction with Internal Training

Professional Development Survey - Level of Satisfaction with Internal Training described below

Text version - Professional Development Survey - Level of Satisfaction with Internal Training

The bar graph shows the average ratings of satisfaction when survey respondents were asked their level of satisfaction with internal training:

  • The variety/type of training available to me (n=959) with 38% Satisfied, 33% Neutral, and 29% Dissatisfied
  • The frequency of training activities (n=958) with Satisfied (38%), Neutral (35%), and 28% Dissatisfied
  • The relevance of training topics to my work (n=958) with Satisfied (34%), Neutral (31%), and 34% Dissatisfied
  • The extent of non-legal activities offered to me (n=958) with Satisfied (32%), Neutral (35%), and 33% Dissatisfied
  • The extent of legal activities offered to me (n=958) with Satisfied (37%), Neutral (36%), and 27% Dissatisfied

Despite the moderate levels of satisfaction, there is currently no systematic needs assessment conducted by the Department in regards to gaps in PD. For example, portfolios generally conduct internal consultations, surveys and research to identify training gaps. Some of these needs have already been identified in many of the Department’s legal services evaluations. These evaluations have identified a need for more advanced and specialized legal courses, and recommended improving training opportunities for counsel to better meet their learning needs.Footnote 10

Generally, key informants reported that employees have access to a large variety of training and PD activities. However, respondents also reported some barriers to accessing training. As shown in Figure 3 below, respondents to the online survey highlighted internal and external barriers of budget constraints, operational constraints, dates and times for training, delivery methods, technological barriers, and training locations that require travel. The most widely reported barrier of access to external training was budget constraints, while operational constraints were seen as the biggest barrier to internal training. Key informants mirrored some of the challenges of the online survey by citing logistical issues, such as IT and connectivity, the availability and the limited size of meeting spaces, time zone differences, as well as budget constraints and the limited time available of in-house legal experts.

Figure 3 : Barriers in Accessing External and Internal Professional Development Activities

Barriers in Accessing External and Internal Professional Development Activities described below

Text version - Barriers in Accessing External and Internal Professional Development Activities

The bar graph shows the percentage of how many respondents selected had experienced different categories of barriers in accessing external and internal PD activities:

  • Budget constraints - 43% Internal and 85.6% External
  • Operational Constraints - 53.2% Internal and 40% External
  • Data/Time training not suitable - 37.7% Internal and 22.5% External
  • Delivery method not suitable - 27.2% Internal and 13.3% External
  • Training location not suitable - 31.9% Internal and 42.7% External
  • Other - 33.6% Internal and 14% External
Regional Barriers

Key informants interviewed also highlighted barriers to access to PD in the regions. According to the Justice Learning Policy (2017), managers and supervisors are required to use a set of common criteria as a guide to ensure fair, objective and transparent treatment of employees, and to establish a consistent approach across the Department to justify decisions when selecting employee participation in learning activities. However, it was observed among key informants that there is a lack of integration of the regions in the PD framework; consequently, training offered by HQ is not tailored to regional realities and needs. There are limits to virtual participation because technology platforms are not established within the Department to fully support distance or repurposed learning materials, which may make regional employees reluctant to participate. Technological issues, such as video connectivity, and the difficulty of time zone differences, as noted in the above paragraph, were emphasized as challenges for the regions. Furthermore, there is no strategy to match content with the optimal learning channel.

4.1.3. Alignment with Priorities

The current approach to planning and alignment with departmental and governmental priorities is primarily informal. There is no systematic strategic approach to planning and priority setting at a departmental level.

A review of relevant documents and interviews with individuals in the Department indicate that while there are mechanisms to provide PD in governmental and departmental priority areas, these are ad hoc in nature and not systematic. Key informants stated that learning needs of employees as well as departmental and federal priorities are key drivers in training development. Key informants noted that there is a common understanding that training should align with priorities, and one way that PD aligns with priorities is through mandatory training offered by the Department. However, there is generally no strategic planning to ensure that overall departmental training aligns with high level priorities. This need is outlined in the Department’s Human Resources Management Plan 2017-2020, which encouraged the development of “a departmental learning strategy and plan for business-specific learning priorities in alignment with the CSPS enterprise-wide approach”.

The Department recognizes the importance of legal training for the development of knowledge and competencies, and the maintenance of excellence in the delivery of legal services to the federal government. In order to help the Department’s legal community meet its continuing PD obligations, CLEP ensures that Justice Canada legal learning activities are accredited with the relevant law societies. The Program also seeks accreditation with law societies, where applicable, for LPD and CSPS training. Requirements around accreditation are always considered; however, a more strategic approach could be beneficial in the development of content, the delivery of activities and the sharing of courses amongst the main delivery providers, sectors, portfolios and regional offices in the Department.

Beyond what is driven by law society requirements or required across the federal government, PD at Justice Canada is also directed by individual employee and manager preferences. The Learning and Development Plan is a tool embedded in performance agreement templates for all Justice Canada employees. The Plan identifies learning objectives relevant for the employee and support for meeting those objectives. However, PD decisions do not reference a departmental strategic plan.

4.2. Efficiency

The evaluation examined a number of measures related to the management of the PD function at the Department, focusing in particular on the delivery model and governance, as well as the cost effectiveness of PD development and delivery. The evaluation sought to identify challenges the Department faces in effective and efficient development and delivery of the PD function, and to identify any best practices that might help guide future planning.

4.2.1. Governance and Coordination

There is no overarching governance of the PD function that encompasses the departmental delivery providers, portfolios, sectors and regional offices. Although there is some communication of PD activities across these areas, this could be increased to improve efficiency and facilitate sharing of training resources and activities. Roles and responsibilities were not always clear.

To a great extent, PD is decentralized in the Department. Although there are two strong loci of planning and delivery with central functions (CLEP and LPD), training is also developed and delivered by CSPS, individual portfolios, sectors and regional offices, and external sources such as the law societies. The absence of an overarching corporate approach and strategy to training has resulted in a lack of coordination of PD activities across the Department. The current approach exposes the Department to the risk of inefficiencies and potential duplication of services. Some key informants believed there was little overlap, although others highlighted potential issues such as overlap across portfolios, sectors and regional offices, overlap between CSPS and other courses in the Department, and overlap related to administration and coordination of training between CLEP and LPD. In addition, their mandates are unclear for clients and they are confused as to what support each can offer in terms of in-house PD activities.

Planning occurs generally at the corporate, portfolio, sector and regional office level. The majority of key informants noted that there are unclear roles and responsibilities, and insufficient departmental-level planning. For example, although CLEP has a role to provide legal training, a number of key informants noted that individual portfolios, sectors and regional offices are responsible for identifying their own needs and developing curriculum. However, it was also noted that some ad hoc coordination takes place among portfolios, sectors and regional offices as well as CLEP or LPD to deliver training (e.g., invitations to employees in other portfolios to attend training that may be relevant to them).

Furthermore, regional roles and responsibilities, as well as the approval process for the planning and delivery of PD, are unclear. Some respondents indicated that approval was the responsibility of the Regional Director Generals, along with their management teams. In some regions, there is an active Training Development Committee of lawyers and paralegals. Regions may consider departmental priorities, but usually training is operational in nature and focuses on the provision of high-quality legal services. Training tends to be more practical, dealing with day-to-day practices, meeting regionally identified needs to fill knowledge/skills gaps and law society requirements.

Despite some challenges, the Department has made efforts to coordinate planning and delivery of training. Some examples of best practices were identified below by key informants for the evaluation. The impacts of these practices may be enhanced by sharing information on the content or processes of these initiatives across the Department.

4.2.1.1. Recent Initiatives in Governance

Documents shared by CLEP and LPD indicate that steps are being taken to promote greater coordination in training development and delivery across the Department. For example, a departmental Learning Steering Committee is being proposed that could provide oversight and high-level guidance for the development and implementation of departmental learning programs and activities for all employees. The mandate of the proposed committee would be to ensure the delivery of an efficient, cost-effective and coordinated national approach to learning. In addition, a Directive on Continuing Legal Learning in the Department of Justice is under development.

A key ingredient of the proposed enhancements would be much-improved communication across the Department, so that the main delivery providers, portfolios, sectors and regional officers would know what was happening at each level in order to help consolidate training needs and availability, initiate training to fill gaps, and maximize sharing of resources. The recent reorganization of CLEP’s structure in 2017 may assist to facilitate this collaboration. The reorganization includes a review of processes, including the creation of detailed roles and responsibilities for the development and coordination of legal learning across Justice Canada legal portfolios to promote better coordination of training activities.

4.2.2. Efficient Use of Resources

The absence of a coordinated approach to drive alignment of limited resources with departmental and governmental needs and priorities may limit efforts to deliver training to Justice Canada employees in a cost-effective manner.

As highlighted in the Limitations section, examining and comparing costs for PD at the Department is challenging. Due to the way that financial systems record training-related expenditures, combined with the uncertainty regarding the completeness and consistency of training activity records, it would be extremely difficult to conduct a meaningful financial analysis on a cost comparison between CSPS, external and internal training. As a result of the governance structure discussed above, there is a lack of central oversight to drive alignment of expenditures with priorities and needs, and to develop and communicate best practices in recording PD activities and associated financial coding.

Table 1 presents overall Operations and Maintenance expenditures for employee PD.Footnote 11 These expenditures include only the cost of the courses, registration fees, conferences, training equipment and supplies, travel for training, and associated logistical expenses. Expenditures for language training have been excluded. As illustrated by Table 1, expenditures for internal and external training courses have decreased in FY 2016-17, although Table 2 demonstrates that the flat rate transferred to CSPS annually has increased over time. These expenditure shifts align with the new role of the School beginning in 2014-15 as the common core learning provider. Additionally, the payment to CSPS based on courses taken has declined as the change to the flat rate transfer was gradually implemented.

Table 1 : Operations and Maintenance Expenditures for Employee Professional Development
Fiscal Year 2012-13 2013-14 2014-15 2015-16 2016-17

Internal and external courses (excluding language training)

$3,744,443

$3,097,399

$3,788,035

$3,677,680

$3,420,697

CSPS (based on payment per course taken)

$262,186

$223,847

$226,702

$92,456

--Footnote 12

Table 2 presents the A-base funding transferred to CSPS to provide training. The flat rate transferred to the School annually is calculated based on the number of departmental employees, and the reference level has been adjusted to reflect an annual transfer of $1,015,450 to CSPS on an ongoing basis beginning in 2016-17.

Table 2 : A-Base Funding Transfer to the Canada School of Public Service
Fiscal Year 2012-13 2013-14 2014-15 2015-16 2016-17

Canada School of Public Service (based on reference level)

n/a

n/a

$242,825

$485,650

$1,015,450

According to CSPS, data for 2016-17 shows that 56.6% of Justice Canada employees accessed CSPS products, which is lower than the participation rate for the public service more generally at 72%.Footnote 13 The report also notes that one of CSPS measures of success is the extent to which employees take courses related to the four main government priorities, which include healthy and respectful workplaces, official languages, service excellence, and values and ethics. Their numbers show that 13.7% of departmental employees have accessed learning activities at the School related to one of the identified priorities offered by the School, while 22.9% of Public Service-wide employees have. The report does not identify why Justice Canada employees may have a lower participation rate. However, the lower rates may be influenced by the fact that CSPS does not offer legal training coursesFootnote 14 and the majority of Justice Canada employees are legal professionals.

Based on available data, it is not clear if the Department is using CSPS courses to the fullest extent possible. It will be important for CSPS courses to be used strategically as part of an overall departmental training strategy in order to make the most of the training dollars that the Department will transfer to the School annually on an ongoing basis, as part of the Government of Canada’s enterprise-wide commitment to learning with CSPS.

As noted in the section on Governance and Coordination, the decentralized nature of PD poses a real risk of overlap and budgetary inefficiencies, if there is insufficient coordination and communication on the availability and prioritization of PD activities. Key informants indicated there may be some duplication in the types of PD activities offered amongst portfolios, sectors and regional offices, and between the Department and CSPS. The extent of duplication is difficult to determine due to the decentralized manner in which PD is organized, delivered and tracked. Budget constraints on training, noted by key informants and many respondents to the evaluation survey, highlight the importance of departmental PD priority setting so that the limited available resources are directed to critical needs. Strategic planning and oversight may help to keep managers informed of all available courses and maximize the cost effectiveness of courses.

However, throughout the evaluation, several efforts to deliver training in a cost-effective manner within the Department were also highlighted. CLEP and LPD have a weekly stand-up meeting to discuss operations to ensure coordination. In addition, some coordination exists between portfolios with regard to delivering legal training (i.e. invitations to employees in other portfolios to attend legal training that may be relevant to them), although this could be expanded. Some examples of cost-effective solutions include the use of WebEx videoconferencing, the expansion of online training and supporting technologies, the use of internal expertise to design and deliver training, and the pooling of resources with the Canadian Bar Association and provincial justice departments.