Annex A – Methodology

Background

The CFS has concluded its fifth cycle, with the first cycle having taken place from 2006 to 2009, the second from 2009 to 2012, the third from 2016 to 2019, and the fourth from 2020 to 2022. Each survey cycle concludes with the publishing of a report that includes survey findings for all five Justice Canada portfolios, as well as results from the Privy Counsel Office Legal Services Sector.Footnote 14 In addition, aggregate reports are created for the National Litigation Sector (NLS), the Public Law and Legislative Services Sector (PLLSS), and the Centre for Labour and Employment Law (CLEL). In advance of initiating Cycle V, the Department established the CFSVWG to fine-tune the CFS project and make improvements where possible. As a result, the CFS has evolved since the last cycle and continues to generate feedback from clients on the Department’s performance against its Standardized Legal Service Agreements.

With the guidance of the Modern Statistical Methods and Data Science Branch at Statistics Canada, the Department developed a standardized questionnaire and methodology for collecting client feedback on the degree to which the delivery of legal services is meeting the needs and expectations of legal service users. Over the years, Statistics Canada has played an important role by reviewing and challenging the proposed approach throughout the survey design and implementation stages, vetting the analyses of survey data and reviewing and commenting on the presentation of findings contained in CFS reports.

Survey Administration

The survey was administered via a web-based questionnaire. In total, 71,876 invitations to complete the questionnaire were sent to potential users of the Department’s legal services. Of this population, 6,831 service users (out of 19,551 respondents) reported having used JUS legal services in the 12 months prior to the survey.

For this cycle, mailing lists consisting of potential service users were developed using two methods: the traditional census method and the LEX targeted pilot method. For the traditional method, invitations to participate in the CFS continued to be sent to all employees at the EX-minus-1 level and above in the NCR and the EX-minus-2 level and above in the regions, as has been done in past survey cycles. For the LEX pilot method, survey invitations were extended to all client contacts identified by the Department’s legal case management system regardless of their occupational group and level. Contacts identified by both methods only received one survey.

Interpreting the Results

The survey collected feedback, in the form of satisfaction ratings, from clients using a 10-point Likert scaleFootnote 15 with two anchors: not at all satisfied (1) and completely satisfied (10). Feedback was sought along three key dimensions of service quality as per the Department’s Service Standards for the Provision of Legal Services to Government (see Annex G): responsiveness, timeliness, and usefulness. Each service dimension is composed of individual elements pertaining to client satisfaction, many of which relate directly to the Department’s Service Standards. Furthermore, service users were asked to rate their level of satisfaction with the overall quality of legal servicesFootnote 16 and Overall Considerations elements.

In reviewing the results presented throughout this report, it is important to note that survey ratings represent estimates of client population perceptions of service delivery by the Department. As such, the margin of error must be considered. The margin of error traditionally reflects the sample-to-sample variability in the use of a sampling methodology. The magnitude of the margin of error is generally affected by the extent of variability in respondent feedback, the overall size of the respondent group and the confidence level chosen by the survey team.

Results in the report are presented in the form of rating (± margin of error). This range of values is called the confidence interval and for the purpose of the CFS, a 95% confidence intervalFootnote 17 is used. As an example, in this report, client satisfaction with the overall quality of litigation services is presented as 8.5 (±0.1), which implies that the 95% confidence interval for the mean rating of the overall quality of litigation services obtained from this survey for this population is from 8.4 to 8.6.

The CFS has traditionally used a census approachFootnote 18 in which invitations to participate in the survey are sent to all potential users of legal services. This approach was chosen for the CFS largely because departmental rosters, including the Department’s legal case management system, are limited in terms of identifying all users of legal services and because with this approach, potential sources of error associated with sampling can be avoided. In this case of no sampling, margins of error account for variability related to non-response to the invitation to complete the questionnaire. That is, the respondents to the CFS are treated like a random sample from all potential users of the legal services, assuming that the respondents were representative of the population of interest, which is all potential users of the legal services. Had all potential users responded to the survey, there would have been no variability, and the margins of error would have all been zero, as all ratings/perceptions would have been accounted for. The Finite Population Correction (FPC) Factor was also applied as part of the calculation of margins of error to take into account the sizes of the number of potential users and number of survey respondents.

Lastly, to compare ratings between current and past surveys, as well as various categories of service users, t-tests for pairs of independent samples were undertaken. All t-tests conducted were based on the null hypothesis of equality of two mean ratings against the alternative hypothesis of mean ratings not being equal. In other words, by identifying which of the two hypotheses one fails to reject, one can determine whether the difference between the two mean ratings is statistically significant.