Aboriginal Justice Strategy Mid-Term Evaluation, Final Report

3. Methodology

The methodology used to conduct this mid-term evaluation is comprised of four main components: a review of documentation and administrative records, project file review, key informant interviews, and a survey of police and Crown representatives.

3.1. Review of Documentation and Administrative Records

A review of program-related documents was used to explore the clarity of AJS objectives and expected outcomes, as well as the roles and responsibilities of various stakeholders involved in the delivery of the AJS. The document review not only provided context for the interpretation of information collected through other methods, it also provided the means for determining whether the AJS, as delivered, is aligned with its original intended objectives and goals. Documentation reviewed included:

  • 2007 Results-based Management and Accountability Framework (RMAF);
  • Aboriginal Courtwork Program (ACW) Evaluation;
  • 2006 AJS Case Studies Report;
  • AJS 2005 Mid-Term Evaluation;
  • AJS 2007 Summative Evaluation;
  • financial data;
  • terms and conditions (grants and contributions);
  • one-year bilateral agreements;
  • one-year tripartite agreements;
  • operational guidelines and policies; and,
  • communications documents (e.g. meeting minutes, communications plan, Website information, etc.).

3.2. Project File Review

All Community-based Justice Programs and Capacity Building Funds funded in the 2007/08 and 2008/09 fiscal years were reviewed for the evaluation. A total of 232 project files were examined: 188 Community-based Justice Program files and 44 Capacity Building Fund files.[13] The review allowed for a comparison of project files and the objectives of the AJS's current mandate. This comparison provided information as to whether the AJS is targeting appropriate projects, the level of access and availability these programs provide (e.g., community sentencing, mediation, etc.) and the extent to which planned activities are implemented.

3.3. Key Informant Interviews

Sixty-nine semi-structured interviews were completed with 83 individuals representing three different stakeholder groups: Department of Justice staff (AJD and ALSP); government representatives from the provinces and territories; and individuals involved in the management/delivery of Community-based Justice Programs or Capacity Building Fund (funding recipients). Most interviews with provincial and territorial partners were completed during August and September 2009. Interviews with Department of Justice staff took place in September and October 2009, while interviews with funding recipients occurred between November 2009 and January 2010. The interviews with key informants were used to address all evaluation issues. The number of key informants interviewed in each stakeholder group is summarized in Table 2.[14]

Table 2: Distribution of key informants consulted
Key informant group Number of individuals consulted
Department of Justice Staff Members 19
Provincial/Territorial Partners 20
Funding Recipients 44
Total 83

3.4. Survey of Police and Crown Representatives

A Web-based survey was designed to explore the awareness of Community-based Justice Programs among Crown and police representatives. The survey was sent to all RCMP members nationwide, via a link provided in a RCMP electronic bulletin in September 2009. Crown counsel that work with the communities delivering a Community-based Justice Program were identified by the provincial/territorial partners and they received an email inviting them to participate. At the outset of the survey, respondents were screened for participation based on if they worked or had worked sometime since 2007 in or near a community delivering a Community-based Justice Program. Those who did not satisfy these criteria were thanked for their interest, but excluded from survey participation.

Summarized in Table 3 is the number of respondents to the police and Crown surveys respectively. The exact number of eligible participants is not known; therefore, response rates and sampling error cannot be calculated.[15]

Table 3: Number of Police and Crown Survey Respondents
Key informant group Number of individuals consulted
RCMP/Police 35
Crown 10
Unknown 1
Total 46

Date modified: