Project Managers' Guide to Performance Measurement and Evaluation


9.1 Laying the Groundwork

Notwithstanding the challenges, realistic, thorough and timely planning helps to ensure that the benefits of performance measurement and evaluation are realized. Developing a Logic Model and Performance Indicators are key tasks that help to focus the Family Violence Initiative and related activities. These tools set the boundaries of your accountability and serve as a ‘touchstone’ for mapping your progress. Determining how and when you will collect and report information – as well as the resources that you will need to do this - is the next part of the performance measurement and evaluation strategy.

Laying the Groundwork – First Steps

The Family Violence Initiative has a long history of working together. When you prepare your annual workplan, it is also a good time to plan and coordinate performance measurement and evaluation activities with your colleagues in the Family Violence Initiative. Coordinated planning will increase the likelihood of success – and is particularly critical when there are partners involved.

Collaborative Approaches – A Unique Challenge

In many cases, the Department of Justice Canada is one of several contributing partners in a project. It such cases, it is important to ensure that all partners are clear on the expectations of the project. Ensure that all partners agree on the evaluation plan and on who will be responsible for the evaluation plan and its implementation.

Some performance measurement and evaluation questions to consider during the FVI workplan development include:

Considerations – Policy Activities

Policy activities pose challenges for performance measurement and evaluation. Policies may be tied to multiple, high-level and distant outcomes. The policy development process itself is subjective and iterative in nature. Time frames are uncertain and ever changing. Problems with measurement, causation and attribution also exist. Yet policies influence and even drive change processes. For example, changes to the Criminal Code related to stalking and criminal harassment and victim protection legislation have had a significant impact on improving the criminal justice response to family violence issues. Knowing what’s working well and what requires improvement is critical.

Sound Familiar?

“The causal gap creates uncertainty on two levels. First, no matter how good your planning and analysis is, you can rarely be certain that today’s policy inputs and outputs will in fact lead to the desired social and economic outcomes. Most of the high-level outcomes targeted by government policies are affected by many factors outside of the government’s control….even if the desired outcomes occur, we can rarely be certain that the government’s intervention (as opposed to other unrelated factors) was the primary cause.”[3]

Tools such as the logic model and performance indicators help to develop a picture of how policies are expected to influence change. Monitoring the policy development process through tools such as policy reviews, scans and opinion research are potential strategies. Policy research and evaluation – along with program development and testing – are ways that cumulatively, help to measure the effectiveness of policies.

Considerations – Research

Collecting performance measurement and evaluation information on the Family Violence Initiative research function is also challenging. Some ideas to consider include:

Considerations – Projects: Justice Partnership and Innovation Fund – Programs and Public Legal Education

Project-level evaluations are one of the key building blocks of the performance measurement and evaluation strategy. The Justice Partnership and Innovation Fund requires proposers to prepare an evaluation plan as part of proposal development. Typically, proposals should include an overview of the project; its target audiences; expected outcomes; indicators of success/impact; data sources and methods; who will conduct the evaluation; how stakeholders and partners will be involved; the time frame, and how the evaluation information will be used. Logic Models – or project “roadmaps” are increasingly important tools.

What is a Logic Model?

A logic model is a picture of how a program or project works – including the theory and assumptions behind it. Where feasible, proposers should be encouraged to learn about and apply this tool[4]. Logic models can take many forms. They may focus on outcomes; activities; concepts or theories; or a combination of the above. According to the Kellogg Foundation, a logic model is a learning tool that helps you to learn about your program or project and make continuous improvements: “A logic model’s value is in the process of creating, validating, and then modifying the model.”[5]

Some questions to consider when you review proposed evaluation plans include:

Plan – and allow sufficient time for - evaluations! For example, if a training project takes one year to implement, consider doing a follow-up survey six months after completion to find out how the training has impacted participants.

Quantitative versus Qualitative Data….Not an Either/Or Situation Quantitative information are things that can be counted and measured, like the number of participants in a project, the number of brochures distributed, the number of locations at which a workshop was given. Qualitative information provides in-depth, descriptive information that can give context or meaning to the experience, such as people’s perceptions or feelings about what happened and how the project affected them. Ideally, you will want to gather quantitative and qualitative data on family violence issues.

9.2 Implementation & Monitoring

Planning is only the first step. It is important that evaluation plans be activated as soon as a project commences. As a project moves from proposal to implementation stage, it is likely the project objectives and activities – and potentially its outputs and outcomes - will be fine-tuned. Keeping track of this evolution will help the project and ensure that the evaluation has the correct focus and asks the right questions in the right way. When monitoring the implementation of the evaluation plan, consider the following questions:

Encourage projects to provide evaluation updates, along with any modifications to their evaluation plans when they submit their formal progress and/or interim reports. In addition, ongoing communication, as well as with informal “check-ins” and on-site visits are ways to determine how a project is doing.