We’ve noticed you’re visiting from NZ. Click here to visit our NZS site.
A monitoring and evaluation framework built with your team
A national plan, a policy, a portfolio of programs, or a new function – after you’ve agreed the goals and actions, you need to be able to measure them. We build you the evaluation framework that gives you those measurements. A theory of change and a program logic set out how your initiative is meant to produce its outcomes. Key evaluation questions, indicators and data collection tools then test whether it is doing so.
-
- We draw it with your team, so it describes the initiative as it runs.
-
- We set out each measure in a data matrix and choose measures your team can collect easily.
-
- We write the framework so your staff can adjust it as the initiative matures. The guidance lays out when a change is due and how to make it.
How can you show progress on a plan or policy when the outcomes are years away?
Your plan, policy, portfolio or new function may have agreed, but outcomes that are still years away. In the meantime, you need a way to show senior leadership whether it is moving towards those outcomes. We build a framework your staff can use to measure that progress and report on it.
- Define the outcomes, then the indicators that measure them. A list of goals and actions does not show whether outcomes are improved. We define the outcomes your initiative is meant to produce, and set out the indicators for those outcomes and the data source for each indicator. Each indicator measures change in either direction, so the framework shows where an outcome has stalled or slipped as well as where it has improved.
- Draw the logic the way the initiative actually runs. Most program logics show A leading to B leading to C, but real programs and plans rarely run that way. We build into the logic the different starting points and paths of the people the initiative serves, and the system around it, so the logic shows how change happens.
- Agree the questions and indicators before any evaluation starts. Any evaluation of your initiative, during its delivery or after it, needs agreed questions and indicators to measure against. We design the framework before any evaluation begins, so every evaluation of your initiative measures it against the same questions and indicators.
- Size the framework to the time your team has. Your staff may have little time to collect data or update a framework on top of their other work. We fit around their availability, and choose a small number of measures they can collect without much extra effort.
How we build a framework
We start with what your initiative is meant to achieve, then build the theory of change: how the initiative contributes to change over time. From that theory comes the program logic – a diagram connecting the initiative to its short, medium and long-term impacts. Then we set the key evaluation questions for the framework, and the indicators, data collection tools and processes that will answer them.
-
Spend the time on the theory of change. It is the part of a framework we give the most time to, because the key evaluation questions follow from it.
-
Work through the logic with the people who run the initiative. For Cancer Australia's 10-year Australian Cancer Plan, which has 2-year and 5-year goals, we began by reviewing the existing frameworks, strategies and program materials. We then ran workshops with Cancer Australia to support priority setting and develop the logic model, theory of change, key evaluation questions and practical data collection tools.
-
Describe the different groups you serve, each with their own path. With a residential service for people with physical disabilities we built four personae: typical residents with different starting points and needs. The logic model then showed several paths rather than one.
-
Model the evaluation system around the initiative. We map four parts of the system: the people involved and what they care about, the tensions between them, outside influences, and the assumptions the logic rests on. The logic then reflects the people and conditions around the initiative, not only one path from activity to outcome.
-
Design indicators for the portfolio and for each program in it. For the NSW Department of Education, we developed the Evaluation Strategy for the Early Years Commitment, a $15.9 billion investment over 10 years made up of many programs. The Strategy's indicators and evaluation questions measure what the whole investment achieves and what each program in it achieves. In the Strategy, we also linked each program's outcomes to the Commitment's overall objectives.
15 Topics
+90 Resources
Need a way to show what your program is achieving?
Book time with an expert.
+40 Webinars
Our process
We build a framework before anyone evaluates your initiative, and your own staff run it after we hand it over. We run most of the work below with your staff, and the work ends with a handover, not a report of findings.
What we deliver:
- A framework document. The theory of change, the program logic, the key evaluation questions, and the indicators and data matrix that answer them.
- The tools and guidance to run it. The data collection tools, an analytical approach, guidance on updating the framework, and the triggers that tell your team when to monitor and when to evaluate.
Understand
We go through your initiative's objectives and actions with you, and find out how much time your staff have to collect data and report on it. If your initiative's objectives are unclear, or no data shows how things stood before it began, we develop a theory of change or identify indicators before setting the questions.
Plan
We read the existing frameworks, strategies, policies and program materials, then design the workshops for your staff: who takes part, in what order, and what each session has to produce. If your staff have limited availability, we fit the workshops around them.
Deliver
We run the workshops, develop the logic and the key evaluation questions with your staff, and build the indicators, data matrix and collection tools. In the workshops we also take your staff through the principles and processes of monitoring and evaluation, so they can run the framework without us. If the framework needs a survey tool, we test how people understand each question before the survey goes out.
Review
We take the draft framework through sense-making sessions with you, and with your stakeholders if the initiative involves any, and revise it from what you and they tell us. We also check that each measure is practical and that your team can get the data each question needs.
Close
You receive the framework, the tools and the update guidance, everything your team needs to apply the framework. If what you need next is the evaluation itself, learn more about our mixed-method evaluations here.
Related Projects
Related Projects
Projects
Less than a minute to read
Review of Medicare Benefits Schedule Continuous Review function
Read More
Projects
Less than a minute to read
Delivering clarity and purpose: a monitoring and evaluation framework for Australia's Cancer Plan
Read MoreFAQs
Frequently Asked Questions
Our program logic is too linear. How can you deal with that?
We build several paths into it. A linear logic model shows one path from activities to outcomes, but real programs serve people with different starting points. We describe each main group you serve as a persona with their own path to the outcomes. As your team measures the outcomes, the results show whether each path is working as intended or whether the activities need to change.
How do you test the assumptions behind the logic?
We set out the assumptions in workshops with your staff, who know from their work whether each one holds in practice. Assumptions are the things everyone hopes are true but may not have been tested. A systems checklist covers the questions we ask about the system around the initiative, and a persona canvas describes each group it serves. Both downloads are available below.
How can you help ensure our teams use the evaluation framework?
We build it with them. We develop the logic, the questions and the tools in workshops with your staff, because people are less likely to use a model they did not help create. For one client, we developed the theory of change and the logic model with their team as part of a learning plan for their program.
Can one framework cover many programs delivered by different providers?
Yes, one framework can cover them. The indicators for the Early Years Commitment measure outcomes consistently across provider types, workforce groups and regions. If you need to know what each provider contributed to the outcomes, we trace each one's contribution through the theory of change.
Will the framework add to our reporting burden?
Not if we build it around the time your team has. A framework adds to reporting when it includes too many measures, often of activity rather than outcomes; we would rather measure fewer things and measure them well. For the Department of Health, Disability and Ageing, we used a data matrix to choose practical measures for a new Medicare Benefits Schedule review function. Those measures kept extra reporting to a minimum.
15 Topics
+90 Resources
Need a way to show what your program is achieving?
Book time with an evaluation expert.
+40 Webinars
Learn More
Related resources
Past webinars
45 minute watch
Rethinking the Logic Model: Personae, systems, and what actually drives outcomes
Read More
Past webinars
45 minute watch