We’ve noticed you’re visiting from NZ. Click here to visit our NZS site.
An evaluation framework built for your own team to run
You have agreed what your policy, programme or strategy will do – the next question is what to measure to show what it is achieving. We build the monitoring and evaluation framework that helps you answer that question. A theory of change and programme logic set out how the activities in your initiative lead to change over the short, medium and long term. Evaluation questions, indicators and data collection then measure how much change is taking place.
-
- If your programme runs through several plans or sectors, we set indicators that report on the whole programme as well as on each part of it. One framework then shows you the trends across the programme and what is happening in each part.
-
- The framework comes with a plan that says who does what and when, so monitoring becomes part of your team's regular work rather than a separate job.
-
- We write interview guides, surveys and other data collection tools that your team can run without us.
What should you measure, and who will collect the data?
Once you have instituted a policy, programme or strategy, you need to know whether it is working. That means deciding what to measure, how often, and who collects the data. We build those measures with you, in a framework your own staff can keep running.
- Bring plans at different stages under one monitoring approach. If your programme runs through several plans, each with its own vision and initiatives and each at a different stage, it can seem impossible to measure them accurately. We build one monitoring approach that fits the full programme and still reflects each plan's context.
- Track change happening this year and towards a goal decades away. Reporting on this year's activity says little about whether your initiative is moving towards a vision decades ahead. We set indicators that record movement towards or away from each outcome, so your framework shows what is falling behind as well as what is improving. For the Ministry of Business, Innovation and Employment's 30-year Industry Transformation Programme, we designed a framework that gave immediate feedback and assessed long-term impact. The Ministry used it straight away to collect baseline data, the starting point for ongoing assessment.
- Reflect the worldviews your strategy adheres to. If your strategy commits to te ao Māori as well as Pākehā policy frameworks, the measures must come from both. For Manatū Taonga, the Ministry for Culture and Heritage, we co-developed Te Waka Hourua, a wellbeing framework that draws from two knowledge systems. We worked on it with Ministry leaders, and Ngā Uri o Kiwa – representatives of Māori and Pacific peoples. The theory of change states the values and assumptions behind its measures of cultural wellbeing, and the Ministry uses the framework to guide policy analysis, programme design and evaluation.
- Build a framework your team has time to use. If your team don't have time for intensive monitoring, or don't think they're measuring the right things, even the most intricate framework is likely to go unused. We plan the implementation around the time your staff have, and your staff help choose the measures in workshops. We also build their skills while we develop the framework with them.
How we build a framework
The first thing we build is the theory of change: how the activities in your policy, programme or strategy will lead to desired outcomes over time. The logic model then lays out that theory, connecting your initiative to its impacts over the short, medium and long term.
Following that we set evaluation questions, key performance indicators and data collection points, and plan for who collects what and when. If the framework will sit inside a strategy you are still writing, learn more about our strategic planning here.
-
Shape the plan around the community's own values. For the Kiribati Federation of Aotearoa we co-designed an evaluation plan around the Kiribati concept of maroro (discussion), and made sure to centre cultural values. The maroro approach gives community members a way to share their experiences and say what success looks like. With the plan, the Federation could run its own evaluation and show its funders structured, objective measures.
-
Include the people and external influences around the initiative. A logic drawn as one line from activities to outcomes leaves out people with a stake in the initiative, the tensions between them and the conditions nobody controls. We list who is involved and what each of them cares about, then set out the tensions between them.
-
Test whether the outcomes reach the people you designed the initiative for. Outcomes change in quantity and quality over time, and the people who benefit most can be those the initiative was not initially designed for. We write the groups the initiative serves into the framework, so the indicators show whether outcomes are shared equally.
-
Decide on indicators that stay useful. An indicator shows movement towards an outcome, and it is not the outcome itself. Once an indicator becomes a target, it stops being useful, because it no longer shows whether the outcome has changed. So, we keep the set of indicators small, and we flag any indicator that could stop reflecting its outcome as circumstances change.
20 Topics
+210 Resources
Need a framework that shows what your programme is changing?
Book time with an expert.
+70 Webinars
Our process
Your staff will run the framework once it is built, so they help us write it. The stages below end with an implementation plan that names who collects each measure and when, not with findings about your initiative.
What we deliver:
- The theory of change, the programme logic, and the questions and indicators that follow from them. All in a single document, with the data source for each indicator.
- The data collection tools and the implementation plan. Interview guides, surveys and other tools, and the plan for putting the framework into everyday use.
Understand
We go through what your policy, programme or strategy sets out to do with you, and learn who receives reports on its progress. We also find out how much time your staff have to collect data. If you have no theory of change, or your objectives are unclear, we develop one with you before we set any questions. If you already have one, or a logic model, we build the framework on what you have.
Plan
We read your strategy, plans and existing reporting, then design the workshops and validation points – who takes part, and what each session has to produce. If you also need a dollar value on the outcomes, the theory of change we build with you is where a social return on investment analysis begins. Learn more about our social return on investment work here.
Deliver
We co-develop the programme logic and the evaluation questions in workshops with the people who will run the framework. Then we build the key performance indicators, the data collection tools and the implementation plan. Alongside this we build your team's skills in evaluation, so your staff can run the framework without us. If the workshops need to reach beyond your own staff – to a community or a sector – learn more about our stakeholder engagement here.
Review
We bring the draft framework to sense-making sessions with you, and revise it from what you tell us before it is final. We also check that each indicator has a data source and each collection task an owner. If the initiative involves communities or stakeholders beyond your organisation, they take part in validation workshops too.
Close
You receive the framework, the tools and the implementation plan, and your team starts collecting against the plan it helped write.
Related Projects
Related Projects
Projects
Less than a minute to read
Anchoring Strategy in Wellbeing: Framework for the Ministry for Culture and Heritage
Read More
Projects
Less than a minute to read
Measuring Success: Designing a Monitoring and Evaluation Framework for Industry Transformation
Read More
Projects
Less than a minute to read
Creating Safer Families by supporting culturally authentic evaluation: Evaluation Framework for Family Violence Prevention
Read MoreFAQs
Frequently Asked Questions
Can one programme logic show groups with different starting points?
Yes, one programme logic can show each group's starting point as a separate path inside a single model. We work with you to identify the three or four main groups your initiative serves, and what each one needs. We draw a path from activities to outcomes for each group, then combine the paths into a single overall logic. The outcomes can differ by group too: one group may aim to become more independent over time, and another to keep its independence for as long as possible.
Does each sector in our programme need its own framework?
You do not need a separate framework for each sector. The Ministry of Business, Innovation and Employment had plans across eight industries, each with its own vision and initiatives and each at a different stage. The one framework we built for all eight showed the Ministry the trends across the programme as well as insights into each industry.
What does an evaluation plan contain?
An evaluation plan sets out the questions the evaluation has to answer, a logic model of how the programme works, the data collection tools and the evaluation's phases. The phases can run from design through data collection and analysis to reporting. The plan also sets the points at which you check findings with the people concerned.
Will a framework add to the reporting we already do?
Clarify in the terms of reference what is fixed, what is contested and what you could realistically change, and say why you want an outside view. Inside those parameters, findings follow the evidence: we report what evidence supports, whether or not that is what you expected. Once you have agreed the scope, let the reviewer work, because a review you intervene in is no longer the independent view you paid for. It adds as little as possible, because we size the framework to the data your team can collect. A few measures collected well tell you more than many collected poorly. For Tūao Aotearoa - Volunteering New Zealand, we set clear criteria and indicators to guide data collection without overburdening programme participants.
20 Topics
+210 Resources
Need a way to show what your programme is achieving?
Book time with an expert.
+70 Webinars
Learn More
Related resources
Past Webinars
45 minute watch
Rethinking the Logic Model: Personae, systems, and what actually drives outcomes
Read More
