We’ve noticed you’re visiting from NZ. Click here to visit our NZS site.
Mixed-method evaluations of what your programme achieved, and for whom
For every programme, fund or strategy there comes a decision: continue it, change it or stop it. Before you, your board, your council or your Minister decides, you need evidence of what it achieved. We gather that evidence with you, through qualitative research with the people involved and data analytics from your records, surveys and financials. Our New Zealand evaluations range in scale from work finished in a matter of weeks to a three-year evaluation of the $1.19 billion Jobs for Nature programme.
-
- The evaluation plan sets out which documents and people, which data and which analysis each finding will rest on. For the Child and Youth Wellbeing Strategy we developed clear evaluation criteria, and our findings informed its statutory review.
-
- A programme can work well overall and still miss some groups it set out to reach. An evaluation that looks only at the overall result will not show that.
-
- If an evaluation involves Māori, we make sure people who whakapapa Māori hold leadership roles on the team. Senior Māori researchers led our assessment of how effective the Child and Youth Wellbeing Strategy was for tamariki and rangatahi Māori.
Did your programme work for the people you meant to reach?
Judging a pilot, a fund or a strategy takes more than a count of what you delivered. You need to know what changed, for whom, and whether the way you delivered it helped or hindered those changes. We build the evaluation with you so it best answers those questions.
- Rate how you delivered it as well as whether it worked. If the evaluation only asks whether a programme worked, you learn nothing about which parts of its delivery to keep or change. For the Ministry for Culture and Heritage's $51.6 million Innovation Fund, we designed a hybrid evaluation of how the fund worked in practice and how effective it was. That let us recommend improvements for each phase of its funding process.
- Assess implementation while effects are still to come. A new programme may not show its outcomes for some time. If it is too soon to measure outcomes, we say so and focus on assessing implementation, which tells you how programme delivery is going now.
- Learn who it is not reaching, and why. The numbers show who used a programme and who did not, but not why. For the Ministry of Social Development, we built on its survey of 1,852 people to understand why some people were not accessing their full entitlements. Teaming up with All is for All, a specialist disability consultancy, we ran 35 in-depth interviews. We turned the findings into ten personas – portraits of typical experiences – that policy teams can use in their decisions.
- Cover every agency, fund and region in one evaluation. Jobs for Nature spanned 25 funds across five government agencies, so no single dataset covered all of it. We combined location studies with studies of themes across the whole programme. That combination showed how the programme moved from creating jobs to lasting environmental restoration.
- Scope the evaluation to the decision you face. You may think a small programme cannot afford an evaluation, or that a large one has to measure everything. The number of questions our evaluations ask depends on the decision, not the budget. With you, we set one or two key evaluation questions for a simple decision and six or seven at most for a complex one. We then answer those questions well.
How we evaluate
The key evaluation questions come first, and the research design follows from them: documents that need reviewing, people to interview or survey, and data to analyse. We combine qualitative research and quantitative research in one design and analyse the results together, so each finding rests on both the numbers and what people told us.
On the quantitative side, data analytics covers your service records, financials and survey data. For the Ministry of Justice's review of alcohol licensing fees, we analysed financial records from all 67 councils and modelled statistically how different fee approaches would affect outcomes. That gave the Ministry a clear understanding of why many councils struggled to recover their licensing costs.
On the qualitative side, thematic analysis sorts what people said in interviews, focus groups and open-text answers into themes and explains what the numbers cannot show. Learn more about how we run stakeholder engagements here.
-
Have senior Māori practitioners lead the aspects that involve Māori. On our evaluation of the Ministry for Culture and Heritage's Innovation Fund, senior Māori practitioners led key aspects so Māori perspectives shaped the entire process. The work followed the principle of mahi tahi, meaningful collaboration. No single Māori perspective represents all Māori, so we plan for the diversity of iwi, hapū and communities a programme involves.
-
Analyse survey data you already hold. The Institute of Public Administration New Zealand had already surveyed 771 public servants, and we analysed the rating-scale answers statistically. We also assessed the open-text answers for themes and compared both with earlier surveys. Our report gave the Institute evidence to advocate for improvements in public service culture.
-
Model what happens without the programme. What would have happened without an intervention is part of judging its value. For the Health and Safety Association of New Zealand's scholarship programme we interviewed recipients, staff, ACC coordinators and industry representatives. We then surveyed the unsuccessful applicants to learn what happens without a scholarship. Our evaluation gave the Association clear evidence that the programme filled a gap in New Zealand's health and safety research.
-
Record the evidence and criteria behind each finding. We judge each finding against the criteria agreed with you in the evaluation plan. The report sets out the evidence behind each finding, why one kind of evidence carried more weight than another, and which frameworks we used. It also says when another explanation fits the evidence.
20 Topics
+210 Resources
Need an independent assessment of what you achieved?
Book time with an expert.
+70 Webinars
Our process
Every evaluation goes through the same stages. We work from the decisions that our evaluations will inform – this sets how many questions we answer, and whether the evaluation needs an economic analysis.
What we deliver:
- An evaluation plan. The key evaluation questions, criteria for judging each answer, research design and the data collection tools, with an ethics assessment if the work needs one.
- An evaluation report. Findings against each question, the evidence for each finding, and evidenced recommendations, with a summary or slides for the people who only need the findings.
Understand
The first conversation is about the decision: what it is, when it falls due, and who is in charge of making it. From there, we agree the key evaluation questions with you. If no theory of change or programme logic sets out how the programme should produce its results, we build one first to evaluate against. Learn more about our monitoring and evaluation frameworks here.
Plan
We plan how we will answer each key evaluation question. Each question gets a method, which goes into the research design, and criteria and indicators for judging its answer, which form the evaluation framework. The plan also covers the people taking part – we assess any risk to them and agree with you whether it calls for ethics committee review. If an answer depends on an economic analysis, the plan checks first that your data can support one.
Deliver
We review the documents, run the interviews and focus groups, and collect the survey and service data. Then we analyse the qualitative and quantitative results together, one key evaluation question at a time.
Review
Before the final analysis we hold sense-making sessions with you, sharing the early findings and asking what we have missed and what else could explain them. If the evaluation has stakeholders beyond your organisation, they join those sessions.
Close
You receive the report and the summary or slides that go with it. If your team will evaluate the programme next time, we develop the data collection tools with them during the evaluation, so the tools stay with you.
Related Projects
Related Projects
Projects
Less than a minute to read
Restoring Nature and Community: Mixed-method evaluation of Jobs for Nature
Read More
Projects
Less than a minute to read
Revitalising the Arts Sector through COVID Cultural Recovery: Evaluation of Cultural Investment Fund
Read More
Projects
Less than a minute to read
Building stronger volunteer leaders: How peer support transformed confidence and capabilities
Read MoreFAQs
Frequently Asked Questions
What does an evaluation tell us that our monitoring reports do not?
Monitoring reports usually describe what was delivered. An evaluation steps back, looks across what was delivered and asks what change the programme can claim. It then states an explicit judgement of the programme's merit, worth or value, measured against criteria agreed with you.
How do you limit bias in the findings?
We keep the people who judge a programme separate from the people who deliver it, even if the team does part of the work. That distance means the people who run the programme do not judge their own outcomes. The report also records why one kind of evidence carried more weight than another, and says when another explanation fits what we found.
Who sees the findings before the report is final?
You see them during sense-making sessions part-way through the evaluation, along with anyone else who will act on the findings. Those sessions test findings while there is still time to collect more evidence. If the people delivering the programme need more than numbers, we write short accounts of people's experiences from the qualitative data. Those accounts show them why we made each recommendation and what acting on it would change.
Can we evaluate a small programme without a big budget?
You can evaluate small programmes without a big budget, because the number of key evaluation questions depends on the decision, not the budget. For a simple decision, one or two questions may be enough, such as how the programme is running and how to improve it.
We also start from the data you already hold. For Volunteering New Zealand we analysed the application and exit surveys from its peer support and mentoring programmes. Those surveys set the starting points for measuring change, before we added anything new. Volunteering New Zealand began improving the programme based on our findings, trialling more group-based peer support.
Does every evaluation need ethics committee approval?
Every evaluation needs ethical practice, but not every evaluation needs formal committee approval. If you think your evaluation needs approval, we can propose an ethics assessment first. In that assessment we work through the questions a committee would ask, and decide with you whether you need formal approval or just agreed safeguards. If agreed safeguards are enough, you avoid the time and cost of a formal application.
Can the evaluation say what the programme is worth in dollars?
It can if we plan for it at the start, because there is no meaningful way to do an economic analysis afterwards. At the Plan stage a data feasibility assessment determines whether your data can support an economic analysis before anyone commits to one. If the value is social rather than financial, the method is a social return on investment analysis; learn more about our SROI work here.
20 Topics
+210 Resources
Need an independent assessment of what your programme achieved?
Book time with an expert.
+70 Webinars
Learn More
Related resources
Past Webinars
45 minute watch
Evaluation Myths Debunked: Reflections from 200+ evaluations
Read More
Past Webinars
45 minute watch