We’ve noticed you’re visiting from NZ. Click here to visit our NZS site.
Mixed-method evaluations that show what changed and what to do next
A service, a pilot program, a change to the way you work or to a whole strategy – at some point someone decides whether it worked and what happens next. Understanding what changed and why takes two kinds of evidence: the numbers, and the accounts of the people involved. We run the independent evaluation that ties both together, with qualitative research and data analytics in one design. Our teams have delivered more than 200 evaluations across Australia and New Zealand, from rapid work done in weeks to multi-year, multi-site projects.
-
- The evaluation plan names which documents, which people and which datasets will answer each question, and we agree it with you before we collect any data.
-
- Developmental evaluations run as your program does, so you can act on our findings before the final report.
-
- For the County Court of Victoria, we evaluated its case management initiatives, and our findings informed their continuation and expansion across Court Services Victoria.
Want to find out what your service, pilot or strategy changed, and why?
Your activity reports answer one question: what happened. Those reports do not answer how a change affected the people involved, or why it happened in one place and not another. Whoever decides what happens next needs answers to both and the evidence behind each.
- Explain why the numbers changed – or did not. Before-and-after analysis of your data shows whether outcomes changed; it cannot say what caused the change. We put interviews, focus groups and surveys beside that data, so the findings say what changed and why.
- Evaluate policy measures, a pilot or a new way of working. What you need evaluated may not be a program. It may be a set of policy measures, the pilot of a new digital tool, or a change to how a court manages its cases. We have evaluated all these, and each time we built the design around the questions the evaluation had to answer.
- Start before the initiative ends. If the evaluation only starts when the initiative ends, there is nothing left to adjust. We run evaluations alongside implementation, in cycles of evidence gathering and feedback, so you can change the initiative while it is still running.
- Build your team's evaluation skills. For the next evaluation, you may want your own team to find out what changed, and why. We develop the evaluation tools together with your team, so your team can use them again after we finish.
How we evaluate
Every evaluation starts with the decision it will inform and the key evaluation questions behind that decision. From those we build the research design: which documents to review, who to interview or survey, and which datasets to analyse. For the Napthine Measures in regional education, we reviewed over 60 documents, ran 22 engagements with more than 50 stakeholders and analysed national tertiary education datasets. If the question is whether a policy, scheme or operating model is still fit for purpose, learn more about our reviews here.
Our quantitative research applies data analytics to your service, administrative and survey data, to find patterns and measure change over time. Our qualitative research gathers what people tell us in interviews, focus groups and workshops.
Thematic analysis of what people tell us draws out the reasons and experiences that the numbers do not show. We analyse the two kinds of evidence together, so each finding rests on both.
-
Measure change in the data you already hold. For the Youth Crime Prevention and Early Intervention Program we compared police attendance data, police records and referral data before and after the program. That comparison let us put numbers to the changes.
-
Ask the people involved what the data means. For the evaluation of Star Ratings for residential aged care, we held workshops with specialist providers, including Aboriginal and Torres Strait Islander aged care services. We also consulted current and prospective residents and ran national surveys of consumers and the sector. Through engagement with Aboriginal and Torres Strait Islander residents we found gaps in how cultural safety is measured, and we recommended integrating cultural safety processes.
-
Add an economic analysis if the decision depends on cost. For the Youth Crime Prevention program our economic analysis put the program's savings to the criminal justice system at about five times what it cost. For the Disability Liaison Officer Program, a lapsing program evaluation, we ran a cost-benefit analysis to show its value for money.
-
Design the evaluation with First Nations co-leads. On many of our projects First Nations leads support engagement with Aboriginal and Torres Strait Islander peoples. Aboriginal co-leads designed one large Aboriginal primary healthcare evaluation with us, which ran across several sites over a number of years.
-
Make it safe for people at increased risk of harm to take part. For national evaluations of family violence programs we spoke with service providers first, asked each victim-survivor what they needed to take part safely, and arranged support if issues arose. We include people with lived experience on the evaluation team from the outset and make materials accessible, including Easy Read and languages other than English.
15 Topics
+90 Resources
Need to know what your initiative changed, and why?
Book time with an expert.
+40 Webinars
Our process
Our evaluations run through five stages, and the numbers and the accounts of the people involved stay together through all of them. Two things change from one evaluation to the next: whether the initiative needs one round of fieldwork or several, and whether the decision calls for an economic analysis.
What we deliver:
- An evaluation plan agreed with you before fieldwork. The key evaluation questions, research design, data collection tools and, if the work needs one, the ethics assessment or ethics committee application.
- A report of findings and recommendations. Our findings against each key evaluation question, the evidence for each finding and our recommendations on what to do next.
Understand
We start with the decision your evaluation will inform and who will make it - you, your CEO, a council or the Minister. From that we decide on the key evaluation questions: one or two for a simple decision, six or seven at most for a complex one. If there is no program logic to evaluate against, we build one first.
Plan
We match each question with a method and set these out in the research design. We assess any risk to your participants and decide whether it calls for ethics committee review. If the decision needs an economic analysis, we run a data feasibility assessment first. That is an early check of whether the data you hold can support one, so we know before fieldwork whether to commit to the analysis.
Deliver
We review documents, run interviews, focus groups and workshops, and collect survey and administrative data. That happens in one round if the initiative is stable, and in several rounds if it is still changing. We test and pilot surveys before they go out and keep them to about ten minutes. Then we analyse the quantitative and qualitative data together against the key evaluation questions.
Review
We test the early findings with you in sense-making sessions before the analysis is final, and with your stakeholders if the evaluation has them. Then we draft the recommendations and test those with you the same way.
Close
You receive the report. If the initiative continues, we can leave your team a measurement framework to keep monitoring it, as we did for Safer Care Victoria.
Related Projects
Related Projects
Projects
Less than a minute to read
Evaluating the Napthine Measures for Regional Education Success
Read More
FAQs
Frequently Asked Questions
Do we need an independent evaluator, or can our own staff do it?
Your own staff can do part of the work, as long as the people who evaluate the initiative are not the people who run it. Keeping the evaluation at arm's length from delivery reduces the risk of bias and blind spots in the method.
A hybrid model, with your team doing part of the work and an external evaluator the rest, keeps that distance and uses your team's knowledge of the initiative. The Australian Centre for Evaluation found that about one in five Australian Government evaluations now run this way. On one multi-agency grants program we set a learning plan with the client, co-developed the tools, and worked with each grant recipient on their own program logic.
Will our evaluation need ethics committee approval?
Your evaluation may not need formal committee approval, although every evaluation needs to be done ethically. The 2025 National Statement removed mandatory ethics review in most circumstances. Even so, we expect to seek review in the same circumstances as before, because our own assessment of the risk to participants calls for it.
If your brief assumes committee approval, we can propose an ethics assessment first: the questions a committee would ask, worked through with you to decide whether you need approval. That step matters because a formal application can add three months or more to your timeline.
Our initiative is small and so is our budget. How do you size the evaluation?
We size the evaluation to the decision it informs, not to the budget. A simple decision needs one or two key evaluation questions, and even a complex one should be no more than seven. Once we have set the questions, we use the data you already hold before collecting more.
For the Older Persons Advocacy Network we were able to run a new analysis on existing post-training survey data. That analysis answered every key question and reduced the burden on aged care providers. In three months, our evaluation team conducted a document review, a follow-up survey and interviews, and reported to the client.
Do we have to wait until the initiative ends before evaluating it?
No, the evaluation can start while the initiative is still running. For the Child Link pilot we evaluated progressively, giving the Victorian Department of Education insights when it needed them rather than in one final report. If the initiative is still changing, a developmental evaluation gives you findings you can act on while it changes. And if it is too new to show outcomes, we say so and focus the evaluation on how implementation is going, which gives you feedback you can use now.
Can the evaluation include an economic analysis?
Yes, if we plan for the economic analysis from the start. The data feasibility assessment at the initial stage tells us whether the data you hold will support one. If it will, the analysis is part of the evaluation, so you read the dollar figure beside what people told us about the initiative. If it will not, we tell you before fieldwork, so no one commits to an analysis that will not work.
What if the findings are not what we hoped for?
You will see them before the final report. We involve you, and the people who will act on the findings, from the start. We then test the early findings with you in sense-making sessions part-way through.
If the numbers alone do not persuade the people who deliver the initiative, we turn the qualitative data into stories that show why the recommendations matter. On one program, those stories made the findings usable for the people delivering it.
15 Topics
+90 Resources
Need to know what your initiative changed, and why?
Book time with an evaluation expert.
+40 Webinars
Learn More
Related resources
