Welcome and Acknowledgement of Country
Before we begin, we would like to acknowledge the traditional custodians of the land we work on and the communities we work with today. Emma and I are on the beautiful lands of the Wurundjeri people of the Kulin Nations here in Melbourne. Chris is joining us from the lands of the Gubbi Gubbi people at the Sunshine Coast. We know that you are joining us from the lands of different traditional owners and custodians around the country, and also from New Zealand. We acknowledge their history, culture, and elders past and present.
Introductions
I'm Linda Gyorki, the Director of Consulting here in Allen + Clarke's Melbourne office.
And hi everyone, I'm Emma Keleher, the Strategy and Planning Lead also in Melbourne's Allen + Clarke.
For those of you who might not be familiar with who we are, we are a consultancy that specialises in evaluation, strategy, and policy. We deliver work that ensures complex, high-stakes decisions can be made with evidence, defended with confidence, and are built to work for the people and communities affected. We work almost exclusively with government, community, and not-for-profit clients, which means that lapsing funding evaluations are a pretty significant part of our work.
Thanks Emma. We're also joined today by Chris Doran, a professor of health economics who has been a close partner of ours across many of the evaluations we'll be drawing on today. Chris brings the technical economic lens that Emma and I would freely admit sits outside our core expertise. So you're getting the full team today.
Thanks Linda. It's great to be here. I've worked with Allen + Clarke across a range of complex evaluations over the years, and I always find these conversations really useful. The economic and evaluation disciplines reinforce each other more than people sometimes realise. I look forward to the discussion today.
Thanks Chris. Between us, we've spent a long time on both sides of the lapsing funding evaluation coin — both inside government, commissioning and managing these processes, and also as evaluators sitting across the table from program teams asking for data and documentation. Today we want to share what we've learned, and in particular the kinds of things that, with hindsight, program managers often wish they'd thought about a bit earlier.
This session is for anyone who might commission, manage, or deliver a program that is subject to lapsing funding arrangements and will at some point need to go through a Treasury process. Whether your evaluation is six months away or two years away, there are things you should be thinking about and putting in place right now.
Just before we go any further, a quick word on terminology. We will be using "lapsing funding evaluation" throughout today's session because that's the language used in the Victorian government context, and that's where much of our experience sits. But if you're joining us from another jurisdiction or New Zealand, you'll have equivalent processes under different names — scheduled funding reviews, sunset provisions, periodic investment reviews, whatever the language is where you are. The underlying challenge is the same: it's a point in time where a funding decision has to be made, and your program needs to demonstrate its value with rigorous evidence. Everything we're covering today applies to that situation regardless of what it might be called in your system.
We also recognise that people in this room will be coming from really different starting points. Some of you will have programs with a really strong performance story, and you'll be thinking about how to demonstrate that compellingly. Some of you might have programs where the picture is more mixed — some things are working well, some areas need strengthening — and you'll be thinking about how your evaluation can help make the case for a redesigned approach. And some of you might be sitting with even harder questions about whether the current program model is the right one going forward. The guidance we're sharing today is relevant across all those situations. Good evaluation preparation helps regardless of where your program ends up. A well-prepared evaluation is one that can genuinely support whatever decision needs to be made, rather than one that scrambles to justify a position after the fact.
Today we're going to move through a series of practical areas including data, ethics, stakeholders, evaluator selection, and recommendations, with a deeper dive into economic analysis presented by Chris towards the end. At each stage, we'll flag three practical things you can act on immediately, and we'll be sharing a full self-assessment checklist after the session so you can work through the detail with your team. Let's get into it, Emma.
Section 1: Understanding What You're Working Towards
All right. We're going to kick off by talking about understanding what you're working towards. It's worth grounding ourselves in what a lapsing funding evaluation actually needs to demonstrate. At its heart, whether that's Treasury in Victoria, a central agency in another state, or Treasury in New Zealand, the decision makers want to know the same things: whether the program achieved what it set out to do, whether that investment represents value for money, and whether ongoing funding is warranted.
Some jurisdictions set out a formula for how to respond to these questions and require evaluators to follow a step-by-step approach. That's the case here in Victoria, where the Department of Treasury has mandatory questions for lapsing programs. But broadly, the framing across jurisdictions is pretty similar.
Those key questions might sound straightforward, but in practice, evaluations can succeed or fail on the quality of the evidence base that programs have built up over their life. That doesn't just materialise out of thin air when evaluation time comes around. It has to be cultivated from the beginning, or at least from the moment you realise an evaluation is coming.
It's also worth saying something at the outset about what good evaluation is actually for. The best lapsing funding evaluations we've been involved in have been ones where the program team genuinely wanted to know what was working and what wasn't — not just to make a case, but to understand and adjust their work or approach accordingly. That openness to findings, including unexpected or possibly uncomfortable ones, is what makes evaluation useful. And frankly, it's also what makes it credible to decision makers. In our experience, Treasury can usually tell the difference between an evaluation that was designed to genuinely answer a question and one that was designed to defend a position that had already been decided before the evaluation started.
Yeah, absolutely Emma. Before we get into the practical preparation, it's worth flagging one thing that underpins all of it — your program logic or your theory of change. Different organisations use different terms, but they're describing the same thing. It's a structured account of how your program is supposed to work: how your inputs and activities are expected to produce outputs, and how those outputs are expected to lead to outcomes and impact. If you don't have one, or yours hasn't been revisited since the program was designed, that's the first thing to address.
The reason it matters so much for evaluation is that a lapsing funding evaluation is essentially asking whether your theory of change held. Was there a demonstrated need? Did the program deliver what it said it would? Did those outputs lead to the outcomes you claimed? Every evaluation question maps back to that program logic.
If that logic is vague, if your outcomes are aspirational rather than measurable, or the causal links between activities and impact are unclear, then your evaluator will be working without a map. And your evidence base will have gaps that are really hard to fill late in the process. There's also a practical diagnostic value here. Sitting down with your team and working through your program logic before you engage an evaluator will surface exactly where your evidence is strong and where it isn't. The places where you can't point to data are the places that need attention now, not at evaluation time.
Thanks Linda. I'd also add an economic perspective to that. The evaluations that produced the most credible economic analyses are almost always the ones where the program team comes in curious rather than defensive. When teams are genuinely open to exploring data options, they tend to track costs and outcomes more diligently and comprehensively because they want to know the answer rather than defend a position — and that rigour really shows in the analysis.
That's certainly been my experience as well. Another big challenge we often see is program teams arriving at evaluation time with good intentions but patchy data — missing baselines, or data that exists but can't actually be shared with evaluators. Every one of those problems is solvable, but much harder to solve under the time pressure of an active evaluation. Think of today as a prompt to get ahead of some of those issues.
Section 1 — Three Things to Act On Now
First, pull out your program logic or theory of change and review it with evaluation in mind. Are your outcomes measurable? Are your causal links clear? Are your timeframes realistic? If it needs updating, do that now before you engage an evaluator.
Second, have an honest conversation with your team about what you genuinely know and don't know about your program's performance. The gaps you identify now are the ones you can still do something about.
Third, check whether your central agency or Treasury has published mandatory evaluation questions or standards for lapsing programs in your jurisdiction. Your program logic doesn't need to mirror those questions — they serve different purposes — but you do need to be confident that your evidence base can speak to them when the time comes.
Section 2: Data
All right, moving on to our next theme. This is around data, because this is where most of the groundwork for the evaluation happens.
The first question to ask yourself is: what are the key data sets that are going to tell the story of your program? A good starting point is mapping out what you actually have. That could be administrative data like participant numbers, service delivery records, and outputs. It could be outcomes data you've already collected, such as surveys, assessments, or referral results. And any external data sets that might help demonstrate impact — things like hospital admissions, employment records, or education data, depending on your sector.
The really important follow-on question is: do you actually have permission to use that data for evaluation purposes? This is a scenario we have seen happen before.
Imagine a primary care navigation program that has three years of really rich participant data — engagement records, referral outcomes, everything you'd want for a good evaluation. The data mapping looks solid. But when someone checked the original consent framework, they found that the data permissions only covered program improvement purposes. The team had quite reasonably assumed that would include evaluation, but in practice it didn't meet the threshold. Getting a consent variation through was required, and that can add weeks to your timeline before data collection can even begin.
That's an example of the kind of issue that, if caught early before an evaluator is engaged, is a pretty straightforward fix — update your consent form for future participants, or get the permissions in place. It's easy to resolve when you have the time, but genuinely disruptive when you don't.
It's something that catches a lot of program teams off guard. Your program might be collecting rich data on participants, but the consent framework under which that data was collected might not extend to evaluation. Or you might want to link your program data with another agency's data — say, linking service delivery records with employment outcomes — but you haven't yet established that data sharing arrangement with that agency. These aren't quick fixes. Data sharing agreements, privacy impact assessments, and ethics approvals take months to negotiate. If you leave them until the evaluation has started, you'll either delay the evaluation or end up working with a weaker evidence base than you need.
Right now, map out your data sets, work out who owns them, and start asking: what would we need to put in place to make this data available for an evaluation?
Data feasibility assessments
That's right. We're going to talk about a couple of those things in a bit more depth now, thinking about data feasibility.
One thing we routinely do at Allen + Clarke as part of our evaluations — particularly where an economic analysis is going to be required — is what we call a data feasibility assessment. It's a structured piece of work delivered at the start of an engagement, done in close collaboration with our clients, and before the evaluation methodology is finalised. The purpose is to map what data exists, what form it's in, what permissions apply, and critically, whether it's actually fit for the analytical purposes the evaluation needs it for.
The reason we find this so valuable is that it shifts the conversation from "what story do we want to tell about our program?" to "what story can the data actually support?" — because those two things are often not the same. Finding that out at the data feasibility assessment stage, before the methodology is locked in, is a very different situation from finding it out six weeks into an evaluation when it's much harder to change course.
Absolutely. From an economic perspective, the data feasibility assessment is generally indispensable. Before I can design any economic analysis, I need to know what cost data has actually been tracked, what outcomes are measurable, and whether there's any basis for a credible counterfactual. A data feasibility assessment gives you that picture systematically rather than through a series of increasingly uncomfortable discoveries mid-evaluation.
In practical terms, what it tells us on the economic side is which analytical approach is the most useful and actually viable — whether it's a full cost-benefit analysis, a more limited cost-effectiveness approach, or whether the honest answer for the client is that the data won't support quantified economic claims at all. In that case, we say so clearly and focus on building the strongest possible qualitative value case instead. That's a far more defensible position than constructing an economic argument on shaky foundations, which an experienced reviewer at Treasury will pick apart.
Definitely. And for programs where outcomes are long-term or preventative in nature — and I know from the questions many of you have sent through that this is a genuine frustration — a data feasibility assessment also helps identify what early indicators or proxy measures might credibly stand in for outcomes that won't be fully visible for years. Used carefully, that's a legitimate and well-established evaluation practice, not just a workaround.
The other thing that catches teams off guard is the complexity of government administrative data sets. Those systems were generally built to run a program — to process payments, manage caseloads, track outputs — not to answer evaluation questions. My recommendation is to broker early introductions between your evaluator and the data analysts or data managers who actually own and understand those systems, before the evaluation starts if possible. It doesn't need to be a formal data release; just a preliminary conversation. An evaluator who spends a couple of hours with the right data manager will save significant confusion later.
Also factor in data release processes. Even once you've got permissions and agreements in place, actually extracting and releasing data from a government system takes time — sometimes weeks, sometimes longer — and you need to build that into your timeline.
That's definitely true. And these systems often need quite a lot of manipulation from someone who knows them inside out.
It's also worth being clear that the data question looks different depending on where your program sits. If your program has been performing strongly, your focus is on making sure the data you've got to tell that story is accessible, linkable, and compelling. If your program has areas of mixed performance, you want data that's rich enough to distinguish what's working from what isn't — that distinction is the foundation of a redesign case. In all situations, the preparation is the same, but knowing which story you're most likely trying to tell can help you prioritise.
Section 2 — Three Things to Act On Now
First, map your data sets on a single page: what you have, who owns it, what consent framework applies, and whether it covers evaluation use. If there are gaps, start those conversations with data custodians now, not once the evaluation is underway.
Second, check your consent forms and data collection frameworks. If they don't explicitly cover evaluation use, take advice on what a variation would require and how long it would take.
Finally, identify your data managers and analysts — the people who actually understand your administrative systems day to day — and plan to introduce them to your evaluator at the start of the engagement.
Section 3: Data Gaps and Additional Data Collection
Building on our data theme, the next topic is data gaps and additional data collection, because existing administrative program data is rarely enough on its own. Most lapsing funding evaluations require some new primary data collection — that might be interviews or focus groups with participants, service providers, or community members; surveys of staff or stakeholders; or case file reviews.
The question worth spending some time on is: who specifically needs to be heard from, and how hard might it actually be to reach those people? This can be something people underestimate. If your program serves people experiencing homelessness, people in the justice system, or people from culturally and linguistically diverse backgrounds, you can't assume that an evaluator will be able to parachute in and collect meaningful data at short notice. These things take time and relationship-building to do properly.
You really need to think about your program infrastructure — your frontline workers, your community partners, the networks you're working with — and how you might work with them to support additional data collection.
A hypothetical example that captures something we see regularly: imagine an employment support program targeting people with complex barriers to work. The program has strong relationships with participants built over two or three years. The evaluator comes in, writes a survey, and sends it to the program's contact list — but only gets a 12% response rate, heavily skewed towards the most recently engaged participants. The people who completed the program two or three years ago, including many of the early cohort who have the most interesting outcomes, simply aren't reachable through that channel. The evaluation ends up telling a partial story — not because the data doesn't exist, but because there wasn't enough time and consideration given early enough to how to reach the right people.
That's so important. And timing matters too — that's an underappreciated constraint. If your program is delivering into schools, fieldwork in the last two weeks of term is basically impossible. If you work with peak bodies, budget season and planning cycles will affect availability. If your program has seasonal delivery patterns, your evaluator needs to know that from the start, so data collection is scheduled around reality rather than discovered mid-evaluation to be in a blackout period.
From an economic perspective, the decisions you make now about data collection will either strengthen or weaken your economic analysis options. We often get brought in at evaluation time and have to work with whatever data exists. The programs where we can do genuinely rigorous economic analysis are almost always the ones where someone made good data collection decisions two or three years earlier — often because it seemed like the right thing to track, not because anyone was thinking ahead to the evaluation. That foresight is worth a lot.
Section 3 — Three Things to Act On Now
First, map the cohorts your evaluation will need to hear from and rate each one for accessibility. For people whose voices are seldom captured through standard methods, identify who holds the relationship and start thinking about how to formalise that access.
Second, mark your program calendar with blackout periods — school holidays, peak body planning cycles, seasonal delivery windows — and share it with your evaluator early so fieldwork is designed around reality, not assumption.
Finally, start collecting data with evaluation in mind now. If your current data collection isn't capturing the outcomes your evaluation will need to demonstrate, it's not too late to start tracking. That will improve your position in 12 or 24 months.
Section 4: Ethics and Privacy
Ethics and privacy are areas where program managers often assume it's the evaluator's job to sort out. And in a technical sense, it is — the evaluator prepares the tools, submits the application, manages the process. But the timeline and cost implications are shared. In our experience, the teams who find themselves in difficulty are often the ones who didn't know enough about the process to have the right conversation with their evaluator at brief stage, before the contract was signed and the timeline was set.
Any evaluation involving personal information will require a privacy impact assessment, or PIA. Sometimes that's called a privacy threshold assessment or privacy review depending on your jurisdiction. That's largely a planning exercise that can begin once the evaluation scope is clear, and it's generally manageable. The part with real implications for your timeline and budget is formal ethics review.
Evaluations that involve primary data collection — particularly from priority cohorts such as children, people in institutional settings, or people with cognitive impairments — will require ethics approval through a Human Research Ethics Committee, or HREC. Your choice of pathway matters significantly, both for timeline and for budget. Ethics approvals through an HREC can take anywhere from a few weeks for low-risk exemptions to several months for full review. An experienced evaluator will know this landscape well, will have existing relationships with committees, and will be able to advise you on the right pathway for your situation. But our advice is to make this part of the conversation when developing your brief, not a problem you discover six months into the engagement.
For evaluations involving First Nations communities, it's really important to be across the IATSIS Code of Ethics for Aboriginal and Torres Strait Islander Research. That sets additional requirements regardless of your HREC pathway. For evaluations involving Māori communities in New Zealand, Te Ara Tika provides the relevant ethical guidelines. In both cases, early engagement with community governance bodies is not optional — it shapes the evaluation design, not just the approvals process.
It's also worth naming upfront that these processes aren't just compliance exercises. Ethics and privacy frameworks exist to manage real risks — to participants, to your organisation, and to the integrity of the evaluation itself. Getting them right protects everyone. Getting them wrong can delay your evaluation, compromise your data, or in serious cases expose your agency to reputational or legal risks.
Yeah, that's absolutely right Chris. And I think the other thing to add is about the spirit of ethics, not just the process. Good evaluation ethics means ensuring that the people you're asking to participate experience it as a respectful and worthwhile process. We need to think about informed consent as a genuine conversation, not just a form to be signed. We also need to think about whether participation could create risk or distress for people and what supports might need to be in place. And we should consider whether the communities you serve could have a say in how the evaluation is designed, because genuine co-design significantly strengthens both the quality and legitimacy of findings. These are all things you can start thinking about well before an evaluation formally begins.
Section 4 — Three Things to Act On Now
First, find out whether your agency has a standard process for privacy impact assessments and who owns it. You won't be able to complete a PIA until your evaluation is designed and your data collection tools are defined — but knowing the process, the timeframes, and the right person to involve means you won't lose time when you get there.
Second, if your evaluation will involve primary data collection, research your HREC pathway options before you brief an evaluator. Know whether you're likely to need a departmental committee or an independent HREC, and factor the cost and timeline into your evaluation budget.
Third, if your program serves or works in collaboration with First Nations communities or Māori communities in New Zealand, initiate contact with the relevant community governance bodies now. This is not a step that can be compressed. Early engagement shapes the evaluation design itself.
Section 5: Evaluator Selection and Independence
All right, moving on to our next segment: who does your evaluation, and how do you ensure sufficient independence?
Treasury requires an appropriate level of independence from program delivery. What that means in practice really depends on your agency's size, structure, and the profile of the program. For some agencies, an internal evaluation unit that sits separately from the program area will satisfy the threshold. For others — particularly where programs are high profile or politically sensitive, or where internal capacity is limited — external evaluation will be expected.
That's right. And if you are going to go external, procurement can take longer than people sometimes think. Developing a solid brief, running the process — whether that's through an open tender, a panel arrangement, or a standing offer depends on your jurisdiction — and then actually onboarding a provider before meaningful work can begin can realistically take two to four months.
Teams that haven't factored that into their timeline often find themselves in a situation where the evaluation already needs to be producing findings before the evaluator has been properly set up to produce them. So it's worth putting a lot of effort into the quality of the brief. An evaluator can only design a rigorous evaluation if they understand the program logic or theory of change, available data, stakeholder landscape, and the evaluation questions that actually matter to decision makers. A weak brief produces a weak evaluation — not because the evaluator isn't capable, but because they'd be working with incomplete information. Everything we're discussing today — data mapping, consent frameworks, priority cohorts — all directly feed into your ability to write a brief that will get you a useful evaluation.
When you're selecting an evaluator, independence is the threshold criterion, but it's not the only thing that matters. You want genuine expertise in the methods your evaluation requires. If you need economic analysis, ask specifically for demonstrated capability — and we'll talk more about what that means in a moment. If your program is for First Nations communities, look to engage First Nations evaluators — people with genuine cultural knowledge, community relationships, and lived experience in that space. A methodology that merely mentions cultural sensitivity is not a substitute for that.
That's right. And for all evaluations, you want someone who will be a genuine partner through the process, not just a report writer coming in at the end. The best evaluations we've been involved in have had evaluators who maintained clear independence but still worked really collaboratively with the program teams to understand context, navigate data challenges, and develop findings that are genuinely useful. That collaborative dynamic doesn't compromise independence. It makes the evaluation better and stronger.
Absolutely. The other point worth making is to be wary of selecting purely on price. Lapsing funding evaluations have real consequences — they can be significant for funding, program delivery, and for the people your program serves. Quality really matters, and that shows up most clearly in what comes next, because a good evaluation doesn't just hand you findings — it helps you think through what to do with them. And that brings us to recommendations.
Section 5 — Three Things to Act On Now
First, work out your procurement timeline from the end backwards. Think about when findings need to be with decision makers, subtract two to four months for procurement and onboarding, and then factor in fieldwork and analysis time. That's when you need to start the briefing — it may be sooner than you think.
Second, start drafting your evaluation brief, or at the very least its data profile. The discipline of writing down what data exists, in what form, and what permissions apply will immediately surface the gaps that some briefs leave unaddressed.
Third, if your evaluation requires specialist capabilities — such as economic analysis, First Nations evaluation, or sector-specific methods — identify evaluators with demonstrated experience in those areas before you go to market, so you know what to ask for.
Section 6: Recommendations
Recommendations are where the real value of an evaluation can be realised — or lost. You can have a really rigorous, evidence-rich evaluation but still end up with a set of recommendations that nobody quite knows what to do with. And that does happen more often than it really should.
Good recommendations are specific, grounded in the evidence the evaluation produced, and oriented toward improvement and strengthening — what the program could do differently, better, or more efficiently. In a lapsing funding context, the question of whether funding continues is ultimately a decision for government, not the evaluator. But what decision makers do need is the clearest possible picture of what's working, what's not, and what would need to change for the program to deliver better outcomes.
There's a practical dimension to this that program staff can directly shape. Without compromising evaluator independence, recommendations don't emerge from findings alone — they're shaped by the evaluator's understanding of what's actually feasible. An evaluator who doesn't understand the program's operating context, its workforce realities, its funding constraints, or the policy environment it sits in, can produce recommendations that are technically defensible but practically impossible to act on. The way to avoid that is to give your evaluator genuine context — not just a program description, but a real understanding of what kinds of changes are actually in scope.
If there are structural or financial constraints that would make certain recommendations unworkable, your evaluator should know that before they write the report — not so they'll soften the findings, but so their recommendations are aimed at the right targets. That requires being genuinely open with your evaluator about what's not working, not just what is. It means making sure they have access to frontline staff and implementation-level knowledge, not just senior management perspectives. And it means thinking ahead about what improvement looks like, so that when the evaluation produces findings, there's already a frame for translating them into action.
And there's a really important distinction here worth calling out. Program staff legitimately know things that should inform recommendations — and that's an even bigger reason why the right model is a collaborative one. Program staff provide that implementation context; the evaluator retains independence over the analysis and conclusions. Providing that context is really appropriate and valuable. What's not appropriate is applying pressure on how findings are worded in a way that might ultimately undermine the credibility of the evaluation.
Absolutely. Finally, expectations about recommendations are set at brief stage, not at report stage. If you build in an expectation that recommendations will be prioritised, specific, and assigned to an accountable party, you're far more likely to get a small number of well-targeted actions than a long list of things someone should probably look into at some point in the future.
Section 6 — Three Things to Act On Now
First, before your evaluator starts, brief them genuinely on your program's operating context — not just what's in the program description, but the workforce realities, funding constraints, and what kinds of changes are actually in scope. Do it early, when it can shape the design of the evaluation.
Second, build recommendation expectations into your brief. Specify that you want a small number of prioritised, actionable recommendations, each with a clear accountable owner. Don't leave that format to chance.
Third, think ahead about how your organisation will receive and respond to evaluation findings. Who needs to hear them? Who has the authority to act on them? Getting that governance clarity in place before the report lands means findings are more likely to lead somewhere.
Section 7: Economic Analysis
All right, we've saved the section that will probably get the most questions about for last. We're going to hand to Chris to lead us through it, because this is firmly his territory.
Thanks Emma. Economic analysis — whether it's cost-effectiveness, cost-benefit analysis, or value-for-money analysis — is becoming a standard expectation in lapsing funding briefs, and central agencies are getting more sophisticated about what they'll accept. But it still remains one of the most commonly misunderstood parts of the evaluation process.
Let me be direct about the thing that matters most: meaningful economic analysis cannot be done retrospectively. It requires data and cost tracking to have been in place throughout the program. If you're trying to build an economic case at evaluation time from data that doesn't exist, you're not doing economic analysis — you're doing creative accounting.
Thanks Chris. For those of us a bit less familiar with these concepts, can you explain what that might look like in practice?
Yeah Linda, thanks. There are generally two sides of the equation.
On the cost side, most programs only track their direct program budget. That's not the full cost. You also need overhead and administration — the share of HR, finance, and executive time — office costs that support the program, in-kind contributions such as staff time or space contributed by other parts of the organisation, and co-investment such as funding or resources from other agencies, partners, or philanthropic sources. Most programs have a reasonable handle on direct costs and almost no handle on the rest. That means the cost figure that ends up in the evaluation is systematically understated, which makes any cost-benefit ratio misleading.
To make this concrete: imagine a youth mentoring program with an annual budget of around $800,000, supporting around 200 young people at risk of disengaging from school. The program team comes to evaluation knowing their direct costs, but when we work through the full picture, we find another $180,000–$50,000 in overhead — a share of corporate services, program management time partially absorbed into another cost centre, and office space. Then there's another $60,000 in co-investment from a philanthropic partner that's never been formally counted. So the true cost isn't $800,000 — it's over $1 million. That changes your cost per participant from around $4,000 to $5,200. It's not a disaster, but it's a meaningful difference when you're building a value-for-money case.
On the outcome side, which of your program's outcomes can actually be expressed in dollar terms, and do you have the data to support that? If your program reduces hospitalisations, supports people into employment, or keeps children at school, there are well-established unit costs that can be applied — but you need the outcome data to plug in. And you also need a credible comparison point: what would have happened anyway without the program? Without that counterfactual, you can't show that the program caused the outcomes, which means the economic case can't be made convincingly.
Using the youth mentoring example again: say the program has data showing participants are significantly less likely to disengage from school than a comparable group who didn't participate. There are published unit costs for school disengagement — the downstream cost to government of a person leaving school early, including welfare, health, and justice system impacts — and they're substantial. If the program can credibly demonstrate it kept even a portion of those 200 young people in school who would otherwise have disengaged, the economic return can be significant. But "credibly demonstrate" is doing a lot of work in that sentence. Attribution requires the outcome data tracked over time and a comparison group. Without those, you have a story but not an economic case.
Thanks Chris. I know someone attending this webinar is thinking: what if my program is already well underway? Is it too late?
It's a fair question, and the broad answer is sometimes no, but the analysis will usually be somewhat weaker. In certain sectors — health particularly — retrospective analysis is sometimes viable, because we have well-established unit costs and administrative data linkage. If you can link your program data to health, employment, or other administrative records through infrastructures like the Australian Institute of Health and Welfare data, Stats New Zealand's integrated data infrastructure, or state-level data linkage units, you can sometimes construct a reasonable picture after the fact. But it's almost always weaker than prospective analysis.
The two things that are generally hard to reconstruct retrospectively are overhead costs — because people's recollections of how much executive time or corporate support went into a program are rarely accurate and easily challenged — and the counterfactual, because without a comparison group or baseline, you end up relying on assumptions that a good reviewer will pick apart.
So if your program is underway, start now rather than waiting for the evaluation to be commissioned. Imperfect prospective data is almost always better than retrospective estimates.
Section 7 — Three Things to Act On Now
First, start tracking your full program costs now, not just the direct budget line. Create a simple cost register that captures overhead allocations, in-kind contributions, and co-investment from partners. Even an imperfect record started today is better than a retrospective estimate at evaluation time.
Second, review your outcome data collection and identify which outcomes could, with the right data, be expressed in dollar terms. If you're not collecting what you need, work out what it would take to start.
Third, if economic analysis is going to be a requirement, flag it explicitly in your brief and ask candidates to demonstrate specific capability. Ask to see examples of economic analysis produced for comparable programs.
Closing Summary
Thanks Chris. We've covered a lot of ground in this webinar: data, consent frameworks, ethics pathways and their costs, evaluator selection, brief quality, economic analysis, and recommendations.
But if there are four things we want you to take away, they're these.
One: start earlier than you think you need to. The groundwork for a good evaluation takes time that program teams sometimes underestimate. If your evaluation feels a long way off, that's actually the best position to be in. Use that time well.
Two: map your evidence base now. Understand what data you have, what you can actually use, and what gaps need to be filled. Start those conversations with your data custodians, data managers, and privacy advisors early to enable the data access you'll need.
Three: treat economic analysis as a planning discipline, not an evaluation task. The cost and outcome data you're collecting right now is either building the foundations for credible economic analysis or not. Start tracking overhead, in-kind, and co-investment. Be explicit in your evaluator brief that economic analysis is required, and ask specifically about capability.
Four: think about recommendations from the start. Build your evaluator's understanding of what's actually going to be feasible. The value of an evaluation is ultimately measured not by how thorough the analysis was, but by whether the findings led to better decisions and a stronger program.
Q&A
Q: How do we demonstrate value for money and impact when we have limited time, budget, and data?
Thanks Linda. This tension came through loud and clear in the questions we received, and it's one of the most common real-world challenges evaluators face. A few practical principles to keep in mind.
First, start with what you've already got. Case notes and service records are often underutilised. You don't always need a new data collection system — you need a smarter lens on existing information.
Second, be clear on the difference between outputs and outcomes — this comes back to understanding your program logic model. Funders increasingly want to see what changed, not just what you did. Even with limited resources, frame your story around outcomes.
Third, triangulate rather than just measure. A mix of quantitative data and qualitative evidence — client stories, practitioner observations — is often more persuasive than numbers alone, especially for social programs where impact is hard to quantify.
And finally, be credible. The rigour of your evaluation should match the scale of the decision. The key message: "good enough done well" is better than "perfect, done never."
Q: Lapsing funding is often provided for short timeframes. How do you frame and measure progress towards outcomes that can only be measured in one to two years, when impact takes much longer?
This is one where there are a couple of different approaches, but proxy indicators come to mind in particular. The use of proxy indicators is really useful in exactly this scenario — where you've got shorter-term program delivery but it will be a longer timeframe before outcomes are fully visible.
If you can't measure an outcome directly, you can identify what might credibly predict it. In the youth mentoring example: school attendance rates and disciplinary incidents could be proxies for educational engagement before longer-term outcomes like graduation or employment come through. In a primary health navigation program: a reduction in local ED presentations or an increase in GP referral uptake could be proxies for longer-term health management and improvement.
That's probably one of the key evaluative methods used in this scenario. An important caveat though: proxy indicators are only legitimate if you can make a credible argument for why that proxy predicts the outcome you care about. You need some evidence or established theory to support that link.
Q: Often the decision to turn off funding has already been made. How can you use evidence to influence decision makers as early as possible in the process?
We recognise that's a really frustrating reality for some people in this space, and it does point to a structural problem when evaluation is treated as a reporting exercise rather than a decision-support tool. But there are ways to shift that dynamic.
The key is to embed evaluation in the commissioning cycle from day one. If evaluation is only resourced and activated at the end, you'll always be playing catch-up. The conversation about what success looks like should happen even before funding starts, so you can regularly check in and demonstrate the impact the program is having.
That leads to the second point: build in interim reports. Don't wait for the funding to end to share evidence. Short, regular updates that highlight emerging outcomes keep funders engaged, build a record of progress, and support continuous improvement and agility in program design and delivery.
Third, know your decision-makers' timeline. Budget cycles have fixed windows. If you want to influence the decision, you need to get evidence in front of the right people before those windows close, not after.
And finally, reframe the conversation. Rather than defending a program — which can come across as defensive — position that evidence to help funders make a better decision, whether that's continuation, redesign, or transition. That shift in framing reduces defensiveness on both sides.
Q: What value do you place on estimations of attribution and drop-off through surveys and prior evaluations in similar sectors? Many social return on investment analyses use estimations where an organisation hasn't been resourced to create and update a monitoring and evaluation framework. How do you approach attribution in those scenarios?
Thanks — that's a good question. First of all, social return on investment is a different methodology to cost-benefit analysis, which is generally considered the gold standard. Social return on investment tends to involve a lot more stakeholder engagement, which provides an opportunity to explore a wider range of data sources.
Attribution is a key question that comes up in all economic evaluations where there isn't solid evidence on causation. And the larger the cost-benefit ratio or social return on investment, the more the quality of the attribution evidence matters. If you're demonstrating a five-to-one or ten-to-one return on investment, the evidence for attribution needs to be really strong.
It comes back to the program logic model — making sure you're collecting the right data and implementing the right measures to inform your outcome and impact analysis. There are a lot of different ways to approach that, which I'll save for another day — but it's a key question, and one that's particularly shared with stakeholder engagement through a social return on investment framework.
Thank you everybody for coming along. If you'd like to reach out to us about preparing for an upcoming lapsing funding evaluation or scheduled funding review, please don't hesitate. We're always very happy to continue the conversation. Thank you again for joining us — enjoy the rest of your day, everyone.
