Kia ora koutou, welcome everyone to our webinar on how to focus on the fundamentals of credible evaluation without extending your resources. So I'm Marnie Carter, I'm based here in Aotearoa New Zealand and I've been working in evaluation since 2008, so I've got about 15 years of experience and I'm mainly a qualitative evaluator and I'm joined by my esteemed colleagues and fellow veterans Ned and Caroline. Kia ora, I'm Ned Hardyboys, I'm also based here in Aotearoa New Zealand.
I have been practising evaluation for around 25 years now, so I guess that makes, you know, gives me some practise. At the same time, at times, I feel quite new to it. Hi everyone, I'm Caroline, I'm an evaluator and senior consultant at Allen and Clark.
I started working in evaluation a little bit more recently than Ned in about 2014 and mostly specialise in M&E and data analytics. Thanks team. So for about a quarter of you that are joining us today, this is your first time to one of our webinars, so welcome.
You might not be familiar with Allen and Clark, so I'll just give you a brief introduction. We're an Australasian consultancy and we're focused on making a positive impact in Aotearoa, Australia and the Pacific. We have a range of areas that we do, including strategy, change management, programme delivery, policy research and of course, evaluation.
As an organisation, we do give a damn about empowering you to help overcome society's challenges and that's why we regularly run these free webinars. We've also got desk guides and we're here to provide expert advice wherever we can. We are going to have a live Q&A at the end of today's session, but we are also taking any questions as we go.
So if there is a question that occurs to you as we're talking, please pop it in the chat and we'll do our best to answer it. At the end of today's session, we'll send you a copy of the recording, the slides that we've used and also a summary of the key points. But basically, we're really thrilled to have you here today as we dive into our discussion that I think is incredibly relevant to all of us working in the evaluation space at the moment.
Before we do though, I'd like to take a moment to acknowledge the traditional custodians of the lands on which people are joining us for this meeting today and to pay my respects to their elders, past and present and extend that respect to all Aboriginal and Torres Strait Islander peoples that are here today. We also recognise Māori as tangata whenua of Aotearoa, so kia ora, welcome, kia ora koutou e te whānau. Ne te mihi e mohoe ki a tātou katoa e hui tahi nei e tēnei rā.
Okay, so to kick things off, before we start our session, we're actually really keen to hear from you. Shortly a poll is going to appear on your screen and the question that we're keen to hear from you is what are your top evaluation challenges that you're focussing at the moment? So the options that we have are, do you have challenges in measuring outcomes and impacts? Is the main challenge attributing those impacts to the intervention? Is it sourcing relative data, relevant data? Is it finding evidence for a return on investment? Perhaps you think a lack of people or skills is the biggest challenge, or the fact that we're facing smaller budgets. So while that's happening, Ned, what do you reckon is the major challenge? Well, I guess it's tempting to select all those items, but I'd say it's an interaction between a couple of areas for me.
I mean, on the one hand, there does seem to be a higher demand for evidence, both to inform investment decisions and to demonstrate impact. On the other hand, operating in this kind of environment of constrained resourcing. Okay, what about you, Caroline? I don't know.
As an evaluator, I always feel like I would just like more time and more people. So I'm going to go with the bottom two, all about resourcing. Let's take a look.
So it looks like the top two are actually the main challenges that people think we're facing. So that focus on measuring outcomes and impacts and then attributing that impact to the evaluation. So actually, we're going to talk quite a lot about that today.
So it's certainly a challenge that we agree we are facing. So digging into that a little bit more in depth then and thinking about the current evaluation landscape in Australia and Aotearoa. Ned, if I asked you to think about the landscape in New Zealand, what would you say are the key things that come to mind? I mean, that's really interesting, those poll results.
But I'd say, you know, back to in Aotearoa, I guess, I do find this sort of challenge of the resourcing constraints to be quite a key challenge at the moment. So we have observed evaluators, you know, working in government agencies have been directly affected by cuts to the public sector. And I guess we've also seen a steady decline over some time in the amount and the type of evaluation work that's been commissioned.
And this has flow on effects to external evaluators and consultants such as ourselves. So I haven't quite seen that impact in the past in my 25 years. Yeah, me too.
And I think for me, there's probably a bit of a tension, because you know, not arguing, we're definitely seeing a decline in resources. But I would argue that the biggest impact or change that we're seeing at the moment is the current government's quite laser focus on outcomes and what works and to what extent can it be attributed to the intervention. So much less emphasis on process evaluation or developmental evaluation, or those sort of more qualitative perhaps approaches.
And I do think that we are seeing much more demand for economic analysis, return on investment. And that's really flowing through even into what service providers are being asked to demonstrate in terms of, you know, we've seen some RFPs that have said, you must show the potential economic benefit of this intervention. So I reckon that's something we're really going to have to prepare for.
Yeah, I mean, absolutely. I've been around long enough to seen a few cycles of this. And I guess I've seen that, you know, we've moved through or governments have moved through those cycles of really honing in on, you know, outcomes, funding approaches, etc.
But, you know, it is that coupled with this resource constrained environment that we're in, which is really into play. Yeah, I mean, it's really, it's interesting to hear you both from across the pond talking about some of those challenges in New Zealand, because I think it really parallels the experience in Australia as well. And at the same time, we're also seeing a more kind of embedded approach within government in Australia towards evaluation practise.
So, you know, the establishment of centres like, for example, the Australian Centre for Evaluation at the federal level, or the Centre for Evidence and Evaluation in New South Wales are really good examples of this. And I think it's interesting as well, that increasingly, these, you know, these centres of practise are being housed centrally within government and often within Treasury. And I think that kind of signals to us that there's a real importance in demonstrating impact.
You know, as we move towards a trend around more kind of outcomes based funding models and decision making. Yeah, it's interesting that you mentioned that there's more central agencies that are tasked with sort of supporting evaluation, because we are seeing something similar here with the kind of reframing of the Social Investment Agency. It's not in Treasury, but it is also very much a central agency.
It's intended to work across governments and really encourage that use of evidence to make investment decisions. So I guess, guys, my question then is, you know, what does this all mean for evaluators in the field of evaluation? Well, I guess in theory, you know, it puts us in a good place. You know, as evaluators, we're good at measuring how things work, we're good at, you know, understanding how well things work.
We're also good at synthesising evidence, we're good at developing theories of change on how initiatives are supposed to work, and also for identifying the data and evidence needed to understand how well things are working. So, you know, we're well positioned to support this increased evidence on measuring impact. But again, we are operating in that environment of constrained resourcing.
Yeah, I think that's right. And I think, you know, within that environment is at the same time that heightened demand for evaluation, which you mentioned, Ned, and that means that we need to upskill as evaluators. And, you know, particularly in areas, I feel like data analytics, and those more kind of sophisticated, quantitative methods, you know, and also there's this emerging opportunity around big data and integrated data, which can be really powerful, but also requires quite a high level of expertise to use effectively.
Yeah, I mean, we've had an experience with integrated data, with the integrated data infrastructure, the IDI. And we've seen it provides both, you know, opportunities and challenges for evaluation. So on the, you know, the opportunities, it provides a relatively easy way to follow up on a participants over time, when that might otherwise be very difficult.
And it also provides an opportunity or a way to statistically develop control groups, which, you know, which might not have been set up from the beginning of a programme. But the IDI, I mean, the challenge of the IDI is it has a lot of breadth, but not much depth. So, you know, there's lots of administrative data from lots of different areas in that data set.
But it's mostly proxies for the things that we're really interested in. Yeah. And what do you mean though, Ned, proxies like? Yeah, good question.
I mean, we might be interested, for example, in the mental wellbeing of a population, for example. Well, in the IDI, we might, we haven't got data on that, but we might have data on access to mental health services. So we take that as a proxy for mental wellbeing.
But it's not particularly great proxy because of course, if you're seeing increased mental health service access, yes, that could be translated to a higher demand. So more need for those services, but it also, you know, equally could mean that there's just greater access opportunities to services. Yeah.
And I think that's where I'm going to like put my flag out for the value of qualitative evaluation and techniques, because that gives us the opportunity to dig into why that is and understand the explanatory factors and get kind of the human experience behind the data. Now we do have a question that's come through from Jane. So Jane asks, in your experience, is this shift also related to the services and interventions that are necessary to create the meaningful impact at the root of complex challenges? So I'm going to say as a simple answer, yes, that the shift to being more outcomes focused, seeking more economic evidence, but in a constrained risk environment is very much at all levels.
So from the smallest service providers to the biggest government funders, I think we are absolutely seeing a desire for services to be designed with impact in mind, for the funding to be tied to that. And, you know, it's not enough to kind of, you know, know or think that we might be doing good things at the service level, but actually have the systems to enable us to capture and kind of prove that. Yeah, absolutely.
I mean, I'm working with a couple of not-for-profit organisations at the moment, and what we're hearing from their sort of peak bodies, but what we're hearing from their member organisations is that same demand for having to demonstrate their impact and to demonstrate that kind of return on investment from those investments. Yeah. So I think it's pretty clear that we are working in an environment of constraint.
So it's going to mean we need to make some tough decisions. What I'm keen for us to think about first, though, is what are the aspects that we absolutely cannot compromise on? So the things that we really need to keep or that are vital for evaluation? Yeah, it's a really good question. You know, I think at the end of the day, you're trying to answer the question, you know, does it work? But it's not always as straightforward as it may seem to figure out what the it is.
And so I think at the very beginning of a project initially, where we should really spend our time is, you know, working with the programmers or the policy implementers to really understand the intervention, you know, and typically policy implementers or programmers will have a pretty good idea of what the types of outcomes they're hoping to achieve are and what working means to them. But then it's a process of kind of formalising that and translating it and mapping out those kind of tangible, measurable outcomes for their communities. And I think, you know, as evaluators, we think about this in the context of a logic model, or a theory of change.
And I think that's kind of what you shouldn't compromise on. That's where you should spend your time. Yeah, I mean, I don't disagree with that.
But I'd also add, I guess, that understanding the intervention or the programme in itself is not sufficient. It's also really important to understand what's going on around the programme, you know, including the institutional context, this broader system context, in which that programme is operating, and in which people are experiencing things the way they are experiencing things. So, you know, if I think some of the evaluations we've done, we've evaluated programmes where people's experience have been hugely affected by wider contextual factors, whether they're political, economic, or environmental factors.
And we can't ignore those factors, you know, if we're wanting to get a good reading on the performance of the programme. So have you got an example, Ned, where you did an evaluation where the context was really vital to know about? Yeah, one that I can think of is would be, you know, an evaluation of a child health check programme within Australia for indigenous First Nations populations in Australia. Now, that initiative was set up in a very kind of heavy political context.
And that really influenced the way that the programme and the services were accepted by both those providing the services, but also by the community. Okay, so if you hadn't known about that context, you might have read the data quite differently. Yeah, that's right.
And you know, in that situation, it's quite a large evaluation. So we did quite a thorough contextual analysis, if you like. But you can't always do an extensive sort of political economy analysis.
Particularly not with the resource constraints we've got. That's right. But it does mean you need to consider what else is going on, and how that might explain what you're observing.
Okay, so guys, I mean, I'm not disagreeing, but I think we've had a conversation that's quite theoretical at the moment. So yes, you need a theory of change, yes, you need to know context. But I'm going to say that the thing that we absolutely cannot compromise on is culturally safe and ethical practise.
So in Aotearoa, you know, we really need to practise in a way that works with Te Ao Māori and in Australia with First Nations. And I mean, I think if we don't get that right, we can have a shiny theory of change, and we can know all about the context. But if we don't practise in a safe way, then to be quite frank, we're probably going to get crap data.
And there's a risk that we might actually undermine the communities that we're trying to serve with our evaluation. Yeah, no, absolutely. We can't compromise on this sort of do no harm principle.
You know, that goes across our practise. You know, we do engage often with very vulnerable populations in our evaluation work. And I mean, the thing that often gets forgotten, ignored, is that, you know, the time and investment needed to run through formal ethics review processes.
Yeah, I don't know if it gets forgotten, but maybe just like we put our blinkers on and forget how long it took, or how much it needs to do it well. So yeah, again, I think if we're thinking about where we're putting our time and money in a limited context, yeah, that ethical, the review process, but also the practise part is pretty vital. Okay, so we've got a few ideas on things that we can't compromise on.
But let's think about some tips for what we might actually be able to do. So we've got resource constraints, what are we going to do if our aim is set kind of the serviceable Toyota of evaluation, rather than the, you know, the shiny Ferrari? I can probably kick off. So I think like I'm going to, there's obviously not a direct relationship between the size of the programme and the scale of the intervention.
But I do think that it gives you some broad parameters that you can consider. So we've delivered evaluations at kind of both ends of the spectrum. I can think of evaluations that you know, the programme itself might have had a budget of 100,000.
And then our evaluation budget might have been a 10th of that. So what we did in that instances was really tightly focus the evaluation on like one or two questions that might cover, you know, how the programme was implemented, but how that could be improved. That's it.
And in those instances, we'd set up a really small team or maybe even an individual person that's doing the whole evaluation and, you know, a small amount of effort and you do what you can with those resources. What about the other end of the spectrum? Yeah, I mean, I could add an example on the other end of the spectrum where, you know, we've evaluated investments, you know, in the vicinity of $4 billion. Yeah, so that those involve, they will often involve evaluations over multiple years, you know, and thousands of hours of effort put into the evaluations.
This particular example I'm thinking of involved, you know, 17 evaluation sites across a wide geography. But still that evaluation, I guess, only had six or seven of those big meaty questions, but they were quite meaty. So they're covering things like the relevance of the programme and the investment, its effectiveness through to the impacts and also, you know, in terms of whether those impacts might last.
Yeah, and I think in the environment we're currently seeing where we are at that more, you know, pointy, smaller end of the triangle, it's interesting that you said even in a big evaluation, you're probably going to only focus on six or seven key evaluation questions. When we are down that smaller end, it really is, you know, what is the thing that you absolutely need to know? And let's focus our resources on that. We have a question.
So we've got something from Jesse, who's asked what creative ways have you gone about showing the value of qualitative data that you've gathered? For example, if people have said they might be feeling less satisfied in their role, but the business doesn't really care because they're still performing well. So any ideas about value of qualitative data? I mean, if I take that one to start with, I mean, I find qualitative data, obviously, it's more in the area of practise, can be extremely impactful and can be very powerful. But it does require a certain depth of that qualitative data.
So it's not, in my mind, good enough just to list, for example, qualitatively, like what the impacts might be. But it's, you know, it's really helpful to be able to dig deeper into those, to look at qualitative data that comes from multiple sources, you know, for example, multiple stakeholders, analyse that data, look at what impacts you can identify in that, but also look at the significance of those impacts. You know, what do those impacts really mean for those participants, for their organisations? And also look for opportunities, you know, what the learning that comes out of that data might be.
In a way, this is, you know, looking at sort of qualitative impact stories, which can be very powerful. Yeah, I'd agree. And in terms of Jesse's point, if there's a quantitative indicator that suggests things are all good, and the qualitative data suggests that it isn't, to me, that would be an indicator to, or a prompt to kind of broaden the search, because it might be in that business instance that you mentioned, Jesse, that, you know, they're performing well in terms of economics, but maybe turnover is really high, or there's other indicators that suggest things aren't, or there's a lot of absenteeism.
So the qualitative data would probably prompt me to look a bit further into the quant and see what that means. So I think this really demonstrates, again, that you can't just have one or the other, they've both got value. Yeah, exactly.
And I think the value is demonstrated there, because if you're capturing something qualitatively, that's not part of your code frame, sorry to be such a quanty, maybe it should be. And also, I think her point speaks to, you know, how can we use qualitative data persuasively, as well. And I think, Marnie, your suggestion to, you know, build it into your capture is a really good one.
Nice. So I'm keen to hear your thoughts on what we can do in this environment of constrained resources. What are some strategies that we can use to manage resource and costs? Yeah, I think when it comes to conducting evaluations in a constrained environment, one opportunity you have is to just look at what data you already have and consider what you can reasonably answer.
So I would urge that programmers actually do generate a lot of data in the course of implementing their programmes. So, for example, something as simple as programme enrolment, participation, equity and inclusion, those are things that you can potentially answer or infer from your existing implementation data, if you've got it set up and capturing it in the right way with the right kind of things that you want to know about, without actually spending any additional resources, that's just drawing on good programme management. So that would be one tip that I would suggest.
Yeah, I mean, that's a very sort of pragmatic solution, but I also, you know, it can be quite problematic. Spicy. I mean, because I think we've got to avoid that sort of pitfall of only asking questions that are easy to answer, because they're not always the right questions.
You know, credible evaluation hinges on having the right questions to guide the work. And, you know, those right questions might be hard to answer. We might only be able to get approximate answers.
But still, it's really important to ask the questions that really matter. Yeah, yeah, I definitely don't disagree with that. I think it's also a consideration for you to, you know, what can you reasonably demonstrate within the timeframe that you've been operating in? And it might be that, you know, you can't really do a good job of speaking to outcomes and impacts at the particular point in the programme that you're at.
And you might want to pivot that focus to something that's a little bit more temporal to your programme, and something that gives you a bit more immediate feedback about how you're implementing. So, you know, is your programme being implemented well? You know, is it kind of operating in the way that you'd expect it to? And I think those kind of things are really important to answer at the beginning of the programme as well. What I'm hearing from both of you really is that it's kind of that staging and trade-off.
So what do we have? What can we do with it? What do we need to know? And what can we tell within the timeframe? Absolutely. Yeah. And I think building on that, it's also really important to know where your gaps are, and just focus on filling that.
So if we have a vaccine programme, for example, I mean, you're not going to need to evaluate the efficacy of the vaccine, because it's already been established. So instead, you'd probably focus on the vaccine implementation and how it interacts with the context of the implementation site. So just really tightly focussing on what is most crucial.
I'm keen to hear you guys' thoughts on if I was a small team in like a NGO or local government, and we don't have, I might be the person on the evaluation team, how can I evaluate the work that we're doing in a way that's really cost effective? Yeah, it's another good question. So for, I think, local government teams or small programmers looking to evaluate their work effectively, my advice would probably be the same. There are several strategies that can help.
For example, I think, you know, in an ideal world, you may want to conduct a evaluation independently, but that might be cost prohibitive for you. So I would think about either whether you can bring that evaluation internal, or conduct components of the evaluation internally, or take over the kind of data collection component and provide that to an external evaluator at a later point. And I think one of the key things we've been really talking about is trade-offs, because there are trade-offs associated with that as well, though.
It's obviously cheaper to do it internally, but you've got not necessarily bias, but perhaps a perception of bias with having an internal evaluation. Yeah, yeah, I think that's right. And I think it's important to just be transparent about that when you're discussing it and designing it and reporting on your results that, you know, that this was, you know, an internal evaluation.
But I would also urge that if it's a choice between, you know, no evaluation and getting the information that you really need to do a good job for your programme, I think it's a worthy trade-off. Yeah, and I think, you know, I agree with the comments about potential conflict around independence and bias, but, you know, you can also achieve a degree of independence within an internal evaluation, depending on, you know, who's doing the evaluation internally and where do they sit in the organisation versus those that may have designed or be implementing the programme. So we've got a question here, and this is from Tracy, and she asks, what about the approach for evaluations for small organisations that are only funded for programme delivery with little or no budget for evaluation? So have we got any really sort of cost-effective or cheap methods? I mean, you know, I hear that challenge and it's a real issue at the moment.
Some of the tips for that, I guess, I mean, yeah, you want to think about the methods. As a smaller organisation, you might already have some, be collecting good data alongside your programme or the initiative is that you're wanting to evaluate, that can be a really important input. I think the issue around sort of, you know, there are certain efficiencies you get over having a small team or one person focussing on the evaluation.
If they're internal, I guess the other benefit is they come with existing knowledge of the programme. But yeah, I mean, that is, it is a really big challenge for small organisations. So what would you do in terms of methods if you just need some quick data or cheap data? Well, I mean, you've generally got to talk to people.
So focus groups are a good way of collecting data relatively efficiently from a group of people. And you can do those focus groups online these days, which, you know, leads to efficiencies. And, you know, online methods, I guess, can also be, you know, as well as being cheaper, sometimes they can be preferable as well for certain stakeholder groups.
But I know you're probably going to raise the issue about trade-offs again, because it is an issue of trade-offs. Yeah, I mean, I won't try not to be a broken record. But I think that is part of the compromises that we're talking about making, that you may well get better data from going face to face and talking to people.
But if it's online, that's what you can afford, that's what you should do. I mean, another possibility there is to look at joined up evaluations. So to either work with other organisations and doing combined evaluations, or to look at several of your programmes at the same time.
So that can be quite a good way to, you know, to save resources. And it can also be a good way of, you know, bringing in greater learnings by looking for, you know, how things might be working in different contexts. But, you know, again, that sort of collaborative evaluation, collaborating with another organisation, or even across your organisation, you know, introduces some complexities.
I'd say so, in terms of a relatively cost effective method, surveys is always a good one, because you can get a lot of data from quite a lot of people, potentially quite cheaply. Again, if you're using online methods, and mostly quantitative, it can be a really good cost effective one. So I'd encourage using that as well.
Yeah. And sometimes we take an opportunistic approach to data collection. So I mean, for example, we're currently evaluating, and this isn't a small programme, or small evaluation, but it, you know, demonstrates efficient methods, I guess.
So we're currently evaluating a palliative care programme. It's across Australia. So, you know, potentially very expensive data collection.
But what we've done is we've aligned that data collection to, you know, existing conferences that are happening in centres around Australia, that we know all the key stakeholders are going to be attending. So in this way, we can focus data collection, you know, around two or three days around each conference, and it also really limits sort of travel expenses. I guess the other thing I'd say is, you know, the methods we might use for data collection, you know, need to be adaptive, and to make sure that we, you know, the data we're collecting remains relevant.
Yeah, I think those are all really, really good suggestions. And, you know, I think also, if you are interviewing, which can be really time consuming, one option is to use what's called a convergent interviewing technique. So that's where you analyse the data continuously as you go through the interviewing process.
And it's helpful because it also, you know, prompts you to be reflective on what you're hearing. But it also enables you to be a little bit agile about where you're directing your interview focus to, you know, perhaps move the interviews in a direction towards kind of, you know, plugging gaps or kind of those more strategic insights that are emerging as you as you go and uncover them. So under that technique, you're doing a few interviews, analysing the data and then using your next set of interviews to more fill gaps rather than kind of do the full interview.
Yeah, that's right. And I think another point is that it's quite possible that you will reach data saturation earlier than you expect to. And so you know, that can enable you to be a little bit strategic about where you focus your efforts and consider where you might be able to rebalance efforts if you start.
And by data saturation, I mean, hearing the same thing and not really uncovering new insights from your engagements, you might be able to invest your time elsewhere. Thanks guys. So I guess looking ahead then, you had to cast a crystal ball.
What would you say is going to be important in the future of evaluation? Well, I'd say in terms of looking for the future, there's going to be the stronger emphasis on demonstrating impact and attribution. I think that'll continue. And that's going to require us to work in interdisciplinary teams.
So, you know, that's bringing on data analysts, economists, et cetera, you know, embedded in our evaluation teams. And that's about giving us a more well-rounded perspective and, you know, more well-rounded interpretation of the results we're seeing. And, you know, I guess for a similar reason, I'd also say that it's going to see us working more in collaboration with local evaluators or locally based evaluators to ensure that our evaluations are both contextually and culturally relevant.
And, you know, that can also help to manage travel expenses. Yeah. So by locally based, do you mean kind of in country when you're doing international development evaluation or more broadly? More broadly than that.
I mean, you know, we often do evaluations covering large geographies, whole of New Zealand, whole of Australia, having locally based evaluators or kaitiaki to help guide our work in those locations can be really valuable. What about you, Carolyn? What do you think? Yeah, I don't disagree with any of those things that Ned said. I also think increasingly we're going to be moving towards big data and open data and, you know, using those data sets, integrated data sets to demonstrate and direct policy.
And I think in that space, data visualisation and data translation will increasingly become important. So it won't just be about collecting it and analysing it. It will also be about communicating it better and more effectively to a broader audience.
Yeah, I mean, that raises a question about audience, which we haven't really covered yet. But that's, you know, I guess some audiences for our evaluations expect quite technical outputs. But generally that is on top of, you know, very accessible products that can be understood by a variety of audiences.
Yeah, so that sort of speaks maybe to the democratisation of evaluation or moving away from seeing ourselves as sort of technical experts and the specialist skill that no one else can have and really making sure that what we are doing and saying and presenting and reporting is really accessible. Yeah, that's right. Yeah, yeah, I think so.
And I think, you know, AI is going to affect everybody and evaluators are not going to be immune. So, you know, AI is already being used in the space of text analytics and that kind of thing. And so I'm interested to see kind of how that how that evolves, because it could have a really significant impact on our work.
Yeah, what kind of work do you see AI doing for us as evaluators? Well, I mean, you know, not AI necessarily, but like we're seeing increasing use of machine learning programmes, not just by people like Google, but it also has application to, you know, to policy settings where you're using linked data like hospital records or social service data, census data, you know, where it could potentially predict the better targeting of programmes and interventions to the people who need it. So I think, you know, the predictive power of data in terms of how we direct and design our programming interventions could be huge. Yeah, and I completely agree, but it really, to me, brings up the importance of having those safeguards in place, because it's something that just feels a bit big brotherish about using data to predict who might need services and when and what time.
So I think we're going to need to see a real need for some strong conversations. And for evaluators, it's also about data sovereignty and how we can use data and who owns the data. And there's some big conversations that we've started to have, but I think we need to have more of.
Yeah, yeah. Yeah, I mean, to me, AI also has the potential to support evaluations in real time. And I think that's going to be a feature of future evaluations or a stronger feature of future evaluations.
So, you know, I think the days of having evaluations of programmes, say once every three to five years, looking back in terms of how backwards that a programme in terms of what's been achieved, I think those days are over. I think the demand is shifting towards evaluations that generate feedback and insights, you know, to inform adaptations and improvements in real time. So, you know, as evaluators, we become facilitators of change, if you like, which, you know, that's something that really appeals to me.
So that's going to require evaluations to be more embedded in evaluations and evaluators, to be more embedded in programme planning, design and implementation, and also to be nimble, timely, flexible. Yeah, but how do we do that? Well, if we've still got this resource constraints, which I think will actually continue into the future as well. Yeah, I mean, on the one hand, you know, real time evaluation can be expensive.
On the other hand, it allows you to look at, you know, particular, I mean, as we've been discussing today, really, focussing on particular issues which are relevant to the implementation stage of your programme. So if you're in an early implementation stage of a programme, and for example, it might be a service that's recruiting people into the service, well, you might look at, you might have a focus on, oh, how well are our referral mechanisms into the programme working? Yeah, and I suppose you mentioned that real time evaluation can be more expensive, but it's less expensive to invest in it, find out if things are going wrong and correct them rather than get to the end of a five year programme and find out actually it didn't work. Yeah, precisely, I mean, you know, that those sort of potential savings and opportunity costs are massive.
Yeah, I would agree. I would agree with that. And I would also say, like, to your point earlier, Marnie, evaluation is no longer going to be the domain of technocrats.
And so people are really time poor as well. It's not just evaluation that's becoming resource constrained, it's everybody. And so we, I think, will increasingly see, you know, a demand for snappy, accessible, short form action and insight rich reporting.
And so I think, you know, our ways of communicating need to become, you know, more comprehensible to people who are not from a technical background. And just deliver those key actionable insights. Yeah, and I guess I'd add that, you know, we does feel like we're in quite a different economic context.
So, and I don't see these financial constraints going away any time soon. So it's going to remain a real priority for the government and for other funders to be able to demonstrate, you know, what's working for whom and in what context, and equally to know what doesn't work. So I think that understanding of cost-effectiveness of policies and programmes is going to continue to be important.
And possibly also cost-effectiveness of evaluations. That's true. I mean, we should really account for what we're doing and its costs.
That's right. Okay, thanks. Well, look, we've got heaps of questions that have come in.
So we'll now move into our question and answer session so we can get through them all. So first up, we have a question from Stuart. And Stuart is asking that governments are increasingly focused on social investment.
And Stuart's wondering how as a not-for-profit, could we evaluate our programmes to take this into account? It's a really good question, Stuart. I think, so assuming this question is about demonstrating the, you know, social impacts and, you know, the relative, against the relative investment of the programme, I think it is a really big challenge, particularly for smaller or not highly funded not-for-profits. I think starting with a good theory of change is a great place to start.
I love how you're constantly flying the theory of change, Mayor Caroline. I know, I am. Sorry.
But I completely agree. The challenging piece is, you know, arriving at clear, defined outcomes. And once you've got those, can you put a value against them? And I think... Can I ask that by Caroline? Do you mean like by a sign of value? Do you mean a financial value or more broader? Yeah, potentially a financial value.
So for example, if you were looking at a youth justice prevention programme, a youth crime prevention programme rather, you may, you know, one of your outcomes might be reduced justice contact. And one of your financial indicators of that might be, you know, reduced time spent in remand, which has a financial cost as well as a social cost. And the financial cost is something that you could perhaps establish through research.
So what does it cost the government to hold young people in remand? And then the social cost is something that you might want to define through your process of developing that theory of change. What is the cost to a young person who is, you know, having school disrupted or having, you know, escalating contacts with the justice system? And then if you can define those monetary values and be upfront about your assumptions around those, you then kind of want to pitch that out to a time horizon. And, you know, then you can kind of look at what, you know, what money you may have saved by diverting young people from that particular pathway.
So that's how I would go about it. But it's complex. Thanks, Caroline.
We've got another question here from Sonia. And that's about the independence of data and power imbalance is hard to manage when doing evaluations internally. And Sonia's wondering if we have any tips around that.
Yeah, I mean, I think we just started to discuss this earlier. I mean, I think that it is a key challenge. I mean, we mustn't forget about the potential bias from external evaluators too.
Yeah, I think we sort of have this assumption that if we're external to the programme, we're going to be completely objective. But, you know, we're human too. Yeah, that's right.
I mean, I'd go back to that, to I think what I said earlier, that, you know, even within an organisation, you can have a degree of independence from the programme, you know, if you haven't been involved in that programme design, if you're not involved in implementing the programme, then you can sort of sit outside that to some extent. I guess other strategies are around having an external sort of person that could act as more of a kind of an advisor or a sounding board, help with some of the interpretation and just also sort of safe, you know, providing advice on sort of how you might safeguard that some of those processes around potential bias. Yeah.
And we have a question from Georgia. And that's for Aotearoa evaluators, but I think it probably is also relevant to Australia as well. It's how do we ensure that resources are still given to whanau ngatanga and koha or the kind of for Australians, the relationship building and reciprocity component during the widespread budgeting constraints.
I can kick that off. I reckon it's just about advocacy. Like if we are, you know, as evaluators, whether we're independent consultants or embedded within government departments or organisations, I think we still have some power to be able to say, look, based on our expertise and our practise, that this is important and needs to be assigned resources and kind of selling it in a bit of a hard way that if we don't actually invest in that relationship building, the data that we are able to obtain is not going to be very good.
So you might want a hard economic answer, but you're not going to be able to get the data to do that if you don't invest in some of these more so-called sort of soft investments around relationship building. I've got another question here from Kirsten. Kirsten's asked, how do you use qualitative information when there's a bias towards quantitative analysis? Okay.
I mean, that's an interesting insight on this bias towards quantitative analysis. You know, I feel this too. But I also think there's an understanding that it's not always feasible to demonstrate and attribute outcomes using quantitative data or quantitative methods.
So, you know, back to our, I think you were saying earlier, you know, you can't ignore mixed methods as the kind of gold standard in my point of view. And then I'd say, you know, how can you use that qualitative information? I guess this relates to an earlier question I think we had from Jesse or another listener, which is, you know, that opportunity to, you know, really focus down on collecting rich qualitative information, you know, developing those into, if you like, impact stories, which really hone down on, you know, what changes have occurred, looking into how the programme has contributed to those changes, looking into the significance of those changes to the people, to the organisations, et cetera. And then also, you know, identifying your lessons learned from that.
So, you know, I think that that can actually build, you know, really powerful contribution, you know, perhaps not attribution, but very powerful kind of contribution stories. Yeah, and I guess I'd maybe put that to you a bit, Caroline, as a kind of resident quanty. What kind of convinces you that there is some value in still using qualitative information? Yeah, well, I think it's certainly complimentary.
Like, I think Mani used the phrase, you know, the human experience. And I think those quotes and those stories do bring to life the human experience and help, you know, any evaluation storyteller tell the story of what it is to, you know, what does the statistic mean when we're looking at one person experiencing this? And that can really, you know, help relevance and help translation of meaning and insight. And I think it can also be a really powerful advocacy tool.
I think it's more than that as well. It actually helps to understand the quantitative information in some cases. So I'm thinking of an evaluation we did of an early childhood intervention, where we found that families that were engaged with the programme were more likely to be reported to Oranga Tamariki, or to engage with Oranga Tamariki, and more likely to be using mental health services.
And on the surface, that looked kind of bad. Like, why are they engaging? Does it mean they're actually in a worse place to start with in the control group? Or was there something else? But the qualitative information helped us to really explain that it was actually about access and connection. And that was sort of our hunch, but we would not have been able to demonstrate that if we hadn't done quite significant qualitative research as well.
So, yeah, I think there are stories we can tell to keep the value of qualitative there. We've probably got time for one more. We've got one from Pip, who's asking how to best prioritise numerous or competing demands for evaluation data in Intel? I think we've talked about ways of how we might prioritise evaluation resources in terms of really honing in on one or two key evaluation questions and framing your data collection and analysis around that kind of quite narrow focus if you've got limited resources.
So that's really going for the depth and trying to answer a single one or two questions as well as you can, as opposed to trying to sort of more of a scatter in terms of like trying to collect a whole broad range of information that only gives you sort of real partial answers to many questions. Okay, well, that brings us to the end of the formal content of our webinar. I am keen to get your thoughts, though, on what's your number one key message for doing a credible evaluation in a context of constrained resources? I've been banging on about it the whole time.
So I'm going to just buckle down and say I think it's really important to set your foundations up right. And that means understand your intervention and understand what your outcomes are and what you might already be collecting to demonstrate those. And I'll probably jump in a step before that even, as part of your framing and evaluation, by really focussing in on what matters and what matters at the particular point in time where your intervention is at.
Yeah, and for me, it's about understanding that you have to make trade-offs. So going right ahead and doing those, but being explicit in any reporting that you do about what compromises that you've made and any effects that they might have. Yeah, but don't get into the trap of not doing evaluation.
If you've got some data, it's better than none. So thanks, everyone. Well, that's us.
So Ngā mihi, thank you very much for joining us, and we hope to see you all at the next webinar. Kia ora.
