Turning Vision into Value: Practical Tools for Social Return on Investment Analysis

Tēnā koutou katoa. Welcome to this Allan O'Clark webinar, a two-part series on social return on investment. So ko Brennan Stephen ahau and it's my pleasure to welcome you to today's kōrero.

For some this is your first time joining Allan O'Clark webinar so you may be thinking who are we? Well we are an Australian-based consultancy working with government organisations and communities throughout Aotearoa, Australia and the Pacific. Our areas of speciality include strategy, change management, evaluations, research and impact assessments like an SROI. As I said today we're going to be in depth on social return on investment so we'll cover how to create robust theories of change specifically designed for SROI analysis with ready-to-use templates.

Master practical frameworks for tracking and measuring your social impact and transform your qualitative insights into compelling quantitative evidence using a CBAS compatible tool. Strengthen your funding applications with clear evidence-based impact measurements. There's a live Q&A at the end of each part but we also take questions as we go so pop them in the chat as they come to you.

After the session we'll send a copy of the recording the slides and a summary of the key points. Before we start a few intros so I'll pass to my colleague Fiona. Kia ora Brendan.

Tēnā koutou katoa. Ko Fiona Scott-Malton ahau. Before joining Allan O'Clark I worked for the Office of the Auditor-General where I led audits that focused on the extent to which public agencies were achieving the desired outcomes from programmes or investment.

Through that work I developed a nerdish interest in how do organisations know if they're making the difference they're hoping to make. It's led me to doing things like outcomes evaluations, developing roadmaps to support implementation of strategies and programmes and looking at measuring performance. For me and us at Allan O'Clark theories of change aka intervention logic, outcomes change, play a critical role in helping organisations assess their impact.

A good theory of change is foundational to calculating social return on investment along with a monitoring and evaluation framework. Back to you Brendan. Kia ora anō.

So in my recent working life, I'm going back 25 years, I've worked and continue to work at Massey University in a bunch of roles. Then I went to Te Hiringa Hauora Health Promotion Agency and now I'm here at Allan O'Clark. I'm a mixed methods researcher and evaluator with a stats background and a nerd level interest in things like CBA, CBXs, SRIs, BFIs, RIs, AIs and all manner of acronyms related to numbers.

Our past back to Fiona is going to lead us through this first part of today starting with the SRI specific theory of change. Thank you. Okay in a previous webinar, dollars and cents measuring social impact, we talked about how social investment is about improving outcomes, getting better value for government spend in the social sector.

Today in this session, we're going to take a deeper dive and look more closely at creating a theory of change and monitoring and evaluation framework that lays the groundwork for calculating a social return on investment. For this webinar, I'm assuming that you understand some of the basics such as things like input, activity, output and outcome. Along the way, I'll be sharing a few tips and tricks as well as some key considerations in the role of assumptions.

So to take us away from simply theory and into the world of practise, we've created a fake NGO, Inspiring Youth Mental Health, YMH. Inspiring YMH provides a service for rangatahi 14 to 18 years old who are experiencing anxiety and or depression. These young people are typically not thriving at school, may face multiple social educational challenges, are at a higher risk of dropping out of school and becoming a NEET, so not in employment, education or training, and could experience harm from alcohol and or other drug use.

The service aims to support these young people to improve their mental health and put their life back on track. The objectives of the service are to support young people to develop good strategies for managing anxiety and or depression, to work alongside these young people providing guidance and advice that helps them identify and pursue their aspirations, and to support young people to be their best selves. To do this, Inspiring YMH have a small local team of well-trained counsellors and mentors who specialise in working with rangatahi.

Inspiring YMH will be applying for renewal of their funding in six months and need an SROI to support their application. So just to give you a bit of a high-level overview of how we were going and how we approach an SROI, our approach involves seven key stages or parts, which you can see on your screen. In the first part of today, I'll be going in depth on stages two and three, so creating a theory of change and creating a monitoring and evaluation framework.

I'll also touch on the value of conducting an evaluation if the programme has been implemented and there hasn't been a recent evaluation completed. Then in part two of today's session, Brandon will cover the final three parts of calculating your social return on investment. You'll notice that there are arrows in our process that go backwards from calculating the social return on investment domains to creating the theory of change in each step in between.

This is because it's an iterative process, and so what happens in the social return of investment calculations also impacts on the theory of change and the monitoring and evaluation framework. What we're aiming for is that the whole thing aligns nicely and things emerge over time as we work our way through. So, as I mentioned right at the beginning, you may also know a theory of change as an intervention logic, as a programme logic, as an outcomes change, or something similar to that.

To build a theory of change that supports calculating the social return on investment, I find it's helpful to work from left to right, i.e. you're starting with the inputs, the activities, the outputs, and short-term outcomes. And when I may say outcomes, it's what you expect to achieve in the short term, what difference you expect to make. Getting a range of key stakeholders involved in the process helps ensure a robust theory of change is developed.

We're not trying to capture everything. Rather, we need to identify the key components that are going to go into our theory of change. To help us with the process, I always start with identifying, firstly, the situation.

What problem is the programme trying to solve? Why is the programme needed? And on the flip side, what's the opportunities? Then we look at the goals. These are generally the broad or long-term results or outcomes that the programme is aiming to achieve. They should align with the programme's vision or purpose and or purpose.

And then there's the objectives. These are the measurable actions or steps needed to achieve that overall goal. The situation, goals, and objectives set the scene for developing the theory of change.

They guide the process and provide something to hang the theory of change off. So if we just move along, we're going to now look at the key inputs. For this particular programme, we've identified that they have, at a high level, they have the workforce, such as counsellors, mentors.

They may also have an administrator. Then they need funding as an input. They have relationships with schools, key support services, and families.

And then they have a premises in which they use as a base. Getting this right is important, as the inputs provide the basis for establishing programme costs. So they establish one side of the social return on investment calculation.

The other next thing we have to do is we look at the key activities we've identified. And for this, we identify that counsellors and mentors interact with young people. They have professional development for counsellors and mentors to ensure they remain up-to-date and follow best practise.

We also have obtained funding from government and philanthropic sources. They have built relationships with key stakeholders, and they build relationships with whānau and hapū, because the programme believes it's important to work with families as part of creating an environment that supports young people to have better mental health outcomes. Then, when it comes to the outputs, I try not to merely reflect the activities, such as counsellors and mentors interact with young people, as they don't add much.

I ask myself questions like, what could be counted and or could be added to the picture? For the purposes of today's exercise, we've identified the following outputs. Young people who completed counselling treatment. Young people who have developed a strong relationship with their muenpo and met with them on a regular basis.

Inspiring YMH has built strong relationships with schools, local youth services, and children and adolescent mental health services. So they're able to refer them on and work with other services to ensure people get young people get the support they need. Then we have inspiring YMH has built and established relationships with whānau, families and hapū.

And counsellors and mentors have been supported to develop these skills and to keep up to date. Hi, so just before we move on to outcomes, there was a question from Sam who asks, how would you include kaupapa Māori frameworks into your theory of change? Yeah, that's a good question. When I think about including kaupapa Māori frameworks, I think about the importance of adopting more of a holistic approach to the logics.

So it means there can be a broader spectrum of outputs and outcomes that need to be included, especially the outcomes. One approach is to use more of a systems approach. So you can include an additional section to the theory of change that highlights these broader factors, influences, and depict them in the diagram.

The other thing to do, I think, is I do this even with the template and explainer, is to give an indication of the overlapping and interrelated aspects of the theory of change. Although we've got little boxes, they actually interrelate and interact with each other. They're not all completely separate like they might look.

So what you might want to consider is whether you want to use a different diagram so that visually it's also more apparent these overlaps and how they all interact. I've seen some different diagrams used to depict kaupapa frameworks that also carry broader metaphors, which is really awesome. The other thing is that when you're thinking about calculating the difference it's making, so the performance measures, a single measure may not capture it because you've got all these different factors involved.

And so you may need to use more of a clustered approach. Yeah, there's always a tension between a holistic measure within kaupapa Māori frameworks and when you start pulling out individual ones. So you're starting to disentangle that, but you're also missing the whole point of holistic measures.

So they don't necessarily play as well, but they can be done. It's just you have to do a lot more thinking and at the measurement end especially, whether it's even possible to do any sort of SRI ratio, any kind of monetising of it. And if you disentangle it here, can you pull apart little pieces inside a holistic framework? Yeah, so hopefully Sam, that answers your question or gives you some indication at least in how we'd approach it.

So for this, in terms of the short-term outcomes, I think of them as within the severe of control of the programme. So what the programme directly contributes to. So these usually emerge within 12 to 24 months of somebody being part of the programme.

Or you could capture these by doing a before and after survey. So when before they join, start using the service, to afterwards they've used it. They need to be measurable.

So ideally you both want quantitative, so numbers, to support the SRI. Qualitative, so think words or stories, which will help tell a rich story about the value of the programme. For this programme, we identified that we expected it would lead to reduced anxiety and depression symptoms.

Young people would have improved self-esteem and self-confidence. Young people have improved engagement in education, training and or employment. And there's also a short to medium term outcome, so they don't always stick in the short term, sometimes they're a bit longer, is that young people have improved social connections with their peers, family and broader communities.

But that also could mean that for some young people, it could be that they have better connections with their marae, with their hapu and with their iwi as well, as part of strengthening their cultural identity through the programme. What could lead on to that? I see the medium term outcomes as those outcomes that are within the sphere of influence of the programme, and long term outcomes as those outcomes that are of interest. There's a broader range of factors that influence these outcomes that are outside of the control of the programme itself.

And just to add to the qualitative evidence, the qualitative evidence also is a check that the quantitative measures are still relevant and still appropriate. So it's a way of checking over time that they should still be the same things. Remember, we're happy to take questions as we go, so just pop them in the chat.

We have another question here, this time from Sarah, who asks, how do you ensure your theory of change outcomes directly map the measurable indicators that can feed into SRI calculations? Yeah, we've been thinking about this quite a lot, quite a lot. So I think the outcomes provide both a basis for developing performance measures. So going back to the point Brandon was just making, those performance measures, when you have those quantitative and qualitative, you're able to check, they're about tracking over time, are you achieving what you're hoping to achieve, and to what extent are you achieving it? So it supports that sort of thing.

The other side of the outcomes is about identifying benefits that can be monetised. So these are kind of your indicators, benefit indicators that can be used. To do this, I use tools like the Treasury CBEX tool and all the Australian Social Value Bank to identify indicators that could be used to calculate a social return on investment.

So these already have monetised values that you can use to help track it. So the Treasury has some of the Australian Social Value Bank measures, and they're available for free, but if you use the Australian Social Value Bank, it's about, I think for not for profit of memory, it's maybe three and a half thousand dollars for a year or something like that, might be a bit more. There are other ways that you can go about this too, but I find these to be a really good starting point and relatively economical.

So a tip, at this point, to guide developing the outcomes, I start thinking about what would you expect to logically flow. So I'm looking to answer the question, if such and such was true and such and such was true, then what would it be? So it's not a one-to-one relationship necessary between each short-term outcome and the medium-term outcomes. And we link here also about what does the evidence tell us and insights from key stakeholders.

So here we've assumed that if young people have reduced anxiety and depression symptoms and have improved self-esteem and self-confidence generally, it will lead to improved mental health outcomes and physical outcomes probably too, but particularly mental health, contributing to improved quality of life in the long term. We've also assumed that improved engagement, education, training and employment will lead to or contribute to these young people having sustainable and meaningful employment, leading again, contributing to improved quality of life because it's one part of that, and also contributing to higher productivity at a society level in the long term. And then the short to medium-term outcome, improved social connections expected to contribute to improved civic engagement.

So just to summarise, here are my top tips for an SROI specific theory of change. One is capture all your inputs, clearly define all resources, staff, funding, partnerships, buildings, infrastructure, systems, as it helps ensure accurate cost calculations and a social return on investment. Then focus on the key outcomes, identify the outcomes that can be measured, but also the ones that are key for you to capture, you don't need to capture everything, you don't need to measure everything.

Then you want to define your assumptions, explicitly state why and how each outcome leads to the next. And where possible, you want to be using evidence and stakeholder insights to support that. And then you want to differentiate between contribution and attribution.

So you want to identify what change is directly caused by the programme, that it's directly contributed to, versus external influences. So the reason why I differentiate between short, medium and long is by the time you get to long term, your attribution is probably really small, especially if it's just one programme, unless you're serving a vast number of people, it'll be a small contribution or attribution that you can put to that long term outcome. And then you want to design for iteration, adjust the theory of change, don't treat it as though it's absolutely fixed forever, as you learn and understand more, or as part of your social return on investment process, you adjust it where needed, such as based on our evaluation findings and or to ensure the programme remains relevant.

You wouldn't expect huge changes, but you might need to make tweaks here and there just to make it more relevant and make sure that it's up to date. Thanks, Fiona. Before we move on to the SRI M&E framework, here's another question from Toby.

How can we develop our SRI theory of change in a manageable way? Many of us have very limited resources. Yeah, yeah. Perennial problem.

Yeah, it is a perennial problem. And it can be, I've encountered this, it can be quite a barrier for people as well. So I think there's a couple of things you can think about.

The first is just to start with what I call a skeleton theory of change. So it's really more of an outline than any real detailed one, wouldn't be nearly as detailed as what I've showed you today. And you can include just the most critical inputs, the most critical activities and outcomes that you need to identify track over time.

So just remember, you can always come back and build on it and develop and refine further at a later date. But what you're thinking about is, if you might want to ask yourself a question, if there was only one or two little things that I was going to track, what would they be? What would be the most important things to start supporting me to tell a good story of actually the difference that this programme is making? And you want to be testing to that it actually is making a difference. It may not be making a difference that you hoped for, in which case you might want to know, oh, this is not working.

What do I need to do? To simplify, you could also just have outputs instead of activities. So you don't have to do activities and outputs. You could just say, oh, look, I'm just going to go for the outputs.

And those are things we know we for sure deliver. So you could simplify in that way. Other options include look on the internet.

There are lots of really good examples there that you may be able to gather some ideas on. So you can think about your particular area and you can go and look and say, what's somebody else done? I use that often to actually kickstart my thinking. And another one is that some places like the Ministry of Education have provided theory change building tools that you might find helps aid the process.

Yeah. Also, I think you do your best, but sometimes you have to live with a greater level of uncertainty because you just haven't put the resources. So just accepting that broader range of estimates.

Yeah, I'd agree. Agree. So once your theory of change is in place, the next stage involves developing a monitoring evaluation framework.

So by clearly defining what success looks like when we're developing the theory of change for each identified outcome, the M&E framework establishes what data needs to be collected, both quantitative and qualitative. The framework will also set out where the data will come from, how often it should be collected, and will consider the feasibility of data collection, such as how much will it cost. Investing in the M&E framework is important for two key reasons.

The data you collect can be used to further refine your SROI and to conduct outcome impact evaluations. This information can be used to support future funding applications as it'll help you demonstrate the difference you're making, including whether the programme is serving the desired population groups. Secondly, it gives you the data and evidence to track your progress against the desired outcomes, enabling you to determine whether your tracking is hoped, like I mentioned before.

If you're not, you can make changes to the programme early so that you can achieve what you'd set out to achieve, or make adjustments to the outcomes if needed. In the process, you can reassure funders you're on top of what's happening. So this approach ensures that SROI is not just a retrospective analysis, but an ongoing measure of progress that informs decision making and future investment.

While collecting good data is crucial in any kind of investment activity, we also need to understand who is at most need, what works for them, and monitor progress. And the M&E framework is a place to have these discussions. For inspiring YMH, we want to know not only whether it's achieving its desired outcomes, but also for whom.

So you can look at gender, you could look at ethnicity, whether they're Māori, Pacific, or some other group. You could look at things like, are they neurodiverse? Do you know if they're neurodiverse? Are they from the rainbow community and needing particular supports? Is it working for them? And that will give you an insight into whether your service is equitable or not. So that's something you really need to do.

The M&E framework is not just about numbers, which can provide hard information about outcomes and support monitoring progress over time. We also want to collect, as we mentioned before, qualitative data, which not only provides rich stories, but it also captures the difficult stuff to quantify. There's some things that you can't really actually quantify.

And it's often a check on whether what you're measuring makes sense, as Brenda mentioned before, and also can help you interpret your numbers as well, so all that quantitative evidence. Part of the monitoring evaluation framework involves preparing to carry out evaluations, either now and or in the future. The evaluation part of the framework is where the approach to the evaluation is set out.

We're not going to spend time on this today. We cover approaches to evaluations in another webinar on Alan Clarke's website that's available, which is called Credible Evaluations. But the evaluations part of the monitoring evaluation plan covers things like identifying key evaluation questions, criteria, so areas of focus to determine how well the programme is performing, and indicators of success and approach to data collection.

Oh, we have a great question on this from Jenny. How do you balance rigorous evaluation with the reality of limited budgets and time constraints? Well, I reckon the trick there is that what you want to do is size your evaluation accordingly. So there are different sizes.

You don't have to do a really massive evaluation. You could do quite a rigorous evaluation, but have it very focused. So you could either do that by actually looking at quite focused areas.

So you could look at specific short-term outcomes that you're particularly interested in. You could look at broader outcomes. But also you can do things like instead of, say, visiting three locations to try and collect evidence, you could reduce it to one location.

You could use documents, which may not cost as much to review. You could do things like, I was also going to suggest a more selective number of interviews, so just really be really targeted in who you're talking to. Having said all that, you then need to have to acknowledge the limitations of it, that it may not be quite as robust as you'd hoped, may not carry quite the same level of certainty as you'd like, but it should give you a pretty good indication.

Have you got any thoughts? Yeah, and I agree. I mean, you're clear in when you're reducing your sample size, essentially. You have to understand what biases you might be incorporating into your findings.

So no who's now doesn't have a voice, who you're excluding. So at least it's clear who you are representing in your findings and your evaluation. Also, if you have been creating quantitative data, you could actually use that to guide you in it.

So you could be saying, I want to hone in on this group. So I want to know, particularly for rainbow community, how well it's working, or particularly for Maori communities or Maori and Pacific communities. So you could use and say, they don't seem to be getting such a good service.

It doesn't seem to work for them. We want to know why. And you could go and use the data for that purpose too.

So it'd be another approach to actually really just focussing your evaluation. So if the programme has already been implemented, we recommend having an evaluation completed as part of the process. And what hasn't been done recently is it helps inform the SRI, proving and writing a clearer picture of what is actually being achieved.

Otherwise, the SRI could be a bit of a theory. It could be in theory you're achieving this, but how would you know if you don't do an evaluation? So a key component of the monitoring plan involves establishing the performance measures. These should be specific, well-defined and feasible.

A good starting point for developing performance measures includes considering what data do you already collect? What measures do you currently have? Could some of these measures be used or just need a bit of adjustment to actually make them fit for purpose? You're aiming to have a mix of, as we've already said, quantitative and qualitative. So you're trying to have both to get that really rich picture. So let's have a look at some potential measures for inspiring YMH, in line with our theory of change.

My approach to the performance measures involves applying the logic established in the theory of change. For example, we have counselling and or mentoring leads to the short-term outcomes of reduced levels of anxiety and depression, improved self-confidence and self-esteem. These two short- conditions needed to achieve the medium-term outcome improved health, which leads to improved quality of life.

What we've developed is just to give you a bit of an idea. So developing the performance measures is the complicated bit. It's when we look to engage with community key stakeholders to understand in more detail what success would look like from them.

We also might engage with experts or look at, as there are often pre-existing measures and standardised questions that can be used to collect evidence. We also draw on interviews, hui, to help determine what measures count. We may complete an evidence scan to help inform our thinking.

Or we may look to existing survey tools to guide us. For example, the New Zealand Health Survey has some questions that cover depression and anxiety. This data can be disaggregated for age group and gender.

Using the data collected can enable us to compare the prevalence of anxiety and depression in the young people accessing Inspiring YMH with a broader population in their age group. It can also provide the basis for a before-accessing-the-service and after-accessing-the-service analysis. So you could say before the service, there's a high, obviously you're going to have, say, 100% of having that experience.

By the end of it, you could find that it's significantly reduced. But then if you compare it against the average, that might give you a bit more of a richer picture of how well are you performing overall? Are you leading towards that? So the sorts of messages that we've come up with was the proportion of young people that report reduced anxiety or depression. It's a reduction in AOD use.

So what we're thinking about there is actually young people self-medicating and masking it. Or that could be also a contributing factor to the depression. A reduction in self-harm and a reduction in rates of suicide ideation.

Some of these measures, if you can keep track of these young people over time, which can be tricky because some people are transient, but could be used again if the information was collected, say 12 months later, post-receiver counselling and or support from a mentor to indicate improved mental health outcomes. Or could be based on a straight self-report. So just to summarise, a few points here.

Here are my main tips for developing good performance measures. Measure what matters. Align your performance measures with key theory of change outcomes, ensuring they attract meaningful change.

Balance quantitative and qualitative data where possible. Use a mix of that will support SOI calculations and qualitative stories for richer impact insights. Use existing data if possible.

Identify what's collected and decide if it could be used. Consider using current measures or adjusting them for continuity and also tracking progress over time, but only if they're relevant. Ensure feasibility.

Choose measures that are practical to collect, reliable over time and not overly burdensome. So although I gave you a list of measures there, you may not want to use that many. If you've got really limited resources, have a look and think what's the one or two that would actually cover it sufficiently to at least get us going.

You can link to social return on investment metrics. Select measures that will support your SROI calculations, e.g. changes in employment, wellbeing or educational levels. There's already some measures for those that can support it.

And then you want to enable continuous learning. So you want to use your performance data to refine programmes, adjust strategies and strengthen future SROI cases. So I want to just recognise that we've covered a lot of ground here today.

And some of you may have limited resources, time and access to people with the right skills, as one of the questions showed. To do this type of work, if you can't do all of this, all good. I'd encourage you to start with at least an outline of the theory of change.

Just make sure it captures the super important things. Similarly, at a minimum, think about what are the really, really important things you need to measure to be able to demonstrate the difference your initiative is making and to track progress over time. Check how they align with your theory of change.

If they're not captured in the theory of change, then adjust your theory of change so it does. So again, coming back to the point, everything should align. You can then add, refine, improve your theory of change and your performance measures as you get better at doing these things and as different outcomes etc.

become more apparent. Thank you. Right.

That's a pretty in-depth view of creating a theory of change in the modern evaluation framework for the SRI analysis. Now on to questions relating to what we've covered so far. If you have any questions, please add them to the chat and we'll do our best to answer them.

First we have from Ken, what are the essential elements a small organisation should focus on when developing their first theory of change? We've kind of asked parts of these questions. I think we have. So the essential is you need to know what you're trying to achieve.

So if you're not clear about your purpose or what you're trying to achieve through an initiative or service, then you need to crack that because otherwise you can't land your theory of change. It requires that because otherwise how would you know what outcomes you want if you don't know what you're trying to achieve at a really high level. The next thing you need to do, like I said, was you want to capture the key inputs and you just group them really up.

That could be quite light touch. I've seen some theories of change that don't do inputs at all. You can do one column that's just say activities or outputs, whichever you prefer.

And then you don't have to necessarily do short, medium and long term outcomes. You could do short, medium, medium, long, and you could just capture the really key things for you. I think that's the essential things.

And the big test for me always is when you've identified these things, check, could you measure it? How would you know that you're tracking that direction or achieving it? Yep. So I found us having a drink of water. This next question is from Molly.

How would you go about creating your first theory of change in M&E framework if your organisation doesn't have that expertise in-house? Well, one thing I would consider is whether you want to, if you've got any funding available, to buy in some support. So you get some support for somebody who's actually going to teach you and walk alongside you while you do it, but like a critical friend, that'd be one option. Like I said earlier, use Google or some other search engine to actually see what else has been there, because that can give you a good outline.

There's all sorts of things already been done before. For some things too, you can actually ask, try and start using AI and see what it comes up with. And you could say that you're buying this sort of service, what sort of outcomes does it think you might get through them? And you can sort through and it can actually do some of that mahi for you.

Similarly for the monitoring and evaluation framework, you can find examples of monitoring and evaluation frameworks. We're going to be providing some templates after this that you could use as a guide to support you to start building these things. And as I mentioned before too, things like Ministry of Education actually have a tool to support you to build it.

And final question from Rebecca. How can organisations effectively measure long-term outcomes when funding cycles are typically short-term? I think the measure of long-term outcomes actually period is really difficult, just because you've actually got an attribution problem. A lot of the long-term outcomes are really big, like higher productivity for a region that could be, or it could be something like improved quality, adjusted life years, quality of life.

These are really big things, which means that there's a lot of other things in there that can impact on those outcomes. So there'll be a lot of factors involved in these things that are actually got nothing to do with actually what your organisation is delivering. So you don't have that same direct line of sight.

But I think some of the things you can do is think about your logic. So one of the ways I've got people think about their long-term outcomes is that the assumption is if you're achieving your short-term and you're achieving your medium-term, then the assumption is that it would lead or contribute to the long-term. The attribution may be small depending on your scale.

But yeah, that's the assumption you're making, is that it actually will. And you can kind of keep track of some of the measures that statistics and others actually publish to see if there's any adjustments. Sometimes you can actually disaggregate that data down to things like age groups, in this case would be age groups, locality, that you might be able to hone in a bit more specifically and to see if it's changing over time.

But you just may need to be mindful attribution, like for the one that we did today, could be really small. Over time, it's much harder to determine what impact you've had. And that's why ratios that calculate do have a component of that, that's the long-term impact you're having.

There's a built-in assumption there. Oh, from Jose, how do you engage busy, frontline staff in theory of change development without disrupting service delivery? Yeah, that's tricky. I don't know that you can not disrupt at all.

What you can do is have quite a good preparation on the theory of change so that you've already got a skeleton so you minimise that disruption. The other thing you can do is most service deliveries have things like, say, team meetings where people do gather. So sometimes you can piggyback on some sort of mechanism you've got where you all come together to discuss it.

One possibility I've worked with is you develop theory of change with a small group of people who are available, and then you actually bring it through, say, to the service itself for them to feedback on it rather than be actually part of the actual building of it. So they get an opportunity to say, yeah, that resonates with me. And that can be quite a quick process.

Yeah, sometimes you can take advantage of existing team meetings where you can piggyback on those. Another question from David is what approaches work best for involving service users in outcome measurement when resources are limited? Okay, I guess the question I have here is the who and the service users can tailor it to those particular people's need. So one of the things that we've done, more on identifying what the outcomes are, is we've used visual approaches to actually help people engage and say, oh, these are the things that are really important for me.

So I think the great thing about visual approaches is that you don't have to have a specific language. It takes people into a different space. I've used this really widely with different groups, and it has real benefits.

You just really need to be capturing what's being said in the room, because otherwise what you've got is pictures, but you don't know what's attached to those pictures. So one thing is even getting images off the internet, icons, or you could also do videos, so you could actually collect it on a video form. It depends on what you've got and your limited resources as well.

But they'd be some of the things that I think about using to include service users. Yeah. It's minimising burden on them as well, and make it as accessible as possible.

I agree. I mean, I guess one of the things I think about some of these things, one of the reasons why I often come with a skeleton or develop something is because for creativity, we know that if you've already got some sort of solution in front of you, then you can actually bounce off that. So it's easier to say, no, that doesn't look right at all when you've already got a solution in front of you.

Whereas if I come with a blank page, it's really hard for people to know where to start and what to do. So I typically don't come with a blank page because it doesn't work very well. So how to reduce that burden is come with some ideas, and people could have done some of that work with them beforehand.

Hopefully you're collecting from them what's important to them, and that could be fed through so it's already in the theory of change ahead of time. And a question from Matt, how do you balance the comprehensiveness of your theory of change with the practical limitations of what you can measure for social return on investment? Look, I think we've talked to that to a degree anyway, and like when we're doing social return on investment, I work quite closely with Brandon to make sure that the whole thing aligns. So you're looking for where it can, where possible you can get those measures, but this is part of, we also have kind of a qualitative value statement that's also attached with it.

So I think it's important to measure the social return on investment is not the summation of everything. It's just one part of the picture of how well a service is performing and how well it's doing. So it's just one part of it.

And so you might say, look, these two things, outcomes are really important, but they're not actually quantifiable. And that's okay. And a related question in the chat was from Raymond, said it can be expensive and difficult to perform quantitative return on investment calculations.

Do qualitative outcomes need to be monetised in order to be of value? Is there value in interpretive logic? Good question. I don't think, I think there's value in qualitative period, because I think it provides part of that richer picture. So I don't think, qualitative collecting, that can be quite expensive.

It's quite time consuming. So I don't think it necessarily needs to be monetised. It might give you some thinking about some things that you could use to monetise.

But I don't think you necessarily have to. Yeah, and it is a check of what you're monetising. So there's always that interaction with what you're trying to do with any quantitative measures.

So definitely of value, you have to almost have to do them together, unless you're limited in resources, perhaps. So there's another question here around, that's quite a long one from Tom. So programmes situated within a range of interrelated contexts, social, economic, political, etc, where things go going outside the programme and direct to support or hinder the success of a programme, how would you recommend incorporating these external factors into a theory of change? Okay, so I have done that for a sector that was actually, some sectors in New Zealand, they're actually really carved up, you know, they're really piecemeal.

But we only look at one part of it. And so what I tend to do is, in my theory of change, I'll have extra lines that are in a different colour. So they're clearly designated as system-wide or an influencer, I've labelled them as influencers.

And in there, I would track actually these other influencers, so that you can actually track and say, oh, actually, these are all the other factors that are outlying to our control. But they might also highlight where you need to work with others, and try and influence actually what's happening. Yeah.

Here's a question from Tara. So social investment is a method, and all methods have limitations. Yep, absolutely.

Can you give any examples where you've tried to use this method, and it hasn't actually been useful or added value, or this value has only been very limited? I don't think we've had that experience yet. But I think again, I agree with you, it's got limitations. And I think that seeing it as just part of the picture is really key.

I don't think, you know, there's a lot of talk about it at the moment, it feels like there's quite a lot of hype in this area. And I think the risk is, is that everything gets weighed on just that number, rather than actually thinking of all these broader evidence, such as the qualitative that tells the story of the value itself. So I think it's really important to kind of capture a broader picture, not just the SROA, and rely on that.

Yeah, that's right. The numbers that pop out of an SROA calculation, which is just a part of the whole theory of change and M&E framework, it's part of it. So you have to talk about the whole thing together, you don't just fixate on that single ratio.

And again, you acknowledge the limitations, acknowledge the uncertainty inherent in these sorts of valuing of things. I think so. And going back to the previous question as well, I was thinking that, yeah, I think some of this is in the narrative that goes with these things and acknowledging that there's all these other factors.

So you need to actually take that into account when you're doing calculating social return on investment. Next question is from Lauren. Does internally done SROI have the same weight as externally commissioned SROI? Does bias come into play? Question.

I have seen for the, like for the Mental Health Innovation Fund, they required it was an external SROI. So if you're going for funding applications, sometimes at the moment they're requiring an external social return on investment for independence. There's actually pros and cons of doing things internally versus externally.

I mean, that's true of evaluations, but some of the benefits of doing it internally is the internal expertise and knowledge that you already have. On the external, you've got some fresh pair of eyes and it gives a sense of independence. I think actually, whether you're working externally or internally, you always have to be wary of bias.

So I'm suspicious of anything like if a whole group agrees on something just straight off, I'm like, really? If it keeps on happening, I'm like, hmm, because that's probably groupthink. So you want to create the conditions for discussions, ideally robust discussions. You want to ideally have some people there who can go, does it mean this? Or does it mean that? Or here's a different way of thinking about it.

Could it mean this other thing? So I think it's about how you interrogate things too, and not just take them at the face value and track that possible bias. I think the fact that you're asking the question is great because you're aware of it, that we all are prone to cognitive bias. Ideally, it would be done internally as well, because there is a buy-in to that and there's a feedback mechanism to track your own progress, that you're doing the right sorts of things.

But it is the environment that can force those external components. There's one quick question I'll quickly answer from Christine. What format do you use to follow to determine how many qualitative interviews that you try to complete? I remember finding some guidance on that, so I'll try and find it and we can share it.

There was a very long question here, but we need to start wrapping up. So thanks everyone. We know there are a bunch of questions relating to the calculations we'll cover in part two.

You can leave this window open. There'll be a handy timer so you know when we're back with our CBayX enabled sheet, and take you through inspiring the youth mental health calculations and how you can use the tool to calculate your own SRI values. So speak soon.

And welcome back for part two of Vision to Value. This time we'll be discussing how to calculate your impact. This will be getting into a gnarly spreadsheet.

I'm so excited about this. You'll get a copy of this sheet after the session so you can see exactly how this has been calculated and use it to calculate your own numbers. You'll also get a copy of the recording and I'm happy to catch up with people to discuss anything we may have missed or lost over.

I have pre-recorded some of this to make it easier to follow along. As we go, feel free to ask questions. Apologies for any video quality issues.

It was late. I had unexpected technical problems like realising I had my headphones on backwards during my first five takes, which meant the microphone was pointing behind me. Really human.

The first thing we're discussing is the investments or inputs part of the SRI. So these are the resources put into the activity we are doing the SRI on. They should be all the resources used, not just those directly related to the outcomes.

So we've got financial inputs, grant donations, funding, revenue generated, if they're reinvested into the programme, and in-kind contributions. So the non-financial inputs, things like staff time, volunteer hours, which is considered a cost, equipment, materials, utilities, property usage, and opportunity costs. Now, into the spreadsheet.

So this we're going to talk briefly about your base inputs, investments, resource invested. That's kind of what this part of it is. These are resources put into the activity that we're doing the SRI on.

They could be financial or non-financial resources. Crucially, they should all be resources used, not just those directly related to the outcomes. Examples of these are financial inputs, grants, donations, funding from government, other agencies, revenue generated, if they're reinvested in the programme, in-kind contributions.

We've got non-financial inputs, so like staff time, volunteer hours, equipment and materials, facilities usage, land property usage, opportunity costs. So a bunch of things in here that are costs here. Now, in-kind is a cost because there is what's called an opportunity cost in that they could be doing something else, some other activity unrelated to this.

What we get up here is the number of people. We've got 20 volunteers, at a cost of $20,000 a year. So this is either an actual cost or could be just delivering their time.

We've got counsellors paid at $80,000 per year, and there's 20 of those, and mentors at $20,000, and we've got admin costs at 10%. There's the salary costs, there's a rough rule of thumb, rates of power, they would just say $22,000 a year, but you can get actual costs, and consumables are 5% of salary costs. So we've got some additional numbers up here, so in case you're doing a more complex SRI ratio, you can actually see if the intervention is being run every year, there is a cumulative, there'll be additional people brought into it, and additional costs.

So you'd, they were just, got an automatic wage growth adjustment coming up here for counsellors, and if you happen to be purchasing properly as part of it, so you're doing a house initiative, and you might use capital gains as well and add that to it. So those are the base costs. Kia ora.

Thanks, Brendan. We know there is a lot to take in, so don't worry, you'll get a copy of the recording and of the calculation sheet Brendan is using. Brendan, we have a question from Priya.

In co-location scenarios, how do you accurately apportion shared costs? So there is no easy answer to that. Ideally, that would be part of the tenancy or contract with the other parties. There may also be an overarching governance group that manage things like this.

In my example, I just sort of halved some of the costs related to the shared activities, but you could split it other ways. Okay. Also, we've got another question here.

What methodology do you use to calculate opportunity costs? Another tricky question. The rule of thumb is to treat it as at a paid role in the cost part of the SRI, so even though it may be volunteer hours, you would have to produce an hourly rate and the number of hours those volunteers would be contributing to the programme. So overall, SRI aims to quantify the social value created by an intervention, not just financial return.

Therefore, the outcomes considered go far beyond civil profit. So the kinds of outcomes we'd look at are changes for beneficiaries or clients, individuals, and these are most crucial and often the most complex. They represent the direct impact on the people the intervention is intended to serve.

Examples include improved health and wellbeing, increased skills and knowledge, enhanced social connections, greater self-esteem and confidence, improved living conditions, increased safety and security. There are also spillover effects. So SRI recognise that interventions often have ripple effects beyond the immediate beneficiaries.

For example, the changes for families, communities, staff and volunteers, or the environment. So what's called the spillover effect, which is these ripple effects at 2%. Other financial outcomes can be included, especially if they are linked to the social outcomes.

For example, reduced costs due to improved health or high earnings due to job training. Remember, this is all crucially part of what should be done with the theory of change in the MD framework. So now I'll take you through calculating the social returns and outcomes.

So now we're looking at the outcomes, which is the exciting bit, but there's some ways that there's the other part of the SRI ratio. We've got the cost and now these are the benefits or the outcomes, or the social returns part of the analysis. So this aims to quantify the social value created by the intervention, not just the financial return.

The kind of outcomes, repeating what we might have talked to earlier, that we'd look at are things like changes for the beneficiaries or clients or individuals, which represent the impact on the people the intervention is intended to serve. Examples include improved health and well-being, reduced illness, increased life expectancy, improved mental health, increased skills and knowledge, enhanced social connections, greater self-esteem and confidence, improved living conditions. So we click over to basic outcomes and see what it looks like here.

So this is the SRI ratio calculated live at the top there. So as we change things, that number will also change. These are your social outcomes here, and this is what measure we're using for it.

So these are pulled from the CBAX, using TrueView CBAX. And we go over here. And we've also got what's called a primary, secondary and tertiary areas, which is a sphere of influence.

This also has the spillover effect, which recognises that interventions often have ripple effects beyond the immediate beneficiaries. For example, changes for families, states, communities, staff, volunteers or the environment. So we've got a spillover rate at a very conservative 2%.

Some key considerations for defining the outcomes, which would be done when you develop the theory of change in the monitoring evaluation framework. So as a reminder, better focus on outcomes that are significant and meaningful to stakeholders. Avoid including trivial changes.

Outcomes should be measurable, either quantitatively or qualitatively. It's important to be able to reasonably attribute the outcomes of the intervention. This is challenging, it's very important, essential.

And also engage stakeholders throughout the process to identify the outcomes that are important to them. You're also going to be careful with double counting. So always make sure that measures are very discrete, they're measuring quite different parts of the outcomes.

Obviously for the SORI ratio, you need to monetise the outcomes. So we're trying to give everybody the same sort of value. So instead of trying to turn everybody into an apple, so you can compare them.

If we're lucky, something like Treasury New Zealand CBX tool will have that estimate. If not, you can use an estimate that is adjacent or overlaps. An example we have in this spreadsheet is social inclusion, which is this being the value of being a member of a volunteering group.

So we're not necessarily saying that we want a young person to be part of a volunteering group, but it has the same value to society, the same social value. So however they're being involved in society, they're medically participating and engaged with society. So we're sort of picking that up.

So if we don't have a measure from say CBX, you could trawl through literature to see if someone has done the work to find that estimate or calculate it. Because in cases usually of the individual that's occurred, you would have direct data to make the calculations yourself. Examples of this would be in the health system, where you have access to those direct costs to society and the changes in cost as a result of a particular intervention.

We go up here, we've got, I'm going to go over these again, we did talk about in the last webinar. So the data is what would have happened without the activity. So the outcome would not have occurred with that intervention, would be 0%.

And 100% meant the outcome would have occurred regardless of what you did. Now the important part is attribution. So what proportion of the changes that were attributable to your intervention? So the flip side to this is what else would also be making the same or similar changes to you? So who else contributed to the change? So we're saying whatever changes is made to the young people's lives, 30% is due to us, our intervention.

Another key factor is what's called the displacement. So the benefits claimed by our intervention, our project, it could be others in the same working with the same population. So of those 100 young people, other social organisations also competing for them essentially.

So are we, if we're taking 100 people into our programme, are we depriving of another programme of those same clients? So we've set 25% as the sort of a low realistic figure for that one. And finally we've got the discount rate, which is slightly more complicated. This represents the rate at which future costs and benefits are discounted to the present value.

So basically this feature, because money available today is worth more than the same in the future due to its potential incapacity. So we would have earned, there would have been additional work done if we spend it now than in the future. So current treasury in these other devices is 2%, which we just accept and go with.

So example 0% means that can last for the whole, sorry, for the whole 25 years, which is what we're calculating it for. The impact of our intervention is the same throughout, whereas 100% means it drops off after one year basically, one year and there's no trace of intervention remains. If we go down to the multipliers, this is where we've got these numbers in here.

So you can change those and it'll change the value of the entire spreadsheet. These, the number affected is 100, so we're just saying we're working with 100 people, 100 young people. And so you can change that here.

These other two, the SRO ratio importance rubric, that it was evaluative component, the SROI. I was so young and naive back then. So some key considerations for defining outcomes, which will be done when you develop the theory of change in the modern financial framework.

So as a reminder, you need to focus on those outcomes that are significant and meaningful to stakeholders and avoid including trivial changes. The outcomes should be measurable, of course, either qualitatively or qualitatively, so you can track progress and calculate the value created. It's important to be able to reasonably attribute the observed outcome to the intervention.

This can be challenging, but it's essential for credible SROI analysis. Of course, engage with stakeholders throughout the process to enjoy the outcomes that are most important to them. This ensures that the SROI is relevant and meaningful.

Obviously, for the primary SROI SROI ratio, we need to monetise the outcomes. If we're lucky, something like Treasury NZ's CBAX tool will have that estimate. As Fiona pointed out, there are other sources.

If not, you use an estimate that is adjacent or overlaps. An example we have in this spreadsheet is the social inclusion, with the CBAX estimate being the value or being a member of a volunteering group. So now on to setting up the different estimates, like the counterfactual, high and low estimates, and other scenarios.

So in this case, the counterfactual represents what would have happened had the intervention not taken place. It's a hypothetical scenario. It's crucial for isolating the impact of the intervention from other factors that might have influenced the outcome anyway, so you don't add or subtract it to your calculations.

Essentially, you're comparing the differing intervention scenarios to your counterfactual scenario. This is also where you do your sensitive testing, i.e. what levers have the largest effects. So to talk about the base outcomes, now we're going to talk about the other tab.

So this is for counterfactuals and other estimates. So the counterfactual, which is very important, represents what would have happened had the intervention not taken place. It's a hypothetical scenario.

It's crucial for isolating the impact of the intervention from other factors that might have influenced the outcomes anyway. So essentially, you're comparing the differing intervention scenarios to the counterfactual scenario. So in this case, we've said, so those 100 young people that we say we're going to work with, things get worse for them.

The drug and alcohol problems in Auckland get worse. Nobody engages with the society or the community. There is no change in income, in the jobs, in employment, nor in education.

I think it's an organisational component, so that's largely unchanged. There's the counterfactual and then we start setting up, trying to set up the ranges. So you've got to remember the SRI is a range of values.

And with more information, with more and better data, you can narrow that range as you get better estimates of what the actual values are between. However, so what we set up for the low estimate is of those 100 people, only 10 young people have been impacted by the intervention. In this case, we've got only 50, so half of the young people have been affected positively by the intervention.

And high ball is, of course, all 100 have been impacted positively. A final scenario, I mean, you can change any of these as you want. I mean, you could make these all different scenarios, rather than going for a range.

But in this case, we've looked at what it looks like, what happens based on their mid-range, 50% have been positively impacted. What happens if we change the costs? In this case, we only have to carry half the number of counsellors, because we're assuming we're co-locating with an organisation with their own counselling service. So we'd be part of this.

There'll be some savings around administration, half in consumables. And rates there don't change, but they could as well. So what would they look like? Fortunately, we have, of course, got a lovely little summary here.

So you can see the counterfactual is things get worse over time. We've got your low point, so I don't need to break even at a dollar. I only choose something that quite makes it to break even point.

The midpoint scenario reaches it after six years here. The high point scenario reaches it after two years. We've achieved over a dollar, $1.19. And the co-location after three years or so, but a higher ratio.

So you can start pulling the levers and adjusting your estimates of how many and to what degree you're affecting. Because not only can you, for example, set how many you're impacted, but you could also do a fraction. The quality might only be improved by 0.3, for example.

Zero means you made no change, so 0.3 is a 0.3 improvement. So you can test incremental changes. And you can start seeing what would give you the best value for your investment.

What things could you concentrate on by looking at these and basically changing these numbers around. And that's it for this one. OK, back to us.

A reminder, you'll get a copy of the recording in the sheet. We've got a question from Thomas here. Could you elaborate on sensitivity testing across scenarios? The simplest way to just make a copy of each scenario and put it in your spreadsheet and make different changes to the cost and outcome to see how it affects your SRI ratio.

This could be by varying the proportion of people impacted, how much they are impacted, and even how long the intervention activity lasts at the discount rates and attribution, the other multipliers. Remember to label each scenario clearly and take notes about what changes you are making and why. So you've got to lay out those assumptions really clearly at each scenario.

Thanks. I've got another question for you. When is a good time to calculate an SRI? Projects often have a mix of part-time and full-time employees during the construction phase.

Workforce may shift to primarily full-time employees once the project is complete. What are your thoughts? It's harder, which means more complex, but you can put lag effects. So your SRI might take in the first year or two, those part-time estimates, and then you can bring in at a later point in time the full-time and change it to full-time.

So it is possible to do both at the same time within the same ratio, and then the SRI calculation will take into account both that part-time and the impacts of that. And then as you switch to full-time mode, for example, that's going to have a slightly different estimate, but it will bring those together in the final calculation. So I think the next one is moving into the evaluative component.

So this is thinking more about how we bring in the evaluation, the qualitative ways of thinking about it. Does the parallel or alternative way of understanding your impact incorporates more qualitative or difficult-to-quantify measures? It's the criteria specific to the programme. As such, it's not really suitable to compare between different initiatives or programmes, unless you're running exactly the same programme, then that becomes viable.

But it is very useful comparing between different scenarios, your sensitive testing, what levers you want to pull, and monitoring the impact of progress over time. Within it, we've set up as a primary, secondary, tertiary little areas, so primary is your main outcomes. The secondary and tertiary capture those spillover effects, so they're calculated at a much smaller rate.

So we're going to go back to the past, Brendan, and the spreadsheet. Right, so now we're going to talk about the evaluative ratio component of our SROI approach. So the SROI evaluative ratio as a parallel or alternative way of understanding your impact incorporates more qualitative or difficult-to-quantify measures.

And as the criteria are often specific to the programme, it means it's not really suitable to use to compare between initiatives or different programmes, because the criteria would be quite different for those. It is very useful in comparing between different scenarios within your programme, your initiative. It's good for sensitive testing again, just pulling those levers and seeing what works best, what has the best impact, and monitoring your impact progress over time.

So once you've set it up, you can say OK, this is what we're achieving, and you can see what progress is being made. You can see it's quite different here. Just in what values it generates, but that's purely how we have set it up in the value part of it.

We jump to our base one. There's a ratio there. And what we're doing here is, again, this comes from the theory of change and the M&E framework, and we set up things like what does good look like? So in this case, where we've got a quality at the SROI ratio, we're asking, we're saying, yeah, there's a young person living their best life.

And how important is the stakeholders? So one to 10, 10 is best. And every, will we achieve those goals? So zero is not achieved, three to four, all goals exceeded. We're set up right in the middle of two, the goals are being met.

The young person is meeting the best life, is living the best life. What's the saying? Young person addicted to drugs or alcohol, and so on and so on. Importantly, unlike the SROI ratio, we can incorporate qualitative or other non-monetised measures.

So in this case, for so-and-so governance, we've got organisation as part with other community groups. So you could add other qualitative or more relative measures in here. So important stakeholders, we've got a seven, and right in the middle again.

As an example, we've got 50 people here, and we've said it's very important, and we've got a four there. So we're just changing numbers around there. So similarly, the SROI ratio, we can test things to see what works.

Now, the SROI relative ratio is calculated slightly differently, because it also uses these primary, secondary, and tertiary groupings. So it takes the weighted average from within each of those groupings. So it combines all the primary, all the secondary, and all the tertiary, and add them together.

But this can be adjusted depending on what's important to the programme. So this is just how we've implemented it. But again, it's something can be done differently, depending on the requirements of the programme that is being evaluated.

And I think that's about us. Thank you for attending, and back to the studio. Hi, thanks for that, Brendan.

So just got a question for you. How do you decide between using practitional versus binary outcome measures? Fractional? I hope I interpret the question correctly. So inside the SROI, it doesn't make a huge difference.

So a binary outcome might best be kept as a binary, and you might vary the number affected, while a fractional outcome might be treated differently. It relates to reporting and interpretation. For example, you could model a partial success as impacting fewer people, or as a fractional outcome affecting the same number of people.

So in practise, that would make sense at the measurement end, and help best explain what is happening to your stakeholders. So there's that interpretation. You might choose how to frame that change inside the SROI to match how you want to report on it.

Importantly, it's about what best captures the type and degree of change, which of course could be identified at the theory of change and M&E stage of the SROI. In a similar vein, I've got a question here from Catherine about how do you accurately calculate the size of the change? For example, if your outcome is quality of life improved, and 80% of service users report improved quality of life, would your improvement value then be 0.8? Again, in that instance, I'd probably vary the number. So say you've got a population of 100, which we had in our example.

In this case, only 80 people would be impacted by it, even though the costs would remain the same, trying to impact 100 people. Or you could say 100 people had a 0.8. So if 100 people all increased their quality of life by 0.8, which would make sense, but in this case, 80% had their quality of life improved, while 20% didn't. So you'd probably model it as only 80 people had their improved quality of life.

Okay, great. Daniel says, sounds like more of an art than a science, this SROI business. Who gets to decide what are reasonable assumptions? Yes, I think it's both.

Perhaps at the early stages, when you lack evidence, probably more at the art end of things, because you're relying on insights from experts and others. And as you gather more evidence and insights, you're getting more science, I suppose, but certainly more mathematical as you're starting to refine your measures. What do you think about that question? Well, I think it's a good question about who gets to decide.

It's something that we definitely consider. I think, coming back to the theory of change phase, when you're building up these assumptions, this is where lived experience is so important. And those who actually deliver services being in the room or having some say is important, because then you can test your assumptions and sense test it.

Does it make sense from their point of view? Does it align with their views? And in that, you want to get a bit of diversity there. So you want to actually get different perspectives, because different groups have different emphasis in different places about what's important. So I always rely quite heavily on sense testing my assumptions and go, does that make sense to you? Does it resonate? It's part of how I look at the assumptions.

And I tend to forefront lived experience, first and foremost. Best insights. That's right.

Even regardless of how good your estimates are, it's a lived insight that actually ensures that, again, those numbers are real. There's another question here. How did we decide on a discount rate of 2%? We didn't.

Treasury New Zealand did, and they've got lots and lots of resources. That is what we base it on. But it can be changed if you've got good rationale for doing so.

For example, the outcomes deteriorate very quickly. You might have a higher number. But we just stick with 2%, because it's easier.

And what would come out of your theory of change in your engagement with the stakeholders, the M&E framework might start telling you something different, and you might adopt different numbers at that point. But it's a standard number that we sort of start with. Then you might adjust it.

Yeah, it's standard for New Zealand, isn't it? It's been set up for a New Zealand context. Next, everyone knows, would the team advise setting up a base spreadsheet with the impacts common across programmes, if you have a number of different programmes and interventions that are trying to affect change in similar impact areas, e.g. improving quality of life, reducing injury rate and severity? Say that again? Yeah, sure. Would you advise setting up a base spreadsheet with the impacts common across programmes, if you have a number of different programmes and interventions that are trying to affect change in similar impact areas? You could.

And if it's the same, again, it's that theory of change stuff, if it's the same outputs, same outcomes, the same sort of cost basis, but there might be variations in how it's implemented, you probably could. I personally might set up as different tabs to keep it separate rather than in a single sheet, and then you can compare and contrast different programmes and tweak them individually. But there's a number of ways of doing it, so that's my personal way I'd approach it.

Because the question was to reduce the time needed to build this information in, would it be a valid approach? Yeah, absolutely. So another question for you, how do you validate duration assumptions for long term impacts? So many hard questions. I rely on expert knowledge and public literature, especially to start with.

Understanding the cohort being affected is really important. For example, an initiative involving secondary school children might see an abrupt change in the predicted outcomes when they leave school. For example, that training kicks in when they move into the employment space, or those drilled homework habits suddenly mean very little.

And another example would be age. So if you're talking about a much older cohort, you might calculate out to 25 or 50 years. So there's really, you've got to understand who it is you're talking about.

Can you explain your approach to measuring cumulative effects? Right. The SROI inherently captures a cumulative effect of outcomes over a person's life, which is why we set it at 25 years. But again, there's assumptions built into that, which we talked about earlier.

How large that accumulation is, is dependent on things like the discount rate, and for how long you believe the initiative will impact people. If you're talking about the cumulative effect of multiple initiatives, activities that are due to your programme, so you're doing a bunch of different activities, all at the same population, you're probably at each the SROI calculation, avoiding double counting, of course, and you'd further nuance each of those estimates using those dead weight attribution and displacement multipliers to account for the slight differences in those different initiatives. Nice.

So, of course, the old adage, rubbish in, rubbish out, holds here. So there's a little graph, which I actually enjoyed making, because it's kind of cool. The better your data and evidence, the more you're able to reduce the certainty and range of the SROI estimates.

So in our spreadsheet, we've modelled that as a low, high and medium, so we can squash those right down and get just more confidence in what we're saying. For our inspiring mental health example, we are near the bottom of our, hence our agency, which is very wide. We are basing our measures in the CBIX coefficients on best guess, evidence and insights from our engagement with the stakeholders as we go through that process.

Again, you always go back to being the change in framework tight, so that they're consistent across right into this measurement stage. So with further work, you'd understand things like how many young people would actually be involved in service each year, keeping in mind that demand would drop after the first few years if you're successful in meeting any unmet demand. Actual admin and salary costs and availability of counsellors and mentors, which would impact the number of young people you could actually help.

Research and the increase in wellbeing following similar initiatives or similar populations. And how long does the impact of such an initiative last and for how many people? So we've covered a lot so far. I have a list of 20 plus tips, which I'll add to the summary from the session.

That said, the ones on your screen are the main ones everyone should focus on. So I'll call out a few, include both your direct and indirect costs, like volunteer hours, which are your opportunity costs. Another thing to remember, excessively large SRO ratios, which we won't be guilty of, but they're set at a higher end, usually means something is wrong with your cost or outcome estimates.

Be honest about the time at which your intervention will last. Obviously avoid double counting by ensuring outcomes entered into your SRO calculations are conceptually distinct. Can I ask you a question, Brendan? How would you know it was excessively large? Treasury needs this guidance.

Between one and two means you're probably right. This is a realistic estimate. Between two and five, better check some things, check your costs, check your estimates.

And then five or higher, seriously look at what you're trying to do here. So you start questioning the validity of the findings at that point, over five. But that's kind of the advice from Treasury New Zealand.

And of course, if you're very certain about your assumptions, then perhaps that is a real number. Those big numbers could be true. You know, if you're absolutely solid about what you're doing, what you've done.

Thank you for taking us through that, Brendan. So now on to some questions or questions. So our first question is from Araha.

How does the model quantify structural systemic impacts? This is so hard to do. So the SRO ratio is inherently an individual focused technique. You could try to estimate the degree of impact of a structural systemic level change in a size and demographics or the population affected, but it'd be really hard.

You need sort of big population level data like statues and an IDI, and it would still be very noisy as there are typically a whole lot of other things going on at the same time, globally, nationally, and at individual and family or whanau level. So the more evaluative techniques could accommodate these kinds of impacts, but it'd be very hard to monetise them with those big level data sets. And even then, again, I'd be cautious about saying it.

But the qualitative stuff comes through this stuff. This is where the stories are. Yeah, I think actually it's one of the limitations of SRO from my point of view.

It's not really good at getting that because of its strong individual focus. I think one of the things, one of the ones I'm working on at the moment, what we're doing is actually treating equity itself as an outcome. So against the short term outcomes, we talk about desegregating data.

So you actually start identifying how well something's serving people to try and give some reflection back on is it actually, is there a structural systemic issue here? Is the service designed as it should be or needs to be to meet different groups' needs? Yeah, I mean at the programme delivery and there might be some structural components in there that you'd have to pull apart and you might be looking at how the outcomes differ for different populations and that give you clues about what's going on there. Yeah, yeah, but it's not really designed for it really. What methods isolate programme impacts from systemic changes? Yes, so I would do it.

I mean, again, that comes back, again, there's a whole package going on here. So there's a theory of change, all this, you know, you'd hopefully consider these things as well. But in SRI calculations, I'd probably simply adjust for those differences using the deadweight attribution displacement multipliers that would require some heavy thinking to determine which direction to take those.

Running different scenarios with different values would give you an idea of what range your programme sits within as well. Yeah, I think another thing too, when you're developing a theory of change, when you're identifying outcomes, one of the things you can think about is that broader systemic picture and think and start identifying as part of your assumptions even or even conditions for success is what would need to be in there. So what are these broader factors that could actually impact on the outcomes? So that's why we say some of those long term you can't really measure easily, because actually getting a lot more systemic kind of changes potentially captured in there.

Mary asks, how can we incorporate kaupapa Māori frameworks while maintaining validity? Well, we did talk a bit earlier in your section. So that's very much setting it up, again, setting up at that theory of change, the MNI framework. So absolutely, you can maintain ability while doing that, but there is the framework will be slightly different.

You have to be more cautious and you have to be careful how you're measuring your outcomes within kaupapa Māori framework because you're incorporating, for example, whare tapa whā, you'd be incorporating spirituality, wairua, te nana, inumāro, other whenua as well. So those are all measurable things, but you always have to be careful not to start pulling apart the thinking and turning it into individualistic measures. So there is that, again, that tension between pulling it apart and measuring bits of it, and then losing the holistic nature of it.

What do you think? Yeah, I think when you start pulling apart, it actually becomes the whole thing. That's the problem. When you do that, you can take little slices of the pie.

Whereas I think the beauty of some of these things is that holistic nature and how they bring into account all the different ranges of things. So I think it is actually trying to get or capture as well as you can, all those different elements and think how do they all work together in this context. Yeah, and that's where the value of the qualitative, the stories have to be in there as well.

So you never lose sight of what you're trying to do with any quantitative evidence. So how do you adapt calculations for community recycling centres? Again, it comes back to that. You have to talk to stakeholders and try and understand who's being affected, who the beneficiaries are.

It's all that theory of change, M&E framework stuff. That's where you do your heavy lifting so that the SRI calculations make sense and you're doing the right sorts of work. I'm not entirely sure how you'd do that, to be the example, without actually delving into it.

You'd have to understand the cost of the programme and there are environmental costs. So you can pull in adjacent measures from CBX around environment and a safer or a better environment for the community. So again, you're probably looking at those adjacent measures that are related to a quality of life at the community level rather than individualistic.

So it's possible, but you'd have to do thinking. I think this is where the theory of change comes in, right? Because if you're able to do that, you can't do that without actually even going through a proper process of identifying not just the outputs of it, but what difference is it making? And you need to really be asking actually some of these things, are there any unintended consequences or is there anything going on that might actually detract from some of these things? Such as the amount of energy involved maybe in some recycling could be something to think about, embodied energy. Yeah, the counterfactuals and those other unintended consequences in it all.

But it requires a full process. Here's another one for you. What techniques measure settlement outcomes with New Zealand data? Not 100% sure what we mean by that one.

If we're talking about colonisation, that'd be really, really complicated. I mean, it's, well in the past they wouldn't have very good measures, but it is possible through a life course theory and other epidemiological approaches and changes in land ownership over time. If we're talking about, say, an iwi settlement, possibly assuming here, assumptions are dangerous.

Then you look at the outcomes for the beneficiaries of that, but knowing that there is an overlap between iwi and this is, you're making up, we're getting what, five cents to the dollar for what Māori lost. So the counterfactual is, the other scenario is if we retain all that land, if Māori retain that land, then what would the outcomes look like then compared to what the settlement has given Māori? So it could be an interesting analysis to do, but it'd be very complicated. I think it would.

How do you effectively monetise qualitative impacts? Again, theory of change, emini framework, and that will tell you if it's possible, if it's possible through adjacent measures, or it just remains as a qualitative story, and perhaps use evaluative techniques. Okay. Well, that's us.

Nga mihi for joining. See you at the next one. And thank you to the team behind the screens as well, which we encourage you to look at.

Nga mihi nui. Ka kite.