Kia ora and welcome everyone to today's webinar, Dollars and Cents, Measuring Social Impact. I'm Jason Carpenter and it's a pleasure to be with you today on behalf of Allen and Clark. Well, we're passionate about making a difference in Australia, Aotearoa and beyond.
For some of you, this is your first time joining us, so you may not be familiar with Allen and Clark, but we're an Australasian based consultancy focussing on change management strategy, programme delivery, policy, evaluation, research, and of course, cost benefit and social return on investment analysis. Today, we're diving into a crucial and increasingly relevant topic, social impact measurement and social return on investment or SROI. Recent government announcements have put a spotlight on these methods, recognising their value and gauging the effectiveness of policies, campaigns and social initiatives.
But understanding the intended impact, the mechanism behind it, and being able to substantiate real measurable change is where the challenge lies and that's what we're here to discuss. And so whether you're brand new to social impact measurement or have years of experience under your belt, this session is designed to provide insights for you. So no matter where you work, you'll walk away with some practical knowledge that you can apply to your projects.
And we know governments are increasingly relying on these assessments to inform their investment decisions and so the stakes are high. But a well-conducted analysis can guide evidence-based policymaking and ensure that social investments deliver meaningful results. So we really hope today's session helps you build a strong foundation in these approaches and equips you with tools to deliver impactful evaluations.
So without further ado, let's get started. Alongside me today, I have two very experienced and knowledgeable colleagues and I'll now ask them to introduce themselves. Fiona.
Kia ora, ko Fiona Scott-Malton ahau. My interest in tracking the different services made started back when I worked in the community delivering support services to people with intellectual disability who could be violent and or had mental health need. I grew further as a senior performance auditor for the Office of the Auditor General where I led audits that had a focus on whether the desired outcomes were being achieved.
And over the past six years, I've been at Allan Clark as a senior consultant with a focus on impact and performance. Brandon. Tēnā koutou, ko Brendan Stephen ahau.
So I've worked at Massey University, Te Hine Ka Hauora Health Promotion Agency and at Allan and Clark. I'm a mixed methods researcher and evaluated with a stats background and I knew to deliver interest in things like CBA, CBXs, SRIs, VFIs, RIs, AIs, in all manner of numbery things. Ngā mihi.
Thank you. And so on your screen now, you see that we've put up a sort of approach to SROI assessment and we're using that to guide our discussion today. And to help bring the process to life in a practical way, we've actually created a fake NGO that will take you through some of the key steps and raise some of the key considerations and risks at each step as we work through.
And so as we said, social investment is a hot topic. It's really about improving outcomes and getting better value for government spend in the social sector. But more specifically, the social investment approach uses data and evidence to understand people's needs.
That's how you go about getting better outcomes, identifying and building on what works and to stop doing what doesn't. So here's our imaginary NGO up on the screen. We've called it Homestrong New Zealand.
And the mission that you can see there is to enhance the quality of housing for vulnerable populations in Erewhon City by improving access to healthy, secure and affordable homes. So they have three core initiatives. The Healthy Homes Initiative, where they retrofit older homes with insulation, ventilation, et cetera, to ensure they meet the healthy home standard.
The Accessible Homes Project, where they modify homes to make them accessible for people with disabilities, including installing ramps, widening doorways and adding accessible bathrooms and kitchens. And the third one is the Safe and Secure Housing Programme, where they provide long-term tendencies for families transitioning out of emergency housing to offer them stability and a pathway to more secure housing solutions. So the scenario that we've got for you today is that there's an application in the offing for a $1.5 million grant from the Better Futures Housing Fund that we have invented.
And they're gonna use that to scale up their Safe and Secure Housing Programme. So the goal here is to secure housing for families currently in emergency housing or facing homelessness through the purchase of existing homes, retrofitting them to meet the healthy home standards. And then, you know, so this grant will really help them to expand its housing portfolio by purchasing more homes, retrofit the existing properties to ensure they meet the standards.
And as no surprise to anyone, our fake application requires an SROI. So we are gonna go through the process with you today for Homestrong New Zealand. And so the first step in any SROI is to do a theory of change.
So Fiona. Oh, kia ora, Jason. So developing a theory of change, which is also known, you might know it as an intervention logic or an outcomes chain, is the foundation of any social impact assessment.
A well-defined theory of change articulates what the initiative aims to achieve and outlines the pathways to get there. It's more than a roadmap. It clarifies the assumptions underpinning the initiative, identifies critical conditions for change, and helps ensure all stakeholders have a shared vision of success.
A couple of tips for developing a theory of change. Don't overcomplicate things. Just capture the most important things.
You don't need to capture everything. And the other one I'd say is just make sure your outputs and outcomes are measurable. This is crucial for the purpose of completing an SROI.
You'll see up on the screen that here's a completed theory of change that we've done for Homestrong New Zealand, and it sets out the key inputs, outputs, and outcomes, short, medium, and long-term. For Homestrong New Zealand, the expected outcome is clear, to provide secure, healthy, and affordable housing to vulnerable populations, such as homeless families and adults with disabilities. The primary outcomes associated with these, those in the yellow in the theory of change, are directly linked to more stable housing, healthier homes, reduced health issues amongst residents, such as respiratory illnesses.
There are also a range of follow-on or secondary outcomes. These are in the dark green boxes in the theory of change. And these assume that maybe for many of the families moving in, they might receive additional support services to help realise these benefits.
And they include things like improved educational outcomes for residents' children, more stable employment, increased social inclusion, and improved quality of life. There's a few key decisions and considerations that you need to take into account when you're developing a theory of change. One of them is about engaging stakeholders.
We strongly recommend involving as many stakeholders as you can, especially those with lived experience in developing the theory of change. This will strengthen its accuracy and relevance, and ensure the theory of change addresses real-world needs and factors on the insights of those directly impacted. You also need to be clear about your assumptions.
You need to identify and document these assumptions, the assumptions underpinning the initiative. For example, Homestrong New Zealand assumes that retrofitting homes will lead to better health outcomes. It's important to challenge these assumptions by asking, what needs to happen for this to be true? And what could go wrong? But also, a key part of theory of change is about providing pathways to change.
It should map out the logical, ideally causal links between the key activities outputs, such as retrofitting homes, and the short, medium, and long-term outcomes. This is essential for clarity and for tracking progress over time. For Homestrong New Zealand, this would involve linking home retrofits to improved health, increased housing stability, et cetera.
But there's a few things to consider when you're preparing an SROI, there's a few risks. So while social investment is primarily focused on individuals, people are part of what's being referred to as an ecosystem, the whānau, family, community. To optimise social investment, we need to think about the role of whānau, family, community, the system more broadly, to ensure improved outcomes are optimised and sustainable over time.
There's another bit, which is about policies and or initiatives that are based on the assumption of employment tend to benefit those who are better off. This is known as the Matthew effect, where an initiative disproportionately benefits those who are already doing well. Could you give a quick example of what you mean by the Matthew effect? Yeah, sure.
As an example, funding for specialist departments within schools that aim to serve a broader range of children with learning needs could have the Matthew effect. Such a programme is typically designed for children who aren't receiving additional support from their whānau, family at home or in the classroom, but would really benefit from more support. More educated parents are likely to benefit from such an initiative more because they are more likely to A, know about the initiative, they have the flexibility, they're more likely to have the flexibility in terms of working hours and the confidence to advocate for their child at school.
They can also afford an educational psychology assessment, which means they can provide evidence of their child's needs. In contrast, under-resourced, poorer families face more barriers. They are less likely to know about the initiative.
They're also more likely to have inflexible working hours or may have limited transport options. They may also lack the confidence and language to advocate for their child. And they couldn't afford any type of assessment to demonstrate their child's additional learning needs.
In addition to that, anything like unstable housing means that they probably aren't in an area long enough to be able to organise such support or that just as it's begun, their child has moved to another school and the process has to start all over again. As a result, the programme disproportionately benefits families who are already in a better position to begin with. This is where being clear about what outcomes we seek but also understanding what success would look like from the perspective of the people who it is supposed to serve and what needs to happen to enable that change is so important.
Yeah, and that's great. And I think a reminder straight off the top that an SROI will only ever measure what you have decided is important. And so I'll pass back to Fiona to kick off with some of the discussion about measurement.
Yeah. So once your theory of change is in place, the next step is developing and monitoring and evaluation or what we know as an M&E framework. By clearly defining what success looks like when developing the theory of change for each identified outcome, the M&E framework establishes what data needs to be collected.
This is both quantitative, so think numbers and qualitative. The framework will also set out where the data will come from, how often it should be collected. The process should also involve considering the feasibility of data collection such as how much will it cost and how much do you need? The framework will also set you up for future evaluations.
Brendan. Right. So collecting good data is crucial in any kind of investment activity.
We need to understand who is at most need, what works for them and monitor progress in the M&E framework is the best place to have these discussions. It's not just about the numbers. While quantitative data can provide information about outcomes and progress over time, the qualitative data provides a rich story, so difficult to quantify evidence and there's often a check on whether what you're measuring makes sense for those intended to help.
However, when we talk about data and quantification, there are a few risks and considerations we need to keep in mind. Choosing the right metrics. So if we focus too narrowly on certain indicators, we might miss broader social impacts.
What we choose to measure is inherently a value judgement. There's always a risk of overlooking what's truly important. Cost is also a big factor.
How much time and resource do you commit to identifying and designing a metrics? The data availability and quality. So while large data sets like administrative data, government statistics or survey data are really important, they're not without limitations. So the data might be incomplete, out of date or biassed.
Additionally, some data points that are critical for social investment decisions may not be available at all, forcing us to rely on proxies that might not accurately reflect reality. There's also a quantification bias that creeps in. So there's a tendency to prioritise what can be easily counted, which can lead to overemphasis on outputs rather than outcomes and simplifying what are typically complex social issues.
There's a tendency for individualistic versus collective measures. So while most quant measures are individual, most of what we're really interested in are sort of more collective measures and designs. So the impact on families and whānau, for example.
And sort of ethical and data sovereignty is a key part of this. As we collect and analyse the data, we want to consider the ethical implications. This is a whole seminar on its own and it's covered much better by others so far clearer than I. So we'll share some links with you that deal with this in more detail.
Finally, an over-reliance on data. So while sort of numerical data is essential, it's important to avoid relying too heavily on it, the exclusion of other forms of evidence. Not everything that matters can be easily measured or counted.
So in short, while these kinds of data are powerful tools in social investment, they need to be used thoughtfully. We must constantly ask ourselves, are we measuring the right things? Are we being mindful of the limitations? And most importantly, are we using this information in a way that genuinely benefits the people and communities we're trying to help? So in my head, when we're talking about an M&E framework, we're talking about going forwards from the activities and the inputs through to the outcomes, but also being really clear that you're doing those things for a purpose. So you already have those outcomes, whether you've explicitly written that or not, but working backwards from the outcomes you're trying to achieve as well.
And so when we're talking about this, is that this really is an iterative process. So with the theory of change going through to the M&E framework, going through to considerations about data and what you're going to measure, you're always going to be going backwards and forwards. And as you think about what you're going to measure, what outcomes you're trying to achieve and what are the measures to show progress towards those outcomes, you're probably going to have to go back to your theory of change and have a bit of a look at whether that still holds true, whether you need to rethink about how you've set that up and whether that really is an accurate representation of what you're doing.
We have had a question come through, which is coming through, do we feature impact in the theory of change? So how would you think the question of, you know? Impact's always an interesting one because this is where we can get tied up with terms, okay? So some people talk about impact, some people talk about outcomes. And so impact is sometimes defined as a thing that happens in the really long term. And so definitely you would want to capture impact.
The way I differentiate between outcomes and outputs is outputs is all those things that you've done and that you can tick a box and say you've done. Outcomes is the difference you make. And if you wanted to define impact, then you'd say that's the really long-term, long-term goal and impact difference you want to make.
Also, when we were talking about including stakeholders in that theory of change discussion, part of the impact conversation is what does good look like for the groups that you're actually trying to do this for? So having everyone on the same page for a really good theory of change process means that you have a really clear understanding about what you're trying to do, what impact you're trying to have, and then how that maps onto the outcomes that exist in the CBACs and the other tools that we will talk through in a minute. Yeah, and sometimes actually what government or some of us might imagine would be the desired impact can be quite different from the group that you're actually trying to serve. And so it's really important to capture from their perspective, what would success really look like? And what difference would it really make for them by their own definition? And it's particularly important when you're looking at different population groups across different ethnicities and cultural backgrounds.
Yeah, that's a really good segue into the SROI itself. When we are talking about the SROI calculation, we are talking about assigning monetary value to some of those social, economic, and health benefits of the programme. And so the whole process is your theory of change, M&E framework, and then taking forward that thinking, you can start the jump to the impact and look at the domains of benefit that we're gonna include in our SROI calculation.
So that helps us to make decisions on what to include in our M&E framework. So if we wanna measure loneliness, then you need to have a measure that tracks how well you're progressing towards addressing those identified issues. So if you hold on to that thought around the M&E framework, we're gonna jump into some spreadsheets.
So I'll hand over to Brendan. Hey, so for Homes Wrong New Zealand, we set up a very simple SROI analysis in Excel that does incorporate the CBA as well. So there's sort of two numbers that pop out of this.
We'll add in some inputs like admin, upgrade costs, the number of houses we've bought and upgraded, and outcomes like improved health and wellbeing. These inputs, outputs, and outcomes in various ways sort of modify and combine these data are also part of the development of the theory of change and the M&E framework. We'll differ depending on the initiative, the kinds of data you have, and for what purpose the SROI will be used.
So I'm gonna throw to past Brendan. Kia ora. Hi, future Brendan, looking sharp.
So this is us talking through the spreadsheet, which is a sort of a combined SROI piece of work. Even a simplified example is complicated to talk through, so apologising for rushing the stream of this. So at the very top here, we have our CBA calculation and the SROI calculations.
They're live, so as we change the slides, these things are also update. Next along is the outcomes. So as Fiona noted, we've got the primary and secondary.
And if we go further down, there's also a TDC domain as well. Next one along is the number who will be affected by initiative. So this is number of houses we've bought or number of families that will be affected by our initiative.
Next along is the CBA. So all the coefficients here are from Treasury's CBAX spreadsheet, where they're available. And we've also got the size or change of the improvement.
So generally, these are sort of yes, nos, discriminated, but satisfied with life. It's got a zero to 10 and some others are one to five. So that part of the coefficient.
And if we go a bit further along, we move into these outcomes being mapped to value using more evaluative techniques. Which have been identified as part of the theory of change and in the M&E framework. So this is what good looks like.
This is the importance of it to the stakeholders. And there can be a range of numbers. At the moment, I'm just giving a 10 for the primary and five for the secondary outcomes.
And then we've met those goals. From one to not achieved, three to five, all goals have been exceeded. Before I progress further, I just note that the CBA and related techniques are additive.
So adding more indicators and the related monetising coefficients does inflate the CBA ratio. Just gets bigger and bigger the more you add to it. So while it makes sense to add in the outcomes your initiative will affect, it makes it problematic when comparing between initiatives, unless they have exactly the same indicators and coefficients in their model.
This may be the case, for example, if an organisation simply didn't have the time or resources to identify and find the coefficients for all the outcomes, or an organisation had the resource to find and load in as many relevant indicators as possible. What they are also useful though, regardless is modelling different scenarios and testing different levers to see what would give the best outcomes for investment. The SROI and our implementation in this example sort of smooths that curve though.
So the actual SROI calculation takes the value score for each domain and divides by the maximum possible score and then multiplies by 10 to get sort of a relative score. So it's not affected by the number of indicators. It's always a ratio of relative measure.
Then we calculate average for the primary, secondary and tiered domains and add them together. This also means that we can add more indicators to improve the resolution of the model without inflating the SROI ratio. Although a key estimate has to be taken is adding more variables can reduce the sensitivity of the model to changes in a single indicator.
So cruising on the top here, we've got our modifiers and multipliers. So data weight, what would have happened without the activity? It's largely quantification of the counterfactual. The 0% means nothing would have happened until we came along.
And 100% is where it would have happened regardless of what we're doing. Attribution, so who else contributed to the change or what isn't within your sphere of influence, for example. 0% would reflect the situation where you're the only activist in the space and the number gets higher as you see others in that space.
So others not providing social housing, for example. Next one along is displacement. Right, so there's a displacement or substitution effects.
These are benefits claimed by the project at the expense of others outside the project. For example, are we competing with other social housing providers for limited housing stock? And then discount rate, so this is how swiftly does the impact of your initiative diminish over time? I.e. for each house we buy and upgrade will have a larger impact on the health and wellbeing of a family immediately after moving in than it would 10 years later. And we've done this over a 10 year period, but you could go on for much longer, obviously it's just, you just extend things out.
This is, these are the CBA calculations. And here we've got the value calculations. So you can see the separate and done independently.
And that's the space spreadsheet. Kia ora, back to the studio. So thanks for that pass, Brendan.
And I understand that the audience may have got a bit of a peek behind the curtain of the magic that goes into pre-recording. So while we were doing that, a few questions have come through. One of them has come around, does the spreadsheet link to the CBACS tool or are the CBACS values entered manually? Entered manually, you could link them automatically, but it didn't make sense for this sort of size of calculation, perhaps if you're doing something much bigger or you're doing lots and lots of these sorts of things, you could do something more automated.
And one of the other questions that's come through is, is SRI more appropriate for some initiatives than others? How does it work with complex, more systemic issues where causation is hard to establish linearly and the research may be limited? It still kind of is, because it's about the story. It's when you get bogged down in doing these calculations and that being the only measure of its impact. So if you're going more broadly, what we're trying to do with the theory of change and the M&E framework, it's part of the limitation of the sort of work, but I think as long as you hold to the truer process of developing one, then it's still appropriate.
What do you think? Yeah, I think, look, I think one of the tricks is to be really where you can't capture everything, right? So sometimes when I'm building a theory of change, I actually take a bit more of a systems view and I'll have things in there that are labelled clearly that are maybe within the organised level of interest or actually under the, that's something that they could influence. So it might be a signal that they need to partner with others to actually realise those benefits, but often you're right. Outcomes are actually often not linear.
They don't follow this lovely linear line. They're actually far more circular and they could be dependent on a lot of different things. Some outcomes have to be realised first before others can.
So I think one of the tricks with that is to be, to remember that the SROI, while it's a useful tool, it's not a panacea. It can't be used for everything unless you'll cover everything and one of the things I think you need to do is to actually put in the limitations. So if there's weak research that shows the linkages between this, then that should be one of the things that you need to highlight.
And one of the things, what's a now theory of change that you'd normally have is assumptions. So the assumptions is that the evidence actually is there for SROI purposes. If it isn't, then I think you just need to be clear.
And some of this is to where qualitative can become so important where you're building up additional. If you're doing the full M&E process as well, if the data doesn't exist, having that discussion about do you have funds to go out and do primary research to help fill in some of those gaps? And really is a lot of it is around like how much money and time and effort do you have to do this evaluation SROI of your initiative? And so with more time and effort, you can narrow the uncertainty, which Brendan will talk about a little bit later. But one of the really key challenges in all of these, even the simple ones, but especially the more complex ones is around attribution.
And so if you wanted to touch on attribution is one of the key challenges that we do cover. Yeah, so thanks, Jason. So one of the most challenging aspects of any SROI analysis is determining how much of the change can be attributed to the programme or initiative itself.
This is where we carefully estimate attribution, how much of the improvement is due to your programme and how much stems from external factors such as economic shifts, government policies, or even anticipated changes in community behaviour. Even with strong data, it can be difficult to say with certainty that a positive outcome is a result of the programme itself and not other factors. For example, if employment rates improve in a community, it may not just be due to the programme you implemented.
It could be influenced by broader economic conditions or other local initiatives. Being aware of this risk helps us to avoid overstating the programme's impact. One way to manage this complexity is to develop a counterfactual scenario, estimating what would likely happen if the programme were not implemented.
This helps sharpen the accuracy of the SROI by ensuring the programme's specific contributions are well understood. Yeah, and a question just come through on how do you calculate dead weight and attribution? And so we will get onto that very shortly. So keep that question in the back of your mind.
So in this one, remember, it's at the beginning of the process of making funded application, which would be different from if we're monitoring or doing something more, at the end of the initiative, for example. We don't actually know what will happen, but we're setting the logic for what we think is going to happen. So for monitoring progress, we would start collecting real-world data, which would improve our confidence in the value estimates over time.
These would get better. So incorporating uncertainty and confidence is, so a single SROI number can be misleading, especially given the uncertainties inherent in any complex social initiative. So a more nuanced approach is called for that factors in uncertainty through a variety of techniques, such as scenario analyses, where you're sort of developing different scenarios based on different assumptions, like high or low estimates of key input variables, which can give you a range of possible outcomes.
It is important to test how the results of the SROI analysis looks from different stakeholder group perspectives. For example, does it differ from Maori or for Pacific peoples? There's sensitivity analyses, testing how small changes to input variables affect the outcomes, which allows us to identify the biggest levers that drive impact. And range estimates, so calculating your upper, middle, and lower bounds with SROI provides a more reliable range of impact, because you're going to sit somewhere in there.
As this lovely little graph shows, reducing uncertainty over time. The more effort, time, and resource invested in the SROI analysis, the more refined and accurate the estimate becomes. With deeper data collection, more precise measurement, and continued stakeholder engagement, your SROI moves from being an educated estimate to a well-supported figure with less uncertainty.
Simply put, the better your data and process, the better your SROI calculations will be, both in terms of accuracy and confidence. One of the big things is around the period and the discount rate. So the time periods in the CVACS tool are often quite long, so they can be lifetime, sort of 30 plus, 50 years kind of measurements.
And so that can be complex for, when you're talking about attribution of initiative that may be short term, or may only have funding for a certain number of years. And so when we've done CBAs in the past, for example, in the natural environment sector, some of the measures in the CVACS tool around things like preventing natural forest loss, increasing biodiversity, those sorts of things have absolutely huge benefits over sort of 45 years. And so there's a real risk of over-attributing your initiative to biodiversity, and then claiming this huge chunk of this massive benefit over a 45-year period.
But in the same sense, there's also a risk of under-cooking it. So the thing that you're doing is to increase biodiversity, prevent virus loss, et cetera. Legitimately, you can look at how your initiative is going to contribute to that, and look at what might be a normal amount to be able to attribute to that.
And someone just questioned, so the 25% discount rate that was in the spreadsheet. Like I said, at the beginning of the process, this is an example. So normally you would negotiate that with the stakeholders and research.
So this is something you would iteratively develop as part of the theory of change in the M&E framework. So there's, but there's always going to be an estimate. So it's just a number we picked for this example, but it possibly would be longer depending on what your assumptions are.
And Treasury has defaults, you know, 6% or 7%. And so it's just, I think for illustrative purposes to show the spreadsheet and how things work. So withhold belief if that seems too high for you, and we can have a chat about that later.
So with the next step around the calculations, we're going to go back to past Brendan, hopefully just once this time, to talk us through in another video of Brendan showing some of the shifts when you do edit that sheet. So we'll play the video now. Kia ora anō.
So this is where we get to play with the spreadsheet. Before I begin, make sure you pay attention to the top left corner where we've got the CBA and the SROI estimates calculated live. So as we sort of tweak the levers, you can see them being updated as we go.
So I'm going to use this as an example, the primary health outcome of the house isn't cold. Right, if we go across here, this is, at the moment, the indicator is basically yes or no, the house isn't cold or it is. So if we go from a yes to a no, we drop down from 1.8 to 1.6. Conversely, if we say we're going to get funding for two houses and we couldn't upgrade three houses, we go from 1.82 to 1.69. So you can see how you can play with the different scenarios and save those.
As for CBA analysis, if we go across to the value estimates, note that you can link these to the, say, if you've got a survey measure, you could use that and have that update the value estimate instead. So what we're doing here is, let's say, you didn't achieve any of your goals. You basically didn't make the houses warm and dry.
So put a one, and you can see that drops down from 5.18 on the top left. Go back to three. And if we say we exceeded all our goals, the houses were just beautifully warm in the summer and beautifully warm in the winter and beautifully cool in the summer.
And we'll do a five, and it jumps up to 6.33. So you can see by playing different scenarios here and across the other indicators, you can see these players and save them independently. And then what I've done is I've run three scenarios. One is the low with all the, basically you failed at everything.
And the high one is where you blew everything into the water, it was amazing. And the mid one is just what I was demonstrating before. What you can see here is how the estimate range or confidence interval increases over time, i.e. confidence decreases.
Another key point to make here is the diminishing returns on investment over time. So if they flatten off, in this case after 10 years, that will change depending on your figures in your main spreadsheet. And these figures are part of the entire SROI.
So we take, say, the 10 year social return on investment and feed these back into the entire SROI alongside the theory of change, the M&E framework, and all the other insights that have been gathered. And that's the SROI there, the whole thing, not just this part of it. There is another feedback loop here too.
So while setting up the SROI analyses, some components of the M&E framework may need to be adjusted such as indicators that can't be actually identified or monetised or sensibly turned into value, or even what other primary outcomes may change as you discover some are bigger levers than you thought and some may be smaller. And that's the SROI CBA magical spreadsheet. No meaning.
So thanks very much for that pass, Brendan, again. And so that was a brief tour of the ANC SROI process. As I said, all the templates and slides used today will be available to download and look at in more detail.
So let's go through some quickfire SROI questions. So the question come through from Paul is, how do you deal with the risk of double counting in an SROI? Well, double counting happens when the same outcome is valued more than once under different categories. For example, if Homestrong New Zealand improved health outcomes and increased employment, both of these could be traced back to better housing.
However, attributing both fully to the SROI could lead to inflation of the benefits. To avoid this, we need to carefully track each outcome and its drivers, ensuring that we don't attribute multiple benefits to the same underlying cause. So question come through from Jenny is, how is qualitative data used in SROI? I love qualitative data.
So qualitative data helps explain the how and the why, and it gives us more depth of insight. While quantitative data tells us what happened. Oh gee, how many homes were retrofitted? So qualitative data gives really important context.
It can help answer questions like, what's working well? What's the factors involved that make it well? Or is it not working well? Is there opportunities to make improvements where you could get even more benefits out of it? So things like, it enables you to also tell stories that can really actually elucidate the difference it's making, such as like in the housing homes. Well, like tenant stories that explain how having a secure home improved their family's emotional wellbeing or ability to focus on education. And this adds richness and depth to your SROI analysis.
Yeah, and it's a check on whether you are counting the right sorts of things, or that you might be missing something, that you're not measuring the right sorts of things. There is a lovely feedback loop in there between those sorts of measures. So segue to that is Mark has asked, how do you decide what to measure in an SROI? Well, theory of change.
Yeah, I think the decision starts with actually where you develop the theory of change, which is when you go to determine what's really important here. Okay, so you wanna get rid of the noise. You don't wanna go for the detail.
It's about really highlighting what are the key outcomes you want to achieve and choose indicators that are linked to those outcomes. So one of my tests with outcomes is, is it measurable? Because you need to be able to measure it. And then, for how strong New Zealand case outcomes, which improved health and reduced homelessness.
So we could measure things like reductions in DB visits, hospital admissions, and instances of homelessness. These are all things that can really be measured. It's also important to choose metrics that are both meaningful and feasible to collect.
One from Kelly is, how do you calculate SROI for long-term outcomes when the benefits take years to realise? Well, when calculating SROI for long-term outcomes, we use technical discounting. There was feedback that we use quite high one in our example, but it did make a nice graph which sort of flattened off quickly, so that's good. But there are smaller amounts.
So discounting adjusts for the future benefits to reflect their present over time, to reflect their present value, which allows us to account for long-term benefits like improved educational attainment or reduced homeless over time. Now, predicting the exact timeline for these benefits is challenging. So that's where you use things like sensitive analysis or scenarios just to test what that looks like.
And so that might be where you assumed a scenario where you're very effective or not very effective and looked at what that meant for the calculation as you walked through. Yeah, it's a one and done sort of thing where you just have that one impact and then you have no control and it has no further impact on their lives. Yeah, and also I think something about attribution, you might disagree with this, but attribution is that the longer, further away that outcome is from realisation, the less you can attribute your programme now to actually its realisation.
So you need to take that into account and factor that in when you're actually thinking about calculating an SROI that involves really long-term outcomes. Yeah, and as I said, over time, your range of estimates, your range of uncertainty increases. So your certainty decreases.
So it does get bigger over time. So you need to remember that. And Paula has asked, how do you ensure that SROI captures cultural and community-specific outcomes, particularly for Māori and Pacific populations? Well, capturing cultural and community-specific outcomes, you need to be working with them.
There's no sort of, you can't be outside of it and do this. You have to be part of the community, ideally. So there's, earlier I did mention the difference between individualistic and sort of holistic or collective outcomes.
So there is, you're measuring different kinds of things and you're measuring in different kinds of ways. So these are really important to remember. How it's done is key.
And often, it's you guys are the ones with the connections to do this. So you can't expect other data sources or other people, externals perhaps, to do it as well as you, even though they could try and actually provide an independent perspective. So it's important to incorporate the different ways of thinking.
But again, you have to bring, you have to work with the communities and those people who work in those spaces. I think it comes back to our point that we made around the importance of engaging and having key stakeholders involved in the development of the theory of change, because actually what it looks like could be quite different, okay, than what maybe somebody from Pākehā background or some other background might actually create. So you actually need to, so what might be in and what might be out of scope might also be very different.
So that's where I think really working closely and getting insights from them, how do they define some of the terms? What does it look like from them? How would they define success? So you start capturing success from their perspective and talk through with them, how would you capture that? How would you measure it? Yeah, and one of the examples we've had working with Māori businesses, right off the bat, that what they value is long-term intergenerational, that we're doing this to get profit for long-term investment, for all sorts of outcomes. It's not just return to shareholders kind of thing. So right off the bat, when you talk to them, what they value is just very different.
And so, again, it can be mismatched from funders, but by having a theory of change that shows that explicitly upfront, you can then put it into your system and measure it well. The indigenous system adopted an infinite timeframe. This isn't within three or five or 10 years.
This is infinite. This is generational. So we'll have a few questions come through on the chat.
So one of the questions is around, how do you quantify the impact or monetise more abstract outcomes like leadership skills, wellbeing, confidence, et cetera? I think that's a really good question. So I think one of the things to remember is that not everything can be monetised. And sometimes it's some aspects of something that can be monetised and other aspects can't be.
So I think one of the things about when you're using terms like wellbeing, is that part of the trick of doing the theory of change is actually in there to define what do you mean by wellbeing? The Living Standards Framework itself, hopefully you might be familiar with that, talks about the whole range of domains that can be involved in the concept of wellbeing. So you need to be clear about what you're talking about within that. So within ours, we looked at, say, social inclusion would be one where we saw that there is measurements for loneliness.
There are other aspects of social inclusion which are not so easily measured. And those is where the qualitative and more value statements that go alongside your SROI are really important. It's important to remember too that some things may not even be ethical to try and actually monetise it.
And finally, you could actually, sometimes you can find some evidence in literature reviews or a scan of what's out there to find actually some source or indication of how it could be potentially valued. So hopefully that answers the question. And then I think linked to that a little bit is someone said, not everything that matters can be easily counted.
How do we measure these things? Can outcomes be stories, qualitative, or is quantitative data king? Ideally, they're together. I mean, you don't ever separate the two. For certain purposes, the numbers are useful for making nice graphs, for example, but there's always a story behind it all.
And the qualitative data is always to check the numbers are right and the numbers are always just supporting the stories. So there is that interrelatedness to it. Different path analyses might prioritise numbers and the more narrative stories.
So, you know, there's shifts and moves. And linked to that, so one of the, I mean, we were talking about doing an initial analysis at the start of a process for getting more investment. One of the questions is, how feasible is it to do an SROI on a new innovative programme? And so I think that's one of the cuts to the heart of some of the things we're doing here is like, yes, absolutely.
And it really is about setting out the logic for why you think your project is gonna have an impact, what the impact is likely to be. But then also taking that more future focus look about how you're actually gonna measure those things that you're doing. So if it's something innovative, you're probably, you've got an agreed outcome you're trying to achieve, but the mechanism by which you're doing that may be something slightly different.
So being really clear about what you're measuring, how that's gonna then contribute to the outcome is the key part of that. And then, Fiona? I was just gonna say, that's where I also think an M&E framework comes in, right? Because there may be very limited data at the moment. So actually, your ability to state what it's likely to be in terms of quantitatively may be very difficult.
You'll have those logical links about what you're imagining it will achieve. So an M&E framework gets you to think about, well, what data do I need to collect? What could be my possible sources to actually start collecting that evidence base that would indicate? Because one of the questions is, if you're doing something really innovative, how do you know it's tracking in the right direction? How do you know that it's delivering what you're planning for it to deliver? And how do you know it's serving the group that it's actually targeted at? And different funders will have different requirements for that as well. So especially if you've got an innovation fund that's deliberately trying to do things differently, to be effective, that has to be linked to an investment framework that allows for those more innovative approaches.
If you're trying to innovation and then require the same, how many people came through your programme, you end up with that same systemic sort of issue where you're not actually then being innovative because you're still requiring the same counting that you've always required. And so you get the same things that you already got. So it really is context dependent around what's actually required, how you're gonna do it.
But the benefit of doing it and having it all stated with the theory of change, the M&E framework, these are the domains of benefit, this is the qualitative narratives that we're gonna try and capture and how we're gonna get them. You can start to tell that story for funders, for your stakeholders, for yourselves around what is the impact you're trying to have and show that thing. And then as you go over time, show whether or not you're achieving it or show the willingness to twist and course correct as you're going if you find that the data isn't quite showing what you're doing.
And sorry, just final thing I'd say there is a theory of change when you do it at the beginning, especially for something innovative, it isn't set in stone. So there'll be key points along the way that you might wanna come back and say, we made all these assumptions and that's why it's important to document them. Do they hold? If they don't hold, what does that mean for the theory of change? Is there some adjustment here that you need to make to actually reflect the fact that it's not actually quite as you thought it was, which I would imagine for many innovative things, actually, it's quite plausible.
Yeah, and that's why the engagement with the stakeholders and other people, experts in this space is so important at the beginning because you're trying to get a clear picture of what the impacts could be and pick and untangle that and then you can do your other pieces to it. Yeah, and someone's here asked about the most smaller for-purpose organisations are unlikely to have the resources or skills to do the SROI as outlined in spreadsheets and can't afford to pay a company to do it. Are there simpler options? It's a, I think that is a good question.
And so there is ways to do it easier. You just get less of the number. So I think the non-negotiables is having that really strong theory of change shows what you're trying to do.
What are the assumptions? How are you gonna do it? That is something that you can find a lot of information online about how to run a theory of change workshop and how to build those. I think the second one, the M&A framework, again, if you search for how to do those things, there's a lot of information, a lot of guidance on how to develop one of those. And so that is something you can do relatively cheaply or with a very small amount of specialist help, you can go quite far along the journey.
When you start getting into the CBACs tool and those things, I understand it is a little bit daunting. And so I think part of the whole process is just trying to, I don't know, have a think about when you're doing that theory of change, what are the outcomes that you're trying to achieve? Have a look at whether they're in the CBACs tool and then have a discussion around if we are trying to support loneliness, for example, there is a measure in the CBACs tool for loneliness. You can start to have a play around with what would it mean if I did do this and sort of, I don't know, it was like have a go, but I think focus on the theory of change, the M&A framework and the qualitative and how that's gonna tell your story and then be aware of the other parts, but aware that it's not super straightforward and not everyone has a Brendan on staff to build a spreadsheet, et cetera.
Yeah, and you have to be honest about the uncertainty in those cases. If you're just having at the time a resource to actually try and figure that stuff out, you just have to be honest, we're in here somewhere. And one of the ways we do do it as well, especially so innovative programmes, new things, if you're not sure about the level of attribution, so things around smoking and those things, it's very hard to attribute a specific programme to that change.
But if you do things like cost even analyses or break even analyses, so if you take that domain benefit for loneliness, it's monetised so you can show that if we were successful and all these things were true and we shifted people one step up the wellbeing framework, whatever it is, that would be a benefit of this, which means that we would only have to help five people to achieve a CBR of zero or CBR of one, sorry. And so there is ways of trying to make the process a bit simpler. I understand they're not always easy, but again, we are happy to have a chat about your specific circumstances and what you may be able to do without needing to go too far down the investment route.
Can I just say one thing then? For smaller organisations, don't be overly ambitious about what you're gonna capture. Make sure that your measures in the M&E are really targeted and they're really the key ones that you can actually achieve. And think about whether for you it's feasible or not, because qualitative can be quite expensive.
You might find if you do a quick search that some of your quantitative can be actually sourced through other sources or could be set up quite easily. So the thing is, start simple. Make sure it's really well targeted.
You can always build from there as you actually grow in confidence and capability, et cetera. Yeah, and someone's asked, any ideas with the new government's social investment approach? Would theory of change slash M&E be sufficient or would you need the monetary value of outcomes? And so I think there's probably talking about some of the investment funds we have seen come through where they have required an SROI to be done by an independent evaluation. So its context is specific because for all of the ones we've seen that have required that, there are still other funds out there that are specifically looking for innovation and aren't asking for that quantification.
So it's, yeah, I think when people are talking about SROI, they really are wanting that sort of shortcut. I wanna understand a dollar amount at the moment based on all the uncertainty to know roughly what I'm looking at. But I think a big thing with all of this discussion is that that is always gonna be reductive no matter what you do.
And there's a big focus on what are the other things that have gone into that, being really transparent with the process of how you got to that number. You'll find that all of those qualitative narratives, all of the things that contributed to that are just as important. And so long way to say, I think sometimes they are actually just wanting that number and that's what they're looking for.
But I think most of the time, most people that are active in the space are aware that that number is just part of the story and a lot of the stuff that goes around it is just as important. So having the narrative, having the other parts that go along with that number is just as important. A number might come from a related programme, you just quote, this is a similar programme, return these sorts of values, and then you can build the narrative and other things around that and not have to go down the strong.
And again, having the uncertainty, we have based this on this. I think the big thing is, as long as you're really clear about where everything came from, that's the really key. Someone's asked a question about, is there any negative view about collecting our own data versus accessing data collected by an independent third party? Not really.
I mean, if you follow, there's good quality data. I mean, sometimes you're the only ones who can collect the data. You're the only one with access to those households or those communities.
So sometimes that trust is where the best data comes from. So you could ask an independent person to collect the data, which is useful as well. If you've got the resources, but it's again, it's context dependent.
There's nothing wrong with collecting your own data as long as it's done in the M&E framework. Those in theory of change does help make sure you've got a really good sort of instruments and tools to do that and how you're going to do it. Yeah.
And one of the things Brendan raised earlier was the risk of using other people's data. So, you know, there's examples where we've looked at other people's research and then you talk to the people involved and they'll tell you, oh, well, I wouldn't actually rely on that because the question they asked was actually this. And they didn't, you know, this other really important change in the sector hadn't happened yet.
So if you ask those same people now, they'll have a very different response. And so I think it's just, again, being really clear about what you're trying to do, what evidence you need to show that you're working towards that. That's sort of the key takeaway.
Someone's asked, some funder objectives for a programme can be too broad and unhelpful. How do you counter this in creating a theory of change? Yeah, that's really tricky. Experience has taught me, it is absolutely crucial that when you do your theory of change, you are clear about what you're trying to achieve.
Otherwise you can't tend to spin your wheels going round and round and round. So I guess you either need to try and get them to clarify what they mean in those objectives, or you need to clearly state, this is what we've understood it to do. And this is what we're setting out to do, because it is absolutely crucial that you're clear about your purpose, because that's what the whole theory of change kind of hangs off.
And when you're talking about big system sort of changes, like your organisation may only be responsible for one small sliver of the actual system and process. And so if you're actually talking about the outcome of safe drinking water or something big and grand, your part of that will be actually really, really small. But when Fiona's talking at the start, you can do broader ones that show the interrelation with other people and where it may be, I don't have direct responsibility for that, but I may show that I want to spend some time and effort in influencing what they're doing to make sure that our work's aligning with them to contribute to that goal.
You can go broader with some of those types of things, even if they aren't directly in your control, but it does get bigger and unwieldier and harder to manage when you do push out. Yeah, I mean, what I would say is actually I think it's desirable to do some of that. I think that actually we're at a time when we don't want to go back into silos.
A lot of the problems we're trying to address are really big and complex. So what you need to be honest about is if you look right across the system, what's your part of the pie? What's that part that you're contributing to? When I include those broader things, they're not actually particularly for measurement. They're actually more highlighting maybe critical conditions for change.
So actually what else would need to happen for our vision or mission to be really realised? Or who else? Yeah, who else do we need to work with? What else do we need to take into consideration? Is there anything over there that could really interrupt actually and undermine us achieving what we're setting out to achieve? And if you're doing the calculations, those feed into the attribution and the substitution effects. Who else are you affecting or impacting? Who else is in that space? And we've had a question come through. How do you handle outcomes that can't be easily monetised? I think actually you mentioned it earlier, there is a component of going out and just doing some research and finding if people have measured it.
There's also where you're measuring something adjacent to it. So like social inclusion, we've got loneliness, but it could be something around how many people are a part of a club? Are children able to go to a club? So there's other measures that are related to it, but not a direct measure of it. And they can also bring their own biases and uncertainties in there, which is why it's so important to actually understand the range of values.
Yeah, I also, I mean, I would be personally very concerned if all the outcomes are driven by what can be monetised. And I think that's a real risk at the moment is that we're getting into the monetisation of everything. Going back to a point I made earlier, not everything can be monetised, OK? It doesn't necessarily have to be.
It does need to be measurable, but that doesn't mean it has to be quantitative or monetised. It could be qualitative evidence that actually helps boost the understanding. And in the homes for example, we talked about increased family stability.
And so we didn't attribute a value to that, but we talked about we can tell the story of how stable housing has led to better life outcomes, for example, by interviewing the people that are in housing, you can ask questions that would help you show that sort of evidence without quantifying it as a... We didn't monetise it, but you can give a value to it and say this is important and how important it is. And even that whole purpose, if they're included, what is important? And you can then weight things as more important. So stable family home may be really, really important to the people that you're trying to measure.
So you can then adjust your sort of... And you might be able to take into account things like family harm and whether it's actually been reduced within that family. If they were under... Has it reduced stress levels? Has it reduced likelihood of family harm? Which is actually being derived from stable housing. Unfortunately, we have run out of time to answer all the questions, but we are happy to catch up with you at any time to answer any more questions you have and discuss potential ideas that may support you further.
Our emails will appear at the end of the recording so you can get in touch with us for a discussion. And so with that, I'm keen to go back through and ask people, what's your key takeaway from everything that we discussed today? Well, just reflecting on the bit I did, the past me and the present me, it's just a tool to inform the entire SROI. This is just something that you do to refine your thinking and it can generate some interesting numbers and useful numbers, but it's just... Don't use it in isolation.
It's part of the whole package of what Fiona talked about. Yeah, so thinking about what I talked about, I think for me, one of the key things is you need to get that theory of change and do it well. You need to be clear about what you're trying to achieve because if you're not clear about that, then the question becomes, how can you possibly measure it? The other thing I think is that when you're doing that, think carefully about the change you're trying to make and the who, and just be careful about the Matthew effect.
That's where people already doing well tend to benefit actually quite often from initiatives that were designed for a group that aren't doing well. And so I just want to really make sure when you're collecting data that you're able to capture that and test for, is it the group that you meant is benefiting? I think my key takeaway, we talked about it a bit, but the iterative process. So just starting the process of mapping out your outcomes, your activities, your outputs, and then working through that, how are you going to measure it, looking at the outcomes you're trying to achieve, and then going back and working through again and understand that this can be some quite complex parts to it, but just giving it a go and just really sort of starting to document your assumptions and the things that you're doing and how they're supposed to work and just see where you get to with that and then go from there and then reach out for a chat if you need any more help.
But that's that's all today. So Nga Mahi, thank you very much for joining and we will see you at the next one.
