Evaluation Myths Debunked: Reflections from 200+ evaluations

Good afternoon everyone and thank you for joining us at our webinar today.

Before we begin, as is respectful in Australia, I would like to acknowledge the traditional owners, the lands which we meet. I'd like to pay my respects to elders past and present and extend that respect to any Aboriginal or Torres Strait Islander people who are joining us today. Thanks Will.

And here in Aotearoa New Zealand, I'd also like to acknowledge tangata whenua, especially those iwi of Te Whanganui-a-Tara where we're based here in Wellington. So I'm Marnie Carter and I'm a Senior Evaluator here at Allan & Clark. I'm based in our Aotearoa New Zealand office and I've been doing evaluations for about 17 years now.

And I'm joined today by my colleagues Emma and Will. Thanks Marnie and good afternoon everyone. I'm Emma Kelleher, I'm a Senior Consultant and Evaluator here at Allan & Clark as well, but in the Melbourne office.

I've also done probably about 17 years worth of evaluations in my time across government, in the social and community sector and not-for-profits as well. And my name is Will Morgan, I'm an Intermediate Consultant here at Allan & Clark. I've worked in evaluation now for about four years across a variety of projects, largely centred around social and health policy and programmes.

Thanks team. So together throughout our organisation we've done more than 200 evaluations across New Zealand and Australia. And that's covered everything from housing programmes to family violence initiatives, community services, transport policy, everything in between.

And that's covered different types and sizes of evaluations from large multi-year, multi-site projects to small rapid reviews that were done in a matter of weeks. And over the course of doing all these evaluations, we have seen that there are quite a lot of myths and quite a lot of assumptions out there that on the surface sound completely logical. But when you dig into it or try it, it doesn't necessarily always work in practise.

Yeah, that's right. So we thought there'd be some benefit in this session today where we are going to challenge some of those most persistent myths that we come across or common beliefs that we encounter with evaluation. So we'll share what the conventional wisdom says.

We'll talk through some examples of how it doesn't always work in practise and offer some ideas about what you can do instead. Before we start, though, let's see what the audience thinks and run a quick poll. So you'll see on screen we've put three common beliefs about evaluation.

So if you touch your voting buttons and let us know which of these common assumptions do you disagree with. Emma, looking at the ones that we've got up there on screen, what do you think most people in the audience are going to disagree with? I think most people will disagree with number two, that all evaluations involving people require formal ethics approval. Interesting.

I reckon the strongest disagreement is going to be with number three, that evaluations can be impartial and apolitical. Well, let's have a look at the results. Oh, it's getting pretty close.

I'm going to claim that for a win because number three does have probably the strongest number of people disagreeing with it with about 53% or 63%. But the other two also, look, pretty much all of them, about 50% of people disagree with. So, yeah, interesting results there.

Good opportunity now for us to dig into these a little bit deeper. So starting with the first one, one that I have heard a lot of talk about recently is the idea that evaluation should only be conducted by qualified and trained evaluation practitioners. And some people are even suggesting that that should include mandatory certification or credentialing before you're allowed to call yourself an evaluator.

So, Emma, what do you think about this one? Yeah, it's a really interesting question and I can understand why it's a bit of a hot topic in the sector. My view is I don't think you necessarily need to be credentialed to do a credible evaluation. But you do need to have you need to sort of have some grounding and understand the science that's behind it because it is a discipline.

It's a lot more than just storytelling or doing a review or collecting feedback on your programme. And evaluation is about being able to design an approach that fits the purpose. And there are different methodologies to evaluation that all have different approaches and purposes.

That could be like a developmental evaluation, a process or an impact eval. So it's really important to understand. You wouldn't necessarily know them unless you had some sort of formal training, right? Yes.

Yes, I would think so. Or, you know, learn that from somebody that has had that formal training. So I think the other aspect of the sort of science of evaluation that's really important to have a good understanding of are concepts like causality, sampling, attribution.

What can actually be attributed to your intervention and what can't and bias, which we'll talk about a bit more later on. And then the really important thing is understanding the difference between just measuring your activity, what you delivered versus what actually changed. So measuring what's changed for whom, how and why.

So that's really technical work. It does take skill and practise. Yeah, I agree.

But I worry we're making it sound more exclusive than it needs to be. And I'm worried that sort of an emphasis or overemphasis on certified evaluators is exclusionary. So not everyone has the time or money to go to university or to do a certificate.

So, for example, I know one course in Australia, a year course costs 16 grand. I mean, it's not cheap. So while professionals certifications, they're valuable, I 100% agree.

I think it can overshadow practical experience. I mean, from my personal experience, I did go to university, but I didn't study evaluation. I learnt evaluation through on-the-ground practise and actually getting my hands dirty, granted.

And that was done with mentors, many of whom were trained evaluators or who had significant experience in evaluation. Yeah. And well, look, I would say that that's worked really well for you.

So you're definitely an excellent evaluator. My experience, though, was a bit of the opposite. So I am a trained, qualified evaluator.

And the reason that I pursued that path is because of my experience trying to do an evaluation without training and realising it just was an abject failure, to be honest. So prior to Ellen and Clark, I worked for an NGO and we were delivering a programme and we wanted to know, is it working? And we thought, look, we're practical people, we're intelligent, we know the programme. We'll be able to, you know, read a few books and hack out an evaluation.

And honestly, we did not get any useful information from that at all. So that's what prompted me to actually do the training and learn how to be an evaluator. And secondly, in that case, the next time we wanted to evaluate, we did hire a trained, qualified evaluator.

And that's when we learned, OK, how to actually measure change, not just kind of count outputs, which is what we've done previously. So, Marnie, with that example, do you think it was the qualification and credentials of that person that you brought in, or was it their skill set that could be learnt in other ways? Yeah, good point. Probably the skill set.

Yeah. But possibly a bit of both. Yeah.

So, I mean, I think that skill set can come from formal training or from experience, but the professional development and mentoring, I think, is a really important aspect to try to ensure. Someone in the registration for this webinar today asked whether programme staff can evaluate their own work and included the quote, which we loved, because we know where the bodies are hidden. And while that's true, that insider knowledge is incredibly valuable, there does need to be, I think, a bit of arm's length distance between the programme delivery and the evaluation, because it can otherwise really introduce bias or blind spots in the methodology.

So, I think that what matters is that whoever is doing the work has a bit of independence from what was delivered and also has enough methodological capability or is working alongside someone who does to ensure it's going to be a robust and sound evaluation. Yeah, look, I want to push back a little bit on you both, though, because I think we need to be careful. So, I'm going to agree that we shouldn't include people because they're not formally trained or they don't have the piece of paper that says they're a qualified evaluator.

But I also think it's quite dangerous if we're saying we don't need formal training at all. So, if we say, you know, anyone can do evaluation if they have kind of good intentions and lived experience. I think that's a risk in undermining the fact that evaluation is a discipline and it's technical work.

And as you said, Emma, you really need to understand that design and causality. And that's not things that you pick up just by kind of being thoughtful and having lived experience. So, I guess what I come down to is, you know, does everyone need a degree in evaluation? Well, no.

And Will's point about learning through mentorship is very valid. But I mean, I would argue that somewhere in a team there needs to be foundational knowledge. And that probably has some type of book learning or some type of qualification.

I mean, I agree. And I think it comes to the question of who can conduct credible evaluations. So, for me, credibility comes from experience of evaluations and knowledge of practise.

And equally, it also comes from first-hand knowledge or an experience of whatever and whoever is being sort of evaluated. So, I think this allows you to both value the academic knowledge that you would get for a degree, alongside that more sort of softer abstract human experience, which can't just be learned or legitimised for a degree or certification. So, what comes to mind here is ensuring input at all levels.

And that includes, I suppose, for in a better term, citizen evaluators. So, people who don't have that background, might not have any background remotely close to it, but they can help ensure that the evaluation you're doing is credible to the communities you're working with and working in and working alongside. So, it's not just credible to broader principles like academic and scientific rigour, but also credible to the community.

But equally, good evaluation practise does involve that co-design so that they're not in opposition to each other. And it's part and parcel, one and the same, really. Yeah.

And look, I do want to make it clear, I'm not necessarily arguing for formal credentialing. If we said that, I think we'd have quite a big problem, actually, because there aren't really that many formally trained evaluators out there. And secondly, in Aotearoa New Zealand, we actually don't have a postgrad level evaluation qualification.

So, that would be a bit of a problem if we said you had to be qualified. I don't think the real question comes down to is about how do we ensure that quality without kind of gatekeeping and say, oh, no, no, everyone has to have a qualification. So, I would say, well, I'll concede that you're right.

It doesn't have to come from formal credentials. And what we do need to look at is a way to build and recognise different types of capability. So, that's probably mixed teams with both that lived experience and practical experience that you were talking about, Will, and also some people with qualifications or substantial experience in training.

So, not credentials, but not anything goes either. Yeah. So, where have we landed on this one, team? I think we're effectively saying it's meth is busted.

Effective evaluation, it requires both technical knowledge, it requires practical skills, but these don't only come from formal qualifications. No, definitely not. And so, we put some practise tips on the slide here.

So, the first one would be to mix your team up. So, have that mix of formal training with practical experience and sort of just lived expertise and also recognise the intersection between all three that can exist. So, you'll get better evaluations that way.

The second one is ask a capability question, not a credential question. So, instead of do they have a degree in evaluation, be more so do they have methodological knowledge? Do they understand the sector? Do they have the relationships with the people that we need to engage with? Yeah. And finally, it is about feeling assured that there is methodological rigour somewhere in there.

So, whoever's leading the evaluation either needs to have that technical capability themselves or they need to be working alongside someone who does. And that's really the non-negotiable bit for me. Yeah.

Agreed. So, that brings us on to the next common belief. And that's that all evaluations involving people require formal ethics approval.

So, this one's a bit of a challenge, of course. As evaluators, we have an obligation and we want to keep people safe. And ethics committee's approval is one of the main ways of doing that.

However, going through that process, I mean, we've all done it. It is complex. It's lengthy.

I can put evaluation work on hold for months. Marnie, what do you reckon? Okay. So, I'm going to give a slightly controversial opinion on this one.

But I have to say that I am a bit of an ethics committee sceptic. And here's my reasons why. So, in my experience, it often takes three months minimum, sometimes six months or more to your evaluation timeline.

And that's a big chunk of the project spent waiting and not actually doing any evaluation work. And then there's the cost. So, it can consume a lot of your evaluation budget before you've even collected one piece of data.

And then I have to get to the fundamental question, which to me is, does it actually improve the safety of participants? And I've got to say, in most cases, honestly, I don't know if it does. And the reason I say that is that all the evaluation practitioners that I know and work with, we apply ethical principles as a matter of course. So, I'm going to put it out there.

I reckon in about 90% of evaluations, ethics committee approval is a box that most evaluations probably don't need to tick. I think that is probably a pretty controversial view, Marnie. But I can understand your reasons.

But I think we do still need to be careful because it's important to separate ethics approval, ethics committee approval, from ethical practise. Ethics committees, of course, they play a really vital role in research and in evaluation, especially for higher risk or really sensitive work. But you're right, Marnie, that a lot of evaluations, they just don't sit in that category.

I think a lot are low risk, they're focused on programme improvement, and they're not necessarily collecting identifiable or sensitive information. So, you know, putting an evaluation like that through an ethics committee process can definitely add delay and cost without actually increasing participant safety. But I agree with you, Marnie, that the key is having evaluators who can make those sound judgments about when that approval is genuinely needed and when strong ethical practise is enough.

So, you know, those concepts like informed consent, confidentiality, do no harm, are really fundamental concepts to understand. I agree, but to recognise, I suppose, the strengths of ethics committees, I think they serve an important purpose in that they give us a sort of mandatory checklist of points we do need to hit. So it's, I suppose, almost like a stopgap for when things do go wrong or when you do miss stuff.

So it's a second pair of eyes that we keep, almost a secondary peer review to make sure nothing is missed. Yeah, I mean, I'm hearing you, but I reckon most evaluators would be doing all that anyway, just by a matter of course. So I guess the question for me is when you think in your evaluation practise, I mean, have either of you actually changed the mental part of an evaluation design or delivery because of feedback from an ethics committee? We did recently.

It wasn't a huge change. At any rate, on a social programme evaluation, we were told that we needed to have both a male and a female First Nations team member to enhance cultural safety. So in our original submission, we didn't have a woman on the team, which we were rightly pulled up on.

It was an oversight. So implementing that feedback allowed us to ensure that our approach was actually more culturally appropriate. And it had the added benefit of ensuring that the data we got was a lot better.

And it allowed us to actually produce a report which is a lot richer in its findings and hopefully once it's implemented, richer benefits for that community in terms of the programme being improved and enhanced. I guess I stand corrected on that one. Yeah, it's a really good example.

And I think it shows like ethics approval isn't always a simple yes or no requirement. The consideration, the core consideration needs to be what level of ethical oversight is proportionate to this evaluation requirement. Because, you know, I've seen examples of where a programme is already underway and there might be some evidence of poor practise or things going wrong and where an organisation has commissioned an evaluator quickly to come in and work out exactly what is happening and potentially intervene before things get worse.

So in that scenario, an ethics committee could introduce additional delays where we know that harm is potentially being caused while we wait for that process to play out. So, I mean, you're right, Emma. And those delays can be harmful, but so is the sort of rushing into the work with safeguards, I think.

So, again, for ethics, I think it gives an independent oversight, which is particularly important, as you touched on earlier, for that higher risk work. So when you're collecting that identifiable information or when you work on this really sensitive topic, we need that external perspective. So I think the committees, they bring people to the table who understand the risks and the protections needed for different communities.

And I think that's particularly important as well when you're working with people and with communities that have been harmed in the past by evaluations and research or overburdened by research or evaluations. So having that sort of independent review can be all the difference between good practise and not as good practise. No argument for me there.

I mean, I do agree that for sensitive, high-risk work, independent oversight is important. And I'm not arguing we should never use ethics committees. Another point I wanted to mention, though, is that I would say inconsistency is or can be a problem.

So I've often found that you get quite different feedback and guidance depending on who's on a committee on any given day, what their particular concerns might be. So just to give you an example, we recently did an evaluation of a national programme that was being rolled out in several different jurisdictions. And so we had to get ethics approval from a few different local ethics committees.

And while the applications that we submitted were pretty close to identical, we got really different feedback, really different requirements on each different committee. And I have to say that made me a little bit sceptical about the process. It actually frustrated Marnie.

And I've had similar experiences, but I think it's an implementation problem at the ethics committee level rather than a problem or fundamental flaw ethics in and of itself as an oversight systemic aspect. So I think the solution isn't abandoning ethics committees altogether. It's having that sort of clearer, more consistent standards internally about what requires approval and what doesn't require approval.

Yeah, we worked through an example of this with a client recently actually and we often see when a client will issue a request for quotation that might specify that they think ethics committee approval is going to be required. In this particular instance, as we worked through shaping up our design, we proposed undertaking a step before that of actually doing a ethics assessment. So it was a deliberate kind of process by which we worked with the client to understand the data landscape, understand the stakeholders involved, really work through all the same kind of elements that an ethics committee would ask you to consider so that we systematically addressed all of the questions about whether formal ethics approval would be required for this project or whether, you know, putting in place all the right safeguards and assurances for ethical practise would be sufficient, which in that case it was and it could save the client a lot of time and money.

Yeah, that's a good example, Emma. I think you've really nailed that critical distinction. So absolutely all evaluations need ethical practise but not all evaluations need formal committee approval.

So what we need is a step or a process to make that risk assessment rather than just kind of defaulting to well, we'd better be safe than sorry and apply for committee approval for everything. So I reckon we can say that myth has been busted and for me the key takeaway here is that we need to be able to make defensible judgments about formal ethics committee approval. So the goal should be to match the level of ethical oversight with the actual level of risk.

Yeah, I agree with that. I think all evaluations need ethical practise, of course, but they don't all need committee approval. And I reckon it can really help to use some of the assessment tools that are out there.

I know some committees have them published on their websites. So it's applying their decision frameworks or flowcharts to determine when formal approval is needed and when it's going to genuinely add value. And of course, as we've said, the principles such as informed consent, confidentiality, participant safety, they should be core practise no matter what you're doing, regardless of whether formal approval is required.

All right, team. That brings us on to our third myth or common belief that we wanted to discuss today. And that is that evaluations are impartial or not political.

So this is an interesting one. Will, what are your views? Well, I reckon all evaluations are political from top to bottom, beginning to end, whether it's a focus of the KQs, the key evaluation questions, selection of the data tools you're using, prioritisation of certain stakeholder groups for input over others. Also, its impacts are political.

So what you're evaluating, decisions you make, or contribute towards, it's going to impact on people's livelihoods and wellbeing. It's going to impact on government budgets and agendas. And touching on that, our clients here, they're predominantly governments.

Governments by their very nature, they're political. They have an agenda they've been elected on and that ministers and the public servants are subject to. And that may favour certain approaches, certain outcomes even, or certain engagement approaches.

So I think claiming impartiality is risky because it prevents examination of that, the reality of that, and the inevitable biases that can come along. So for you then, what does that actually look like in practise when you're doing an evaluation? An example could be a client requesting an emphasis on efficiency metrics. So I suppose privileging cost effectiveness over more abstract qualitative aspects of equity, access, or quality of experience.

So I think you can definitely offset that to a degree by emphasising equity and access in your report. But then it's a matter then of how that will be favoured and weighted in the report and how it's then later used by that client. Yeah.

I've definitely seen examples where evaluations are commissioned by people that already have a stake in the outcome or already have a predetermined view on what the evaluation is going to say. And I've seen cases where the client might want certain findings downplayed, as you say, or others emphasised, depending on what's going to be politically palatable at the time. Definitely.

And I think we also need to be aware that even as evaluators ourselves, as people, we bring biases. Confirmation bias, availability bias, authority bias. I mean, our own political biases.

And these biases are features of human cognition. They're not individual flaws. Okay.

But hold up. I have to disagree a little bit. So I think when people are saying that evaluations are political, I reckon there's maybe a little bit of conflation with the context in which evaluation happens.

With the evaluation itself. So I heard what you're saying. Like, absolutely, programmes exist within a political context.

But I don't think that necessarily makes the evaluation itself political. So I would say that as evaluators, like, isn't that our job? That we want to, we need to apply very systematic, we have transparent methods that are deliberately intended to take the politics out and minimise bias to the extent we can. And we've got a whole canon of tools for that.

We've got standardised methodologies. We've got validated tools. We've got statistical techniques, peer-reviewed frameworks.

And all of these have been precisely designed for that purpose, to remove the politics. I mean, that's true. But at the same time, those methods, they weren't designed in a vacuum.

They themselves were informed by purely beliefs and values of those who designed them. So for example, evaluators who are Aboriginal or Torres Strait Islander, they would probably argue that these standardised methods favour non-First Nation ways of thinking. Okay, yep, so I agree that certainly specific tools have absolutely originated through a Western way of thinking.

But I would argue that the underlying principles, so if we think about evaluation, what's it about? It's about systematic observation. It's about applying logical reasoning. It's about following the evidence to conclusions.

And that's not uniquely Western. I mean, almost every culture has ways that understand and determine what counts as credible evidence and what sound judgement means. So I have to say, I disagree that we're coming from a particular viewpoint and that there is room for different frameworks and ways of thinking within evaluation that doesn't necessarily make it biassed or political.

Yeah, I think that is the critical point, Marty, that impartiality doesn't mean pretending that politics doesn't exist, but it comes from being transparent about our methods, our evidence, those limitations as well. And a credible evaluator recognises all of those dynamics that are at play and can still hold the line on the evidence. Yeah, okay.

Not disagreeing, but if we accept that evaluation is inherently political, don't you think we're kind of cutting our own lunch and in risk we undermine the whole field? So, I mean, if we're kind of saying, oh, look, all evaluation is political, we can't trust the findings, we're kind of offering a door to dismiss our evaluation findings as just political opinion. And I think that kind of framing risks giving people who don't like the findings an easy excuse to just ignore that evidence if they don't like it. Yeah, and I definitely agree with that, Marnie.

I think like evaluations happen in political context, yes, but it's really important to have those rigorous tools, transparency and reflexivity as well in your evaluation approach so that you can produce trustworthy, useful findings, whether they, what they kind of expected to hear or not. And it's also about weighing, a good evaluator has the ability to weigh up the evidence base as well and understand, you know, where a particular finding has come from one strong voice versus another one that's come from multiple different sources. And very few evaluations are a team of one, right? So inherently, if we've put together a team which has the right skills, viewpoints and understandings and worldviews, we kind of to some degree can help to, if not mitigate bias, at least put the different biases on the table.

Yeah, definitely. So as we're kind of touching on now, it's about acknowledging that political context. So it doesn't mean you're abandoning rigour.

It just means you're conscious of the pressures, the biases, the choices that are being made on every step of the way and the impact that's having. So with all that being said, I reckon it's fair to say that myth or the belief that evaluations are completely impartial has been busted. So evaluations, they happen in a real world context with all the political pressures and evaluators are human as well.

We are human. So biases are inevitable. Yeah.

But I think, you know, what we're saying is that doesn't mean you can abandon rigour or transparency because that is the goal with a robust evaluation. So really important to acknowledge your own biases, manage those and deliver findings that people can trust and have confidence in. Definitely.

And so we put up on the screen again, some steps we can take to make sure that our value agents remain robust and credible, even in these complex political contexts. So this includes acknowledging your positionality and potential blind spots you bring. So explicitly stating your assumptions, experiences and limitations throughout the process from design to conclusion.

I think we should also be seeking out sending perspectives so as to deliberately include those voices that challenge those dominant narratives or power structures and sort of pushing for those to be included when they're perhaps not included. Yeah. And I would take that a step further and say that it's really important that we make our evaluative reasoning explicit.

So as we've said, we have a range of choices we can make in terms of the tools or frameworks that we're using, which approaches we're bringing to the table. And so the key really is to be absolutely clear and document what those methodological choices were, why we prioritised or weighted a particular type of evidence, what decisions we made in terms of framing our evaluation, what we reported. And also importantly, acknowledging that there are potentially alternative interpretations or explanations for the findings that we're seeing.

What do you guys think about challenge mechanisms? Well, there's a few techniques you can use, like one that we use often in Allan and Clark is like sense making workshops where, you know, at a particular stage in your evaluation, you're after some data collection and some analysis but not full analysis, bringing together a range of stakeholders or people around the table to share what you've learned so far and really test those kind of early findings that seem to be coming through and asking those challenge questions of, you know, have we exhausted the avenues here that we need to? Are we missing something? That's something, one of the ways we do that. Yeah, and I'd say I've used a similar approach, but I'm using a panel of critical friends. So people that present different viewpoints and might be brought in different parts of the evaluation to say, hey, look, hey, this is how we interpret these findings.

Is there any alternative explanations or is there anything that doesn't ring true based on your experience? And then, of course, you've got peer review as a way to actively test findings and identify where you might not say that you're bringing in bias, but a peer review might help to bring that to the surface. Yeah. All right.

So thanks, guys. That was a pretty robust discussion on a few of those points. But I think that encapsulates our views on those three common evaluation myths that we have busted today.

Before we open up for questions, though, let's try and kind of pull together what some of those key themes that we've covered today are. So, Will, out of the discussions that we've had, which evaluation myth do you say is the most important one that we have to bust? For me, it was the idea that you need to be credentialed to be an evaluator. I think there's a risk of excluding those who have, for whatever reason, not undertaken higher education or not undertaken higher education in evaluation.

So I think rigorous standards are definitely important, but there's value in having a range of voices, including those who are not, I suppose, for want of a better word, certified but may have equally valid knowledge from other sources. Yeah. OK, though.

I mean, I don't want to come back and relitigate what we've already talked about. And I agree that that's a myth that we definitely have to bust. But, you know, what do you think the place of evaluation qualifications are? Like in New Zealand, we don't currently have a postgraduate level qualification evaluation.

Does that put us at risk or are we OK to rely on mentoring and other ways to keep that rigour? It's a good question. I suppose those certifications, they give legitimacy and it's important. I was touching earlier, those standards, like in any profession, you're a scientist, let's say, there are those standards you have to adhere to, to make sure the work is rigorous and it's not doing harm.

And so, yes, there is that risk, I think, to a degree. If you're just relying on sort of non-structured mentor support, sort of like pass the parcel of that knowledge, there is that risk that you could sort of transform that change, not be quite applied quite as correctly. So that risk is very real, I think.

I think a couple of the myths that we've talked about today for me are entangled because I am a qualified evaluator and I'm wondering if that maybe brings a little bit of my bias to the table because I just reflect on my own experience and trying to do an evaluation without a qualification and then becoming an evaluator through doing a postgrad diploma and what I know now and what I can do, I honestly don't know if I could have got to that same point without going through that qualification. Maybe I could have learned on the job, but I think I'm going to keep flying the flag and while I agree with Buster's myth, I'm going to keep flying the flag for qualifications. Emma, what about you? What was your key takeaway from today? I think likewise, as you mentioned, Marty, about a common thread, I think a common thread for me throughout all of these topics today has been that, you know, the strength of an evaluation really comes down to the quality of its methodology and that will enable you to kind of address all of those risks that we've talked about.

On the credential question, I'm just going to say, so I don't have a formal university credential in evaluation. I did undertake like research methods and some evaluation content in my undergraduate and postgrad degree, but not as a formal evaluator, but have done various certified professional development courses as well over the years throughout my career. So that's probably somewhere in between.

Yes. But I think, you know, coming back to understanding the science, understanding the political context in which you are operating, having that clear design that's defensible and robust where you're triangulating data from multiple sources and weighing up the strength of those findings is what gives evaluation credibility. So what are we trying to do with evaluation then? I mean, can we get to the point where we say that if we do all those things and we apply a robust methodology, we have people that have both lived experience and qualifications.

Going back to that last myth, to what extent do we actually say that we can put forward as biassed and apolitical evaluation as possible, or based on the experience as evaluators, how much do you think the clients or the politicians or even the people that are participant kind of put weight to bring in that bias that puts evaluation back in the political space? It's a tricky question, but, you know, I think what we see is that, you know, coming back to the reason why a lot of government departments commission evaluations in the first place is because they need to have a solid evidence base to then make recommendations to their ministers around funding decisions, you know, whether the programme might be quite popular or not. If it doesn't have a clear evidence base to justify that investment, you know, then it warrants questions about its, you know, value and viability in the longer term and really only an evaluation can deliver that kind of evidence. So if we're looking at what I think one of the key takeaways for me today is, I'm going to come back to ethics.

And I know I said I'm an ethics committee sceptic. I think I still remain a bit in that space. But I think what you said Emma really was the key takeaway for me is that it's not just about that knee jerk, we better be safe than sorry and we'll put in an ethics committee application for absolutely everything.

It's about having that step and really going through systematically and thinking what is the risk associated with this evaluation? What are some potential harms? And then therefore what does our ethical practise look like? Maybe that is formal ethics committee approval in some instances, maybe not in others. One thing I am thinking about though is we're evaluation consultants and we typically do kind of consultancy evaluation work for government clients. What about if you're in a different context and you're looking to say get in a paper published in an academic journal? What's the ethics approach there do you think? I think ethics committee approval definitely has a bigger role to play if you are looking to publish in an academic journal or present at a conference.

So that is definitely a consideration to factor in if there is a likelihood of wanting to publish at some point down the track, publish your findings because publishers will look for that ethics approval. But it's still not always necessarily required particularly if you might be relying on secondary data for example or if it's been a government kind of departmental programme improvement activity. Horses for courses then I guess is what we're saying.

So thanks, I think it's a good opportunity now to look at some of the questions that are coming through. So let's see what our audience has got. Okay, so we've got a maybe pretty fundamental question coming in from Paul who's asked is there a difference between evaluation and research? So do we think that might be a myth that evaluation and research are different and are we kidding ourselves when we think we have a specialised discipline in evaluation? I think there's definitely a difference between evaluation and research.

I think you know evaluation is about you know not always retrospectively, sometimes it's alongside programme delivery but it's usually about kind of taking a step away and then looking across what has been delivered and the change that can be attributed to that rather than just asking research questions and exploring those through your research methods. Yeah, thanks and I reckon the biggest difference is the judgement piece. So methods are similar and our skills are definitely cross transferable but in my experience anyway research typically does not involve making an explicit judgement about merit, worth, value.

So I reckon our evaluation and research is the same thing. I would say that myth is busted. What about a myth that AI can do evaluation? So maybe we're going to draw ourselves out of a job as a belief that AI can do perhaps do cost benefit analysis and you know just as good as any living person evaluator.

What do you reckon? Is that true? Is it a myth? I reckon it can in some instances or maybe increasingly even more instances support evaluation. I don't think you can do evaluation. I think humans do the evaluation.

I think AI offers like in the case of a cost benefit analysis cost effectiveness it allows you to do that rapid assessment of like bucket loads of data but you're going to need to you know design and pick the method and the approach and the tool for analysing that data. You'll need to set up the context and whatever else in the AI and more importantly once it's outputted you'll need to then test that, make sure it makes sense, make sure it hasn't sort of hallucinated anything. Yeah.

And then actually apply that and write that. I mean there's been examples recently of when that hasn't occurred with some evaluation practises and I think there's really that risk that you don't have that human insight and oversight throughout the whole process and quite deeply involved still. There's quite a lot of substantial risks.

Yeah. So in my view I think it is true that AI can be a good evaluation support person, can be a good analyst but I would be very sceptical that AI can run the full evaluation from start to finish, develop, understand the nuance enough to really develop robust KEQs to understand what questions to ask in evaluation data collection tools and to really interpret that analysis. In my view that's definitely a myth.

And you generally need to talk to people, right? I reckon most evaluations will require some degree of consultation, human consultation and assessment of those findings as you say Will. Yeah. I reckon we'll put a pin on that and say that myth is busted for now but let's check in in one or two years' time and see how AI has evolved.

Cool. Okay. So I've got a question from Frances.

She asks whether we have any examples of practical alternatives to lived experience consultation that still centre stakeholder voices. So can we do lived experience consultation? Yeah, that's an alternative. So we're not going to communities necessarily that might be overburdened.

Is there a way that we can do that without actually going and talking to people? What are some of the ways you've done that, Marnie, where you've needed to collect voices but don't have the time or resources to go wide? Yeah, so for me it hasn't actually been so much about the time and resources. It's been about the overburden. So we've recently done a piece of evaluation work with a participant group or a stakeholder group that were very much over-evaluated.

So there had been several pieces of evaluation research that took different angles but were around a similar topic and involved the same group of stakeholders. So we needed to do something different there. So rather than going and doing extensive lived experience consultation, first of all, we took what we could know and learn from literature, other documents and other consultations that had been happening, and then we did really targeted consultation which we called kind of high impact, low time Berlin consultation, where we did go and talk to people but it was only a specific number of people on a short number of issues and the rest was more about we've heard this and validating it rather than gathering new information.

Yeah. There might also be benefit in talking to, I know, some communities, sort of community groups or big bodies. So in some instances it might be okay or appropriate to approach them.

They'll have people talking to them and they can then feed that back to you because they'll be connected with the communities that they're a part of. So I think that's potentially another way to do it. Yeah.

Yeah. Or like a survey to, you know, obviously take the point around, you know, there are always some groups that are also over surveyed, but it can be a really effective method to get much wider reach as long as you are super targeted about what you want to ask and why and really minimise, yeah. Yeah.

Yeah. I would say that's a double level of myth busting. So do you have to do, you always have to do extensive lived experience consultation, myth, but you always need to do some stakeholder consultation lived experience is true.

So, yeah, I don't think we can sort of chuck it out the window and say we'll get everything from other sources. Agree. Okay.

So that's our time up today. Thank you very much to our audience for your participation and questions. We've really enjoyed discussing and having a bit of a debate about what we think are evaluation myths.

Thank you very much for joining us. Thanks, everyone. Have a good afternoon.

See you.