‘We know we’ve been making an impact, but it’s hard to capture’: How an outcomes harvest can help your evaluation
4-minute read
Based on our experience working with Scottish Recovery Network on 2-year systems change evaluation, we’ve put together some of our learnings on one of the particular approaches we took during the evaluation. Like many organisations, they knew they were making positive changes but didn’t necessarily have a systematic approach to evidencing it.
As an organisation with national scope, knowing how far their materials, reach and impact spanned as well as collecting this impact can be tricky. Traditional evaluation methods often fall short in capturing the full spectrum of outcomes, especially in complex and dynamic environments. This is why we decided to use an Outcome Harvesting approach as it shines when flexibility and complexity are involved.
The name ‘outcomes harvest’ can put some people off but putting the name to one side, the approach is helpful as it focuses on actual changes rather than predefined indicators, it helps to create a comprehensive and nuanced understanding of a charity's impact.
So, what is outcome harvesting?
Outcome Harvesting is an evaluation method designed to identify, verify, and make sense of the changes brought about by an activity or intervention. Unlike traditional evaluation methods that compare results against expected outcomes, outcome harvesting instead starts by collecting evidence of what has happened and then works backward to determine whether and how the organization’s activities contributed to those changes.
In relation to the Scottish Recovery Network, they have been delivering, working with and alongside communities and organisations in the mental health sector for almost 20 years. They know that the work they do has created positive change, however capturing what has changed, how and why they had an impact was somewhat harder to gather. Particularly, a major goal of the organisation is that their work extends beyond the people they work with. This includes the ways in which they work, the approaches that they use, and the tools they create are further adopted beyond the lifecycle of any given project.
An outcomes harvest approach meant that we could look back on their work over the years and gather the evidence and stories of impact that had already been achieved, and document how Scottish Recovery Network had contributed to those outcomes.
“A space has been created where people who wouldn’t have come together before, are learning from each other”
What’s involved in an outcomes harvest?
With Scottish Recovery Network, we carried out over 50 interviews, focus groups and workshops with past and present partners, collaborators, funders, and stakeholders with varying levels of previous engagement with the organisation. We asked questions such as: ‘Has, and if so how, has Scottish Recovery Network influenced what you do’ to delve into the changes that Scottish Recovery Network have contributed to. We also asked questions such as ‘What has changed as a result of your work/connection to Scottish Recovery Network’ and ‘What could Scottish Recovery Network do more of’ to uncover what changes stakeholders would like to see in the future.
What are the benefits of an outcomes harvest?
We chose conducting an outcomes harvest for this evaluation due to the alignment and benefits the approach offers. Especially, outcome harvests allow for flexibility and adaptability. Given that this approach is at the heart of all of Scottish Recovery Network’s work it made sense as it allowed for real-tine insights to be gathered, which meant that the organisation had timely insights to inform their ongoing activities and strategies rather than having to wait until the end of an evaluation cycle to gain useful information. Also, the fact that outcomes harvests look backwards rather than trying to pre-determine outcomes that might occur meant that it applied to all of the different types of work and activities that they carry out.
This method was precisely helpful given the complexity of the work that Scottish Recovery Network does. As an organisation focused on changing mental health systems, an outcome harvest allows for the complexity and non-linear nature of this work. It provided a comprehensive view of their work, allowed us to document a wide range of outcomes, and uncover the broader impacts of their work including the unintended or indirect outcomes that can often get missed by traditional methods, providing a fuller picture of impact. For Scottish Recovery Network, one of these was in just how far their peer training materials and consequently peer values had travelled across the globe. In only a light scan we found Scottish Recovery Network materials cited in 16 countries on 4 continents (we are confident materials are used in more places).
“You cannot underestimate the impact or role of SRN’s behind-the-scenes role for making a project successful ”
Outcome harvests are noted for the way in which they empower stakeholders, it includes them in the approach, involving them in identifying the outcomes and analysing their impact. This helps to foster both a sense of ownership and engagement. Again, this aligned well with Scottish Recovery Network’s approach in engaging collaboratively with their stakeholders, meaning that our approach as evaluators matched the experience they already know and have with Scottish Recovery Network.
Lastly, this approach provides enough detail and context to guide and shape strategic decision making, refine activities and approaches and trace outcomes back to specific activities in order to strengthen and maximise limited resources.
What did we learn?
As a learning partner to all our clients, we ourselves are always looking to learn along the way too. As such here are a few lessons we learnt about using this method:
All good evaluators will tell you to spend a good amount of time developing clear questions that guide what it is that you’re looking for. What helped in this project was meeting with a few close stakeholders to the organisation to test and refine the questions to ensure that they were getting at what we were looking to uncover and that they were understood.
Matching our approach to how the organisation interacts externally with their partners and stakeholders. Whist we intentionally try to do this for all projects, it was particularly important that when using this method, to match the organisations approach and values to further increase engagement and trust rather than alienate participants in the evaluation process.
Although many of our connections to external stakeholders came through Scottish Recovery Network, we dedicated time to connecting with others named during interviews and focus groups who could add further insight, perspective and validate what we were hearing. This also enabled us to understand the breadth and scale of the impact of Scottish Recovery Network.
The validation of information gathered with Scottish Recovery Network staff was key to understanding how different activities and outcomes were perceived by internal and external staff. For example, some staff identified training as key to sharing knowledge and how to do the practicalities with external groups, however participants of the training identified that they saw it as a learning journey, and most importantly, it was a source of hope for them.
Embracing the learning journey
The outcome harvesting process has consolidated the positive impact the Scottish Recovery Networks’ values, methods and strategic influence are having on individuals and across the Scottish mental health system.
Outcome Harvesting is more than just an evaluation method; it is a learning journey that empowers organizations to uncover the real-world impacts of their work. By focusing on actual outcomes and involving stakeholders in the process, it provides rich insights that can drive strategic decisions and continuous improvement. By embracing this approach, organisations can gain a deeper, more nuanced understanding of how their efforts are making a difference and enhance their ability to create lasting change.
To read more on the specific learning that came from this approach, go to the Scottish Recovery Network’s blog.
At Habitus, we have helped develop and design successful evaluation and research projects for various charities, health organisations and businesses both in the UK and internationally. Find out how we can help your organisation do likewise and for other types of support we offer by clicking the button below.
We are accomplished in peer research, co-production, system change, action-based and anti-oppressive research and evaluation. We are experts in helping organisations to engage wider community participation in their projects so that their work is more inclusive and impactful. Through this approach we are dedicated to increasing lived experience leadership.