The Story of Developing a Realist Evaluation Framework

Flexible, Adaptive, Evolving, Responsive	We have seen the priorities of the LDP evaluation change during the lifetime of the previous evaluation contract and we expect this change and development to continue. Having one large contract with resources tied to a specific number of hours for specific individuals meant that our previous evaluation partner found it hard to flex with the needs of the LDP and plan resources accordingly.
Collaborative	Our previous evaluation partner saw the value of working in a consortium and advised that we consider the ‘mix’ of individuals involved in future evaluation work in order to get the right skill sets – and we realised that we needed a combination of evaluation expertise and on the ground capacity. They also highlighted the challenge they had in accessing the information needed to effectively evaluate the LDP which we hope a more collaborative approach between internal and external would overcome.
Knowledge Retaining & Sustainable	The LDP is a time-limited project and so there is a need to consider legacy and sustainability. In order to embed evaluation practices into Active Essex and ECC, we know we need to upskill members of the team and help them to make evaluation part of their work – this was not possible to achieve with the ‘contracting out’ approach that we took previously.
Context Informed	We have seen Sport England moving away from a focus on measurement and monitoring and moving more towards a culture of learning together using the changes and stories that we are observing locally. In order to make the learnings we share more transferable to other areas and contexts, we need to consider not just ‘what works’ but ‘how, why, in which circumstances and for whom’.
Not ‘Evaluation’	Experience has taught us that traditional evaluation terminology can hinder engaging the system in evaluation practices, particularly when working with ‘unusual suspects’.

The successful bid for an academic partner came from a consortium led by Hartpury University alongside Sheffield Hallam University and Durham University. Their bid set out an approach which is collaborative and utilises realist evaluation.

Realist evaluation is rooted in the philosophical orientation of realism where a key principle is that observing something is not enough, on its own, to establish the causal relationship between variables. Realist evaluation involves three key phases: First, establishing a programme theory, second, testing programme theory, and finally, refining programme theory to result in evidence-informed explanations about how social interventions (physical activity interventions in this case) work, for whom, under what circumstances, and why (Mason, et al. 2021).

Additional capacity was also added to the LDP Insight & Evaluation team through the recruitment of 2 Evaluation Researchers who started in March 2022. It is hoped that by using an embedded researcher approach, the evaluation team will be closer to the work of the LDP and will therefore be more able to ‘notice’ changes, have access to people and the information needed, and influence future decision making based on learning from evaluation. We would be keen to learn with/from other areas using embedded researchers in order to benefit from examples of good practice.

The Story of Developing a Realist Evaluation Framework

What has happened?

What's next?

Activity Finder