Evaluating personal outcomes focused ways of working
Here we bring together learning from evaluation of personal outcome focused interventions to provide some practical guidance for those planning, commissioning and delivering personal outcomes focused approaches and seeking to monitor and evaluate this work. Links to relevant resources and reports are presented throughout.
What do we mean by a personal outcomes approach?
There is a growing realisation across health and social care that if we are to be effective we need to work with people, tailoring support to their circumstances, needs and hopes for the future. This requires health and social care organisations, planners and commissioners to shift from being ‘service led’ to ‘outcome focused’. Scotland has pioneered a range of approaches that focus organisations on what matters to the person and why through:
- Engagement with the person in one or more ‘good conversations’ through which the practitioner supports the person to identify what matters to them and co-produce a plan for support, building on their assets and strengths
- Recording of the outcomes important to the person, the plan for achieving these, including the actions the person will take, and progress towards these
- Use of that data and information to plan and develop support for the person and to gain understanding at an organisational level of the systems of services and supports that need to be in place to meet needs and improve outcomes for people.
Read more about personal outcomes approaches in practice in this example from the Midlothian health and wellbeing practitioner service.
Evaluating personal outcomes focused approaches
Internationally, there has been relatively little work to evaluate the effectiveness of personal outcomes focused approaches (see Miller, 2017 for a review). Qualitative research has shown that these ways of working do deliver benefits for people, however there has been little progress in developing evaluation approaches that are effective for these kinds of interventions.
In the drop down below, three reasons are listed for why little progress in developing effective evaluation approaches has been made.
- Why little progress has been made in developing evaluation approaches
- Peoples’ personal outcomes are hard to measure. There are two issues here, firstly the kinds of things people are working towards are often not readily categorised or standardised, for example being able to take your grandchildren to the park. Secondly it is very hard to get a reliable measure of progress towards a personal outcome as peoples’ aspirations and insight into their situation appropriately change over time. Indeed, practitioners often report that it is only once they have been working with someone for a while that the person recognises how good their life good be and how far they need to go. Whilst Patient Reported Outcome Measures such as the WEMWEB or IROC can be used to capture change in more general wellbeing or recovery over time, they do not capture progress towards specific personal outcomes.
- Personal outcomes focussed interventions are hard to measure. Good personal outcomes focussed practice is tailored to the person and therefore appropriately different every time. For one person a single session might be just the right intervention, but for another this would be woefully lacking.
- There are many different factors that influence whether or not someone makes progress towards a personal outcome over and above the support they receive. It is not possible for services to ‘deliver’ outcomes for people. Instead they can only contribute to change, good or bad.
For a further discussion and a summary of evidence from a recent action research project visit the Meaningful and Measurable page within the personal outcomes collaboration website.
Focusing on contribution as a tool to evaluate personal outcomes approaches
This concept of contribution forms the basis of an approach to evaluation that is widely promoted in Scotland called Contribution Analysis (CA). CA is a theory-based approach to evaluation that involves developing a ‘theory of change’ for the project and using this to structure the data collection and analysis. A theory of change simply refers to the way the project team understand how what they do makes a difference. How the service operationalises this theory of change can then be captured diagrammatically in what can be called a ‘results chain’, ‘logic model’ or ‘outcome map’ The conceptual development and application of CA has been influenced by individuals such as John Mayne and Steve Montague who have described the process as ‘results-based management’ involving the gathering of a range of forms of evidence in order to tell the story about programmes have contributed to outcomes in the short-term, medium-term and long-term.
Over the past four years Dr Sarah Morton and Dr Ailsa Cook of Outcome Focus have been adapting and refining CA to develop an approach that can be used by organisations to learn, improve and tell the story of the difference they make. At the heart of this approach is a seven step process (pictured right).
As organisation, you can produce an outcome map that shows what needs to be in place for your approach to improve outcomes for people. A starting point would be for the outcome map to show what the experience is like from the perspective of the person.
You can audit the information you already gather to see how well you evidence that people were having the experience expressed in the outcome map. You can use or adapt existing information from case notes, activity logs and patient story proformas. For example adding in a field into case notes so that practitioners could reflect specifically on how people felt about the process.
Through focusing on contribution and bringing together qualitative and quantitative data, it is possible for organisations using personal outcomes approaches to robustly and meaningfully evaluate their work. Ensuring that practitioners are closely involved in this process and that there is time for analysis and learning and important strategies to maximise the benefit of this work.
Miller, E (2017) Review of evidence about personal outcomes relevant to the Carers Act (2016). Paper produced for the Scottish Government Carers Policy Team.
Mayne, J. Contribution analysis: An approach to exploring cause and effect, Institutional Learning and Change Initiative Brief 16, http://www.cgiar-ilac.org/files/publications/briefs/ILAC_Brief16_Contribution_Analysis.pdf
Montague S. Practical (Progress) Measurement and (Impact) Evaluation for Initiatives in Complex Environments. Performance Management Network: Performance Management Network; 2011.
The ISM behaviour change tool is an evidence based framework that can be used to understand the system in which any initiative is being undertaken in terms of opportunities and threats occurring at an individual, social and material level. More information from http://www.gov.scot/Publications/2013/06/8511