Much of the six-step approach to programme evaluation developed by HIS is transferrable to learning system evaluation:
Step 1: Planning
How to decide the type and timing of evaluation, who to involve and what will be required
Step 2: Framing
How to clarify what is being evaluated about a programme (using theory of change) and how the evaluation should be structured
Step 3: Focusing
How priority questions are set that the evaluation will be designed to answer
Step 4: Designing
How the methods of collecting and analysing information are developed
Step 5: Collecting and analysing
How information is collected and interpreted
Step 6: Synthesising and reporting
How the evaluation questions are answered and the findings shared
Source: A guide to programme evaluation in quality improvement, Evidence and Evaluation for Improvement Team (EEvIT), Healthcare Improvement Scotland
Collect evidence of impact
Before implementing an idea, we should build evidence of efficacy of the proposed intervention including testing within different contexts.
The added value of a learning system is the ability to share the process which led to change. By testing a range of approaches in different contexts, the learning system can show factors for success.
Consider how you can evaluate the learning system in a credible way. Balance this with the logistics of managing the evaluation. Focus on what is meaningful and realistic.
Promote self-reflection and flexibility in practice
Reflective questions help us learn and can tell us a lot about progress. Reflection can be for individuals and teams.
Consider the following questions:
- what has happened?
- how are you feeling?
- what are you noticing that is working well?
- what are you noticing that is working less well?
- how should this be different?; and
- share findings and experience
Openness and willingness to share progress, even failures are essential to learning systems. The sharing of processes and changes often results in more learning than a focus on impact.
A functional learning system requires a community. People must be comfortable sharing without fear of reproach. Psychological safety is important for this. Here are 3 tangible concepts which make a different to psychological safety:
- frame the work and add meaning
- leaders should model fallibility and invite input; and
- embrace messengers
Source: Three Ways to Create Psychological Safety in Health Care, ihi.org
Assess achievement on aim and effect on inequalities
Think about "opportunity cost" of an innovation. For instance, is one approach preferable to alternative approaches when we weigh up all the costs and benefits? It is easy to make judgements based on a few narrow measures. Considering a variety of lenses reduces the risk of missed opportunities.
Assess each approach for feasibility, cost-effectiveness, equity, cultural appropriateness and community preferences.