Evaluating

Evaluating Image readind evaluating

Much of the six-step approach to programme evaluation developed by HIS is transferrable to learning system evaluation:

Step 1: Planning

How to decide the type and timing of evaluation, who to involve and what will be required

Step 2: Framing

How to clarify what is being evaluated about a programme (using theory of change) and how the evaluation should be structured

Step 3: Focusing

How priority questions are set that the evaluation will be designed to answer

Step 4: Designing

How the methods of collecting and analysing information are developed

Step 5: Collecting and analysing

How information is collected and interpreted

Step 6: Synthesising and reporting

How the evaluation questions are answered and the findings shared

Source: A guide to programme evaluation in quality improvement, Evidence and Evaluation for Improvement Team (EEvIT), Healthcare Improvement Scotland

Collect evidence of impact

Before implementing an idea, we should build evidence of efficacy of the proposed intervention including testing within different contexts.

The added value of a learning system is the ability to share the process which led to a change. By testing a range of approaches in different contexts, the learning system can show factors for success.

Consider how you can evaluate the learning system in a credible way. Balance this with the logistics of managing the evaluation. Focus on what is meaningful and realistic.
Promote self-reflection and flexibility in practice
Reflective questions help us learn and can tell us a lot about progress. Reflection can be for individuals and teams.

Consider the following questions:
- What has happened?
- How are you feeling?
- What are you noticing that is working well?
- What are you noticing that is working less well?
- How should this be different?
- Share findings and experience
Openness and willingness to share progress, even failures, are essential to learning systems. The sharing of processes and changes often results in more learning than a focus on impact.

A functional learning system requires a community. People must be comfortable sharing without fear of reproach. Psychological safety is important for this. Here are three tangible concepts which make a difference to psychological safety:
- Frame the work and add meaning
- Leaders should model fallibility and invite input
- Embrace messengers
Source: Three Ways to Create Psychological Safety in Health Care, ihi.org
Assess achievement on aim and effect on inequalities

Think about "opportunity cost" of an innovation. For instance, is one approach preferable to alternative approaches when we weigh up all the costs and benefits? It is easy to make judgements based on a few narrow measures. Considering a variety of lenses reduces the risk of missed opportunities

Assess each approach for feasibility, cost-effectiveness, equity, cultural appropriateness and community preferences.

Quality Management System Portfolio