Taking a model to scale requires that we not only understand the efficacy of a given program but also look at the effectiveness of the intervention. Intervention studies can be placed on a continuum, with a progression from efficacy trials to effectiveness trials. We define Efficacy as the performance of an intervention under ideal and controlled circumstances, and Effectiveness as its performance under ‘real-world’ conditions.
Determining whether an intervention works is not simply a matter of seeing a change in behaviour. Almost all interventions can be considered to “work” if they have some positive outcome, but that does not mean that the intervention is a good one. The real question is, “Does it work and is it worth the price to use it?”
Determining the costs versus the benefits of outcomes requires an understanding of context as well as the defined parameters you have set for yourself when you are measuring cost/benefit – is it money? Time? Cost/benefit analysis will be different for different contexts and depends on the parameters you set.
The following five criteria are what we consider to evaluate the effectiveness of an intervention and to determine whether the benefits outweigh the costs:
1. Does it work? This requires going beyond the efficacy study to conduct an effectiveness study. For this we may design a Randomised Control Trial (RCT). We may have the experimental group receiving the intervention that is being tested and the other (the comparison group or control) receiving an alternative (conventional) treatment.
2. Will it have any negative long-term consequences? Even if the intervention works in the short term, there is need for some reflection on its long-term consequences for the teachers and learners in school.
3. How does it affect the student’s motivation to learn? If the intervention not only resolves the immediate issue with the student but also encourages him/her to engage in instruction and increases his/her motivation to learn, the intervention can be considered all around beneficial. But, if the intervention only resolves the immediate issue or leads the student to becoming less motivated to learn through other strategies, the cost of the intervention can outweigh the benefit (here we also need to look at how this impacts on other subjects).
4. Is it an effective use of the teachers’ and learners’ time? From the teacher’s perspective, “How much time can we be expected to spend on the one intervention or class when they have more than one subject and class to worry about?” here we would look at issues around system wide mainstreaming of an intervention.
5. Does it look backward or forward? Many interventions focus too much on what the student did and not enough on what she/he will do next.
If we follow this process, a given study will ideally be able to answer the following:
Why are you conducting the intervention – underlying assumption- about the learners, teacher, education system, education philosophy etc.
- What outcomes do you predict or wish to see from the intervention?
- How will you measure these outcomes?
- How will you ensure independence of results?
- What do you want to explore or know about your intervention and its effectiveness?
Reproduced with permission from Alice Ruhweza, Vice President of Programmes and Partnerships at Conservation International, Fellow of the Salzburg Global Seminar; 2017 Henry Arnhold Fellow; 2019 Aspen New Voices Fellow.
She is passionate about the intersection between Nature & Human Capital; Conservation & Development; whole of Systems thinking & the use of the cutting edge technology, science and data to guide and inform investments, planning, policy and decision making at all scales. Director/Board Member – the EverGreening Alliance; Member of UN Stats IAEG Task Team on SDG Interlinkages; Member of Steering Committee-Future Earth Water-Energy-Food Nexus Knowledge Action Network; Member of UNEP WCMC-CONNECT Advisory Group; Homeward Bound 2018 Participant; Relentless advocate for Educating Girls & Empowering Women.