Using the EIF Guidebook to help make decisions: Thinking about fit

The EIF Guidebook is a searchable database of information about early intervention programmes, designed to support policymakers, commissioners, service managers and others to make informed decisions about selecting and investing in programmes that have been evaluated and shown to improve outcomes for children and young people. It is a rich source of information about evidence, child outcomes, cost, implementation, previous evaluations, and more.

The Guidebook provides an evidence rating for each programme, at the top of the page. This rating provides an indication of the quality of evidence for a programme’s effectiveness, based on past evaluations and trials.

But this evidence rating is not the end of the story – it’s only the beginning. A high evidence rating is not a guarantee or recommendation. To use the Guidebook effectively in making vital decisions about services and support, it’s essential to dig deeper, into the particular characteristics of a programme, and how it’s been implemented or evaluated in the past.

“The evidence rating is really important – it tells us which programmes represent a ‘best bet’ about what might work again in the future. And that’s obviously critical information for commissioners, service managers and others.

“But there’s a real risk in relying on this number by itself as a way of making decisions. The evidence tells us whether a programme has worked in the past. But it can’t tell us whether it will work in a new situation, or in all situations. “Of course, evidence matters. But it should be considered alongside a whole range of other factors that are likely to influence whether a programme is right for you and your area.”

Making high-quality, evidence-informed decisions about commissioning early intervention means using professional judgment and local knowledge to ensure that there is a good fit between the circumstances in which a programme has been successfully evaluated in the past, and the particular characteristics of your local area.

“In attempting to make best use of the evidence, we should think about comparability – how comparable, how similar are the conditions in a past trial and the conditions in your area. The more comparable they are, the more likely you are to see similar results from that programme. On the other hand, if you don’t think about comparability when you come to select or implement a programme, you might not achieve the positive outcomes you’re looking for.”

Finding a good fit

The Guidebook provides a wealth of information to help you judge fit between a programme and your area.

We think there are two important ways of looking at fit – population fit, and environment fit. Population fit is all about the needs and characteristics of the people living in your area, who will use or access the early intervention you’re planning. Environment fit is all about the programme itself, how it’s implemented and delivered, and the features of the area it will be introduced to.

Finding a good fit: Population fit

“To understand whether the evidence rating for a programme is a good guide to how it might perform in your area, it’s essential to understand as much as you can about your local population, and how different or similar they are, as a group, to the people who were involved in that programme’s past trials. Differences in some of these characteristics – including age, sex, ethnicity, language, socioeconomic status, and level of risk – may influence the effectiveness of a programme in a specific area, with a specific group of participants. For example, selecting a programme that’s been effective in the past for boys from disadvantaged backgrounds might be less successful – or not successful at all – if it’s provided to girls from middle-class families. “The evidence rating, by itself, doesn’t give you this level of information – you have to dig a little deeper.”

So, how does the EIF Guidebook help you to assess population fit? There are three main ways.

First, you can use the filters. There are over 100 programmes on the Guidebook, and the filters can help you reduce this to a more manageable number. The Guidebook allows you to filter according to the age groups for which a programme has been shown to be effective, from pre-birth to the teenage years. You can also filter according to level of risk – in its past evaluations, was the programme provided universally, for everyone, or in a way that was targeted at a specific group of people considered to be at higher risk, or already experiencing problems? Using these filters, you can begin to factor in population fit even before you start to read about individual programmes.

“There are always wrinkles in the information we can provide. Sometimes there are different versions of a programme, designed for different groups. We can’t assume that something that was effective for one group or population will work for another. Therefore, when we publish information about these variations, we include the group in the name, and add a note to accompany the evidence rating.”

A second way to incorporate population fit is by digging deeper into the information the Guidebook provides about each programme. The key characteristics page provides details about the programme based on its highest-quality past evaluations, including which age groups saw positive outcomes from the programme, and whether the programme was provided in a universal or targeted way.

The third way is to access the evidence page. Here, you can review the detail of each of the studies that EIF used in their assessment of a programme’s evidence, and learn much more about the groups who participated in these past evaluations. This might include information on age, sex, language, socioeconomic status, and more.

“This stuff isn’t easy. It takes time and resources to build up data about the needs of people in your area, and to incorporate that into your decision-making. And it takes work to understand more about when and where a programme has been effective in the past. But these are both crucial parts of the commissioning puzzle.

Finding a good fit: Environment fit

“The second type of fit that really matters here is environment fit. This is about the fit between the context or situation in which a programme was evaluated previously, and the context or situation in your area. Overall, this is about implementation: how the programme was delivered in the past, in other places, and how it might be implemented here and now. If there are big changes in how a programme is going to be implemented, then that might compromise its potential to be effective in your case. For example, a programme that was shown to work when delivered to groups of children in an afterschool club might not be so effective if it’s delivered to individual children via home visits.

“Environment fit is not so much about the people who going to be receiving the support. In part, it is simply about whether a programme has been implemented in the UK before. It’s not uncommon to introduce an intervention that’s been successful overseas, but you should be aware that’s you’re trying something that’s new in the UK. It’s also about the setting – the physical place where a programme is delivered. It’s about how a programme has been delivered in those past trials showing positive impact – things like group size, duration and frequency. And it’s about whether there are any specific implementation requirements that you would need to meet, such as money for materials or staff with certain skills or qualifications. Lastly, it’s about considering added value – how is this new programme likely to compare to the services and interventions you’ve already got in place.”

How does the Guidebook help you to assess environment fit? Again, there are three main ways.

First, you can use the filters to narrow down the list of programmes you might focus on. The Guidebook allows you to look only at programmes that have been implemented in the UK. You can also filter according to settings, to focus on the venues that you are considering or might already have available. Or according to the delivery models, which gives you a range of common options for how a programme operates. Using these filters, you can be thinking about environment fit in the process of narrowing down a list of individual programmes to consider.

“Sometimes there are different versions of the same programme, delivered in different ways. When this is the case, we include the version in the name. For example, the Nuffield Early Language Intervention has been delivered in several different versions, that are delivered in different settings and over different time periods. The guidebook clearly labels the programmes and draws attention to the different implementation features of each.”

A second way to consider environment fit is by digging deeper into the information the Guidebook provides about each programme. There are two important sections here. ‘Key programme characteristics’ provides a summary of how and where the programme has been delivered when shown to be effective in the best available studies, which countries it has been implemented in, and specifically whether it has been implemented and evaluated in the UK.

Then, ‘About the programme’ provides a wealth of information submitted by the developer of the programme. It focuses on delivery and implementation, including important details on training and qualifications, supervision, maintaining fidelity, and licensing. The third way is to look at the evidence, for the details of the trials and evaluations that have shown positive impact in the past.

“By looking at the details of these studies, you can learn what a programme has been compared to in past trials, and consider how your current situation matches up with that comparison. This can help you to reach a view on how likely a programme is to add value, over and above the services that are already available. For example, a programme might have demonstrated, in evidence from the US, that it is effective compared to the support available as standard. But what’s available as standard in the US is likely to be different from what’s standard in the UK, where existing support is often more extensive. So again, it’s important to look at how well the evidence fits your situation: just because a programme was found to add value, compared to the status quo, in one study, doesn’t mean it will have the same affect in your area.”

“The Guidebook provides a whole raft of information designed to help you decide whether a particular early intervention programme might be right for your area. It’s important to tap into other sources of information as well: the people who developed the programme, peers from other areas who may have experience with it, the people who would be running the programme, and even those who might be accessing it for support.

“It’s essential to consider a programme’s fit before making any commissioning decisions. If you can identify a programme that is a good fit with your population and environment, then you can be that bit more confident of getting positive results. And if you want to commission a programme that doesn’t have good fit, then it’s important to pilot the programme on a small scale first, and monitor it closely.”