EIF Guidebook: What’s in the Guidebook, what’s not, and why?

The EIF Guidebook is a searchable database of information about early intervention programmes, designed to support policymakers, commissioners, service managers and others to make informed decisions about selecting and investing in programmes that have been evaluated and shown to improve outcomes for children and young people. It is a rich source of information about evidence, child outcomes, cost, implementation, previous evaluations, and more.

Since it was launched in 2014, the EIF Guidebook has more than doubled in size, to include over 100 early intervention programmes.

“It’s great that we’ve been able to continue to expand the Guidebook year after year, to include more and more of the programmes that are actually being delivered around the country, or might be brought to the UK from other countries.

“But we know there are hundreds of early intervention programmes out there that aren’t included on the Guidebook. This raises some important questions – what does it mean for a programme to appear on the Guidebook in the first place? And perhaps just as important, what does it mean if a programme doesn’t appear?”

What does it mean for a programme to appear on the Guidebook?

The EIF Guidebook provides information about programmes that have at least preliminary evidence of improving outcomes for children, and which are already being used in the UK, or could feasibly be introduced here. It is not a conclusive list of programmes which have the highest quality evidence, have been proven to work, or are endorsed by EIF.

“The Guidebook contains a huge diversity of programmes. It is not a directory of programmes that have the very best evidence. Yes, there are big international programmes with multiple highly rigorous trials – but there are also smaller, local programmes with some early pilot testing.

“Being on the Guidebook is not, by itself, an indicator of quality or effectiveness – all it says is that a programme has at least some useful early evidence of positive impact. This is why the Guidebook provides more, deeper detail on how a programme has demonstrated effectiveness in the past, and how strong that evidence is.”

Being on the EIF Guidebook does not mean a programme is endorsed or recommended by EIF. However, there are clear and important differences between the programmes that are on the Guidebook, in terms of how well evidenced they are. The evidence rating indicates how much confidence you can have that a programme is effective, based on the number and quality of trials and evaluations conducted so far.

Programmes with a level 3 or level 4 rating are considered evidence-based. This means these programmes have a proven track-record of having worked in the past, based on at least one high-quality evaluation.

Programmes with a level 2 rating are considered to have preliminary evidence. These programmes have started to develop their evidence, but because of the way in which they have been evaluated, we cannot say for certain whether they have worked in the past.

“The evidence rating reflects how confident we can be that there is a causal relationship between implementing the programme, on the one hand, and achieving positive outcomes for children on the other. In other words, how confident can we be that there was an improvement in the lives of the children who received an intervention, and that the intervention caused the improvements.

“Crucially, the evidence rating is not a rating of the scale of impact – it doesn’t say how big the improvement was, or might be in future.”

“All programmes on the Guidebook have an evidence rating of at least 2 – they all have at least preliminary evidence of improving outcomes for children. That’s the bar a programme has to get over to get onto the Guidebook.

“So simply being included on the Guidebook – in and of itself – doesn't imply anything about the quality of the programme, or the effectiveness or impact of the programme. To find that out, you always have to dig deeper, into all the other information that the Guidebook provides about individual programmes.”

What does it mean when a programme doesn’t appear on the Guidebook?

Before a programme is added to the EIF Guidebook, its evidence is reviewed by a team of researchers to assess its quality. Programmes that are given an evidence rating of 2 or more are added to the Guidebook.

This means there are two primary reasons why an early intervention programme doesn’t appear on the Guidebook: because it hasn’t yet been assessed by EIF, or because it has been assessed and it doesn’t qualify for at least a level 2 rating.

“The most important thing to remember is this: the fact that a programme doesn’t appear on the Guidebook does not necessarily mean that it doesn’t work, that it’s too small, or too local, or is lacking in evidence. It’s likely there are well-evidenced, effective early intervention programmes out there which don’t yet appear on the Guidebook. “We will continue to work hard to assess the evidence for as many programmes as possible – but for now, at least, the Guidebook shouldn’t be taken as an exhaustive list of early intervention programmes.”

When they are assessed by EIF, some programmes are found not to have evidence that qualifies for a level 2 rating. These programmes are not included in the EIF Guidebook.

A list of these other programmes appears alongside the main Guidebook. If a programme does not appear in the main Guidebook or on this list, then it has not yet been assessed by EIF.

“The fact a programme doesn’t achieve a level 2 rating or higher does not mean that a programme doesn’t work. Many of these programmes are developing and innovating, and are on the way to being established, effective programmes. Indeed, many of the programmes currently on the Guidebook wouldn’t have received a level 2 rating earlier in their lifecycle.

“Being on this list doesn’t mean that a programme does not work or should never be commissioned. It simply means they have not yet conducted an evaluation which meets our standards for preliminary evidence. If they are commissioned, they should be monitored closely to ensure they are providing positive benefits to the children and families who are taking part.”