General

What is a programme?

EIF defines ‘programmes’ as a discreet and predefined set of activities that are offered by a specific provider in a form that can be purchased by a local authority or other agency, or that a local authority or other agency might deliver. Providers of such programmes will also typically have developed the programme’s content and processes for ensuring implementation quality. For example the Incredible Years Preschool BASIC Programme teaches parents strategies for interacting positively with their child. Parents whose children are aged between 3 and 6 attend 18-20 weekly group sessions in community venues led by practitioners qualified to at least level 7 in a helping profession.

What do you mean by ‘evidence-based’?

Programmes that have reached a level 3 or level 4 evidence rating. Level 3 programmes have high-quality evidence of impact from at least one trial that is also the programme’s best trial and had good measures of child outcomes, pre and post data, a rigorous and relevant counterfactual, and good coverage of programme participants, including many of these who did not complete the programme. Level 4 programmes have achieved replication of such evidence.

Why are you not publishing reports or cost ratings for programmes rated as NL2 (‘not level 2’)?

We are not providing programme reports or cost information at this stage about programmes without level 2 evidence because there is insufficient direct evidence about their effectiveness to know whether they benefit children or not. We will work with the developers of these programmes and others to agree robust and fair means of determining and rating the quality of these programmes. Many NL2 programmes are based on good science and robust implementation processes. Given the right kind of support they could become the high-quality, evidence-based interventions of the future.

How do I get a programme included in the Guidebook, or reviewed by EIF?

The key way in which programmes get onto Guidebook is through calls for evidence for reviews on particular themes. We will be updating the Guidebook towards the end of 2016 with new programmes. As part of this we will develop plans for how providers can engage in the new guidebook. We will consult with providers, developers and funders on how best we can enable some form of broader inclusion which is feasible with our available level of resource.

How will the impact of this work be monitored?

EIF will work with commissioners and providers to support the application of this evidence. Ultimately we want to see an increased proportion of spending on interventions that have good evidence of effectiveness. Over time this should lead to an improvement in early intervention and in outcomes for children and families but this will require careful and sustained evaluation over many years. We also want to see innovative programmes which aim to tackle unmet needs commissioned, carefully monitored and evaluated in order to develop the evidence for early intervention.

How many people use these programmes? How much money is spent on them?

This information is held by local commissioners, not by EIF. There are no reliable national estimates of spending on early intervention programmes or other activities.

These ratings are important – to families, and to the businesses that deliver these programmes. How does EIF make sure its findings are accurate and fair?

The EIF is an independent charity that does not provide any specific programme or have any investment in or commitment to any specific programme. The evidence review has been subject to a set of principles and standards of evidence that are clearly located in the broad scientific principles of Cochrane and the HM Treasury ‘Magenta book’, but also recognise the need for innovation and local application. These standards are transparently set out in the report and are openly available. The work has been led by EIF staff subject to challenge and oversight from an independent group of academics. The decision about ratings is made in relation to transparent and clear standards of evidence. Participating programme providers and developers have had opportunities to raise issues, contribute information and make formal challenges.

How does support in the early years link to babies and children’s brain development?

Brain development occurs through childhood and into adulthood. Neuroscientific evidence shows that that brain development is particularly rapid in the early years of life, but that there are other periods of active development in childhood such as during adolescence. There is malleability through life. If a child suffers severe trauma there can be an impact on how the brain develops in ways related to physical, emotional, social and cognitive development. However, it is not true to say that if a child experiences an inconsistent or negative experience with a carer parts of the brain do not grow and may not ever grow. The emerging evidence from neuroscience is exciting and persuasive but is easily overstated.

Evidence

How can you improve your evidence?

The EIF strength of evidence ratings reflect stages in developing programmes from the point of theory through practice to pilot studies, proof of concept, adaptation and replication. These stages are broadly agreed by What Works centres and other evidence rating and evidence advice agencies. All provide advice on the detailed underpinning of the steps required.

Are RCTs the gold standard for social policy evaluation?

No, this is an unhelpful and misleading metaphor. RCTs if done well are particularly good at providing an unbiased estimate of scale of impact because there is a clear comparison in outcomes between those who receive an intervention and those who don’t, that can be reasonably assumed to be the result of the intervention. However, this can be achieved through other means so RCTs are not necessary to achieve a high rating, nor are they necessarily the best or only form of evaluation of impact in every circumstance.

An RCT won’t work for us. Does this mean that we will never be considered ‘evidence based’?

Programmes that are considered evidence based are those that can show a specific positive impact for children when compared with an equivalent group who did not receive the programme. This could be and RCT or an alternative quasi-experimental design. Although programmes which do not yet have this sort of evaluation evidence are not considered to have good evidence of impact, they can be considered promising and science based if they have a robust logic model which draws on other evaluations, or preliminary positive findings from a pilot study. However, just because a programme has a strong science base this does not necessarily mean that it will work when tested more rigorously.

Why do you rate Family Nurse Partnerships at a level 4?

The FNP programme (NFP in the US) is a programme that has been developed very carefully over a long period. It has been tested in many studies in different times and places. It has demonstrated proof of concept and shown that it can be adapted to work in different contexts (see the case study in the full report). It has also shown impact in relation to secondary outcomes on children’s learning and language development. However, as the EIF standards of evidence emphasise commissioning cannot be based solely on the answer to the question what has been shown to work in the past or in other places. There must always be consideration of local costs and benefits and local implementation. The finding that the FNP did not deliver improvements to its primary outcomes in its UK trial is a good example of the challenge of determining the applicability of evidence from one context to another. It does not mean the programme is not evidence-based, but does indicate the case that is required in determining likely local benefits.

EIF process

How did you pick the programmes in your review?

EIF commissioned an independent review of literature and published papers to identify programmes available in the UK that met the scope of our review (see The Best Start at Home). We also requested information from our 20 places about programmes that they might like to commission. In this way we have ensured that the assessments are relevant and useful to commissioners and the overview of findings provides a good indication of the strength of evidence for available programmes of this sort. We do not claim to have assessed every available programme in the early years, but the 75 programmes included in this review are a good enough sample to allow us to comment on the strength of evidence for many programmes in the ‘market’.

What is the methodology for rating programmes?

EIF works with a continuum for rating the strength of evidence that recognises different levels of quality of impact evaluation. We have developed detailed criteria through which the evidence for any programme can be transparently assessed. These have been shared with providers and developers and are described in the full report.

How does the EIF rating methodology deal with mixed evidence?

Evidence can be mixed in that multiple studies about a specific programme might disagree and within the same study some outcomes might change more than others. We prioritise the evidence from the strongest evaluation design and within studies prioritise outcomes that are well measured and are most likely to translate into long term impacts. There is an element of judgement involved.

How do programme ratings keep up to date as the evidence changes?

We will be updating the online Guidebook to ensure that we reflect the latest evidence for programmes that are included.

What works

Are you saying that universal services don’t work?

No. Early intervention programmes are separate additions to universal services for families. Universal early intervention programmes are also different to universal services.

Are you saying that programmes rated as NL2 (‘not level 2’) don’t work?

No. Many NL2 programmes are based on good science and robust implementation processes. Given the right kind of support and with resource to evaluate they could become the high-quality, evidence-based interventions of the future. The rating indicates that they do not yet have direct evidence about impact sufficient for EIF to assess the scale and nature of the benefits for children.

Should programmes with ‘no effect’ be discontinued or decommissioned?

Not necessarily. This primary focus of this review is on improvements to child development. There are programmes that parents might enjoy or benefit from that do not improve outcomes for children. These still might be sensible or appropriate to commission, but it should be clear why they are being implemented and parents should not be given false promises about child impacts that are unlikely to materialise. However, there will also be cases where the evidence of no effect means that a programme in its current form should be discontinued or adapted.

Are you advising commissioners and parents not to use programmes that are below a certain rating?

No. We do not kitemark programmes. Commissioners must make decisions which weigh up evidence with other factors such as local need and circumstances. We offer commissioners information on what the ratings mean, how to interpret them in making decisions and what other information to take into account. Commissioners also need to support innovation and help to develop the evidence of promising and new programmes that cater for an unmet need.

What advice does EIF provide about commissioning individual programmes?

EIF has produced a substantial suite of resources relating to early intervention programmes and regularly receives requests for advice about the commissioning of individual programmes.

EIF publishes information and advice about early intervention programmes at www.eif.org.uk, including details of types of interventions for different children’s needs, details of individual programmes, and advice on commissioning programmes at different evidence levels. EIF does not hold information about or provide advice on all early intervention programmes.

EIF is not able to provide further information or advice about specific programmes beyond what is provided in the published reports or the Guidebook. Further questions about an individual programme should be directed to the programme developer or provider.

EIF does, however, have a programme of work to support the use of its evidence reviews and help local commissioners draw on these as they develop commissioning plans and specifications. To find out more about contracting with EIF get in touch via info@eif.org.uk.