An ‘added value’ score for specialist services

Child and adolescent mental health services, and other specialist services for children with emotional and behavioural difficulties, often want to assess how much they help the young people they see. A simple approach is to administer the SDQ at the time of first assessment and then repeat it after a fixed interval, say 6 months. It is then easy to calculate the change in SDQ score. Unfortunately, this simple approach does not establish how much of the change is due to the specialist intervention – high SDQ scores typically improve with time even when children receive no assessment or intervention, partly as a result of ‘regression to the mean’ (an effect of measurement error) and partly as a result of spontaneous improvement. Using data from longitudinal community surveys of young people whose psychiatric disorders have not been treated in specialist settings, youthinmind have devised a measure of added value for use by specialist services.

The youthinmind formula allows you to calculate the added value of specialist intervention for young people who have been assessed with parent-completed SDQs at the time of initial assessment and again 6 months later. The formula is:

Value added = 2.3 + 0.8*T1Total + 0.2*T1Impact – 0.3*T1Emotion – T2Total

where:

Positive scores represent more improvement than was predicted on the basis of the initial scores. Conversely, negative scores represent less improvement than was predicted, or even deterioration. The value added score is in SDQ points, e.g. a value added score of 3 means that the actual total difficulties score at follow up was 3 points lower (i.e. better) than predicted.

It is possible to turn value added scores into standardised effect sizes by dividing by the standard deviation, which is 5. For example, if the average added value for a particular clinic is 3 points, the average effect size of that clinic is 0.6 (3 divided by 5).

Notes:

1) The added value score is not of much use for a single individual because the 95% confidence intervals are very wide (plus or minus 10 points). However, the confidence interval narrows when results are averaged across a large number of individuals, e.g. the last 100 cases seen by a clinic. For example, the 95% confidence interval drops to plus or minus 1 point for the average of 100 individuals.

2) The youthinmind formula was developed to predict added value for high-risk groups where most children have psychiatric disorders and parents have previously been concerned about their child’s mental health. These two factors (psychiatric disorders and prior concerns) reduce the ‘spontaneous improvement’ seen over 6 months, and this has been allowed for in the youthinmind formula. An important consequence is that if the formula is applied to low-risk children who do not have disorders, and whose parents have not been concerned, the spontaneous improvement is underestimated, leading to an average value added score of about 0.8 SDQ points even if nothing has been done. In other words, the youthinmind formula is not calibrated to measure the added value of interventions for low-risk groups, e.g. a change affecting an entire school or an entire community.

3) At present, added value can only be calculated from parent-completed SDQs. It is not valid to apply the formula for parent-completed SDQs to data collected with teacher of self completed SDQs.

4) Although initial findings on added value scores are promising, they should not be taken too seriously until accumulating experimental data from around the world tells us more about the formula’s own strengths and difficulties!