Letter

Helpful and Unhelpful Risk Assessment Practices

To the Editor: Snowden and associates' ( 1 ) report in the November 2009 issue on a validation study of the Classification of Violence Risk (COVR) is an example of a worrying trend toward the promotion of unhelpful risk assessment practices in mental health services ( 2 ).

The authors' frequent use of the terms "future violence" and "prediction" might lead some readers to conclude that they are describing a method for predicting future violence. In fact, risk assessment does not, and cannot, predict the behavior of individuals but instead places the individual being assessed in a group of persons whose likelihood of violence is similar but of whom only a proportion will ever actually be violent.

This confusion between a population forecast and an individual forecast is then compounded by the authors' use of the area under the curve (AUC) statistic as the main measure of the effect size of risk assessment. The AUC is a single summary statistic of the trade-off between sensitivity (the true-positive rate) and specificity (the true-negative rate). As a consequence the size of the AUC is not affected by the base rate of the harm being considered. In this study the AUC is the probability that when we randomly pick one patient who went on to be violent and one who did not, the instrument will have assigned a higher score to the violent patient than to the nonviolent one. Although this is an interesting property of the COVR, it hardly amounts to a prediction.

A far more useful risk assessment measure is the proportion of high-risk categorizations that are later found to be correct—the positive predictive value (PPV). PPV has the advantage over AUC of being easy to understand, and although it is dependent on the base rate of harmful events, PPV is more useful precisely because the accuracy of categorizing patients as high or low risk is highly dependent on the base rate. When combined with the proportion of correct negative predictions (negative predictive value), PPV provides a comprehensive description of the actual performance of risk assessment categorizations in the population being considered.

The prediction of rare events, such as homicide by a person with schizophrenia, is inevitably associated with a low PPV even when the sensitivity and specificity (and thus AUC) is high, whereas more common events, such as physical aggression in a hospital, could be associated with a higher PPV, even if the risk assessment instrument has a lower sensitivity and specificity. Even after careful review of the paper by Snowden and colleagues, the reader cannot truly judge the utility of the COVR to predict aggression in an inpatient setting because only the AUC and correlation were reported. In our view the AUC should be reported with other measures, including the optimal sensitivity and specificity of the instrument recommended by the researchers and measures of the accuracy of positive and negative predictions in the reported sample.