In Reply: Burns has made three general points in his letter about our article on performance measures for schizophrenia services. First, he has endorsed the use of the Delphi process for engaging multiple stakeholder groups. Second, he has made pertinent observations about the process of using the Delphi based on his considerable experience working with multiple groups, including consumers. More specifically, he has pointed out the potential problem that occurs when groups rate many items as essential, and he has made several practical suggestions about how to encourage the group to be more focused and selective. We recognize the value of his suggestions and, in retrospect, we could have improved our Delphi process.

We have a different perspective on his third suggestion, to return to the Delphi group with improved directions in order to obtain a more focused group of performance measures with the goal of changing the practice of individual clinicians with a set of “performance measures that nobody could ignore.” The reason underlying our perspective is that we see the identification of measures as only a small step on the path to implementing performance measures in mental health services. In a prior systematic review, we examined the facilitators and barriers to implementing performance measures (1). The review identified seven broad factors that influenced the implementation of performance measures, including indicator characteristics, promotional strategies, implementation strategies, resources, individual factors, organizational factors, and external factors. In our article on performance measures, which is the subject of Burns' letter, we focused on one factor only, indicator characteristics.

We have previous experience with use of the Delphi process as part of the process of identifying performance measures for first-episode psychosis services (2). When we moved to implement these measures, we found that some measures rated as essential were, in practice, hard to specify with precision, and some were difficult to collect. When we studied the use of the available measures to compare services, only one measure, hospitalization, was sufficiently robust and widely available to be considered suitable for setting standards (3). We have since sought to increase the potential value of hospitalization as a performance measure for first-episode psychosis services. First, we developed a risk adjustment formula for hospitalization (4), and second, we tested the validity of hospitalization as a performance measure by demonstrating that it correlated with two other important outcome measures, quality of life and global psychopathology (5).

In conclusion, we see the systematic review and rating of evidence as essential first steps in developing performance measures. The Delphi is a useful next step to narrow the number of measures, but it reflects only opinion evidence, one of the weaker levels of evidence in evidence-based medicine. Further practical steps include the rigorous definition of measures and further research to validate the measures. All these steps would still address only one of the seven factors (indicator characteristics) that we identified as facilitators and barriers to implementing performance measures in mental health services.