Challenges in the Operationalization of Mental Health Quality Measures: An Assessment of Alternatives

In 2001, the Institute of Medicine published a series of reports on the quality of health care, including several focused on the quality of care in mental health and substance abuse treatment, that identified deficits in quality and the need for developing a quality measurement infrastructure in mental health care systems (1,2). Recent health care reforms in the United States and other countries have accelerated the need for valid mental health quality measures that are feasible to implement.

The goal of quality measurement is to assess the ability of systems to provide structures, processes, and outcomes of health care to improve the health of the population served (3). Successful quality measurement requires a foundation of evidence that is systematically synthesized and translated into clear clinical guidelines. These guidelines must then be operationalized into reliably defined numerators and denominators that can be populated with feasibly accessible data.

Implementing development of quality measures remains a challenge (4) but not because of a shortage of available measures specific to mental health care. A review of available measures across the globe identified 656 mental health quality indicators that met the criteria of having a specifically defined numerator and denominator and an identified data source (5). In many cases, available indicators assessed the same general measurement concept but varied in their definition. These variations may have different resource requirements and data sources, may result in significantly different findings, and may make comparisons across systems of care difficult. Choosing among alternatives requires decisions about what is feasible versus what is ideal, about what purpose quality measures ultimately serve, about what entities will be held accountable for performance, and about what is supported by empirical evidence regarding reliability and validity of the measure.

The Mental Health Quality Indicator Project

The International Initiative for Mental Health Leadership (IIMHL) has sought to create an international framework for quality measurement. IIMHL is an international organization of mental health leaders seeking to exchange information and collaborate in developing best practices. The IIMHL Clinical Leaders Group has led a multiphase process to create and pilot such a framework. This work began with identifying available quality measures and measurement concepts through a review of existing research and “gray” literature and a survey of IIMHL Clinical Leaders Group members (6,7). The 656 identified indicators were collapsed into 36 broader concepts that were assessed by using a modified Delphi process to collaboratively identify a quality measurement framework; details are described in a previous publication (8). This process was similar to an effort by the Organization for Economic Co-operation and Development that used a Delphi panel of academic and administrative experts as part of the Health Care Quality Indicators project (9).

Quality Measure Alternatives and Operationalization: Delphi Process

The next step for the IIMHL Clinical Leaders Group was to attempt to move from the 36 broader concepts to specific, operationalizable quality measures. We returned to the original list of 656 indicators and identified one base indicator and a number of alternatives for each concept, organized around key operationalization issues. The Delphi panel members rated and discussed these alternatives to develop a consensus on these operationalization issues. Table 1 provides examples of some alternatives identified and highlights the decisions needed for successful operationalization.

Among the issues weighed by panel members were decisions balancing data source with accountability for the quality measure. In some cases, choosing data sources that support more feasible implementation shifted accountability. For example, using filled prescriptions from pharmacy databases to assess medication adherence instead of more labor-intensive chart reviews of prescribed or offered medications shifts accountability away from providers and onto consumers. Similarly, use of administrative data to determine wait times can improve feasibility of data collection, but it comes at the expense of not using more granular data (such as direct surveys) collected more proximal to providers and consumers directly involved and affected by the measure.

In many cases, panel members made choices about segmentation (that is, “splitting” rather than “lumping”) of measures, either by patient or service characteristics. Segmentation results in a greater administrative burden in terms of data collection and analysis, but the improved granularity can provide a better “fit for purpose” to drive quality improvement. For example, segmentation by type of injury on inpatient psychiatric units by separating the clinically distinct phenomena of falls and self-injury can drive more specific quality improvement interventions for each mechanism of injury. However, in cases in which more quantification was required, there was no evidence base on which to set specific thresholds (for example, four versus two medication visits a year). It is unlikely that valid evidence beyond clinical consensus will be forthcoming.

Conclusions

These efforts by the IIMHL Clinical Leaders Group are an important first step in operationalizing a consensus international set of quality measures and addressing key issues regarding data source, frequency, period of assessment, thresholds, segmentation, and breadth of the denominator. However, there are still large gaps in data on the operationalization of mental health quality measures. Empirical data are needed to determine which of the many alternatives can be successfully implemented, how the choice of specific alternatives affects provider and system behavior, and how choices affect health care provision and outcomes. It is unclear who will shepherd this empirical evaluation, given a high level of fragmentation of responsibility and variation in the organization of mental health quality measurement, both in the United States and internationally (7). Data system incompatibility will also pose a significant barrier to international cooperation on quality measure comparison and benchmarking (10). Further complications for mental health services include diversity in loci, funding, and methods of care and administrative separation between mental health and general medical care (1).

Nevertheless, empirical data alone will not provide all the answers needed. Policy makers and mental health leaders will need to make qualitative decisions on how to balance the ideal of evidence-based measures with strong validity with the feasibility of data access, the people or systems to be held accountable for performance on the measure, and the ultimate purposes served by the measures. In particular, stakeholder input will be required to tailor the specific choice of measures to the intended use, whether it is targeted quality improvement, benchmarking within or across systems, accreditation or maintenance of standards, or public reporting to facilitate consumers’ health care choices. The work thus far by the IIMHL Clinical Leaders Group has included only clinical leaders of public mental health systems; input is needed from a broader array of providers, administrators, and especially consumers. However, this work can provide a window into the process of decision making that will be required for the successful operationalization of mental health quality measurement.