Comparison of National Mental Health Quality Assessment Programs Across the Globe

International surveys conducted by the World Health Organization (WHO) demonstrate high one-year and lifetime prevalence of mental illness, with interquartile ranges of 9%–17% and 12%–47%, respectively (1–3). This burden has led the WHO to recommend that countries establish well-formulated mental health policy agendas that include the measurement of quality of mental health services (4). The Institute of Medicine (IOM) has also highlighted the importance of quality measurement infrastructure and accountability in improving mental health service quality (5). However, the assessment of mental health service quality faces multiple obstacles, including the lack of a sufficient evidence base, the lack of sufficient data in existing resources to determine measures of quality, the lack of government prioritization of quality assessment, and the lack of a sufficient quality assessment infrastructure for efficient access to meaningful data (6).

Efforts have been made to establish cross-national or international frameworks for mental health quality assessment to promote standardization of quality measurement and to assist comparative benchmarking. The National Institute for Health and Welfare in Finland (STAKES), coordinating with the European Commission Health Monitoring Program, developed a set of 32 mental health care quality indicators for a European Community comprehensive health monitoring system in the domains of demographic and socioeconomic factors, health status, determinants of health, and health systems (7). The Organization for Economic Co-operation and Development (OECD) Health Care Quality Indicators project used an expert panel to identify 12 key mental health quality indicators for use in international benchmarking across four domains—treatment, continuity of care, coordination of care, and patient outcomes (8). Although not specifically focused on quality assessment, the WHO Assessment Instrument for Mental Health Systems captured structure measures of quality in 42 low- and middle-income countries (9,10). Pincus and colleagues (11) have proposed a mental health quality measurement framework consisting of ten key quality indicators grouped in the IOM quality domains of safety, effectiveness, patient centeredness, timeliness, efficiency, and equity.

The International Initiative for Mental Health Leadership (IIMHL) has also sought to evaluate mental health quality assessment on an international scale. The IIMHL was established initially by mental health leaders in nine developed countries in order to exchange information about effective leadership, management, and operational practices in mental health services delivery and to collaborate in the development of best practices for mental health services. Twelve participating IIMHL countries initiated the Clinical Leaders Project to develop an international framework of mental health care quality measures in order to compare system performance across countries and inform initiatives to transform mental health systems. Prior work by this group includes general descriptions of mental health quality assessment in five IIMHL countries, identification of mental health quality assessment programs in the United States, and an examination of various mental health quality initiatives identified through published articles, government reports, and other gray literature (12–19). This study sought to further investigate ongoing or soon-to-be-established quality measurement programs in the countries represented in the IIMHL Clinical Leaders Project, including programs without previously published information or data. Using a survey, we evaluated the structure of these programs, the development of measures, the collection and reporting of data, and the domains of quality assessed by quality indicators.

Methods

Participants

Respondents were representatives from the 12 participating countries (Australia, Canada, England, Germany, Ireland, Japan, the Netherlands, New Zealand, Norway, Scotland, Taiwan, and the United States). Respondents were asked to obtain additional information about established or soon-to-be-implemented mental health quality assessment programs in their country from individuals affiliated with their mental health systems, including government mental health care leaders, quality assessment project leaders, key academicians or researchers, and leaders of nongovernmental health care organizations. A “point person” was identified for each country to coordinate responses from identified sources. There were no inclusion or exclusion criteria for participants. All respondents were fluent in English.

Survey instrument

A survey to assess quality assessment programs was developed and distributed to participants. [A copy of the survey is available online as a data supplement to this article.] The general structure of the survey was based partly on surveys used in evaluating mental health programs in the U.S. Veterans Health Administration (20). Participants were asked to identify mental health quality and performance assessment programs that met all of the following criteria: the programs were designed not only to provide descriptive information about the state of the system but also to present results that measure the quality or performance of the mental health system; the purpose of the programs is to measure quality or performance of the mental health system, not to guide clinical decisions; the programs provide measurements in an ongoing and organized method, rather than being one-time initiatives or assessments; and the programs are intended for quality measurement and use on a national level, rather than on a state, provincial, or local level. However, given variability in mental health care systems in the participating countries, if national-level measures were not generally available, participants were asked to provide information about major state, provincial, or local-level programs instead.

The survey asked participants to provide a description of each program being reported and the administering organization. Participants were then asked to provide information about the derivation of utilized measures, the health care entities being assessed, the sources and means of data collection, the level at which data were reported, and the use of the data.

Participants were also provided with a list of 15 domains of quality assessment, including several specific subdomains within each domain. These domains and subdomains were adapted from the National Inventory of Mental Health Quality Measures established by the Center for Quality Assessment and Improvement in Mental Health, as well as from IIMHL participant recommendations (21). Participants were asked to identify which quality domains and subdomains were represented by the indicators in each program and were asked to provide specific examples of representative indicators, with the option to provide additional domains or subdomains. When participants provided examples of indicators that did not reflect the aforementioned criteria for quality assessment, the identified domain or subdomain was excluded from the analysis.

Study procedure and data analysis

Participants were contacted via e-mail between September 2009 and March 2010 to complete the survey online, via SurveyMonkey.com, or by annotating a Microsoft Word document and returning it. Participants were asked to submit a separate “template” for each reported quality assessment program. Surveys were provided in English only, and the point persons in each country were asked to provide their responses in English. After responses were collected, participants were contacted via e-mail and conference calls, as needed, to clarify responses. Survey results were analyzed in Microsoft Excel with descriptive summary statistics.

Results

A total of 38 qualifying programs were identified in the 12 participating countries. [A list of the programs is available online as a data supplement to this article.] Seventy-nine percent of programs (N=30) were active and provided data for collection, and 16% of programs (N=6) were being planned for future implementation but were at least in the data-collection phase. Sixty-six percent of programs (N=25) were administered by governmental organizations, and 29% (N=11) were administered by nongovernmental organizations, including independent regulators, accrediting or licensing organizations, health insurance providers, research institutes, and professional organizations. The mental health care entities most commonly assessed were hospitals and outpatient mental health care clinics, with fewer programs or countries measuring private health care plans or insurers (Table 1).

Methods to identify and establish quality assessment indicators included literature reviews, forums or consultations with key stakeholders and experts, or adoption from preexisting quality measurement programs. Many programs had specific inclusion criteria for identified measures and included feasibility studies and field testing of determined measures prior to full implementation. In most cases, the organization responsible for the operation of the program was the organization involved in deriving or identifying the quality indicators used (such as the Council of Australian Governments’ National Action Plan on Mental Health and England’s Care Quality Commission’s Periodic Review).

The data collected for the reported programs most commonly came from mental health care utilization databases or registries or from client survey data, and few programs included data directly obtained from physician surveys or insurance claims (Table 2). Results were predominantly reported on a national or regional level (74%, 28 of 38) or on a clinic or organization level (50%, 19 of 38) and were less frequently reported on the level of consumer demographic cohort (34%, 13 of 38), diagnosis (21%, eight of 38), or individual provider (13%, five of 38). Seventy-three percent of programs (28 of 38) presented their data publicly, compared with 13% that did not (five of 38). Sixty-four percent of programs presenting data publicly (18 of 28) presented individualized quality or performance data about the participating care providers or organizations being measured.

The data collected by the programs were used in various ways, often related to the structure and intent of the responsible organization. At the time of our survey, many programs were still in the process of collecting and generating data and had not become active. Programs variably used quality data to identify national or regional targets for quality improvement, to compare existing services with identified benchmarks, to create interventions to improve quality performance, and to track changes over time. In particular, the Department of Health in England and the Scottish Integrated Care Pathways for Mental Health used quality data both on the national level to drive countrywide health care policy and on the local level to drive quality improvement and redesign. Programs such as the National Psychiatric In-Patient Reporting System in Ireland and the Uniform Reporting System in the United States were used to compare regions within the respective country, whereas programs such as the Taiwan Quality Indicator Project were used to compare individual hospitals. Several programs were geared toward accreditation procedures for health care organizations, such as Scotland’s Electro-Convulsive Therapy Accreditation Network and the ORYX(R) Performance Measurement Initiative in the United States. Programs reporting data publicly were also used by stakeholders such as consumers, providers, and employers to allow for comparison of providers or health plans (such as the GGZ Transparency Steering Group Performance Indicators in the Netherlands, the National Quality Indicators in Norway, and the Healthcare Effectiveness Data and Information Set in the United States). Some programs, such as Germany’s Ambulatory Quality Indicators and Key Measures, Taiwan’s National Health Insurance Bureau quality measurement program, and the Physician Quality Reporting Initiative in the United States, were used to determine financial incentives for systems or providers based on quality benchmarks.

Table 3 presents data for the domains and subdomains of quality assessment measured by the indicators included in the reported programs. The presence of indicators within the listed domains was highly variable among the programs. No domain was assessed by every participating country. Subdomains measured by two-thirds or more of participating countries included symptom assessment domains (such as bipolar or depressive disorder, substance abuse, and suicide risk), efficiency and continuity measures (including duration of hospitalization, utilization of outpatient services, and inpatient readmission), safety and legal issues (involuntary hospitalization and use of seclusion or restraints), total population mental health care expenditures, and access to emergency care. The subdomain most commonly measured was duration of hospitalization, assessed by 42% of programs and 83% of participating countries. Domains measured by half or fewer countries were recovery, cultural competence, evidence-based pharmacotherapy, nonpharmacological somatic treatment, and substance abuse.

Participants provided examples of subdomains not included in the survey prompts, such as adequate duration of antidepressant treatment, use of measurement-based care, self-injurious behavior, elopement, assessment of dementia, attention-deficit hyperactivity disorder or borderline personality disorder, use of supported housing or occupational therapy, access to crisis resolution or home health services, and screening for patient strengths and wellness.

Discussion

This study builds on prior work by the IIMHL Clinical Leaders Project and other groups (the OECD, STAKES, and WHO) in identifying mental health quality indicators and quality assessment programs and in helping to provide a framework to understand the state of international mental health quality assessment. In particular, this study provides a more in-depth and direct comparison of program-level features of quality assessment, such as methods of deriving quality indicators and uses of quality data. The findings of this study are consistent with prior studies, showing a wide variation among established quality assessment programs in indicator derivation, program administration, and utilization of generated data. This may indicate that programs are focused on local priorities rather than on building a consensus framework for mental health quality assessment. This variability may also be attributed to the significant variation in the organization of mental health care systems and quality assessment infrastructure in the participating countries; data collected on these topics will be reported separately. Further study is required to better understand how individual national priorities may be reflected in the choice of quality assessment methodologies, uses, or domains of focus.

Although prior studies by the IIMHL group have broadly demonstrated variety in the domains covered by quality measurement programs, this study extended beyond the prior work by providing an updated and quantified comparison of specific quality indicator subdomains measured by these programs. The domains and subdomains most commonly assessed by the identified programs in this study were largely related to high-acuity mental health care, such as involuntary hospitalization, inpatient readmission, access to emergency care, and use of seclusion and restraints. These areas are consistent with the finding that a high proportion of the identified programs measure hospital-based care and may reflect either a general international consensus regarding the critical domains of mental health quality measurement or a greater availability of data for indicators measured in these domains. The low proportion of recovery-oriented quality assessment measures, despite the high proportion of programs and countries collecting consumer-level data, is notable given the recent emphasis, including from the IOM, on recovery-oriented, patient-centered services (5). The impact of this misbalance of represented quality domains requires further exploration, but it is concerning because of a potential lack of focus of national policies and resources on recovery-oriented, outpatient, and preventive services. These findings may also indicate the need for a more balanced range of mental health care quality indicators and provide further impetus for the development of a consensus framework for mental health quality assessment.

There are a number of important limitations to this study. The surveys and responses were generated in English only because we had insufficient resources to provide translation; respondents were responsible for translation from native languages. This may have placed a larger burden on respondents from non–English-speaking countries and influenced both the interpretation of the survey prompts and generated responses. Furthermore, the study focused on only a subgroup of developed countries; further study is required to determine how these data relate to mental health systems in low- and middle-income countries. Some reported programs did not have distinct elements separating data collection, measurement development, and data reporting, which may be attributable to countries with more centralized mental health quality structures. The analysis included programs that have been implemented and programs that are still in the implementation phase. The yet-to-be implemented programs were included in the analysis because they are still able to reflect national methods and priorities of mental health quality assessment, such as domains of interest, data sources, and uses of data. We did not identify the feasibility of full implementation of these programs, and therefore the results may be skewed away from truly feasible quality assessment. Also, the reported percentages are overrepresentative of countries without a unified mental health quality assessment infrastructure, with countries such as the United States and Ireland having multiple independent groups measuring quality data and thus more reported programs. In addition, participants from countries with decentralized systems may not have fully identified quality measurement programs in their country, partly due to the nature of their positions in the public mental health system, and therefore may have contributed to a skew in program-level reporting. The study was reliant on participants to report only programs and quality domains that met the established criteria for quality assessment, and independent verification of validity could not be reliably established. Additional important domains and subdomains of mental health quality assessment may exist that were not included in this survey and may require further exploration.

Both this study and prior IIMHL Clinical Leaders Project studies have provided mental health leaders the opportunity to exchange information and to obtain peer-to-peer consultations for the assessment and implementation of quality improvement initiatives. Ultimately, the goal of this project is to establish a consensus international framework for mental health quality assessment and thus a means to compare key measures of performance across countries. The second phase of this project involves examining all of the collected mental health quality indicators and creating a set of core quality measurement concepts based on their validity, importance, and feasibility. Furthermore, we intend to create an international network linking quality measurement groups or organizations within each country to help facilitate quality measurement framework development and assist implementation of the core measures.

Further steps in research and practice will be required to improve quality assessment internationally, including establishing tighter links between process and outcome measurements, increasing use of standardized assessments, expanding use of information technology, delineating benchmarks for comparison across settings, increasing investment in quality research, and integrating mental health quality assessment into the broader framework of health care quality (11). The development of a common framework of mental health quality assessment may be an instrumental first step in the process of refining mental health quality assessment. The results of this study will help provide data to better understand shared priorities for mental health quality assessment and may help identify barriers to development of a common framework.

Conclusions

This study sought to assess existing or soon-to-be implemented mental health quality assessment programs among participating members of the IIMHL Clinical Leaders Project. Thirty-eight qualifying programs from 12 participating countries were identified. Most of the programs assessed were active, were administered by governmental organizations, were focused on hospital care, and used encounter or utilization databases as sources of information. Different methods were used by the programs to identify and establish quality indicators. Program data were generally publicly reported and were used for various purposes by their respective organizations or countries. A wide range of domains of quality were represented in the reported programs, although most commonality was seen in domains associated with high-acuity mental health care, with fewer programs assessing domains related to recovery. These data will help future work intended to establish an international framework for mental health quality assessment.