Implementing a Statewide Outcomes Management System for Consumers of Public Mental Health Services

Compelled by numerous forces such as national health care reform (1) and managed care (2), many states are undergoing reforms aimed at improving the organization and financing of public mental health services (3-5). Issues such as access to care, continuity of care, and outcomes of care are also recent concerns for many states (6-9). Although great interstate variation exists in accountability efforts (10), documenting and improving the effectiveness of mental health services is clearly an urgent issue that has affected not only private payers, but public payers as well.

In the past, techniques such as utilization management and utilization review have been the primary methods for system accountability, with an emphasis on utilization of services (11). However, the difficulty with using utilization review as the sole approach to assessing mental health services is the questionable assumption that service utilization is an adequate measure of treatment needs or effectiveness. In addition, utilization review typically examines only a few components of treatment—the location, amount, and, sometimes, type of care. Although utilization review was initially intended to improve the quality of care, it has resulted in the micromanagement of individual cases and often has left clinicians feeling frustrated that outside care reviewers unfamiliar with their patients are directing care (12).

Current methods of accountability are much more focused on demonstrating how treatment affects outcomes, and these approaches include an increased role for consumers' perspectives of functioning and well-being (13,14). For example, the state of Georgia has undergone a radical restructuring of its public mental health system, shifting from a centralized model of service delivery to one that is decentralized and consumer driven with regional boards whose membership is at least 50 percent consumers or family members (13). These new approaches to accountability, particularly the use of outcomes management systems, promise to go beyond a narrow focus on service utilization to provide valid and reliable information on what types of care are most effective for whom and in what settings and circumstances (15-19).

This paper describes a public-academic collaboration to develop a statewide outcomes management system for measuring outcomes and processes of care for public mental health patients at risk for hospital admission. The system was also designed to provide information, such as about patients' satisfaction with their living situations, that could be used to shape policies. Other goals of the system were to promote successful treatment modalities, such as adequate medication dosing for chronically depressed patients, and to enhance continuous quality improvement initiatives.

Very little research has focused on efforts to achieve large-scale statewide system accountability such as the initiative described here, nor has research adequately addressed the effectiveness of such systems of care (20). However, because of recent national trends to shift care from the hospital to the community (4,7) and local trends in Arkansas involving financial decentralization (21), questions about whether patients are receiving appropriate services and about the extent of these services are at the forefront of many stakeholders' agendas. It is hoped that the discussion of our approach, methods, data collection tools, and processes to facilitate continuous quality improvement, as well as issues encountered during implementation, will help others who are involved in clinical services research and accountability efforts in the public mental health system.

The outcomes management system

Background

Our working model for this considerable task relies heavily on the use of outcomes management consistent with the Shewhart-Deming conceptual model of continuous quality improvement (CQI) (22,23). This approach follows a cyclical process that begins with monitoring clinical performance to identify problems that influence clinical practice patterns and the causes of these problems. Once problems are identified, practice modifications can be recommended and introduced, and the results assessed. Most important, by using the CQI process, information can be fed back to clinicians and administrators to improve a system's clinical performance while also addressing issues of accountability.

Key collaborators in this public-academic venture to develop a statewide outcomes management system include the Health Care Financing Administration and the Arkansas Division of Mental Health Services (DMHS), agencies that both set policies and provide payments; the Center for Mental Healthcare Research, an academic institution that provided the implementation project team; and Arkansas community mental health centers (CMHCs), the provider agencies. The project was conceptualized in the winter of 1994 and implemented the following summer, in 1995.

The involvement of senior organizational leaders, particularly from the DMHS in our case, has been noted as a critical factor in the success of an outcomes management system (17). To facilitate this involvement, before implementation of the system, the project team presented the proposed project to the executive directors of the CMHCs at a regular meeting in the spring of 1995. Voluntary support was necessary from each center director because CMHCs operate as private nonprofit organizations in Arkansas and are not under the direct administrative control of the DMHS.

Clinician support, also an important element of success (17), was fostered in the summer of 1995 by two days of intensive training for all CMHC clinicians participating in the project. The training was given by the project team from the Center for Mental Healthcare Research at a central site to allow the greatest attendance possible. Site-specific inservice training was provided on request, and the project team visited the facility to train CMHC clinicians to tailor data collection to their specific circumstances. Throughout the project, telephone consultation has been provided to CMHC clinicians as needed. The executive director at each CMHC selected a clinical staff member to provide additional training to all clinicians and to coordinate patient enrollment and data collection efforts throughout the project.

The scope and impact of this project is quite large in terms of dollars devoted to services and numbers of providers and patients. For example, although state and federal funds are not the only source of income for CMHCs in Arkansas, approximately $20 million of such funds were disbursed to the CMHCs in fiscal year 1996. Slightly more than 10 percent of this budget (approximately $2.6 million) was earmarked for alternatives to state hospitalization. All 15 CMHCs and their on-site staff throughout the state are involved in the project. In 1996, with a state population of 2,225,000, approximately 60,000 consumers received some type of CMHC service in Arkansas.

Project implementation costs are difficult to estimate, and costs of implementing such a project will vary greatly among different systems depending on the technology and resources that are available. Cost of maintaining the Arkansas outcomes management system will range from $100,000 to $175,000 a year, which is less than 1 percent of the state and federal funds spent annually in the CMHC system.

Some factors that would result in lower costs include care providers' collecting data from consumers at the point of service versus the use of special project assistants, less dependency on outside professional interpretation of data, less reliance on sources outside of the CMHC for providing technical support such as data entry, and sampling only as many consumers as are needed to provide an accurate appraisal of outcomes for the system.

System design

A disorder-specific approach to sampling and assessment was used to examine how patient outcomes relate to treatment provided within the state's public mental health system. Consumers at CMHCs were selected to be monitored by the outcomes management system based on their psychiatric diagnosis. An assessment tool specifically designed to measure the salient aspects of their disease and its treatment was used.

This disease-specific approach, which is based on the tracer methodology, has been favored by others evaluating mental health services (17,24), primarily because it provides relevant clinical and prognostic information for a particular disorder (tracer). Also, it uses specific tracers as an indicator of the overall quality and effectiveness of the system delivering that care (25,26); thus this approach is more economical than one that monitors every condition and every patient.

For several reasons, schizophrenia and depression were chosen as tracer conditions in this outcomes management system to assess the quality and effectiveness of our public mental health services. Schizophrenia and depression have been rated as areas of primary professional concern by those involved in public-academic health linkages (27). Depressive disorders in mental health settings are common (28). In Arkansas the prevalence of schizophrenia and depression is quite high, as is the use of services by persons with these disorders compared with patients who have other diagnoses. For example, a 1994 study using a statewide database of patient characteristics and service use in Arkansas CMHCs showed that individuals with schizophrenia or with depression received 46.8 percent of all CMHC services provided (Zhang M, Smith GR, Heithoff K, et al, unpublished manuscript, 1996). Finally, the economic and social burdens associated with both of these disorders (29-32) make accountability and quality improvement efforts imperative.

Procedures

Recruitment of patients for monitoring and evaluation by the outcomes management system began in August 1995 and ended in December 1996. To be eligible, individuals had to have a diagnosis of major depression or schizophrenia, be 18 years of age or older, and be at risk for hospitalization, as initially determined by a clinician. Patients from each of the state's 15 CMHCs were considered for the depression tracer condition. However, a necessary component of our monitoring approach for patients with schizophrenia was a periodic face-to-face interview; given the costs and constraints involved in such interviews, patients with schizophrenia seen only in the eight CMHCs in central Arkansas were targeted for monitoring by the system.

To examine a relatively homogeneous patient population in terms of diagnosis, individuals with depression who were also known to have severe borderline personality disorder, antisocial personality disorder, or bipolar disorder were excluded from monitoring. Similarly, for the tracer condition of schizophrenia, individuals with schizoaffective disorder, organic psychosis, bipolar disorder, or depression with psychotic features were excluded.

Clinicians at each CMHC selected patients who met the above criteria. After appropriate diagnostic screening and patient consent, clinicians and patients completed the baseline portion of the Depression Outcomes Module (33) or the Schizophrenia Outcomes Module (34). These instruments measure disorder-specific outcomes and prognostic case-mix characteristics as well as general health status, functioning, social factors, demographic characteristics, quality of life, and processes of care (for example, quantity and type of services). The modules are designed to measure outcomes from multiple perspectives and from several sources, including consumers' self-reports, clinicians' ratings, and medical records. They are also relatively brief and feasible to implement in routine mental health care settings (26). To ensure patient privacy, the data collected for this project were secured as if they were research data, so they could not be readily linked to individual patients.

Collection of follow-up data

The protocol for the outcome management system called for collecting follow-up data for patients with depression every four months after baseline for up to one year, or until the depressive disorder resolved, and for patients with schizophrenia every six months after baseline for one year. The follow-up times were chosen based on clinical experience and on literature indicating that they would be reasonable periods for observing clinically meaningful changes (35,36).

For individuals with depression, the follow-up forms were mailed to patients, administered by phone, or completed at the CMHC. For those with schizophrenia, face-to-face follow-up interviews were conducted by a trained research assistant at the CMHCs. The CMHCs provided temporary unused space for the interviews. They also helped locate patients and plan specific times for follow-up interviews.

Medical record review, one of the components of the modules, was performed after the last follow-up by trained research assistants. Diagnoses initially made by CMHC clinicians were verified using a clinician DSM-IV checklist diagnostic verification form, and any further questions about the diagnosis were resolved by a review of the patient's medical records by a research psychiatrist. Additional information collected by chart review included quantity and type of services utilized within the public mental health system, such as day treatment, or utilized elsewhere, such as emergency room services. Information on psychotropic medications prescribed was also obtained. Follow-up of service utilization ended in December 1997, one year after the final recruitment month.

Feedback for CQI

Perhaps the most important part of an outcomes management system is the feedback of information for continuous quality improvement. Not surprisingly, CQI is becoming a major goal in state mental health systems (37). Our approach to CQI consisted of a twofold strategy. The first step involved a meeting between the project team and the CMHC directors to mutually decide how data could best be reported. The second step involved disseminating regular reports to participating stakeholders, starting in July 1996, so that they could conduct site-specific CQI initiatives, especially feedback of information to clinicians.

These regular reports provide a written and graphical summary of primary findings, including patient characteristics, changes in outcomes over time, and mental health services received. The reports are sent bimonthly to CMHC executive directors with a request for input on their concerns about patient care, managed care, and state policies. Regular reports are also sent to the director of the DMHS, the Arkansas Alliance for the Mentally Ill, and the Arkansas Mental Health Council.

When enough data are gathered, all individuals and groups that have been receiving regular reports will receive reports that will include comparisons between large subgroups of patients and analyses of patient and treatment characteristics associated with improved outcomes. It is anticipated that these reports will be ready by mid-1998.

Issues in implementation

Others might benefit from a discussion of several issues related to our experiences in implementing a statewide outcomes management system in the public mental health setting. First, we believe a strength of our approach is that multiple perspectives and sources of outcome are taken into account over time and that all parties invested in the process of public mental health are represented. The people who can make a difference in improving the quality and effectiveness of care—consumers, clinicians, consumer advocates, DMHS administrators, CMHC directors, and other state policy makers—are all involved in the project. In addition, our disorder-specific design and measurement tools are economical and incorporate critical components of case-mix that provide sufficient differential data on which to base sound policy.

A limitation of our outcomes management system is that we have obtained fewer referrals for monitoring than were expected, given the number of known cases of schizophrenia and depression in the state. Although the limited number of referrals is a typical problem encountered in large-scale "real world" evaluations of this type (38), the reasons for the low referral rate are not entirely clear. As in other studies of outcomes management systems, it is possible that referral rates are directly correlated with the number of clinicians at each site who are designated to make referrals and the number of staff in the outcomes management system responsible for data collection (18).

However, other factors accounting for the relatively low number of referrals might include the stringent inclusion criteria, the fact that cooperation by CMHCs was voluntary, and concerns about confidentiality by both providers and consumers. Privacy in outcomes management systems is an ongoing concern. Although individual providers were not specifically monitored in this project, clinicians might have been hesitant to volunteer information they perceived could be used against them.

The CMHCs had no financial front-end investment in the outcomes management system and were strictly voluntary participants. Each CMHC agreed to designate time to enter a patient in the project, amounting to about one hour per participant. The CMHC also provided medical records so that service utilization data could be obtained. However, even though the project was described as an outcomes management system from the start, some CMHC staff members perceived that the data gathering was for academic research. This perception may have given the project lower priority in the CMHC system than if it had been viewed as a practical CQI tool. In light of these observations, it is clear that for an outcomes management system to operate to capacity, providers must value and incorporate it into the service delivery system as a necessary means of measuring the effectiveness of treatment. Research is needed to identify specific techniques for gaining providers' cooperation.

Limitations aside, we suggest that it is feasible to implement an outcomes management system in the public mental health setting, despite common perceptions noted by others that public mental health systems are complex and turbulent systems that do not support research efforts (39). The application of clinical services research and outcomes management systems, such as the one described here, will become an increasingly important issue for political and clinical leaders in public mental health settings to consider.

Particularly as competition in the health care industry continues to intensify, public mental health systems must come to terms with these issues to survive. Not only do such state-of-the-art projects add to our current knowledge about processes of care and effectiveness of treatment, but they also provide guidance in solving practical problems such as case-mix adjustment, outcomes specificity, and quality improvement efforts. Thus they help reduce the burden on public mental health systems that many states currently face.

The authors are affiliated with the National Institute of Mental Health's Center for Mental Healthcare Research and the department of psychiatry at the University of Arkansas for Medical Sciences, 4301 West Markham Street, Slot 589, Little Rock, Arkansas 72205. Dr. Booth, Dr. G. Richard Smith, and Ms. Nordquist are also affiliated with the Health Services Research and Development Field Program of the Department of Veterans Affairs.