The Research Potential of Administrative Data From State Mental Health Agencies

State mental health agencies (SMHAs) have long played a pivotal role in serving adults with serious mental illness and youths with serious emotional disorders, but that role has grown increasingly complex. As providers, SMHAs operate state psychiatric hospitals and community-based services. However, other functions now include planning, policy development and implementation, and monitoring and licensing of non-SMHA providers. SMHAs also fund services and oversee contracts with private vendors, who provide an increasing proportion of services. This role often involves the use of Medicaid funds, over which some SMHAs have at least partial control. However, discussions of these broad national trends obscure the fact that SMHAs are as different as the states they serve, and as components of state governments, they are subject to local economic and political forces.

The evolution of 50 different agencies providing services to populations with similar characteristics and needs seems a natural focus for mental health services researchers. However, capitalizing on such opportunities requires systematically collected and routinely available data that track changes in policy, service use, organization, financing, and other domains across states and over time. Since the 1980s, the National Association of State Mental Health Program Directors Research Institute, Inc. (NRI) has collaborated with the Center for Mental Health Services (CMHS) of the Substance Abuse and Mental Health Services Administration (SAMHSA) and SMHAs to collect data on an ever-expanding range of system, utilization, financing, policy, and practice domains. These computerized databases are designed primarily for use by officials in monitoring service delivery. But can they be used for research? This column addresses the research potential of these data sets. The analysis was conducted as part of a larger project designed to increase the use of national and state administrative data sets for mental health policy analyses.

NRI data sets

We focused on three data sets: the SMHA-Controlled Revenues and Expenditures Study (R&E), the State Mental Health Agency Profiles System (Profiles), and the Uniform Reporting System of the Mental Health Block Grant (URS). NRI personnel provided us with general descriptions of the data sets and explained their histories, overall structures, and intended purposes. Documentation included descriptions of collection procedures and categories of variables. The stance of a "research user" was adopted to examine the potential utility of sample data sets as well as the data currently available online. Domains examined included scope, quality, and value to the research community. Scope refers to units of analysis, time periods, and substantive content areas. Quality includes completeness, accessibility, and analyzability. Value refers to purposes for which the data set might be used.

To illustrate patterns of states' data reporting and variation in those data across the three data sets, we compiled sample data on forensic services from the data available online for each data set. Because of space limitations the table is posted in an online supplement to this column at ps.psychiatryonline.org . The table lists some of the forensic variables addressed in each data set. The number of states contributing data for each variable is reported, along with the U.S. total for each variable (calculated from the state data provided), and the mean, median, and range as appropriate.

For example, the R&E data show that 44 SMHAs responded in 2006 to an item on total SMHA expenditures for forensic services provided in state hospitals. The U.S. total was calculated to be $2.49 billion, with a state mean expenditure of $57 million, a median of $29 million, and a range of $2 million to $686 million. The Profiles data indicate that 36 states reported the total number of adults held on forensic charges who were discharged from state psychiatric hospitals in 2007. The U.S. total was calculated as 13,975, with a mean of 388, a median of 180, and a range from 16 to 1,653. URS data indicate that 50 SMHAs responded to an item about the number of adults served by SMHAs in 2007 who were in jails or other correctional facilities. The U.S. total was calculated to be 74,758, with a mean of 1,495, a median of 426, and a range of five to 34,099.

R&E data set

In terms of scope, the R&E data set is a state-level database of information on SMHAs' fiscal dynamics. In addition to online reports from 1997 to the present, annual overviews of national trends are available that could steer researchers toward important topics in SMHA financing. The R&E Study has evolved over time, adding new data elements but not eliminating previous ones. Data are submitted annually to NRI by SMHAs in four tables covering expenditures on inpatient and community-based programs and revenues from state and federal sources.

In terms of quality, documentation provided to SMHA personnel compiling the R&E data includes an overview of the study, explicit instructions for completing the forms, the table shells themselves (with examples), a glossary of terms, sample footnotes, and concise operationalization of terms. Every cycle of the R&E Study includes data from all 50 states and the District of Columbia, although not all 50 states supply information for all categories. Data from 2002–2006 are accessible online at www.nri-inc.org/projects/Profiles/Prior_RE.cfm . To access and utilize data for specific questions, analysts would need to cut, paste, and transpose columns of data from the R&E tables and transfer them to analytic software programs.

In terms of value, R&E data could serve as analytic foci themselves, as a source of fiscal variables for other state-level studies, as contextual factors in multistate person-level research, and as useful background data for identifying states with different expenditure patterns. They are also a resource for describing patterns of financing for specific populations, such as youths and forensic patients. For example, a researcher might examine relationships between community expenditures and state hospital costs, compare states' use of Medicaid or Medicare to deliver services, or assess cost savings associated with various stages of state hospital downsizing.

Profiles data set

In terms of scope, Profiles is a compilation of data describing SMHAs' organization, funding, operation, services, policies, clientele, and the statutory environment in which they operate. Data have been collected since 1992 and are updated every two years. Although some data elements have been in place for more than two decades, others are new, precluding long-term trend analyses for some factors.

In terms of quality, Profiles data are primarily qualitative, including some text describing various programmatic and other details. A particularly useful Profiles feature allows analysts to identify comparator states that are similar or different in various domains. The number of states reporting varies across different components, and in some instances significant amounts of information are missing. For example, as shown in the online table in 2007 a total of 24 states reported the total juvenile inpatient forensic census in 2007, whereas 42 reported the total adult inpatient forensic census. For researchers attempting to understand how SMHAs are managing this population, the meaning of missing data would need clarification. For example, did failure to provide the juvenile forensic census mean that the state does not provide inpatient forensic services to juveniles, or did the state simply overlook this item? Researchers would need to contact individual states, and NRI could help facilitate such contacts.

Profiles data can be accessed online through queries and in the form of reports, which are detailed summaries and a good source of background data for project planning or identifying states maintaining certain policies or practices. The survey questionnaires from 2001 to 2009 are available at www.nri-inc.org/projects/Profiles/data_forms.cfm . Online data queries from 1996 forward can be made at www.nri-inc.org/projects/Profiles/data_search.cfm . Researchers can also request selected Profiles components for specific years in electronic data files by contacting NRI staff (see below).

In terms of value, variables included in the Profiles provide a unique, multidimensional picture of SMHAs that would be virtually impossible to obtain in other ways. As analytic foci, they can provide useful contextual variables. One might examine, for example, whether availability of certain evidence-based practices or use of second-generation antipsychotic medications has reduced state hospital census or length of stay.

URS data set

In terms of scope, the URS consists of data reported by states and territories receiving mental health block grants. Data have been collected annually since fiscal year 2002. Individual-level data aggregated at the state level for each year are presented in 21 separate tables that cover the number of persons in various demographic and other categories served by the SMHA, number of inpatient admissions and lengths of stay, number of persons receiving evidence-based practices, and aggregate results of outcome measures, such as consumer survey responses, changes in social connectedness and functioning, hospital readmission rates, criminal and juvenile justice contacts, and school attendance.

In terms of quality, our review of the data sets showed that few URS data were missing for persons served, tabulated by age and gender, and for other key variables. Some elements, particularly outcome measures, have been collected only recently, whereas others remain works in progress, such as employment and criminal justice involvement, which are based on data collected directly from clients. Specially organized workgroups with expertise in specific outcomes areas are currently testing many of these measures. The URS tables and data definitions from 2002 to 2008 are available at www.nri-inc.org/projects/SDICC/urs_forms.cfm . State-level data reports, with national comparison rates, are posted on the CMHS Web site at mentalhealth.samhsa.gov/cmhs/MentalHealthStatistics . The data files are stored at NRI in Excel and SQL databases.

In terms of value, the blending of demographic and outcome data in the URS constitutes a key strength. Although it is a state-level data set, it allows comparative analyses by state and demographic subgroups of access to evidence-based practices, incidence of criminal justice involvement, satisfaction with services, and other outcome indicators. One could envision, for example, research examining whether certain demographic groups are disproportionately represented among persons who are unemployed or involved with the justice system or whether certain evidence-based practices, such as supported employment or assertive community treatment, affect levels of employment.

Recommendations

The NRI data sets are a valuable resource. They have recently been used by researchers in works published in peer-reviewed journals. For example, Hendryx ( 1 ) assessed the relationship between SMHA expenditures and 21 indicators of performance and access using URS data and data from other national sources. To examine the roles of state psychiatric hospitals in contemporary mental health systems, Fisher and colleagues ( 2 ) used sections of the Profiles describing data on utilization, closure patterns, expenditures, and other factors pertaining to these hospitals.

However, there are a number of areas in which improvements would greatly enhance the attractiveness of these data sets to the research community. First, simple upgrading of variable labeling and other data set features would be enormously helpful. Second, although the ability to access the data sets separately may be important for SMHA officials, an integrated data set incorporating all data elements into a state-level data set would be most useful for researchers. The additional development of "state-period" data sets in which all data for each state are included for the years available would also be attractive to researchers analyzing changes in SMHA operations and outcomes, particularly when examining the effects of lagged variables. Finally, creating versions of these data sets that could be easily downloaded to statistical software or made available as SAS or other such files would go far toward increasing access for the research community.

Conclusions

These data sets offer a unique window into the operations and utilization patterns of America's public mental health systems. Advantages are the comprehensiveness and relevance of domains, a high degree of completeness for the R&E and URS data sets (less for various components of Profiles), consistency in data definitions over time, and the integrity of the data collection process. Disadvantages include challenges in accessing and using these data sets. Because they were not designed as research data sets they lack many user-friendly features, such as mnemonic variable names and labels or formatting that facilitate downloading to statistical software.

SMHAs are evolving in ways that offer interesting and important research opportunities but at a time when research support is diminishing and extensive data collection is less feasible. The low cost and potentially high return for researchers afforded by these data make them an important resource to be explored. For information on accessing these data sets, contact Ted Lutterman at [email protected] or 703-682-9463.

Acknowledgments and disclosures

This work was supported by contract HHSN271200700021C from the National Institute of Mental Health to NRI.

The authors report no competing interests.