The crisis in health care, fueled by rising costs, has led to changing views of the nation's health mission. Included in the redefinition is the important role of outcomes assessment (1,2,3,4,5,6,7). Policy planners, employers, the managed care industry, third-party payers, and the federal government are demanding to know what they are paying for.

Third-party payers and managed care companies are insisting that these "objective" measures of outcomes be incorporated in clinical care, asserting that traditional nonstandardized clinical assessments are no longer sufficient (3,5,8,9). The concept of clinical outcomes in psychiatric interventions (7,10,11,12,7), as with the rest of medical practice, has been broadened to include, in addition to symptoms, health-related quality of life, patient satisfaction with care, cost-effectiveness, and the ability to return to work with acceptable productivity (7,10,14,15,16,17).

The usefulness of patient-based health status assessments depends largely on feasibly incorporating their administration, scoring, interpretation, and use into routine care. Even if patients find self-administration acceptable, an additional administrative burden is placed on clinic staff. For efficient integration, existing technology allows clinics to automate data collection and reporting at reasonable costs (1,7,18).

Clinicians are now challenged to understand the benefit of the information with its regular use. Despite progress in collecting health status information, the large number of available clinician- and patient-completed outcomes assessments has left many clinicians ambivalent about their usefulness (19). In addition, time constraints and administrative burdens are often associated with the use of outcome measures. Many decisions need to be made when choosing a particular tool, including whether to use patient self-reports or clinician interviews, whether to include generic or disease-specific measures, and what other purposes assessments can serve.

Concerns have been raised about the validity of psychiatric patients' self-reports. However, clinician-administered interviews also have their deficiencies. Measurement validity may be assumed, but unless clinicians are trained, the interview data may be unreliable between clinicians. These instruments are also time consuming and expensive and are seldom used outside of academic centers.

Patient self-assessment questionnaires are reported to be practical and easy to use and pose little intrusion on the system. Successful implementation of outcomes monitoring requires a brief measure that is easily completed by patients. Available patient self-report measures achieve the best compromise between psychometric elegance and minimal burden for the respondent. Although disease-specific measures may be more informative for monitoring symptoms, they are not cost-effective when results are applied beyond the disease group. On the other hand, generic measures allow comparisons of emotionally impaired persons, using a common health metric, with both well individuals and patients who have physical disorders. We hypothesized that comparison between different diagnostic groups would foster better acceptance by clinicians and better understanding of differences in scale scores.

Despite concerns that emotional impairments decrease a person's ability to report on health reliably, we sought to test and implement the Medical Outcomes Study Short-Form-36 Health Survey (SF-36) (20) in a psychiatric outpatient clinic in a general hospital. The SF-36 is a general health status self-report measure that assesses eight domains: physical functioning, role functioning due to physical problems (physical role), bodily pain, general health, vitality, social functioning, role functioning due to emotional problems (emotional role), and mental health. Each domain is scored on a scale from 0 to 100, with a score of 100 indicating best health.

The SF-36 was selected because of its demonstrated reliability and validity in a variety of disease groups, as well as in the general population. It has also proven to be useful in estimating the relative health burden of different conditions, including mental disorders, and in assessing the impact of associated treatments (21,22,23,24,25,26,27,28,29,30,31). In addition, because the SF-36 is a widely used generic instrument, numerous comparisons may be made using published documentation support and interpretation guidelines for both the individual scales (32) and two summary scores. One summary score is the mental component summary, and the other is the physical component summary (33).

Our ultimate goal was to use outcomes tools as part of routine monitoring. However, conducting patient self-assessments of health status was—and continues to be—new to most patients, clinicians, and clinic staff, and it distinguished itself as a unique task. The three goals of this study were to determine whether it is practical to routinely administer a generic health status instrument in a busy outpatient psychiatry practice; to confirm that the SF-36 provides reliable, valid, and usable data when completed by patients referred to an outpatient psychiatry setting; and to present preliminary data to corroborate reports of its usefulness as an additional monitoring tool in patient care (34,35,36).

Patients

All patients age 17 and older who presented to the outpatient psychiatry department at New England Medical Center beginning in April 1994, who could read English, and who agreed to complete the instrument completed the SF-36 as part of routine care. Although this paper describes a sample defined by a particular time period, April 1994 to April 1995, the process described below is still maintained. At the time of data collection, 85 percent of all adult outpatient referrals to the clinic were to a general psychiatry clinic module. The remaining 15 percent were assigned to other specialty clinic modules, including clinics for eating disorders, cognitive disorders, developmental disabilities, and children, and these patients were not included in the sample.

In addition to data from the outpatient psychiatry department, data for group comparisons came from two sources. One source was 2,698 elective surgery patients at the New England Medical Center who completed the SF-36 during a preadmission visit (including but not limited to general surgery, cardiothoracic surgery, and other adult specialty services). This first source is referred to here as the elective surgery group. The second source of comparison data was a representative sample of 2,474 persons in the noninstitutionalized general U.S. population between the ages of 18 and 24. Data from the second source were gathered from October through December 1990 in the National Survey of Functional Health Status, which was conducted by the National Opinion Research Corporation for the functional outcomes program at the New England Medical Center. This sample is referred to here as the normative sample. Ware and associates (32) provide further discussion of this sample.

Setting

The outpatient psychiatry department at the medical center serves approximately 15,000 adults and children annually. The services offered in the clinic include diagnostic intake assessment, crisis intervention, individual and group therapy, and short- and long-term psychotherapeutic and psychopharmacologic services. At the time of data collection, no substance-abuse-specific services were offered on site. Due to the financial demands of current health care delivery systems, the structure for delivering care underwent many changes during implementation of this new project, and the project had to adapt to them.

Logistics and implementation process

Patients took between five and ten minutes to complete a two-sided multiple-choice version of the SF-36, indicating their responses by darkening ovals that were later optically scanned. Twenty-seven percent of patients (N=149) did not complete the SF-36, usually because they were seen immediately by their clinician and therefore never given the form. Other reasons included failure to bring glasses and not being able to understand or read English.

Practical and successful implementation of the assessment needed to satisfy sometimes-conflicting goals. Feasibility was the principal goal, which we evaluated using predetermined subjective and objective evidence about administration of the instrument, its psychometric properties, and costs. Examples of the criteria used to evaluate the instrument are listed in the box on this page.

As expected, the procedures for collecting, reporting, and filing data underwent several modifications over a one-year period to be acceptable to patients, administrative staff, and clinicians. With support from clinical leaders, the SF-36 was successfully incorporated into regularly administered intake packages that included demographic, clinical, and billing information forms to be completed by the patient before being seen by the clinician. Early in the implementation process, a staff member greeted all patients, introduced the SF-36, and was available for questions. The SF-36 is now fully integrated into the registration process and does not require a dedicated staff person. This change has resulted in some patients' failing to return the form and a loss of data, which affected our ability to identify diagnostic information.

Upon registering, the patient is handed the intake packet, now including the SF-36, to complete in the waiting room. The SF-36 is then returned for processing. The forms are scanned and scored, and reports are generated. Copies of reports are given to the clinician. The clinic now manages all data collection, scoring, and reporting in order to minimize the expense to 35 cents per administration and approximately one hour a week of dedicated support staff time. Automated processing requires either a Response Technology box (RT-2000) or an alternative scanning device. An RT-2000 is an optical recognition device available at a one-time cost of $4,000 to $4,500. A scoring and interpretation manual is also needed (32), which costs around $50.

Clinic staff members easily operate the RT-2000 into which the data are scanned. This device reads the patients' responses, checks data quality, scores each SF-36 scale using a programmed algorithm, and generates a report for each patient.

The SF-36 results are available for weekly clinical team meetings for discussion of diagnosis, disposition, and quality monitoring of new patients. The team briefly reviews each patient, and the psychiatrist team-leader presents the individual patient's SF-36 results confirming information gathered at intake. After review, the SF-36 results are placed in the patient's chart. Aggregate data from the RT-2000 device are downloaded as an ASCII file and used for group-level analyses.

Psychometric tests of scaling assumptions

The SF-36 scales were constructed by a method known as summated ratings, which allows maximum reliability and validity with a minimum number of questions (37). We conducted tests to substantiate that the SF-36 scales met scaling assumptions in the sample from the outpatient psychiatry department and that scores satisfied the minimum standards of reliability and validity. A multitrait analysis program (38) was used to test several assumptions, which are described in detail elsewhere (39,40). They were completeness of data, item convergent and discriminant validity, reliability of the scale scores, and features of the score distributions (39,40,41).

Validity in relation to clinical criteria

Using a method of known-groups validity (42), we assessed the validity of each SF-36 scale by comparing the scores of patients from the outpatient psychiatry department with those of the normative sample and those of the elective surgery group. The method of known-groups validity tests the ability of each of the eight SF-36 scales to discriminate between patients from the outpatient psychiatry department, the normative sample, and the elective surgery patients.

We hypothesized that the outpatient psychiatry patients would score significantly lower than the norm on the four SF-36 scales measuring mental health domains: mental health, emotional role, social functioning, and vitality. Based on the 25-year clinical experience of the senior psychiatrist, we also hypothesized that outpatient psychiatry patients would have a qualitatively and quantitatively distinct profile from that of the elective surgery patients, and that they would have higher scores on the three physical domains—physical functioning, physical role, and bodily pain—and lower scores on the mental health domains. On the basis of evidence reported in other published studies (26,27), we expected to observe lower mental health scores among psychiatric patients compared with those of the normative sample. We anticipated the physical morbidity of outpatient psychiatry patients to be comparable to that of the normative sample.

Statistical methods

Statistical analyses were conducted as described elsewhere (40,41), using SAS software (version 6.1). Descriptive statistics characterized the psychiatry patient sample, and Student's t test was used to calculate differences in scale scores between the psychiatry sample and the two other samples.

Statistical analyses were conducted to compare mean SF-36 scale scores between the two patient groups and the normative sample. Univariate linear regression analyses were conducted to estimate mean SF-36 scale scores for the outpatient psychiatry group and the elective surgery group adjusted to the age and gender of the general U.S. population. For each SF-36 scale, an F statistic was computed to test the significant differences in mean scores observed across the three groups. Pairwise t tests were used to test significant differences in scores between groups.

Inherent in the use of multidimensional measures of health status is the problem of inflated type I errors (rejecting the null hypothesis when it is true) due to multiple comparisons. To guard against this possibility, we conducted multivariate analyses of variance (MANOVAs) to test significant overall differences observed in mean SF-36 scale scores across all three groups.

Sample

A total of 411 patients completed the SF-36 between April 1, 1994, and March 31, 1995. This sample represented 73 percent of the 560 patients who completed an intake from the general clinic module during the time period, who agreed to fill out the questionnaire, and who were not prevented from doing so by the above-mentioned obstacles. Of the 411 patients in the outpatient psychiatry sample, 255 (61.2 percent) were female and 203 (49.4 percent) were 35 years old or younger. The sample characteristics are listed by DSM-IV diagnostic category in t1. Confirmed follow-up diagnoses were used if patients' diagnoses made at intake were later revised.

Psychometric analysis of scaling assumptions

The results of psychometric tests justified the use of the summated ratings method in scoring the SF-36 items as scales for patients with mental health diagnoses.

Completeness of data.

Missing value rates for the 36 items ranged from 1.6 percent to 6.5 percent—1.6 percent on the vitality scale (the item about feeling tired) and on the physical functioning scale (the item about walking more than a mile), and 6.5 percent on the social functioning scale (the item about the amount of time spent visiting with friends and relatives). The average missing value rate was 3.4 percent. These results are slightly higher than those from controlled clinical trials (41). Computable scale scores, from complete data or proration, ranged from 93.5 percent to 100 percent across the eight scales. According to predetermined algorithms that calculate the percent of inconsistent responses, questions were answered consistently 98 percent of the time.

Item convergent validity.

Each item correlated with its hypothesized scale with a correlation coefficient of .51 or higher, exceeding the minimum of .40 required for the correlation coefficient.

Item discriminant validity and scaling success rates.

The item-hypothesized score correlations were higher than the item-competing scale correlations in 100 percent of all comparisons (scaling success rates) (38).

Reliability of scale scores.

Estimates of internal consistency reliability ranged from .80 to .94, exceeding the minimum recommended level of .7 for group-level comparisons (42).

Features of score distributions.

The full range of score distributions from 0 to 100 was observed for all eight scales. Notable floor and ceiling effects occurred in the physical role and emotional role scales. These role functioning scales are known to be the coarsest of the eight scales, because responses are yes or no only. The floor and ceiling distributions in this population are similar to those observed in other populations (32,41).

Validity

Tests of empirical validity using groups of patients known to differ in psychiatric and physical morbidity demonstrated the usefulness of SF-36 scales by capturing the impact of mental illness on general health status. This impact was particularly apparent in differences in scale scores observed between the outpatient psychiatry sample and the normative sample (see t2). The differences in the mean SF-36 scale scores between the two patient samples—the outpatient psychiatry sample and the elective surgery sample—and the normative sample were statistically significant (for the elective surgery patients, MANOVA F=58.8, p<.001; for the outpatient psychiatry patients, MANOVA F=109.2, p<.001).

For patients diagnosed with depression, anxiety, co-occurring depression and anxiety, or bipolar disorder (the outpatient psychiatry sample), SF-36 scale scores were significantly lower on all eight scales than those of the normative sample after the analysis adjusted for age and gender of the normative sample (32). Lower scores indicate more impairment. As t2 shows, compared with the elective surgery patients, the psychiatry patients were more impaired on five of the eight domains: general health, vitality, social functioning, mental health, and emotional role. Also, no difference was found between the psychiatry patients and the elective surgery patients in mean scores on physical role.

In addition to the directional findings, the magnitude of the mean differences is impressive. The mental health score for psychiatry patients was 29 points lower than the score for the well normative sample, even after the analysis adjusted for age and gender. Other scales show differences in the expected directions and of comparable magnitude.

Our results demonstrate that implementing a process of administering brief health status questionnaires to patients in the waiting room of a busy psychiatry clinic is logistically feasible. The process also proved to be acceptable to most patients, administrative staff, and clinicians. This observational study was instituted as part of routine clinic administration procedures that tended to change. The unanticipated changes in intake procedures made it impossible to identify patients who refused to complete the instrument. The nonresponders remained anonymous except for patients who specifically stated obstacles to completing the form.

Our choice to use a generic health status questionnaire challenged us to prove that psychiatric patients could reliably and validly establish the extent of their illnesses on the same health continuum as those with physical disorders and healthy people. The comparison populations were chosen because of their availability and also because each provided a unique point on the generic health continuum defined by the SF-36 scales. Thus we were able to compare patients with mainly mental distress with those who had identified physical conditions and with a healthy population.

The large number of subjects our samples provided allowed us to detect small differences if they existed. However, these statistically meaningful findings should not detract from clinically meaningful differences and the magnitude of those differences. Mental health clinicians were previously unaware of poor physical functioning of their patients and found the information meaningful. In addition, the mental health patients' physical functioning scores were only slightly higher (7 points) than the low scores reported by elective surgery patients with known physical limitations.

Our results demonstrate that mental illness has an impressive impact on a person's functioning in all health domains, both physical and mental. Our findings reinforce those of Wells and colleagues (43), who found that physical, role, and social functioning scores of patients with mental illness were comparable to or lower than those of patients with many physically debilitating diseases such as heart disease and arthritis.

For the psychiatric clinicians at our clinic, the most striking discoveries were those outside the usual psychiatric formulation. Whereas the differences between the psychiatry patients and the other groups on the mental health and vitality scales were expected, the significantly lower scores on the physical functioning, social functioning, and physical role scales provided additional information about patients' difficulties. Using the same metric to compare patients with emotional distress and patients with physical disorders magnified each group's needs. We were reminded that patients, no matter what their presenting problem, require care both physically and mentally. The intensity of the physical health burden felt by patients with emotional illnesses highlights the need to consider equal reimbursement for treatment of mental and physical conditions.

Despite the controversy surrounding the use of results from patient-based health status assessments as part of an individual patient's care (44,45), our preliminary findings revealed that providing results to clinicians effectively engaged them and educated them about their use. The use of data from patient-based health status assessments has been integrated into our usual clinical practice with little resistance. The main hurdle was fostering the idea that this information could be valuable in an individual patient's care.

Preliminary observations have shown that over time and with experience using the results, clinicians felt less threatened and assumed more ownership of the information. In addition, the use of patient-based health status assessments provided a common language among clinicians from other disciplines for discussing outcomes of psychiatric disorders.

Although we do not endorse the SF-36 as an exclusive source for monitoring individual patients, we found that clinicians were receptive to using its data to corroborate their clinical impressions. Anecdotally, a high degree of congruence was noted between patients' and clinicians' assessment of patients' symptoms. Clinical staff were also interested in the SF-36 results because they are often more clinically relevant than required utilization or financial outcome data.

In conjunction with other clinical information, results from individual patients proved useful to strengthen clinical decision making. The SF-36 provided consistently helpful information on patients' physical functioning that was frequently overlooked by mental health clinicians. Similarly, their colleagues in the dialysis and orthopedic clinics obtained information about patients' mental health problems that they were not aware of (23,33,35), indicating that both types of clinicians tended to overlook information that was not directly relevant to the type of care each provided.

The SF-36 results are part of the team discussion about patients' disposition and follow-up, although the regular use of outcomes results is preliminary in the clinic at this time. Discrepancies between clinical impressions and scores are discussed. Clinicians seem to use the scores as a shorthand method of objectively assessing their impressions across all health domains. Further, the clinicians use the SF-36 results to monitor patients' perceptions over time of both physical and mental health states.

To simplify interpretation of results, we sought to provide a reference point, because clinicians learn to judge clinical results against a norm or a standard similar to reference ranges for laboratory tests. One helpful reference is the cutoff of 52 on the mental health scale. A score below the cutoff indicates a high probability of clinical depression or other psychiatric disorder, and the cutoff gave the clinicians a starting point for using data unfamiliar to them. Although the use of 52 as a cutoff is debatable (45,46,47), particularly at an individual patient level, it provides a concrete way for providers to relate new information to their clinical impressions.

In the sample of patients from the outpatient psychiatry department, 61 percent were at or below a cutoff score of 52 on the mental health scale, which indicates a high probability of a mental health disorder. Patients scoring above the cutoff were scrutinized carefully because they were referred to the psychiatry clinic for a presumed mental health problem.

Advancing the use of patient-based health status assessments from research into clinical practice is challenging and should be tested further. Although the process of incorporating the assessments into existing and changing clinic functions was initially frustrating, the results benefited both patients and clinicians with little burden on them or the administrative staff.

Introducing an outcomes management system into routine practice requires commitment of clinical leadership. The added cost of time and dollars obligates the use of standardized instruments found to be useful in caring for patients (7). Technological advances in computer software and instrument design make the choice of appropriate items, data entry, collection, and results reporting more affordable, cost-effective, and user friendly.

The use of an equivalent metric to compare patients with emotional distress to patients with physical disorders and the normative sample is beneficial to all clinical disciplines. Using a patient's results can promote the development of and extend the art of listening to a patient. In this preliminary outcomes monitoring project, we noted that for psychiatric clinicians, the most striking discoveries were in the physical domains. These domains are not a focal point of intake interviews, but the information gained was helpful.

The ability to demonstrate success in care using objective measures of health status will be important for a health care system continually being requested or required to meet the demands of both payers and regulations, such as those from the Joint Commission on Accreditation of Healthcare Organizations and the National Committee for Quality Assurance. With the successful implementation described in this paper, we envision that the use of patient-based health status assessments will give clinicians and clinics the opportunity to further improve the quality of care and to implement quality management projects that will grow into continuous improvement systems to assist in patient care. The data may even provide additional evidence to prompt payers to set reimbursement levels for treatment of emotional disorders equivalent to those for chronic physical conditions.

The authors thank Nicole Roos, M.A., for her contributions to earlier analyses and preparation of the patient self-report data, Carol Ann Gould, B.A., for her dedication to keeping the process going in the clinic, and William H. Rogers, Ph.D., for guidance on statistical matters.

Dr. Adler is senior psychiatrist in the department of psychiatry at the New England Medical Center, Box 1007, 750 Washington Street, Boston, Massachusetts 02111 (e-mail, dadler@lifespan.org). He is also professor and Dr. Bungay is assistant professor of psychiatry at Tufts University School of Medicine in Boston. Dr. Bungay is also research scientist and Ms. Cynn is research analyst at the Health Institute at the New England Medical Center. Mr. Kosinski is senior director of analytic services at Quality Metric, Inc., in Lincoln, Rhode Island.