Variations in the Clinical Presentations of Children and Adolescents at Eight Psychiatric Hospitals

Variations in the clinical presentation of service recipients across provider hospitals make it more challenging to manage system performance and outcomes (1,2,3). If variations in patients' clinical presentation affect a hospital's performance, then analyses must control for these variations before fair comparisons can be made between provider hospitals. Studies of medical-surgical hospitals have already demonstrated that controlling for the level of medical severity across hospital types can account for large differences in length of stay and charges (3). Less work has been done on variations across psychiatric hospitals, and still less has been accomplished with children and adolescents.

Obtaining reliable information about patients' clinical characteristics and their variations across hospitals can have several benefits. They include creation of provider-specific outcome expectations, a rationale for resource allocation, identification of children who are being underserved, and creation of an opportunity for research to inform treatment matching or develop clinical pathways (2).

When comparing hospitals, Kiesler and associates (4) found evidence of differences in the type of patient treated—those with a primary diagnosis of mental disorder versus those with primary substance abuse—and in the type of hospital—general versus specialized hospitals. Patterson and colleagues (5) studied the characteristics of a single hospital sample and found a wide range of psychiatric and comorbid nonpsychiatric impairment. Other studies have demonstrated changes in patient characteristics over time (6,7). For example, Wright and coworkers (6) found significant changes between 1970 and 1980 in the level of psychiatric severity and complexity of illness of children and adolescents in their single-hospital population.

Research on patients' characteristics has examined differences at either the level of hospital type (4)—general versus specialized—or the level of the individual hospital (6). Studies show diagnostic variation, which could reflect differential reporting practices; however, no research has provided independent clinical profiles of hospitalized psychiatric patients that reflect potentially divergent treatment needs.

Developing methods to describe between-hospital variations in clinical presentations is particularly important now, given recent guidelines from the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) mandating hospital outcomes management. In addition to outcomes, the JCAHO Oryx system manual suggests that health care organizations provide "risk adjustment/stratification," defined as "a process for reducing, removing, or clarifying the influences of confounding patient factors that differ among comparison groups" (8). Clearly, knowledge about how variations in clinical presentation may influence outcomes is crucial to the reasonable and fair evaluation of hospitals. When awareness of these variations is lacking, organizations are placed in the position of integrating performance differences and potentially making allocation and planning decisions under the untested assumption that all of its providers are serving the same types of clients.

This study examined differences in the clinical presentations of patients across eight psychiatric hospitals serving children and adolescents who were wards of the state in a metropolitan county in a large Midwestern state. A reliable and valid measure of psychiatric severity, the Severity of Psychiatric Illness measure, Child and Adolescent Version (910) was completed by independent hospital screeners before each hospitalization. The measure was then used to create distinct scales measuring severity of children's internalizing and externalizing symptomatology and caregiver issues. The scales were then compared across the eight hospitals.

Methods

Setting

This study was conducted through the screening, assessment, and support service program of the Illinois child welfare system, the Illinois Department of Child and Family Services. The program was implemented in 1992 to provide mobile crisis assessment and treatment services to children in protective custody who are referred for or at risk of hospitalization. Referrals are made to program workers when a ward of the Illinois Department of Child and Family Services demonstrates a risk of self-harm or injury to others and might require hospitalization.

Program workers screen the child to assess for appropriateness of hospitalization. The attending psychiatrist is responsible for the final decision. If a child is not considered appropriate for hospitalization and the psychiatrist agrees, a variety of diversion services can then be implemented. Data from eight hospitals in one large metropolitan county were used. All eight hospitals are under contract with the Illinois Department of Child and Family Services to serve state wards and are reimbursed on a per diem basis through Medicaid.

Sample

The sample included all children and adolescents from the eight largest Cook County hospitals who were screened and hospitalized through the screening, assessment, and support service program from January 1997 through September 1997. All data were collected directly from the program workers at the end of the month in which the screening interview occurred.

The overall sample consisted of 875 children and adolescents. Of these, 370 (42.3 percent) were female and 505 (57.7 percent) were male. The mean±SD age was 13±3.4. A majority of children (718 children, or 82 percent) were African American; an additional 96 (11 percent) were Caucasian, and the remaining 61 (7 percent) were Hispanic or Asian.

Diagnoses were made by the workers in the screening, assessment, and support service program. A total of 333 children (38 percent) were diagnosed as having a primary mood disorder; 140 (16 percent) had anxiety disorders, including posttraumatic stress disorder; 158 (18 percent) had attention-deficit disorder and disruptive behavior disorders; 44 (5 percent) had adjustment disorders; 96 (11 percent) had impulse control disorders; and 53 (6 percent) had psychotic disorders. The remaining 61 children (7 percent) had a variety of other disorders.

Measures

The Childhood Severity of Psychiatric Illness (CSPI) measure, a standardized decision-support tool (2,9,10,11), was used. Completing the CSPI involves making ratings on 4-point scales, with 0 indicating no evidence and 3 severe dysfunction, for a series of 25 items. These items were identified by focus groups of various providers and by a literature review.

The items cover five dimensions: symptoms, risk factors, functioning, comorbid factors, and placement or system characteristics. An example of the rating criteria for one of the 25 variables, emotional disturbance, is presented in Table 1. All workers in the screening, assessment, and support service program attended a three-hour training session on the use and implementation of the CSPI. Interrater reliability was tested during the sessions using clinical vignettes, and it ranged from .70 to .80 using Spearman's rho. Recent reviews of program charts indicated that reliability remained strong after training; the average Spearman's rho of .67 suggested that the measure was used properly in the field.

Results

Nine variables from the CSPI were submitted to a factor analysis. These variables were chosen for their relevance to hospital-based services and a minimum of missing data. A three-factor solution that accounted for 44 percent of the variance in CSPI scores emerged. The variables that constitute the three factors are presented in Table 2. On the basis of factor loadings, the factors were called caregiver problems, externalizing symptoms, and internalizing symptoms.

The factor analysis was used to create three scales. These scales had moderate to high internal consistencies. Cronbach's alphas were .8 for caregiver problems, .7 for externalizing symptoms, and .6 for internalizing symptoms. Correlations between the scales were low to moderate, indicating statistical independence. The caregiver problems scale had a correlation of .33 with the externalizing symptoms scale, and .13 with the internalizing symptoms scale. The correlation between the externalizing symptoms and internalizing symptoms scales was .18.

Because the scales each had three variables, scale scores ranged from 0, indicating an absence of disturbance on all three CSPI variables, to 9, indicating ratings of severe disturbance on all three CSPI variables in the scale.

Scale scores were then grouped into one of three severity categories—mild, moderate, or severe. For each of the three scales, children with scores of 3 or less were categorized as having mild symptoms. Scores of 4 or 5 placed children in the moderate category, and scores greater than 5 placed children in the severe category. For instance, a child would receive a classification of mild on a scale by receiving ratings of 1 on each of the three CSPI variables in the scale, for a total score of 3. Moderate ratings on two of three CSPI variables in a scale would place a child in the moderate severity category.

Notable differences were found on the three scales for the 875 children admitted to the eight Cook County hospitals. Caregiver issues were generally mild—470 cases (68 percent) fell into the lowest severity category on this scale. Only 90 children in the sample (13 percent) had caregiver problems that were considered severe. Overall, 282 children (32 percent) had mild externalizing symptoms, and 353 children (40 percent) had severe externalizing symptoms.

On the internalizing symptoms scale, 334 children (40 percent) had mild symptoms, and 184 (22 percent) had severe symptoms.

To assess variation among hospitals, a set of chi square analyses was performed across the severity categories and the hospital sample. Table 3 shows that 13 percent of the overall sample of children had severe caregiver issues, which can be considered the base rate for this severity category. However, rates at the individual hospitals varied significantly from the base rate. For example, only 3 percent of children from hospital E had severe caregiver issues. Children who stayed at hospital D, on the other hand, were more likely to have severe caregiver issues—23 percent compared with the 13 percent base rate. The overall chi square test across hospitals and severity levels was significant for the caregiver problems scale (χ²=54.7, df=14, p<.01).

The chi square test for the externalizing symptoms scale was not significant overall. However, some variation existed between individual hospitals and the base rate for this scale. For example, as Table 3 shows, only 14 percent of children at hospital F were classified as having severe externalizing symptoms, which is only one-third of the base rate of 40 percent. Twenty-six percent of children and adolescents at hospital G had severe externalizing symptoms, which is also substantially lower than the base rate. None of the hospitals had a significantly higher proportion of children with externalizing symptoms compared with the base rate.

The chi square test for the internalizing symptoms scale across hospitals and severity levels was significant overall (χ²=49.4, df=14, p<.01). Again, several hospitals demonstrated variation from the base rate of 22 percent. Hospital C had a rate of 34 percent of patients with severe internalizing symptoms. Hospital G, with a 14 percent rate, demonstrated a different pattern. Therefore, whereas only one-sixth of the children and adolescents admitted to hospital G had severe internalizing symptoms, one-third of hospital C's caseload was made up of patients with severe internalizing symptomatology.

Discussion

This study used a measure of psychiatric severity and its mitigating factors with children and adolescents presenting to psychiatric hospitals. The study demonstrated significant clinical variation across eight hospitals in one metropolitan county. Comparisons showed that the rate of severe disturbance among children at certain hospitals varied substantially from the base rate.

This variation in clinical presentation has several implications. One important potential use of this information is as an adjustment of outcome expectations. Outcome expectations for providers who deliver services to clients with more challenging symptoms should be lower than for other providers. Previous studies have shown that variations in psychiatric symptoms and severity lead to different rates of improvement (12,13), which further justifies continued research in this area and provides a direction for future studies.

Adjusting for severity in order to control for outcome differences may also limit "creaming"—the practice of serving the easiest-to-treat clients (2). By eliminating incentives for treating less severe and complex patients, populations that are harder to treat are less likely to be underserved.

We can also use information on severity to supplement evaluations of quality and to tailor those evaluations to the unique client populations served by hospitals. For instance, hospital F appears to serve a high proportion of children with both internalizing and externalizing symptoms. Quality assurance staff at this hospital may want to ensure that adequate safety precautions are in place for children with severe internalizing symptoms. Because the correlation between the externalizing symptoms scale and the internalizing symptoms scale was low (.18), we know that patients who were rated high on one scale were generally not rated high on the other, which suggests that there is a group of children with internalizing symptoms who are distinct from, and possibly in need of protection from, a group of children with externalizing symptoms.

About one patient in four admitted to hospital D appeared to come from an unstable caregiver environment, compared with only one in 33 at hospital E. Therefore, quality improvement staff may want to focus on ensuring that hospital D has the capacity to work with high-need caregivers.

It is important to note that data were collected from independent screeners with a limited incentive to portray a biased picture of any particular hospital. Considerable criticism has been directed to the use of psychiatric diagnoses to describe the clinical needs of psychiatric patients in situations where particular diagnoses are required for reimbursement. No such conflict of interest existed in this study. It is possible that different screening, assessment, and support service programs varied in their reporting style. However, because of substantial worker crossover in these programs and in the eight hospitals, such variations in practice patterns were unlikely to influence the results of the study.

Future research should establish a link between variations in the clinical presentation of service recipients and outcomes and utilization. This information can then be used as empirical support for planning decisions and quality improvement efforts. Once this goal has been accomplished, we can begin to explore whether providers whose clients have a predominant profile serve those clients better than other providers with a more diverse case mix. The surgical literature has demonstrated that patients are more likely to survive a procedure when it is performed in a hospital where an above-average number of similar procedures are performed (14). We do not yet know whether a similar finding would be made in psychiatric treatment settings.

Conclusions

This study demonstrated that client characteristics may vary notably across hospitals. This variation has important implications for understanding and improving systems of care. Particularly in the face of performance management initiatives, improving our ability to measure, monitor, and control for clinical variations in the populations served by different hospitals represents a significant challenge to the field. Failure to understand these differences creates the potential for untoward consequences of efforts to manage performance and outcome.

Acknowledgments

The authors thank Jennifer Behr, Sharon Fleishman, Purva Rawal, Katie Schafer, and Courtney West for their assistance. Financial support for this research was provided by the Illinois Department of Child and Family Services.

The authors are affiliated with the mental health services and policy program of the department of psychiatry and behavioral sciences at Northwestern University Medical School. Send correspondence to Mr. Leon at the Institute for Health Services Research and Policy Studies, Northwestern University Medical School, 339 East Chicago Avenue, Room 717, Chicago, Illinois 60611 (e-mail, [email protected]).

Enlarge table

Enlarge table

Enlarge table