Measuring Children’s Response to Inpatient Treatment: Use of Practice-Based Evidence

With the advent of managed care and the trend toward community-based mental health treatment (1), inpatient hospitalization of children has declined (2). However, inpatient psychiatric services to manage acute risk and symptoms and provide crisis stabilization remain a crucial component of any successful behavioral health care system (3). In the current behavioral health care environment, the most common goal of inpatient hospitalization is to efficiently reduce the psychiatric distress associated with risk of danger to self or others so that providers in community settings can continue to promote patients’ long-term treatment goals.

Traditionally, acute-stay psychiatric hospital outcomes have been measured by length of stay in the hospital (4–7). Research that has attempted to account for variation in length of stay has had somewhat mixed success. However, the literature on length of stay has been successful in providing a window to evaluate the functioning of a range of inpatient systems. For example, Gifford and Foster (4) found that more than half of the variance in the length of stay of children who were Medicaid recipients in Tennessee could be attributed to hospital factors, a disheartening finding that suggested that providers of care can be a major driver of inpatient service utilization and outcomes.

There are two potential drawbacks to using length of stay to examine inpatient outcomes. First, it is not a direct measure of the primary goal of psychiatric hospitalization, reduction in psychiatric acuity. In fact, acute-stay psychiatric hospitalization is the only service domain of behavioral health care that uses utilization, rather than changes in patients’ mental health status, as the most frequent method of evaluating outcomes.

The second drawback of a length-of-stay approach is that by its very definition a hospital episode must be completed before the outcome statistic is computed. This prevents the type of immediate, concurrent monitoring of the milieu that is necessary, arguably, in the current psychiatric hospital environment.

Recently, some researchers have proposed an alternative outcome measurement strategy that examines change in patient symptoms during hospitalization (8). The adult inpatient and residential treatment literature supports the use of regular measurement of psychiatric symptom acuity to monitor outcomes of psychiatric hospitalization (9,10). Brief measures of psychopathology, such as the Brief Psychiatric Rating Scale (BPRS) (11,12), are now used to measure symptoms among hospitalized patients and to identify moderators of clinical improvement during the episode of care (9).

For example, Varner and colleagues (8) supported the use of the BPRS in an adult psychiatric inpatient population because of its ability to predict clinical outcomes on the basis of early symptom scores and treatment responsiveness. They also found that clinical outcome was associated with subscale scores at intake, suggesting that inpatient treatment response is quantifiable and subject to empirical prediction. Such efforts set the stage for the use of quantitative assessment in clinical settings to support practice and refine treatment on the basis of real-time data (13).

The use of data to support and improve treatment in practice settings is an example of practice-based evidence. The goal of practice-based evidence is to measure enhancement of treatment quality in routine clinical settings (14,15), thereby promoting the external validity of the results (16). Approaches to practice-based evidence focus on measuring the outcomes of as many patients as possible—for example, by using limited or no exclusionary criteria—in actual clinical practice settings. Local practitioners use these measures to continually monitor and improve outcomes through what is termed “clinical practice improvement” (17).

The outpatient mental health services literature has made significant progress in developing and refining practice-based evidence methods, but the psychiatric inpatient literature has done much less. For example, Lambert and colleagues (18) developed a method for outpatient services that relies on practice-based-evidence that could serve as a model of for inpatient care. The method uses four color-coded feedback signals to alert practitioners when clients are not making expected progress during the episode of care. The signals were formulated on the basis of statistically derived, expected courses of improvement developed by using results from the Outpatient Questionnaire 45 (OQ45). The expected trajectories are compared with actual treatment response, and patients whose scores are far worse than expected receive a red code, suggesting that significant changes to treatment are necessary to improve the course of care. These feedback signals are used in conjunction with case review and supervision to improve the quality of client care in real time. Originally designed to improve adult psychotherapy outcomes, this outcome feedback model has also been successfully applied in youth psychotherapy (19).

Borrowing from the practice-based evidence approach of Lambert and colleagues in the outpatient treatment literature, this study aimed to utilize acuity ratings more frequently than in previous studies to provide regular monitoring of patients’ acuity status. Further, the use of growth curve modeling via hierarchical linear modeling (HLM) (20) for data analysis allowed for the development of expected improvement trajectories. Such trajectories can be used in the future to monitor patients’ clinical progress during an episode of care. This statistical technique is particularly suited for dependent data, given that acuity scores are nested within the patient.

Methods

This study was conducted at three psychiatric hospitals with child units (ages 4–12) in Illinois, Missouri, and Kentucky. The hospitals are owned by Psychiatric Solutions, Incorporated (PSI), a national, for-profit corporation specializing in inpatient psychiatric services. A pilot study was conducted between August 2009 and October 2009 to ensure study feasibility and measurement reliability. Data collection for this study commenced October 1, 2009, and ended October 1, 2010.

Participants were recruited upon intake to the hospital. Parents or legal guardians were informed that the study would measure changes in the acuity of their child’s psychiatric condition or symptoms during his or her stay on the unit in order to monitor hospital outcomes and promote quality improvement. Parents or legal guardians who agreed to participate signed an informed consent form approved by the institutional review board at Loyola University Chicago. Participants who were wards of the state received consent from the state guardian’s office. Over 95% of families agreed to participate in the study, a percentage that was consistent across the three participating hospitals.

Participants

A total of 524 children participated in the study, 227 (43%) at hospital A, 139 (26%) at hospital B, and 158 (30%) at hospital C. All children were between the ages of four and 12, and the mean±SD age was 8.92±1.64. Sixty-two percent (N=325) of the sample were males, and 38% (N=199) were females. A majority (N=414, 79%) of participants lived with their biological parents, and 110 (21%) participants were in the custody of the child welfare system. Of those in the child welfare system, 66 (60%) lived with nonrelative foster parents, 33 (30%) lived with relative foster parents, and 11 (10%) lived with their adoptive parents.

Assessment and measures

Psychiatric acuity was measured with the Acuity of Psychiatric Illness Scale, Child and Adolescent Version (CAPI) (21). The CAPI consists of 17 items rated across the following three domains: risk behaviors, for example, suicidal ideation or gesture and aggressive behavior toward others; symptoms, for example, reality assessment, anxiety, and noncompliance; and functioning, for example, nutritional status and self-care. Each item is rated on a scale of 0 to 3, with 0 indicating no evidence of acuity; 1, mild acuity; 2, moderate acuity; and 3, severe acuity. Ratings are based on the past 24 hours. For example, a score of 0 on the CAPI item for self-mutilation indicates that the child did not engage in self-mutilation over the past 24 hours beyond normative behaviors such as nail biting or drawing on the skin. At the extreme, a rating of 3 on self-mutilation indicates behavior with potentially significant medical risk, such as head banging, self-biting, or cutting.

Prior research has found that both the adult and child versions of the Acuity of Psychiatric Illness Scale are reliable and sensitive to change over brief periods (3,21). The alpha reliability of the CAPI as a composite measure of acuity was in the acceptable range (.80). Possible CAPI composite scores range from 0 to 51, with higher scores indicating more severe acuity.

The CAPI was completed by milieu therapists and psychiatric nurses at the end of each weekday shift. The clinicians based the CAPI ratings on their experience with the child that day and on a review of shift notes spanning the previous 24 hours. Staff were required to complete a two-hour training session in the use of the CAPI, which included rating practice vignettes and discussing actual cases. After being trained, staff were required to rate a mock vignette with 85% accuracy to receive certification. The overall kappa reliability across the training sessions was .80. Reliability was further ensured through ongoing chart reviews. A group consisting of the primary author and quality assurance staff rated a subsample of existing charts and provided feedback to staff about their reliability. Staff were awarded incentives of $15 gift cards for demonstrating accuracy rates of 80% or higher. A kappa reliability of .75 was maintained throughout the study.

Data analysis

CAPI scores were nested within child, creating dependencies in the data. A common approach to handling this problem involves the use of multilevel modeling strategies, such as hierarchical linear modeling (HLM). Therefore, the data were submitted to an HLM analysis that used the HLM 6.04 statistical software program. At level 1, days of care and the quadratic of days of care (to test for a curvilinear relationship between days in care and acuity scores) were entered into the model as a predictor of CAPI scores. At level 2, each child’s intercept (intake CAPI) and fitted linear (or curvilinear) slope trajectory were predicted as a function of demographic variables, clinical variables, hospital unit (hospital B and C), and interactions between the hospital and intake clinical variables.

Because the primary goal of this study was to model slopes (changes in acuity over time), when predicting intercept terms we included only demographic variables, such as age, gender, and custody status; CAPI scores at intake; and hospitals B and C. However, in predicting slope terms, we included variables that allowed for a more detailed and possibly nuanced picture of the course of acuity. For example, instead of using the CAPI composite scores at intake to predict the slope of CAPI scores, we used individual items of the CAPI that were not highly correlated (≤.30) with other items and a composite variable of several CAPI items with higher correlations with other items (>.40). The composite variable, which we labeled “externalizing behavior,” consisted of the following CAPI items: aggressive behavior toward people, aggressive behavior toward objects, impulsivity, noncompliance, and activity level—overactive.

Further, we created interaction terms between the dummy-coded hospitals (hospital B and C) and the CAPI items and externalizing behavior composite score to examine whether clinical variables at intake influenced the course of acuity depending on the hospital serving the children. Because of the number of variables tested, a relatively more conservative alpha level of .01 was chosen.

Results

The results of the HLM model are presented in Table 1. HLM generates fixed-effect beta weights as predictors of the intercept (child’s psychiatric acuity at intake) and slope (rate of change in child’s acuity throughout treatment). Not surprisingly, higher initial CAPI scores were significantly associated with higher intercept terms (estimated CAPI scores at intake). Hospital B was significantly associated with intercepts, such that children who entered treatment at hospital B had higher intake CAPI scores than children at hospital A.

Table 1 also presents results of predictors of CAPI slopes. The following intake CAPI variables were associated with the course of acuity throughout the hospital episode: externalizing behavior composite score, self-mutilation, suicidal ideation or gesture, anxiety, peer functioning, and sexually aggressive behavior. Higher scores at intake for externalizing behavior, self-mutilation, anxiety, and sexually aggressive behavior were associated with more rapid decreases in CAPI slope trajectories. The score for suicidal ideation or gesture was associated with less rapid decreases in CAPI acuity slope trajectories.

Several interaction terms involving hospital B and hospital C were significant predictors of slope trajectories, suggesting that a nuanced picture emerges when one examines the role of hospitals in outcome analyses. For example, the interaction term involving hospital B and hospital C and suicidal ideation or gesture suggests that youths with higher rates of suicidal ideation and behaviors experienced relatively faster rates of recovery at hospitals B and C than at hospital A. However, youths with more anxiety had relatively slower responses at hospital B than at hospital A. Youths with more sexually aggressive behavior had slower responses at hospital C than at hospital A.

Discussion

This study examined implementation of a daily measure of psychiatric acuity on three psychiatric hospital units. The study had two aims: to monitor the progress of children on the milieu and to examine the use of psychiatric acuity as a potential measure of patient outcome.

Because HLM was applied to the nested data, the results offer initial support for the goal of monitoring patient progress by changes in acuity over the course of inpatient treatment. Further, although this study was limited in that it was conducted on only three child inpatient units, some evidence was found suggesting that hospitals can vary in the average rate of change in acuity of illness of children under their care. This finding suggested that the approach described here might be applied on a larger scale to examine provider-level outcomes at more hospital units and systems.

However, a more detailed analysis of the data suggested that specific dimensions of psychiatric acuity at intake predict better or worse outcomes for children depending on where they are treated, suggesting that hospital units may have relative core competencies and deficits. From a practice-based-evidence perspective, the results of the data could be used as feedback to help hospitals maximize and expand upon their strengths and improve in areas where they are not as strong as other hospitals. Further, the fact that many of the interactions between the CAPI and the three hospitals involved individual CAPI items might allow for targeted discussions about the efficacy of the milieu in treating children with specific clinical presentations. These discussions would be consistent with a Communimetrics approach to the consumption of evaluation data (22).

Figure 1 presents an example of how the data from the approach described in this study might be used to improve the care of an individual client. Because the child entered care with greater acuity scores, the model predicted a relatively more rapid daily rate of reduction in acuity. However, over the first seven days in care, the child experienced a steady and consistent increase in acuity, followed by a decrease in acuity beginning on the sixth day of care. If this unit had been using a color-coded system similar to the Lambert and colleagues’ model (18), this particular patient would likely receive a red warning code, indicating risk of sustained problems unless changes were made to treatment or milieu functioning. In fact, anecdotal evidence from the staff indicated that the social dynamics on the unit were not favorable for this child, and as a result the child experienced sustained psychiatric distress. Once these dynamics changed, the child experienced a steady reduction in acuity until discharge.

This study was limited in that it evaluated only three hospitals. Future work should study enough hospitals to include hospital as a third level in the HLM analyses, allowing for a random-effects analysis of variability in outcomes at the level of the hospital. Further, this study employed only one measure of outcome, the CAPI, and it was based on the perspective of frontline staff and therapists and not on the range of mental health professionals—for example, psychiatrists—involved in care on a typical unit. An ideal study would involve multiple informants, use more than one measure of outcome, and capture treatment variables possibly relevant to patients' acuity status, such as family functioning as observed during visits, response to pharmacological treatment, and the organizational climate and culture of the unit.

It is also important to note that the CAPI is a composite, broadband measure of psychiatric acuity, and scores are not meant to indicate a child's readiness to be discharged or his or her current level of risk in the community, a decision that should obviously be made by the physician in consultation with the treatment team.

Finally, the literature in general still does not use a consistent conceptual framework for studying and discussing the effectiveness of inpatient psychiatric care. Variables ranging from length of stay, readmission, and structural changes in family functioning all have been labeled “outcomes” in the inpatient literature, when in fact they are used to support the empirical study of very different goals pertaining to the use of inpatient care in the current behavioral health care environment. In this study, our aim was to assess changes in acuity as a measure of response to treatment, given the assumption that effective inpatient care more quickly addresses and ameliorates the distress that led the patient to become a risk to self or others.

Conclusions

Overall, this study suggested that a daily measure of psychiatric acuity can be implemented at multiple hospitals and may be suitable for use on a larger scale such as a statewide hospital system. These data may also be used during the episode of care in actual practice settings to provide real-time monitoring of children’s progress. Further, by measuring actual changes in acuity instead of service utilization, as measured by length of stay, measurement of psychiatric acuity may on its face be a more valid approach to outcomes management. Future research should study a larger number of hospitals and use a multi-informant, multimethod approach. Researchers should also seek to study in greater detail the characteristics of the various hospitals that are associated with more optimal outcomes as a way to develop and share practice-based evidence that supports clinical technologies.