A Meta-Analysis of the Effectiveness of Mental Health Case Management Over 20 Years
Abstract
OBJECTIVE: Meta-analytical methods were used to investigate the effectiveness of case management and to compare outcomes for assertive community treatment and clinical case management. METHODS: Controlled studies of case management published between 1980 and 1998 were identified from reviews and through database searches. The results were quantitatively combined and compared with results of studies of mental health services without case management. Combined effect sizes and significance levels for 12 outcome domains were calculated. Analysis of homogeneity was used to explore differences between models. RESULTS: Forty-four studies were analyzed; 35 compared assertive community treatment or clinical case management with usual treatment, and nine directly compared assertive community treatment with clinical case management. Both types of case management were more effective than usual treatment in three outcome domains: family burden, family satisfaction with services, and cost of care. The total number of admissions and the proportion of clients hospitalized were reduced in assertive community treatment programs and increased in clinical case management programs. In both programs the number of hospital days used was reduced, but assertive community treatment was significantly more effective. Although clients in clinical case management had more admissions than those in usual treatment, the admissions were shorter, which reduced the total number of hospital days. The two types of case management were equally effective in reducing symptoms, increasing clients' contacts with services, reducing dropout rates, improving social functioning, and increasing clients' satisfaction. CONCLUSIONS: Both types of case management led to small to moderate improvements in the effectiveness of mental health services. Assertive community treatment had some demonstrable advantages over clinical case management in reducing hospitalization.
According to Thornicroft (1), case management is the "coordination, integration and allocation of individualized care within limited resources," which includes ongoing contact with one or more identified key personnel. Case management has been implemented in many mental health services in recent years in an attempt to overcome deficiencies in community care, particularly those due to fragmented service systems and lack of continuity of care. Numerous reviews of studies of the effectiveness of case management in mental health services have been conducted (2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22), and the majority have indicated that case management improves outcomes for clients.
However, a review by Marshall and colleagues (18) found that case management "approximately doubles the number of hospital admissions with little evidence of causing an improvement in mental state, social functioning or quality of life." This finding challenged the prevailing orthodoxy and called the effectiveness of case management into question. The review was conducted as part of the Cochrane collaboration, a network of researchers dedicated to "systematically reviewing the effects of healthcare within their areas of interest" (23). The collaboration is widely regarded as one of the best sources of evidence about the effectiveness of health interventions. Hence these findings must be regarded seriously.
Case management has been defined in various ways. Solomon (8), for example, distinguished four types of case management: assertive community treatment, strengths case management, rehabilitation, and generalist case management. Mueser and colleagues (22) described six models: broker case management, clinical case management, strengths case management, rehabilitation case management, assertive community treatment, and intensive case management. They pointed out that the models could be grouped into three broad types, although they acknowledged that "the differences between models within each of these broad types of community care can be difficult to establish."
Rather than describing discrete models, Thornicroft (1) proposed 12 dimensions that could be used to distinguish case management programs. Thus there appears to be little consensus about the best way to specify models of case management.
A commonly articulated conceptualization of case management distinguishes between assertive community treatment and other forms of case management (7,8,17,18,19,21). For example, Marshall and colleagues (18,19) have distinguished assertive community treatment from other models of case management on several dimensions, including lower caseloads, team rather than individual case management, an emphasis on outreach, and an orientation to the teams' providing as many services as possible rather than referring clients to other providers. Arguably, assertive community treatment has some elements in common with other forms of case management as well as some unique features. It is therefore reasonable to investigate both the common and the specific effects on outcomes of these different approaches.
Most previous reviews have been narrative accounts that compare the number of studies finding positive results to those finding negative results in order to determine overall trends. However, as Glass and associates (24) and others have shown, such methods can lead to inaccurate conclusions. There have been four attempts to combine results quantitatively using meta-analytic methods (13,18,19,21). Bond and colleagues (13) examined studies of the implementation at nine sites of the Thresholds Bridge program, which is based on the assertive community treatment approach. Gorey and associates (21) used a meta-analytic approach to evaluate the effectiveness of case management more broadly. However, both analyses included controlled studies as well as studies that used a baseline-versus-intervention design. The latter design confounds change resulting from the intervention with change occurring over time resulting from factors not related to the intervention.
The meta-analytic studies of Marshall and colleagues (18,19) analyzed assertive community treatment and other models of case management separately. The meta-analysis of the effectiveness of assertive community treatment found that clients in these programs were more likely than clients receiving usual treatment to remain in contact with services and less likely to be hospitalized; assertive community treatment clients spent less time in the hospital and had better outcomes on housing status, employment, and satisfaction with services (19). For other models of case management, Marshall and colleagues were able to reach conclusions for only two domains of outcome; they found that a greater proportion of case management clients were hospitalized (reported also as an increase in total admissions among case management clients) and that case management clients were less likely to drop out of mental health services (18).
To compare the effectiveness of assertive community treatment directly with generic or clinical case management, Marshall and colleagues (19) examined only those studies in which clients were randomly assigned to one model or another. However, because only six trials met this criterion, firm conclusions could not be drawn.
We decided to replicate the approach of Marshall and colleagues, but we broadened the inclusion criteria so that more studies and outcome domains could be included. One approach to comparing the effectiveness of assertive community treatment with clinical case management is to examine studies that directly compare outcomes for these two models—the within-study approach. However, Marshall and colleagues found only six such studies. An alternative is to examine the relative strength of the effectiveness of each model compared with usual treatment. This approach entails examining whether, for example, assertive community treatment is more effective than usual treatment on some domain of outcome than is clinical case management. Although this method is less strictly controlled, more studies can be included.
We also decided to include studies using measures for which reliability and validity data had not been previously published, rather than limit the analysis to studies using only established measures. Although the inclusion of some unreliable measures will attenuate the estimated average effect size, it may nonetheless increase the power of the analysis to determine whether there is any effect at all. The combination of these methods allowed us to conduct a broader examination of the effectiveness of case management as called for by Parker (25) and extended the range of outcomes subjected to meta-analysis.
The purpose of this study was to investigate the effectiveness of case management compared with standard community care without case management. We conducted a meta-analysis of the results of controlled studies of the effectiveness of case management. A second aim of the study was to compare outcomes of assertive community treatment and clinical case management.
Methods
Literature search
The analysis included studies of outcomes of case management in mental health services published between 1980 and 1998 in refereed journals. Studies were included if their focus was on the treatment of adults with serious mental illness such as psychosis, affective disorders, personality disorders, or anxiety disorders. Studies in which subjects had a diagnosis of a substance use disorder were included if subjects also had another diagnosis; however, studies in which subjects' sole or primary diagnosis was a substance use disorder were not included.
To be included in the analysis, studies also had to compare outcomes of a group receiving case management with those of a group receiving standard community care but not case management, or they had to compare outcomes of a group receiving assertive community treatment with those of a group receiving another form of case management. The studies included in the analysis had dependent variables that were measures of client outcome, such as hospitalization, quality of life, satisfaction, and level of community functioning.
The first sampling strategy was to examine studies cited in previous reviews of case management in mental health services (2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22) that met the criteria outlined above. Second, MEDLINE and PsycLIT searches were conducted using the terms case management, care management, care programming, or assertive community treatment and mental health, psychiatric, or psychiatry and evaluation, outcome, comparison, or effect.
The combination of these two methods yielded approximately 180 articles in English-language journals. Some of these articles reported comparisons of clients in two or more types of case management with clients in control groups. The results of other studies were reported in several articles, but they are included here as a single article. For our analysis, each comparison was coded and included as a separate study when possible; however, studies for which different outcomes were reported in more than one article were included only once. Studies comparing assertive community treatment and another model of case management were included and analyzed separately.
Measures
Each study was coded for client characteristics as well as several aspects of the study design, including sample size, study period, number of outcome measures used, attrition rates, and the method of assigning subjects to treatment groups. Each study was categorized by an estimate of research quality using criteria similar to those used by Glass and associates (24). These categories were random assignment to conditions with attrition less than 20 percent (high rating); random assignment with attrition greater than 20 percent or differing between groups (medium-high rating); well-designed matching studies or analysis for covariance (medium-low rating); and weak or nonexistent matching procedures (low rating).
Standardized measures of outcome were calculated for each of 12 domains. The domains were improvement in symptoms (excluding the Global Assessment of Functioning and Global Assessment scales), which include both symptoms and level of social functioning; number of hospital admissions; length of hospital stay; proportion of clients hospitalized; contacts with mental health services; contacts with other services; dropout rates from mental health services; level of social functioning, measured as quality of life rated by clinicians and clients on the basis of clients' level of social functioning and improvement in their housing situation; clients' satisfaction; family members' satisfaction; family burden of care; and cost of services.
Statistical analysis
Two methods were used to calculate standardized measures for each domain of outcome—the Pearson product-moment correlation coefficient (r) and the one-tailed level of significance (p). Although several measures of effect size are in use, we regarded r as being a more well-known and interpretable statistic than other effect size measures. Correlation coefficients can be derived from other statistics such as t values, means and standard deviations, and one-factor F values (26). The META computer program (27) was used to calculate r from reported statistics.
When possible, the one-tailed p value was also calculated for each outcome measure independently of whether the actual r could be calculated. One-tailed p values were obtained by halving the reported two-tailed probability if it favored case management and by subtracting the halved two-tailed value from one if it favored the control group. For example, a two-tailed p of .05 in favor of case management was coded as .025, and a two-tailed p of .05 in favor of the control group was coded as .975.
If studies simply reported the result as not significant, p was coded as .5 (26). When a study reported more than one result in domains that were combined, such as two different symptom measures, the mean p or r was calculated (26). All r and p values were calculated by the first author, and after all studies had been coded, each r or p value was recalculated and any inconsistencies resolved. For relevant outcome domains, we recorded whether the measure used in the original study had previously been reported in peer-reviewed journals. Missing data were excluded from the analysis.
The weighted mean r for each outcome domain was calculated by converting each r to a standard normal deviate (Fisher's z) and weighting each z by the study's sample size and research quality (26,28,29). The 95 percent confidence interval was calculated for each weighted mean r (28). As a separate analysis, the combined p value was calculated for each outcome measure by calculating the standard normal deviate for each p reported and weighting by sample size and research quality (26). To investigate the impact of possible publication bias, we calculated Rosenthal's fail-safe N for each combined p (26) and calculated the regression asymmetry test for publication bias first suggested by Egger and associates (30,31). The fail-safe N is an estimate of the number of studies with nonsignificant results that would have to be added to the sample in order to change the combined p from significant (at .05) to nonsignificant. Egger and associates' asymmetry test is a formal statistical test using relative effect size and sample size from each study to detect whether effect sizes are biased.
Homogeneity analysis techniques devised by Hedges and Olkin (29) were used to compare the effect sizes between groups. This technique, which is based on effect sizes, determines whether variance within and between groups is significantly greater than would be expected by chance. Q values can be calculated for the heterogeneity that exists within groups (Qwithin) and between groups (Qbetween). If there is a real difference in outcomes between groups, Qbetween will be significant on the basis of the chi square distribution (29).
To compare outcomes of assertive community treatment and generic or clinical case management, two techniques were used. First, the combined p and weighted mean effect size were calculated for studies that directly compared these programs. Because the number of such studies was small (nine), we also analyzed all studies comparing case management with a control group by calculating Qbetween to compare differences between outcomes of assertive community treatment and clinical case management.
Results
Selection of studies
Table 1 lists the 44 studies and identifies the outcome domains included in the meta-analysis (32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49, 50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85, 86, 87, 88, 89, 90, 91). Thirty-five comparisons of either assertive community treatment or another model of case management with usual treatment were found that met the criteria for inclusion. These comparisons were reported in 50 separate articles (32,33,34,35,36,37,38,39,40,41,42,43,42,43,45,47,48,50,51,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74, 76,77,78,79,82,86,87,88,89,90,91). Some articles appear in the table more than once because they report the results of separate comparisons. Nine studies were found that directly compared assertive community treatment with another model of case management. These comparisons were reported in 11 separate articles (44,46,49,5274,75,80,81,83,84,85), which are shown in Table 1 as assertive community treatment versus clinical case management.
Six studies that met the inclusion criteria were not included in this analysis. One used criminal justice contacts as the only outcome measure, and this domain was not included in our study (92). In another study the case management and control groups received services for different amounts of time (93). In the other studies (94,95,96,97,98,99,100), neither effect sizes nor p values could be retrieved.
The type of case management was recorded according to the definition used by the authors of the report. Initially we used Solomon's typology (8) of assertive community treatment, strengths case management, rehabilitation, and generalist case management models to record types of case management. Nineteen studies compared assertive community treatment with usual treatment (32,33,34,35,36,37,38,41,45,48,50,51,55,56,62,64,65,71,74,77,78,79,82,86,87,89,90,91). Sixteen studies compared another model of case management with usual treatment—one with a strengths model (63), one with a rehabilitation model (42,43), one with a generalist model (40)—and 13 other studies that could not be further classified (39,47,53,54,57,58,59,60,61,66,67,68,69,70,72,73,74,76,88).
We refer to these other models as "clinical case management." They all included a single person responsible for conducting needs assessment, developing individual plans, coordinating access to needed services, and monitoring mental state and social functioning. In addition, most emphasized the importance of the case manager's establishing a therapeutic relationship and providing ongoing care rather than acting merely as an administrative service broker. Thus the models described correspond to Kanter's description of clinical case management (101), although they also seem to share features of the strengths and rehabilitation models described by Mueser and colleagues (22).
The assertive community treatment programs in the studies we analyzed provided intensive support and many of the case management functions listed above, but they differed in several ways: they operated with teams of two or more that were responsible for each client, they had lower caseloads, and they often—but not always—provided more services from within the program rather than referring clients to other services. In 14 of the 19 assertive community treatment programs (74 percent), case managers had caseloads of ten to 19 clients. In the other five programs (26 percent), case managers had fewer than ten clients.
In nine of the 16 clinical case management programs (56 percent), case managers had caseloads of ten to 19 clients; in the other seven clinical case management programs (44 percent), case managers had caseloads of 20 or more clients. Usual treatment was generally provided via outpatient visits to a community mental health facility.
Client characteristics
A total of 6,365 clients were included in the 35 studies comparing case management with usual treatment. Eighty-three percent (N=5,283) were single, which included the categories of never married, widowed, and divorced. Fifty-six percent of the clients (N=3,564) were male. The mean±SD age was 37±5 years. The mean±SD number of previous psychiatric admissions was 6.6±2).
In the 19 studies that reported DSM axis I diagnoses for all clients, 61.6 percent of clients (N=2,023) were diagnosed as having schizophrenia, 7.8 percent (N=257) had bipolar affective disorder, 9 percent (N=297) had another psychotic disorder, 11.4 percent (N=376) had depression, 2.1 percent (N=69) had neurosis, and 8.1 percent (N=267) had another diagnosis. No statistically significant differences between case management and control groups were found for any of the demographic variables or for diagnosis. However, information about previous admissions was unavailable in 25 of the 35 studies (71 percent) because admissions were reported in a format that did not allow comparisons across studies.
Study characteristics
Of the 35 studies comparing case management with usual treatment, 29 (83 percent) employed control groups to which clients were randomly assigned. The mean±SD length of the study period, which was defined as the length of time during which the two groups received different services until the final measure for comparison was recorded, was 16.5±6.7 months. The mean±SD attrition rate was 15.9±9.3 percent for case management groups and 23.4±10.9 percent for control groups. The median size of the study samples varied considerably across outcome measures, from a maximum of 121 clients for the proportion of the group hospitalized to a minimum of 32 for family satisfaction.
Outcomes
The results of the comparison of case management with usual treatment are presented in Table 2, which shows each outcome domain by the number of studies contributing p values, the combined one-tailed p, Rosenthal's fail-safe N, the number of studies contributing r values, and the weighted mean r and its associated 95 percent confidence interval. Case management programs were more effective than usual treatment for most types of outcome, measured both by a combined p of .05 or less and by the probability that r is significantly greater than zero at the 95 percent confidence level.
Specifically, compared with usual treatment, case management was associated with greater improvement in symptoms, fewer hospital days used, a smaller proportion of clients hospitalized, more contacts with both mental health and other services, lower dropout rates from mental health services, greater improvements in level of social functioning, greater client and family satisfaction with care, less family burden of care, and lower total cost of care. Clients of case management programs were admitted to the hospital more frequently than clients receiving usual treatment.
Publication bias
Although most of the relationships were highly statistically significant, it is possible that because of publication bias, studies finding nonsignificant results were not published and thus could not be included in this analysis. Rosenthal's fail-safe N is shown in Table 2. There are no objective criteria by which to judge when the N is large enough to ensure confidence in the validity of the results; however, Rosenthal (26) suggested that a fail-safe N greater than or equal to five times the number of comparisons plus ten means that the results could be considered "robust." By this criterion, six domains could be considered robust against publication bias on the basis of reported p values: fewer hospital days used, a smaller proportion of clients hospitalized, more contacts with mental health services, lower dropout rates, greater improvement in social functioning, and increased client satisfaction.
Egger's test for publication bias (30,31) was calculated for each outcome domain except contacts with other services, family burden of care, and family satisfaction, which had sample sizes too small to plot. Three of the remaining nine domains showed some evidence of publication bias: the proportion of clients hospitalized (p=.015), contacts with mental health services (p=.017), and client satisfaction (p=.05). These results differ slightly from those for the fail-safe N because they are based on effect sizes rather than p values.
Study quality
The Qtotal values and associated level of significance are shown in Table 2. Nine domains had variance greater than that which would be expected by chance. We tested whether the quality of the research design was associated with different outcomes for these measures by collapsing the four categories for research quality into high-quality and low-quality categories and comparing weighted mean r values for these two groups.
Four of these nine measures showed significant differences in outcomes by study quality: number of admissions (Qbetween=14.7, p<.001), days hospitalized (Qbetween=4.6, p=.03), contacts with mental health services (Qbetween=12.2, p<.001), and level of social functioning (Qbetween=4.8, p=.028). However, the weighted mean r values for the high-quality group were almost the same as those calculated for the sample as a whole. For number of admissions, the weighted mean r for the high-quality group was -.14, compared with -.10 for the total sample; for days hospitalized, it was .26, compared with .24; for contacts with mental health services, it was .21, compared with .24; and for level of social functioning, it was .14, compared with .15. Taken together, these figures suggest that including matched-control studies and weighting for study quality increased the power of the analysis while effectively limiting the impact of lower-quality studies on the overall estimates of effect size.
We also examined the impact of using measurement instruments that had or had not been previously described in peer-reviewed journals, because this was one of the criteria used by Marshall and colleagues (18,19) to exclude some outcome domains. We compared the weighted mean r value for studies in which previously reported measures were used with those in which the measures used had not been previously reported in peer-reviewed journals. The number of studies using both reported and nonreported measures was sufficient to enable a comparison for only two domains. For both domains, the mean weighted effect sizes were significantly higher for the studies that used previously reported measures than for those in which nonreported measures were used: level of social functioning (r for reported measures, .18; r for nonreported measures, .13; Qbetween=4.41, p=.036); and client satisfaction (r for reported measures, .38; r for nonreported measures, .19; Qbetween=8.54, p=.003). Assuming that instruments not described in previous reports have lower reliability rates, the findings support our a priori assumption that these measures may tend to underestimate effect sizes.
Assertive community treatmentand clinical case management
We first examined studies that directly compared assertive community treatment programs with clinical case management programs. No differences between programs were found in previous admissions, age, the percentage of clients with psychosis, the proportion of males and females, or the percentage of single clients. All but two domains had fewer than four studies contributing effect sizes—too few to allow any firm conclusions to be drawn. Assertive community treatment was superior to clinical case management in improving social functioning (five studies, r=.18) and marginally superior in reducing the total number of days hospitalized (five studies, r=.08), although the small number of studies limit confidence in these findings.
To investigate this issue further, we compared outcomes for controlled assertive community treatment studies with outcomes for controlled studies of clinical case management. Clients in the assertive community treatment studies had more previous admissions than those in the clinical case management studies (7.4 compared with 4.7 admissions; t=2.35, df=8, p=.047); however, data were available from only nine studies. No significant differences were found in age, percentage of clients with psychosis, proportion of males and females, or percentage of single clients.
The Qtotal scores and the associated levels of significance shown in Table 2 indicate that there was no statistically significant difference in outcomes for improvement in symptoms, contacts with other services, and family burden of care. This finding suggests that on these measures, case management was effective but that assertive community treatment and clinical case management were no different; however, only two studies assessed family burden of care.
The remaining nine domains were analyzed further by comparing outcome effect sizes for assertive community treatment with those for clinical case management. Results are shown in Table 3. For the number of admissions, effect sizes could be calculated only for two studies in the assertive community treatment group, making comparison using Qbetween impossible. However, enough studies reported p values for this domain to show that assertive community treatment was effective in reducing the number of admissions, whereas clients in clinical case management had a larger number of admissions (the combined p being significant in the opposite direction).
As Table 3 shows, for days spent in the hospital, the assertive community treatment studies showed a significant positive effect according to both combined p and weighted mean r; however, clinical case management showed a significant positive effect for weighted mean r only. The weighted mean effect size for days in the hospital was significantly greater for assertive community treatment than for clinical case management.
For the proportion of clients hospitalized, the assertive community treatment studies showed a significant positive effect according to both combined p and weighted mean r, suggesting that a smaller proportion of clients in assertive community treatment than clients in usual treatment were admitted to a hospital. For the clinical case management studies, the weighted mean r and combined p values were not significant, suggesting that clinical case management did not differ from usual treatment. Assertive community treatment was significantly better than clinical case management in reducing the proportion of clients hospitalized.
As shown in Table 3, a higher frequency of contact with mental health services was found for clients in both assertive community treatment and clinical case management, according to both weighted mean r and combined p; however, the number of contacts was significantly greater for clients in clinical case management programs than for clients in assertive community treatment programs.
On the basis of weighted mean effect sizes, clients in both assertive community treatment and clinical case management had a higher level of social functioning, lower dropout rates from mental health services, and higher satisfaction; the total cost of care was lower in both types of program. No differences were found between the two types of program in clients' social functioning; however, for dropout rate, satisfaction, cost of care, and family satisfaction with care, the number of studies was too small to allow comparison using Qbetween.
Discussion
The findings of this study considerably extend our knowledge about the effectiveness of assertive community treatment and clinical case management and the differences between them. We found that clinical case management is generally effective in improving outcomes from mental health services, as measured by clients' level of social functioning, symptoms, client and family satisfaction, and family burden of care. These results directly contradict the claim of Marshall and colleagues (18) that there is little evidence that case management improves clients' mental state, social functioning, or quality of life.
We have also elaborated on Marshall's finding that clinical case management increases the proportion of clients hospitalized, by showing that it increases the total number of admissions but that it decreases the total length of stay in the hospital. This finding means that although clinical case management led to more admissions than usual treatment, these admissions were shorter, which reduced the total number of hospital days. These results suggest that the overall impact of clinical case management on hospitalization was positive, again contradicting Marshall's rather pessimistic conclusions (18).
We have replicated the findings of previous reviews in showing that assertive community treatment programs are superior to clinical case management in reducing hospitalization, both in terms of the proportion of clients admitted and the total length of stay. However, for the first time, we have shown that these two approaches appear to be equally effective in improving symptoms, improving social functioning, and increasing clients' satisfaction with services. The total cost of care was reduced in both types of case management, but different methods of calculating costs limit confidence in this finding. The results should be interpreted with caution because assertive community treatment programs may deal with a different client group than clinical case management programs. For example, clients in assertive community treatment in the studies we analyzed had a greater number of previous admissions before program entry than clients receiving clinical case management.
It is useful to consider the effect of different aspects of case management on outcomes. Assertive community treatment programs often include a specific goal to avoid or at least minimize hospitalization, and staff on these teams may be able to make decisions about admissions, while staff on other programs do not. This decision-making ability may have a major influence on hospitalization as a measure of outcome, independent of other considerations for admission such as mental state or level of social functioning.
A notable feature of the data was that average outcomes for clients in different programs varied widely. One reason for the diversity of outcomes may be the tension between monitoring and support of people with a mental illness. A greater emphasis on monitoring may result in more hospital admissions and less client satisfaction, because clients may perceive that case managers are intrusive and controlling. That clients in clinical case management programs had more admissions than those in usual treatment could be evidence of greater monitoring. However, the shorter total length of stay, together with the overall improvement in symptoms and social functioning, suggests instead that admissions may be more timely and may thus prevent the need for longer stays. Nevertheless, reconciling the monitoring and support functions remains an important dimension of case management programs.
Another reason for differences in outcomes may have been some variation in the difference between the case management programs and usual treatment. For example, some of the usual-treatment programs may have incorporated aspects of case management into their standard practices and procedures.
The main difference between this study and the study of Marshall and associates (18) is that we analyzed a greater number of studies by including matched control groups as well as randomly assigned control groups, by including outcome data regardless of whether the measure had previously been reported in a peer-reviewed journal, and by not excluding studies that used parametric methods with skewed data. It is worth considering the effects of these differences. We have demonstrated that the inclusion of matched-control studies increased the power of the analysis and enabled conclusions to be drawn for a broader range of outcomes than previously considered; this was accomplished without lower-quality studies biasing the results. We also demonstrated that studies using unpublished outcome measures tended to underestimate effect sizes but nonetheless provided evidence in favor of the effectiveness of case management.
Thus our results may in fact underestimate the effectiveness of case management. The analysis of skewed data with parametric statistics could be of concern, but simulation studies, such as those described by Sawilowsky and Blair (102), indicate that tests are robust to skewness as long as the sample sizes are reasonably large (larger than 30). That was the case for the studies included in this analysis.
Some skepticism has been expressed about the technique of meta-analysis (103). For example, some meta-analyses may not take into account the fact that studies finding nonsignificant results tend to be published less often—the "file drawer problem" (104,105). We found that fewer hospital days, more contacts with services, greater client satisfaction, and improved social functioning could be considered to be robust against the file drawer criticism.
An added strength of this meta-analysis over individual studies—even those of the highest quality—is that it includes results from a wide range of services, which allows us to be more confident that our findings can be generalized to the mental health service system as a whole. In comparison, findings from individual studies may have limited generalizability because of the specific nature of the service under examination. Another limitation of meta-analyses is that they rely on the data supplied by original studies, which may not provide enough detail about some areas. For example, the descriptions of case management programs in the studies we analyzed varied from several pages of detail to only a sentence or two.
Although the effects of case management appear to be small to medium according to the criteria advocated by Cohen (106), this finding is consistent with results of studies of many other new social programs or treatments. Citing the results of two analyses—one of 24 meta-analyses (107) and another of 36 meta-analyses (108)—Cook and associates (109) concluded that "one strong finding from various meta-analyses is that most new treatments have, at best, small to moderate effects." Similarly, a meta-analysis of the effectiveness of psychotherapy (not specifically for serious mental illness) found an effect size equivalent to r=.3 (110), which is about the midpoint of the range of effects found for case management. These findings therefore suggest that the effect of case management should neither be overstated nor dismissed.
Three important issues remain to be further explored. First, a limitation of this study was some uncertainty about the extent to which the programs described in the reports analyzed fit the definitions of assertive community treatment and clinical case management provided in the literature. The broad service context in which case management programs are based and the manner in which such programs are implemented vary considerably (111,112), which may be additional reasons for the heterogeneity of outcomes found in this study. This issue requires greater attention in evaluation studies, and the more recent phenomenon of measuring the adherence of programs to model specifications is to be welcomed (112).
Evidence from one study (112) suggests that closer adherence to the ideal assertive community treatment model in implementation of these programs results in better outcomes, although the only outcome considered in that study was hospitalization, and no control groups were used. We believe a fruitful direction for future research would be to measure case management programs on the 12 dimensions proposed by Thornicroft (1) or some adaptation of them and then examine the impact of each of these dimensions on outcomes.
Second, the effectiveness of case management may be related to other factors that may play a mediating role, such as the availability of other community resources, the client's relationship to the treating psychiatrist, staff availability, adequacy of funding for services, and so on. These factors may operate independently of adherence to an idealized assertive community treatment model or to other models and need to be investigated separately.
Third, we cannot necessarily assume that case management is equally effective for all clients; some clients may benefit and others may not. For example, some of the studies included in this meta-analysis targeted people who had co-occurring substance use problems or who were homeless. We did not explore the relative effectiveness of case management for these groups compared with other groups of clients or the impact of different models. However, these questions are obviously important, as are the issues of consumer choice of case manager and client participation in needs assessment and planning. These factors need to be explored in future research.
Conclusions
Case management brings about small to moderate improvements in the effectiveness of mental health services. Assertive community treatment has demonstrable advantages over clinical case management in reducing hospitalization. The two approaches have similar effects in improving clinical symptoms, client and family satisfaction with services, and the client's level of social functioning.
The results of our meta-analysis reinforce the view that assertive community treatment should be targeted to clients who are at the greatest risk of hospitalization and that both assertive community treatment and high-quality clinical case management should be a feature of mental health programs (84). Our findings imply that the widespread introduction of case management in mental health services has been a positive initiative.
Acknowledgments
The authors thank Mary Ann Test, Ph.D., for her suggestions for the study design and Steven Klimidis, Ph.D., and Dean McKenzie, B.A., for comments on an earlier draft of this paper. They also acknowledge the support of the Victorian transcultural psychiatry unit. This work was partly funded by a grant from the mental health branch of the Victorian Government Department of Human Services.
Mr. Ziguras is affiliated with the Brotherhood of St. Laurance, 67 Brunswick Street, Fitzroy 3065, Australia (e-mail, [email protected]). He is also with the School of Social Work at the University of Melbourne. Dr. Stuart is affiliated with department of psychiatry at the University of Melbourne. When this work was done, both authors were affiliated with the Victorian transcultural psychiatry unit at St. Vincent's Hospital in Fitzroy, Australia.
 |
Table 1. Features of studies included in the meta-analysis of the effectiveness of case management and outcome domains examined
 |
Table 2. Results of a meta-analysis of studies of the effectiveness of case management, with outcome domains measured by significance level, effect sizes, and index of homogeneity
 |
Table 3. Results of a meta-analysis of studies of the effectiveness of case management, with outcome domains measured by significance level and effect sizes, by whether the studies examined assertive community treatment or another model of case management
1. Thornicroft G: The concept of case management for long-term mental illness. International Review of Psychiatry 3:125- 132, 1991Crossref, Google Scholar
2. Torrey EF: Continuous treatment teams in the care of the chronically mentally ill. Hospital and Community Psychiatry 37:1243- 1247, 1986Abstract, Google Scholar
3. Taube CA, Morlock L, Burns BJ, et al: New directions in research on assertive community treatment. Hospital and Community Psychiatry 41:642-647, 1990Abstract, Google Scholar
4. Olfson M: Assertive community treatment: an evaluation of the experimental evidence. Hospital and Community Psychiatry 41:634-641, 1990Abstract, Google Scholar
5. Chamberlain R, Rapp CA: A decade of case management: a methodological review of outcome research. Community Mental Health Journal 27:171-188, 1991Crossref, Medline, Google Scholar
6. Holloway F: Case management for the mentally ill: looking at the evidence. International Journal of Social Psychiatry 37:2- 13, 1991Crossref, Medline, Google Scholar
7. Rubin A: Is case management effective for people with serious mental illness? A research review. Health and Social Work 17:138-50, 1992 [correction, 17:238, 1992]Google Scholar
8. Solomon P: The efficacy of case management services for severely mentally disabled clients. Community Mental Health Journal 28:163-180, 1992Crossref, Medline, Google Scholar
9. Test MA: Training in community living, in Handbook of Psychiatric Rehabilitation. Edited by Liberman RP. Needham Heights, Mass, Allyn and Bacon, 1992Google Scholar
10. Damron Rodriguez J: Case management in two long-term-care populations: a synthesis of research. Journal of Case Management 2:125-129, 1993Medline, Google Scholar
11. Dietzen LL, Bond GR: Relationship between case manager contact and outcome for frequently hospitalized psychiatric clients. Hospital and Community Psychiatry 44:839-843, 1993Abstract, Google Scholar
12. Dvoskin JA, Steadman HJ: Using intensive case management to reduce violence by mentally ill persons in the community. Hospital and Community Psychiatry 45:679- 684, 1994Abstract, Google Scholar
13. Bond GR, McGrew JH, Fekete DM: Assertive outreach for frequent users of psychiatric hospitals: a meta-analysis. Journal of Mental Health Administration 22:4-16, 1995Crossref, Medline, Google Scholar
14. Burns BJ, Santos AB: Assertive community treatments: an update of randomized trials. Psychiatric Services 46:669-675, 1995Link, Google Scholar
15. Holloway F, Oliver N, Collins E, et al: Case management: a critical review of the outcome literature. European Psychiatry 10:113-128, 1995Crossref, Medline, Google Scholar
16. Santos AB, Henggeler SW, Burns BJ, et al: Research on field-based services: models for reform in the delivery of mental health care to populations with complex clinical problems. American Journal of Psychiatry 152:1111-1123, 1995Link, Google Scholar
17. Scott JE, Dixon LB: Assertive community treatment and case management for schizophrenia. Schizophrenia Bulletin 21:657- 668, 1995Crossref, Medline, Google Scholar
18. Marshall M, Lockwood A, Green R, et al: Case Management for People With Severe Mental Disorders (a Cochrane review). Oxford, England, Update Software (for the Cochrane Library), 1998Google Scholar
19. Marshall M, Lockwood A: Assertive Community Treatment for People With Severe Mental Disorders (a Cochrane review). Oxford, England, Update Software (for the Cochrane Library), 1998Google Scholar
20. Draine J: A critical review of randomized field trials of case management for individuals with serious and persistent mental illness. Research on Social Work Practice 7:32-52, 1997Google Scholar
21. Gorey KM, Leslie DR, Morris T, et al: Effectiveness of case management with severely and persistently mentally ill people. Community Mental Health Journal 34:241-250, 1998Crossref, Medline, Google Scholar
22. Mueser KT, Bond GR, Drake RE, et al: Models of community care for severe mental illness: a review of research on case management. Schizophrenia Bulletin 24:37-74, 1998Crossref, Medline, Google Scholar
23. Mulrow CD, Oxman AD (eds): Cochrane Collaboration Handbook. Oxford, England, Update Software (for the Cochrane Library), 1997Google Scholar
24. Glass G, McGaw B, Smith M: Meta-analysis in social research. Beverly Hills, Calif, Sage, 1981Google Scholar
25. Parker G: Case management: an evidence-based review fails to make its case. Current Opinion in Psychiatry 10:261-263, 1997Crossref, Google Scholar
26. Rosenthal R: Meta-analytic procedures for social research. Newbury Park, Calif, Sage, 1984Google Scholar
27. Kenny DA: Meta-analysis: easy to answer (META), 1997. Available at http://nw3. nai.net/~dakennyGoogle Scholar
28. Cooper H: Integrating research: a guide for literature reviews. Newbury Park, Calif, Sage, 1989Google Scholar
29. Hedges L, Olkin I: Statistical methods for meta-analysis. New York, Academic Press, 1985Google Scholar
30. Egger M, Davey Smith G, Schneider M, et al: Bias in meta-analysis detected by a simple graphical test. British Medical Journal 315:629-634, 1997Crossref, Medline, Google Scholar
31. Steichen TJ: Tests for publication bias in meta-analysis. Stata Technical Bulletin 41:9-15, 1998Google Scholar
32. Stein LI, Test MA: Alternative to mental hospital treatment: I. conceptual model, treatment program, and clinical evaluation. Archives of General Psychiatry 37:392-397, 1980Crossref, Medline, Google Scholar
33. Weisbrod BA, Test MA, Stein LI: Alternative to mental hospital treatment: II. economic benefit-cost analysis. Archives of General Psychiatry 37:400-405, 1980Crossref, Medline, Google Scholar
34. Test MA, Stein LI: Alternative to mental hospital treatment: III. social cost. Archives of General Psychiatry 37:409-412, 1980Crossref, Medline, Google Scholar
35. Stein LI, Test MA: An alternative mental hospital treatment, in An Alternative to Mental Hospital Treatment. Edited by LI Stein, MA Test. New York, Plenum, 1978Google Scholar
36. Test MA, Stein LI: Training in community living: research design and results, in ibidGoogle Scholar
37. Hoult J, Reynolds I, Charbonneau-Powis M, et al: Psychiatric hospital versus community treatment: the results of a randomised trial. Australian and New Zealand Journal of Psychiatry 17:160-167, 1983Crossref, Medline, Google Scholar
38. Hoult J, Reynolds I: Schizophrenia: a comparative trial of community-oriented and hospital-oriented psychiatric care. Acta Psychiatrica Scandinavica 69:359-372, 1984Crossref, Medline, Google Scholar
39. Santiago JM, McCall-Perez F, Bachrach LJ: Integrated services for chronic mental patients: theoretical perspective and experimental results. General Hospital Psychiatry 7:309-315, 1985Crossref, Medline, Google Scholar
40. Franklin JL, Solovitz B, Mason M, et al: An evaluation of case management. American Journal of Public Health 77:674-678, 1987Crossref, Medline, Google Scholar
41. Bond GR, Miller LD, Krumwied MHA, et al: Assertive case management in three CMHCs: a controlled study. Hospital and Community Psychiatry 39:411-418, 1988Abstract, Google Scholar
42. Goering PN, Wasylenki DA, Farkas M, et al: What difference does case management make? Hospital and Community Psychiatry 39:272-276, 1988Google Scholar
43. Goering PN, Farkas M, Wasylenki D, et al: Improved functioning for case management clients. Psychosocial Rehabilitation Journal 12(1):3-17, 1988Google Scholar
44. Jerrell JM, Hu TW: Cost effectiveness of intensive clinical and case management compared with an existing system of care. Inquiry 26:224-234, 1989Medline, Google Scholar
45. Bond GR, Witheridge T, Dincin J, et al: Assertive community treatment for frequent users of psychiatric hospitals in a large city: a controlled study. American Journal of Community Psychology 18:865-891, 1990Crossref, Medline, Google Scholar
46. Bush CT, Langford MW, Rosen P, et al: Operation Outreach: intensive case management for severely psychiatrically disabled adults. Hospital and Community Psychiatry 41:647-649, 1990Abstract, Google Scholar
47. Bigelow DA, Young DJ: Effectiveness of a case management program. Community Mental Health Journal 27:115-123, 1991Crossref, Medline, Google Scholar
48. Bond GR, McDonel EC, Miller LD, et al: Assertive community treatment and reference groups: an evaluation of their effectiveness for young adults with serious mental illness and substance abuse problems. Psychosocial Rehabilitation Journal 15:31- 43, 1991Crossref, Google Scholar
49. Bond GR, Pensec M, Dietzen L, et al: Intensive case management for frequent users of psychiatric hospitals in a large city: a comparison of team and individual caseloads. Psychosocial Rehabilitation Journal 15(2):90-98, 1991Google Scholar
50. Test MA, Knoedler WH, Allness DJ: The long-term treatment of young schizophrenics in a community support program. New Directions in Mental Health Services, no 26:17-27, 1985Google Scholar
51. Test MA, Knoedler WH, Allness DJ, et al: Long-term community care through an assertive continuous treatment team, in Advances in Neuropsychiatry and Psychopharmacology: Schizophrenia Research, vol 1. Edited by Schultz CT. New York, Raven, 1991Google Scholar
52. Champney TF, Dzurec LC: Involvement in productive activities and satisfaction with living situation among severely mentally disabled adults. Hospital and Community Psychiatry 43:899-903, 1992Abstract, Google Scholar
53. Curtis JL, Millman EJ, Struening E, et al: Effect of case management on rehospitalization and utilization of ambulatory care services. Hospital and Community Psychiatry 43:895-899, 1992Abstract, Google Scholar
54. Curtis JL, Millman EJ, Struening E, et al: Deaths among former psychiatric inpatients in an outreach case management program. Psychiatric Services 47:398-402, 1996Link, Google Scholar
55. Morse GA, Calsyn RJ, Allen G, et al: Experimental comparison of the effects of three treatment programs for homeless mentally ill people. Hospital and Community Psychiatry 43:1005-1110, 1992Abstract, Google Scholar
56. Morse GA, Calsyn RJ, Allen G, et al: Helping homeless mentally ill people: what variables mediate and moderate program effects? American Journal of Community Psychology 22:661-683, 1994Google Scholar
57. Rössler W, Löffler W, Fätkenheuer B, et al: Does case management reduce the rehospitalization rate? Acta Psychiatrica Scandinavica 86:445-449, 1992Google Scholar
58. Burns T, Raftery J: Cost of schizophrenia in a randomized trial of home-based treatment. Schizophrenia Bulletin 17:407-410, 1991Crossref, Medline, Google Scholar
59. Burns T, Beadsmoore A, Bhat A, et al: A controlled trial of home based acute psychiatric services: I. clinical and social outcome. British Journal of Psychiatry 163:49-54, 1993Crossref, Medline, Google Scholar
60. Burns T, Raftery J, Beadsmoore A, et al: A controlled trial of home based acute psychiatric services: II. treatment patterns and costs. British Journal of Psychiatry 163:55-61, 1993Crossref, Medline, Google Scholar
61. Hornstra RK, Bruce Wolfe V, Sagduyu K, et al: The effect of intensive case management on hospitalization of patients with schizophrenia. Hospital and Community Psychiatry 44:844-847, 1993Abstract, Google Scholar
62. Audini B, Marks I, Lawrence R, et al: Home-based versus outpatient/inpatient care for people with serious mental illness: phase II of a controlled study. British Journal of Psychiatry 165:204-210, 1994Crossref, Medline, Google Scholar
63. Macias C, Kinney R, Farley OW, et al: The role of case management within a community support system: partnership with psychosocial rehabilitation. Community Mental Health Journal 30:323-339, 1994Crossref, Medline, Google Scholar
64. Marks IM, Connolly J, Muijen M, et al: Home-based versus hospital-based care for people with serious mental illness. British Journal of Psychiatry 165:179-194, 1994Crossref, Medline, Google Scholar
65. Knapp M, Beecham J, Koutsogeorgopoulou V, et al: Service use and costs of home-based versus hospital-based care for people with serious mental illness. British Journal of Psychiatry 165:195-203, 1994Crossref, Medline, Google Scholar
66. Muijen M, Marks I, Connolly J, et al: Home based care and standard hospital care for patients with severe mental illness: a randomized controlled trial. British Medical Journal 304:749-754, 1992Crossref, Medline, Google Scholar
67. Muijen M, Marks IM, Connolly J, et al: The daily living programme: preliminary comparison of community versus hospital-based treatment for the seriously mentally ill facing emergency admission. British Journal of Psychiatry 160:379-384, 1992Crossref, Medline, Google Scholar
68. Van-Os J, McKenzie K, Gilvarry K, et al: Home-based versus in/out-patient care for people with serious mental illness. British Journal of Psychiatry 166:543-544, 1995Crossref, Medline, Google Scholar
69. Muijen M, Cooney M, Strathdee G, et al: Community psychiatric nurse teams: intensive support versus generic care. British Journal of Psychiatry 165:211-217, 1994Crossref, Medline, Google Scholar
70. McCrone P, Beecham J, Knapp M: Community psychiatric nurse teams: cost effectiveness of intensive support versus generic care. British Journal of Psychiatry 165:218- 221, 1994Crossref, Medline, Google Scholar
71. Wood K, Anderson J: The effect on hospital admissions of psychiatric case management involving general practitioners: preliminary results. Australian and New Zealand Journal of Psychiatry 29:223-229, 1995Crossref, Medline, Google Scholar
72. Ford R, Beadsmoore A, Ryan P, et al: Providing the safety net: case management for people with a serious mental illness. Journal of Mental Health 1:91-97, 1995Crossref, Google Scholar
73. Marshall M, Lockwood A, Gath D: Social services case-management for long-term mental disorders: a randomised controlled trial. Lancet 345:409-412, 1995Crossref, Medline, Google Scholar
74. Quinlivan R, Hough R, Crowell A, et al: Service utilization and costs of care for severely mentally ill clients in an intensive case management program. Psychiatric Services 46:365-371, 1995Link, Google Scholar
75. Chandler D, Meisel J, McGowen M, et al: Client outcomes in two model capitated integrated service agencies. Psychiatric Services 47:175-180, 1996Link, Google Scholar
76. Ford R, Ryan P, Norton P, et al: Does intensive case management work? Clinical, social, and quality of life outcomes from a controlled study. Journal of Mental Health 5:361-368, 1996Crossref, Google Scholar
77. Lafave HG, de Souza HR, Gerber GJ: Assertive community treatment of severe mental illness: a Canadian experience. Psychiatric Services 47:757-759, 1996Link, Google Scholar
78. Herinckx HA, Kinney RF, Clarke GN, et al: Assertive community treatment versus usual care in engaging and retaining clients with severe mental illness. Psychiatric Services 48:1297-1306, 1997Link, Google Scholar
79. Lehman AF, Dixon LB, Kernan E, et al: A randomized trial of assertive community treatment for homeless persons with severe mental illness. Archives of General Psychiatry 54:1038-1043, 1997Crossref, Medline, Google Scholar
80. Morse GA, Calsyn RJ, Klinkenberg WD, et al: An experimental comparison of three types of case management for homeless mentally ill persons. Psychiatric Services 48:497-503, 1997Link, Google Scholar
81. Wolff N, Helminiak TW, Morse GA, et al: Cost-effectiveness evaluation of three approaches to case management for homeless mentally ill clients. American Journal of Psychiatry 154:341-348, 1997Link, Google Scholar
82. Mowbray CT, Collins ME, Plum TB, et al: Harbinger 1: the development and evaluation of the first PACT replication. Administration and Policy in Mental Health 25:105-123, 1997Crossref, Medline, Google Scholar
83. Drake RE, McHugo GJ, Clark RE, et al: Assertive community treatment for patients with co-occurring severe mental illness and substance use disorder: a clinical trial. American Journal of Orthopsychiatry 68:201-205, 1998Crossref, Medline, Google Scholar
84. Essock SM, Frisman LK, Kontos NJ: Cost-effectiveness of assertive community treatment teams. American Journal of Orthopsychiatry 68:179-190, 1998Crossref, Medline, Google Scholar
85. Essock SM, Kontos N: Implementing assertive community treatment teams. Psychiatric Services 46:679-683, 1995Link, Google Scholar
86. Fekete DM, Bond GR, McDonel EC, et al: Rural assertive community treatment: a field experiment. Psychiatric Rehabilitation Journal 21:371-379, 1998Crossref, Google Scholar
87. McDonel EC, Bond GR, Salyers M, et al: Implementing assertive community treatment programs in rural settings. Administration and Policy in Mental Health 25:153-173, 1997Crossref, Medline, Google Scholar
88. Holloway F, Carson J: Intensive case management for the severely mentally ill: controlled trial. British Journal of Psychiatry 172:19-22, 1998Crossref, Medline, Google Scholar
89. Rosenheck R, Neale M: Cost-effectiveness of intensive psychiatric community care for high users of inpatient services. Archives of General Psychiatry 55:459-466, 1998Crossref, Medline, Google Scholar
90. Rosenheck R, Neale M: Intersite variation in the impact of intensive psychiatric community care on hospital use. American Journal of Orthopsychiatry 68:191-200, 1998Crossref, Medline, Google Scholar
91. Rosenheck R, Neale M, Leaf P, et al: Multisite experimental cost study of intensive community care. Schizophrenia Bulletin 21:129-140, 1995Crossref, Medline, Google Scholar
92. Wilson D, Tien G, Eaves D: Increasing the community tenure of mentally disordered offenders: an assertive case management program. International Journal of Law and Psychiatry 18:61-69, 1995Crossref, Medline, Google Scholar
93. Cutler DL, Tatum E, Shore JH: A comparison of schizophrenic patients in different community support treatment approaches. Community Mental Health Journal 23:103-113, 1987Crossref, Medline, Google Scholar
94. Solomon P, Draine J: Satisfaction with mental health treatment in a randomized trial of consumer case management. Journal of Nervous and Mental Disease 182:179-184, 1994Crossref, Medline, Google Scholar
95. Solomon P, Draine J, Meyerson A: Jail recidivism and receipt of community mental health services. Hospital and Community Psychiatry 45:793-797, 1994Abstract, Google Scholar
96. Solomon P, Draine J: One year outcomes of a randomized trial of case management with seriously mentally ill clients leaving jail. Evaluation Review 19:256-273, 1995Crossref, Google Scholar
97. Dincin J: Assertive case management. Psychiatric Quarterly 61:49-55, 1990Crossref, Medline, Google Scholar
98. Mueller BJ, Hopp M: A demonstration of the cost benefits of case management services for discharged mental patients. Psychiatric Quarterly 55:17-24, 1983Crossref, Medline, Google Scholar
99. Mueller BJ, Hopp M: Attitudinal, administrative, legal, and fiscal barriers to case management in social rehabilitation of the mentally ill. International Journal of Mental Health 15:44-58, 1987Google Scholar
100. Aberg-Wistedt A, Cressell T, Lidberg Y, et al: Two-year outcome of team-based intensive case management for patients with schizophrenia. Psychiatric Services 46:1263-1266, 1995Link, Google Scholar
101. Kanter J: Clinical case management: definition, principles, components. Hospital and Community Psychiatry 40:361-368, 1989Abstract, Google Scholar
102. Sawilowsky SS, Blair RC: A more realistic look at the robustness of type II error properties of the t-test to departures from population normality. Psychological Bulletin 111:352-360, 1992Crossref, Google Scholar
103. Bailar JC: Passive smoking, coronary heart disease, and meta-analysis (edtl). New England Journal of Medicine 340:958- 959, 1999Crossref, Medline, Google Scholar
104. Marchioli R, Tognoni G: The yield of meta-analysis (ltr). Lancet 351:915-916, 1998Crossref, Medline, Google Scholar
105. Cooper H, Hedges LV (eds): The Handbook of Research Synthesis. New York, Russell Sage Foundation, 1994Google Scholar
106. Cohen J: Statistical Power Analysis for the Behavioral Sciences. Hillsdale, NJ, Erlbaum, 1988Google Scholar
107. Gilbert JP, Light RJ, Mosteller F: Assessing social innovations: an empirical base for policy, in Evaluation and Experiment. Edited by Bennett CA, Lumsdain AA. New York, Academic, 1975Google Scholar
108. Light RJ (ed): Evaluation Studies Review Annual. Beverly Hills, Calif, Sage, 1983Google Scholar
109. Cook TD, Cooper H, Cordray DS, et al: Meta-Analysis for Explanation: A Casebook. New York, Russell Sage Foundation, 1992Google Scholar
110. Shadish WR: Do family and marital therapies change what people do? A meta-analysis of behavioral outcomes, in ibidGoogle Scholar
111. Brekke JS, Test MA: A model for measuring the implementation of community support programs: results from three sites. Community Mental Health Journal 28:227-247, 1992Crossref, Medline, Google Scholar
112. McGrew JH, Bind GR, Dietzen L, et al: Measuring the fidelity of a mental health program model. Journal of Consulting and Clinical Psychology 62:670-678, 1994Crossref, Medline, Google Scholar
