0
Get Alert
Please Wait... Processing your request... Please Wait.
You must sign in to sign-up for alerts.

Please confirm that your email address is correct, so you can successfully receive this alert.

1
Articles   |    
Generalizability in the Family-to-Family Education Program Randomized Waitlist-Control Trial
Sue M. Marcus, Ph.D.; Deborah Medoff, Ph.D.; Li Juan Fang, M.S.; James Weaver, M.P.H.; Naihua Duan, Ph.D.; Alicia Lucksted, Ph.D.; Lisa B. Dixon, M.D., M.P.H.
Psychiatric Services 2013; doi: 10.1176/appi.ps.002912012
View Author and Article Information

Dr. Marcus and Dr. Duan are affiliated with the Departments of Psychiatry and Biostatistics in the Division of Biostatistics at New York State Psychiatric Institute (NYSPI), Columbia University, New York City. Dr. Dixon is affiliated with the Department of Psychiatry, Columbia University, and with NYSPI, New York City. Dr. Medoff, Ms. Fang, and Dr. Lucksted are with the Department of Psychiatry, University of Maryland School of Medicine, Baltimore. Mr. Weaver is with the Research Foundation for Mental Health, New York City. Send correspondence to Dr. Dixon, Department of Psychiatry, Columbia University/NYSPI, 1051 Riverside Dr., New York, NY 10032 (e-mail: dixonli@nyspi.columbia.edu).

Copyright © 2013 by the American Psychiatric Association

Objective  Randomized controlled trials (RCTs) may have limited generalizability for the community when a high proportion of individuals refuse randomization or otherwise do not participate—a not uncommon phenomenon. A randomized waitlist-control trial of the Family-to-Family (FTF) education program, a 12-week course offered by the National Alliance on Mental Illness for family members of adults with mental illness, was previously reported. This study assessed whether the RCT-derived estimates of effectiveness of FTF were generalizable to individuals who participated in FTF but declined participation in the RCT.

Methods  Propensity score matching was used to create five quintiles, each containing scores for individuals in FTF or waitlist conditions and for decliners; scores were matched on multiple baseline characteristics (N=442) within each quintile. Effectiveness estimates, with standard errors, were derived for the decliner population on the basis of effectiveness estimates derived from participants in the RCT; estimates were weighted to the baseline distribution of quintiles for the decliners.

Results  For each outcome, estimates of the effect sizes observed in the RCT were very similar to the effect sizes observed for the decliner population; confidence intervals also had a high degree of overlap.

Conclusions  This study suggests that the benefits of FTF observed in the RCT are generalizable to the group of individuals who declined RCT participation, providing further evidence of FTF’s effectiveness. Propensity score matching was a useful statistical tool for addressing selection bias resulting from high rates of nonconsent in randomized waitlist-control trials.

Abstract Teaser
Figures in this Article

Randomized controlled trials (RCTs) are considered the most rigorous test of an intervention’s effectiveness. The internal validity of RCTs gives confidence that study findings can be attributed to the differences between the experimental and control conditions. However, RCTs may have limited external validity (generalizability) for the community of potential users of the program being tested if a high proportion of individuals refuse randomization or otherwise do not participate—a not uncommon phenomenon (1).

Individuals may decline to consent to random assignment to a treatment if it differs greatly from those currently received or familiar (medication versus psychotherapy, for example) (1). A similar situation arises when an RCT control group is placed on a waitlist for an experimental intervention; some people may withhold consent for random assignment if they are unwilling to wait for the experimental treatment. In prior work, we proposed the parallel randomized and nonrandomized (PRN) clinical trial design (also known as the partially randomized preference design) as a solution for this problem (1,2). Most RCTs exclude individuals who do not consent to randomization. However, in the PRN trial design, those who consent to randomization are randomly assigned, and those who do not are assigned to their treatment of choice and are followed in a manner similar to those in the RCT. This design can enhance generalizability by enabling the estimation of effectiveness for those who decline randomization (1).

We reported the results of an RCT that tested the effectiveness of the National Alliance on Mental Illness (NAMI) Family-to-Family (FTF) program, a 12-week course for family members of adults with mental illness (3). In this study, 318 consenting participants in five Maryland counties and Baltimore City were randomly assigned to participate in FTF immediately or to wait at least three months for the next available class and to freely use in the meantime any other NAMI, community, or professional supports. We found that FTF participants had significantly greater improvements in coping, family problem solving, knowledge and distress. However, less than one-third of the potential sample was willing to consider study participation. The most common reason for declining was unwillingness to undergo random assignment because of the potential delay in FTF participation (3).

The study’s consent rate created a concern that the RCT participant sample was not representative of individuals who generally participate in FTF. To address this, we offered nonrandomized study participation to a cohort of 124 individuals who refused to enroll in the RCT and who were planning to take the class immediately. We evaluated these individuals (called the “decliner” sample) according to the same schedule as the participants in the RCT. The aims of the study were to apply innovative statistical methods to determine whether the findings of the RCT could generalize to the sample of decliners and therefore potentially to the population of individuals who enroll in FTF through usual NAMI programming.

+

Participants

Individuals were eligible to participate in the primary study if they were between ages 21 and 80, desired enrollment in the next FTF class regarding a family member or significant other, and spoke English. A total of 1,532 individuals who expressed interest in FTF were screened for study participation; 1,168 were found to be eligible. From this group, 318 individuals consented to participate in the randomized portion (RCT) of the overall study; 160 were randomly assigned to FTF, and 158 were assigned to the waitlist. An additional sample of 124 individuals from the 850 who had declined enrollment in the RCT and who were planning to take the class enrolled in the nonrandomized portion of the overall study. This decliner group was recruited approximately midway through the RCT when the need to address the modest consent rate was recognized; of the persons deemed eligible for the RCT, those who consecutively refused to participate in the RCT were offered enrollment as decliners until we achieved our target enrollment. Participants in both the RCT and decliner (nonrandomized) portions of the study completed identical baseline and follow-up interviews. We refer to three groups: decliners, RCT FTF participants, and RCT waitlist participants.

The institutional review board (IRB) at the University of Maryland approved all study activities; because interviews were conducted over the telephone, the IRB permitted consent to be obtained over the telephone after complete description of the study to the participants. Participants were recruited between March 2006 and September 2009.

+

Variables

This study considered three sets of variables. The first set includes all of the variables that were obtained in the participant interview. [This set is available online as a data supplement to this article.] Each variable was tested for inclusion in the propensity score analyses.

The second set of variables, described below, included those that differed between the decliner and RCT samples and therefore were used to generate the propensity scores. These included consumer race-ethnicity, consumer gender, living siblings of the consumer, family member income, family member marital status, consumer hospitalization in the past six months, and information about objective illness burden and required supervision obtained from the Family Experience Interview Survey (4).

Also in the second variable set, the Family Empowerment Scale provided measures of family, community, and service system empowerment (5); the Experience of Caregiving Inventory provided measures of positive aspects of the relationship, need of backup, problem with service system, stigma, and total positive and total negative subscales (6,7); and the NAMI Family Member Questionnaire provided measures of empowerment, coping with consumer’s illness, subjective burden and worry, and understanding of the mental health system (8). We also used the physical composite score of the 12-Item Short Form Health Survey (9), the Global Severity Index of the Brief Symptom Inventory–18 (BSI-18) (10,11), the percentage correct on the FTF mental illness knowledge test, and whether the family member self-reported having ever attended any formal NAMI educational programs (8).

The third relevant set of variables consisted of outcomes that improved with FTF for participation in the RCT. Knowledge was measured with a 20-item true-false test of factual information covering material drawn from the FTF curriculum that tapped general knowledge about mental illnesses (3). The five-item anxiety subscale of the BSI-18 measured psychological distress. It is designed for use primarily in nonclinical, community populations and has well-established reliability and validity (10,11). Family functioning was measured with the five-item problem-solving subscale of the Family Assessment Device, which evaluates family functioning and family relations. It is widely used in studies of family response to general medical illness and has well-established reliability and validity (12). The four-item acceptance dimension of the COPE measures emotion-focused coping (13) and family, service system, and community empowerment as described above. The analyses used both baseline and three-month measures of FTF outcomes.

+

Statistical approach

Although this FTF study began with a traditional randomization process, the addition of the decliners transformed the combined RCT plus the decliner nonrandomized portion into a PRN study, a hybrid of randomized and observational study that is used in effectiveness analyses (14). Observational studies often suffer from selection bias; that is, people who receive treatment may differ systematically from those who do not. One approach is to match those in the treatment group with those in the control group, so that treatment effects can be attributed to the treatment rather than to baseline differences between the treated and untreated participants.

Propensity score matching is a useful statistical tool for adjusting for many covariates simultaneously (15). Matching participants on the unidimensional propensity score between those who receive or do not receive treatment has been shown formally to be statistically comparable to matching separately on each of the multiple covariates used to create the propensity score, but the former is preferable because separate matching becomes infeasible when there are more than a few covariates.

The propensity score is defined as the probability that a participant received the treatment versus the control condition, which is contingent on a set of potential measured confounders. Commonly, the propensity score is estimated with logistic regression to model the propensity of an individual to receive the treatment versus the control condition (16). However, to assess the generalizability of the treatment effect for the decliners, we chose to evaluate the propensity to be in the RCT versus in the decliner sample, in order to clarify characteristics that were associated with being a decliner. This approach has been shown to be mathematically equivalent to the more common method of estimating propensity to receive the treatment versus the control condition but provides more useful information for this study design (14).

First, propensity scores for each person were estimated with logistic regression. Variables were selected for the regression model by using bivariate statistics (chi square and t tests) to compare the RCT sample with the decliner sample for all baseline measures we collected in the RCT, including, for example, consumer and family demographic variables, family member–reported objective and subjective burden, coping, empowerment, family functioning, and other supports (3). Variables showing significant differences between the RCT and decliner samples (23 variables, p<.2) were then entered into a logistic regression model comparing the RCT sample and the decliner sample. Missing data were handled by using missing-data indicators (17). Propensity scores were calculated from this model. Participants with higher propensity scores had profiles more closely resembling RCT enrollees. All participants were placed into quintiles according to their propensity score (18). We then examined the distribution of participants by propensity score quintile for the randomized FTF, randomized waitlist, and decliner samples.

A sample of covariates by quintile and group is listed in Table 1. Analyses of variance (ANOVAs) and chi square tests were used to assess heterogeneity across quintiles, with respect to each selected covariate.

 
Anchor for Jump
Table 1Baseline covariates of participants in a randomized controlled trial (RCT) of the National Alliance on Mental Illness (NAMI) Family-to-Family psychoeducation, by propensity score quintiles
Table Footer Note

a From the Family Experience Interview Schedule. Possible scores range from 0 to 4, with higher scores indicating more frequent assistance in daily life.

Table Footer Note

b From the Family Assessment Device. Possible scores range from 6 to 24, with higher scores indicating worse problem solving.

Table Footer Note

c T scores range from 38 to 81, with higher scores indicating more anxiety symptoms.

Table Footer Note

d T scores range from 33 to 81, with higher scores indicating more global symptoms.

Table Footer Note

e Family Empowerment Scale. Possible scores range from 1 to 5, with higher scores indicating more empowerment.

Table Footer Note

f From the COPE. Possible scores range from 4 to 16, with higher scores indicating better coping.

Table Footer Note

g As measured on the Center for Epidemiological Studies Depression Scale. Possible scores ranges from 0 to 42, with higher scores indicating more severe depression symptoms.

Table Footer Note

h Family Member Questionnaire. Possible scores range from 1 to 4, with higher scores indicating less worry and fewer burdens.

+

Effect size estimates for the decliner sample

Our primary goal was to determine whether the estimate of the effect of FTF versus waitlist observed in the RCT generalized to the decliner sample. We planned to derive estimates of FTF’s impact on the outcomes of knowledge, family problem solving, empowerment, acceptance aspects of coping, anxiety, and subjective burden (worry) for the decliner sample and compare these with the estimates of benefits of FTF observed in the published RCT (3). Our approach was to build on the internally valid estimates of benefit derived from the RCT and enhance external validity by weighting the estimates of effectiveness observed in the RCT to fit the distribution of the propensity score quintiles for the baseline decliner population.

Although similar to age-adjusted estimates that are commonly used in life tables, propensity score matching enabled us to adjust for many covariates simultaneously. Estimates of the effectiveness of RCT FTF versus RCT waitlist were calculated for decliners. [Calculations and corresponding standard errors are available online as a data supplement to this article.] This approach provided an estimate of how individuals similar to the decliners would do if they received FTF versus how they would do if they could also be observed after assignment to (hypothetically) the waitlist. We next used these estimates and standard errors to calculate 95% confidence intervals and corresponding effect sizes regarding FTF versus waitlist effectiveness for the decliner sample. These confidence intervals were then compared with the RCT estimates.

+

Baseline differences

Of the total sample of 409 consumers whose race was reported by family member participants, 260 (64%) were white, 11 (3%) were Asian, 105 (26%) were black, seven (2%) were Hispanic, and 26 (6%) were other. When compared with the decliners, participants in the RCT were significantly more likely to report having family income greater than $50,000 per year (χ2=4.39, df=1, p<.036), to report that the consumer required more assistance in daily living (t=2.13, df=435, p<.033), and to report that the family member had a psychiatric hospitalization in the past six months (χ2=11.35, df=1, p<.001). With respect to study outcomes, decliners had less knowledge about mental illness (t=2.43, df=435, p<.016) and less community empowerment (t=2.51, df=434, p=.012) at baseline.

Figure 1 provides an example of how the propensity score approach allowed the decliner sample to be matched with the RCT sample when one of the variables contributed to the propensity score. An important aspect of matching groups via propensity score quintiles was to examine the percentage of participants within each quintile by sample. Specifically, the critical question was whether there was a comparable percentage of each sample (decliner, RCT waitlist, or RCT FTF) for any particular variable represented in each quintile.

 
Anchor for JumpAnchor for Jump
Figure 1Percentage of participants reporting that a family member had a psychiatric hospitalization in the past 6 months, by propensity score quintilea

a Respondents received or were waitlisted to receive the 12-week Family-to-Family (FTF) psychoeducation program in a randomized controlled trial (RCT). Persons who declined random assignment (decliners) received FTF but did not participate in the RCT.

The figure shows that the variable, percentage of participants whose family member experienced a psychiatric hospitalization in the past six months, decreased from quintile 1 (most like the decliners) to quintile 5 (least like the decliners); in other words, compared with the RCT participants, a higher percentage of the decliners tended to have a family member who had been hospitalized, consistent with the bivariate analysis presented above. However, within each quintile, the percentage was generally similar across the three samples. It is important to note that matching does not require perfect balance. Other covariates were also effectively balanced by propensity score quintile matching. Table 1 gives baseline covariate and outcome data by group (RCT waitlist, RCT FTF, or decliner) and propensity score quintile. ANOVAs and chi squares across quintiles of all covariates demonstrated significant heterogeneity.

+

Generalizability estimates

Table 2 provides mean outcome levels by group and propensity score quintile for three-month outcomes. The three groups appeared to be comparably distributed within each quintile, as shown in Figure 2 for the knowledge test. Table 3 presents the effectiveness estimates with confidence intervals and effect sizes derived from the RCT for comparison of individuals receiving FTF with individuals on the waitlist. It also provides estimates of the RCT FTF versus RCT waitlist effect for the decliner sample—estimates that reflect the capacity of the propensity scoring process and assignment of quintiles to predict what the effects of FTF versus waitlist would have been for the decliners. We note that the effect sizes were remarkably similar despite the selection differences for being in the decliner population. For example, with respect to knowledge, the effect size observed in the RCT was .31. The estimated effect size for the decliner population was .29. Also, the confidence intervals had a high degree of overlap.

 
Anchor for Jump
Table 2Three-month outcomes in a randomized controlled trial (RCT) of participants receiving or waiting to receive Family-to-Family (FTF) psychoeducation, by propensity score quintile
Table Footer Note

a From the Family Assessment Device. Possible scores range from 6 to 24, with higher scores indicating worse problem solving.

Table Footer Note

b From the Brief Symptom Inventory. Anxiety T scores range from 38 to 81, with higher scores indicating more anxiety symptoms.

Table Footer Note

c T scores range from 33 to 81, with higher scores indicating more global symptoms.

Table Footer Note

d From the Family Empowerment Scale. Possible scores range from 1 to 5, with higher scores indicating more empowerment.

Table Footer Note

e From the COPE. Possible scores range from 4 to 16, with higher scores indicating better coping.

Table Footer Note

f As measured on the Center for Epidemiological Studies Depression Scale. Possible scores ranges from 0 to 42, with higher scores indicating more severe depression symptoms.

Table Footer Note

g From the Family Member Questionnaire. Possible scores range from 1 to 4, with higher scores indicating less worry and fewer burdens.

 
Anchor for JumpAnchor for Jump
Figure 2Knowledge scores at 3 months, by propensity score quintilea

a Respondents received or were waitlisted to receive the 12-week Family-to-Family (FTF) psychoeducation program in a randomized controlled trial (RCT). Persons who declined random assignment (decliners) received FTF but did not participate in the RCT.

 
Anchor for Jump
Table 3Effectiveness of Family-to-Family program versus waitlist and estimated generalizability for persons declining random assignment in the randomized controlled trial (RCT)
Table Footer Note

a Comparisons were as follows: RCT, estimate from the RCT, excluding decliners (3). For decliners, the estimate is of Family-to-Family recipients versus the waitlist effect for the decliner group.

RCTs are vulnerable to selection bias that can reduce the external validity of study findings. Without external validity, the overall value of RCTs for informing care delivery and policy is critically limited. Programs that are widely available prior to effectiveness evaluation may face special challenges in avoiding significant selection bias when attempting to conduct RCTs. This creates difficulty in amassing high-quality practice-based evidence sufficient to merit the program’s determination as an evidenced-based practice. This study’s overall significance derives from its development and application of methods to meet that challenge.

Our RCT of NAMI’s FTF education program appeared to be vulnerable to selection bias because individuals could access FTF without participating in our study. We were able to empirically evaluate this threat to our analysis by recruiting a sample of persons (decliners) who declined to participate in the randomization process. We showed that the estimated RCT FTF versus RCT waitlist effect sizes for the decliner sample were quite similar to the effect sizes observed in the RCT in which the individuals randomly assigned to FTF were compared with a waitlisted group. We thus conclude that FTF may indeed be effective for a target population that includes people similar to the decliners as well as those similar to the RCT participants.

This study therefore reinforced our previous findings that the NAMI FTF program is a valuable resource to family members of individuals with mental illness. FTF has been found to increase knowledge about mental illness, improve self-reported family problem-solving skills, and reduce distress. The RCT also demonstrated that FTF improves family members’ coping skills and empowerment (3). The positive and generalizable impact of FTF observed in this study further reinforces the value of this program as an evidence-based practice and the imperative for mental health providers and clinicians to consider it a resource for struggling family members. These findings also underscore the unique contributions of peer-based support programs in the service array for persons with mental illnesses and their relatives (19,20).

The differences between RCT decliners and RCT participants may suggest some unique sampling vulnerabilities for RCTs with waitlisted control groups. As could be expected, individuals with higher indicators of need (greater objective burden and greater likelihood of a consumer’s recent hospitalization) were less willing to take the chance of random assignment to the waitlist condition. In addition, individuals with greater income were more likely to refuse RCT enrollment. Such patterns could have plausibly produced estimates suggesting that FTF would not have been effective with the decliners. The analyses presented therefore underscore the importance of adopting a systematic, empirical approach to evaluating external validity.

This use of propensity scores to evaluate such potential biases in nonrandomized samples was limited by the fact that the estimates for the decliners were unbiased only when we could adjust for all confounders. Thus it is important for confounders to be considered during the design phase, so they can be measured. This involves collecting information about characteristics that might be related to being a decliner, as well as characteristics that are thought to influence the primary outcomes of a study. Collecting more covariates can add cost and complexity, but it is important not to overlook those necessary to determine whether results are convincing. Qualitative methods can be helpful in identifying additional confounders, particularly for areas in which there is not much existing research.

Our approach can be modified to other situations that commonly occur in psychiatric services research, for example, when people do not consent to randomization processes because they have strong preferences about treatment (1,14). Studies comparing medication to psychotherapy, two different medications, or two different psychotherapies exemplify this circumstance. In these situations, it is essential to document reasons for nonconsent and to subsequently collect outcome data when possible.

This study used innovative statistical methods to assess whether the benefits observed in a RCT of NAMI’s FTF education program could be extended to a majority of eligible individuals who declined to participate in the RCT. By including a cohort of decliners and evaluating their status before and after participating in FTF, the analyses suggest that FTF versus waitlist benefits of improving knowledge, reducing distress, and improving family problem solving generalize to the larger group. The significance of this study rests not only in the demonstration of benefits of FTF but also provides an important example of how RCTs of interventions available in the community can address the problem of external validity for the valid designation of programs as evidence-based practices.

This project was supported by grant 1R01-MH72667-01A1 from the National Institute of Mental Health, by grant P20 MH085983 from the Center for Collaborative Inner-City Child Mental Health Services Research, and by grant P30 MH090322-01 A1 from the Advanced Center on Implementation–Dissemination Science in States for Children and Families.

The authors report no competing interests.

Marcus  SM:  Assessing non-consent bias with parallel randomized and nonrandomized clinical trials.  Journal of Clinical Epidemiology 50:823–828, 1997
[CrossRef] | [PubMed]
 
Paradise  JL;  Bluestone  CD;  Bachman  RZ  et al:  Efficacy of tonsillectomy for recurrent throat infection in severely affected children: results of parallel randomized and nonrandomized clinical trials.  New England Journal of Medicine 310:674–683, 1984
[CrossRef] | [PubMed]
 
Dixon  LB;  Lucksted  A;  Medoff  DR  et al:  Outcomes of a randomized study of a peer-taught Family-to-Family Education Program for mental illness.  Psychiatric Services 62:591–597, 2011
[CrossRef] | [PubMed]
 
Tessler R, Gamache G: Family Experiences Interview Schedule (FEIS); in the Toolkit on Evaluating Family Experiences With Severe Mental Illness. Cambridge, Mass, Human Services Research Institute, Evaluation Center, 1995. Available at www.hsri.org
 
Koren  P;  DeChillo  N;  Friesen  B:  Measuring empowerment in families whose children have emotional disorders: a brief questionnaire.  Rehabilitation Psychology 37:305–321, 1992
[CrossRef]
 
Szmukler  GI;  Burgess  P;  Herrman  H  et al:  Caring for relatives with serious mental illness: the development of the Experience of Caregiving Inventory.  Social Psychiatry and Psychiatric Epidemiology 31:137–148, 1996
[CrossRef] | [PubMed]
 
Joyce  JL;  Leese  M;  Szmukler  G:  The Experience of Caregiving Inventory: further evidence.  Social Psychiatry and Psychiatric Epidemiology 35:185–189, 2000
[CrossRef] | [PubMed]
 
Dixon  L;  Lucksted  A;  Stewart  B  et al:  Outcomes of the peer-taught 12-week family-to-family education program for severe mental illness.  Acta Psychiatrica Scandinavica 109:207–215, 2004
[CrossRef] | [PubMed]
 
Ware  J  Jr;  Kosinski  M;  Keller  SD:  A 12-Item Short-Form Health Survey: construction of scales and preliminary tests of reliability and validity.  Medical Care 34:220–233, 1996
[CrossRef] | [PubMed]
 
Derogatis  LR;  Melisaratos  N:  The Brief Symptom Inventory: an introductory report.  Psychological Medicine 13:595–605, 1983
[CrossRef] | [PubMed]
 
Derogatis  LR:  BSI-18: Administration, Scoring and Procedures Manual .  New York,  Pearson, 2001
 
Sawin  KJ;  Harrigan  MP:  Measures of Family Functioning for Research and Practice .  New York,  Springer, 1995
 
Carver  CS;  Scheier  MF;  Weintraub  JK:  Assessing coping strategies: a theoretically based approach.  Journal of Personality and Social Psychology 56:267–283, 1989
[CrossRef] | [PubMed]
 
Marcus  SM;  Stuart  EA;  Wang  P  et al:  Estimating the causal effect of randomization versus treatment preference in a doubly randomized preference trial.  Psychological Methods 17:244–254, 2012
[CrossRef] | [PubMed]
 
Rosenbaum  PR;  Rubin  DB:  The central role of the propensity score in observational studies for causal effects.  Biometrika 70:41–55, 1983
[CrossRef]
 
Stuart  EA;  Marcus  SM;  Horvitz-Lennon  MV  et al:  Using non-experimental data to estimate treatment effects.  Psychiatric Annals 39:41451, 2009
[CrossRef] | [PubMed]
 
Haviland  AM;  Nagin  DS;  Rosenbaum  PR:  Combining propensity score matching and group-based trajectory analysis in an observational study.  Psychological Methods 12:247–267, 2007
[CrossRef] | [PubMed]
 
Rosenbaum  PR;  Rubin  DB:  Reducing bias in observational studies using subclassification on the propensity score.  Journal of the American Statistical Association 79:516–524, 1984
[CrossRef]
 
Segal  SP;  Silverman  CJ;  Temkin  TL:  Self-help and community mental health agency outcomes: a recovery-focused randomized controlled trial.  Psychiatric Services 61:905–910, 2010
[CrossRef] | [PubMed]
 
Brown  LD:  Consumer-Run Mental Health: Framework for Recovery .  New York,  Springer, 2012
 
References Container

Figure 1 Percentage of participants reporting that a family member had a psychiatric hospitalization in the past 6 months, by propensity score quintilea

a Respondents received or were waitlisted to receive the 12-week Family-to-Family (FTF) psychoeducation program in a randomized controlled trial (RCT). Persons who declined random assignment (decliners) received FTF but did not participate in the RCT.

Figure 2 Knowledge scores at 3 months, by propensity score quintilea

a Respondents received or were waitlisted to receive the 12-week Family-to-Family (FTF) psychoeducation program in a randomized controlled trial (RCT). Persons who declined random assignment (decliners) received FTF but did not participate in the RCT.

Anchor for Jump
Table 1Baseline covariates of participants in a randomized controlled trial (RCT) of the National Alliance on Mental Illness (NAMI) Family-to-Family psychoeducation, by propensity score quintiles
Table Footer Note

a From the Family Experience Interview Schedule. Possible scores range from 0 to 4, with higher scores indicating more frequent assistance in daily life.

Table Footer Note

b From the Family Assessment Device. Possible scores range from 6 to 24, with higher scores indicating worse problem solving.

Table Footer Note

c T scores range from 38 to 81, with higher scores indicating more anxiety symptoms.

Table Footer Note

d T scores range from 33 to 81, with higher scores indicating more global symptoms.

Table Footer Note

e Family Empowerment Scale. Possible scores range from 1 to 5, with higher scores indicating more empowerment.

Table Footer Note

f From the COPE. Possible scores range from 4 to 16, with higher scores indicating better coping.

Table Footer Note

g As measured on the Center for Epidemiological Studies Depression Scale. Possible scores ranges from 0 to 42, with higher scores indicating more severe depression symptoms.

Table Footer Note

h Family Member Questionnaire. Possible scores range from 1 to 4, with higher scores indicating less worry and fewer burdens.

Anchor for Jump
Table 2Three-month outcomes in a randomized controlled trial (RCT) of participants receiving or waiting to receive Family-to-Family (FTF) psychoeducation, by propensity score quintile
Table Footer Note

a From the Family Assessment Device. Possible scores range from 6 to 24, with higher scores indicating worse problem solving.

Table Footer Note

b From the Brief Symptom Inventory. Anxiety T scores range from 38 to 81, with higher scores indicating more anxiety symptoms.

Table Footer Note

c T scores range from 33 to 81, with higher scores indicating more global symptoms.

Table Footer Note

d From the Family Empowerment Scale. Possible scores range from 1 to 5, with higher scores indicating more empowerment.

Table Footer Note

e From the COPE. Possible scores range from 4 to 16, with higher scores indicating better coping.

Table Footer Note

f As measured on the Center for Epidemiological Studies Depression Scale. Possible scores ranges from 0 to 42, with higher scores indicating more severe depression symptoms.

Table Footer Note

g From the Family Member Questionnaire. Possible scores range from 1 to 4, with higher scores indicating less worry and fewer burdens.

Anchor for Jump
Table 3Effectiveness of Family-to-Family program versus waitlist and estimated generalizability for persons declining random assignment in the randomized controlled trial (RCT)
Table Footer Note

a Comparisons were as follows: RCT, estimate from the RCT, excluding decliners (3). For decliners, the estimate is of Family-to-Family recipients versus the waitlist effect for the decliner group.

+

References

Marcus  SM:  Assessing non-consent bias with parallel randomized and nonrandomized clinical trials.  Journal of Clinical Epidemiology 50:823–828, 1997
[CrossRef] | [PubMed]
 
Paradise  JL;  Bluestone  CD;  Bachman  RZ  et al:  Efficacy of tonsillectomy for recurrent throat infection in severely affected children: results of parallel randomized and nonrandomized clinical trials.  New England Journal of Medicine 310:674–683, 1984
[CrossRef] | [PubMed]
 
Dixon  LB;  Lucksted  A;  Medoff  DR  et al:  Outcomes of a randomized study of a peer-taught Family-to-Family Education Program for mental illness.  Psychiatric Services 62:591–597, 2011
[CrossRef] | [PubMed]
 
Tessler R, Gamache G: Family Experiences Interview Schedule (FEIS); in the Toolkit on Evaluating Family Experiences With Severe Mental Illness. Cambridge, Mass, Human Services Research Institute, Evaluation Center, 1995. Available at www.hsri.org
 
Koren  P;  DeChillo  N;  Friesen  B:  Measuring empowerment in families whose children have emotional disorders: a brief questionnaire.  Rehabilitation Psychology 37:305–321, 1992
[CrossRef]
 
Szmukler  GI;  Burgess  P;  Herrman  H  et al:  Caring for relatives with serious mental illness: the development of the Experience of Caregiving Inventory.  Social Psychiatry and Psychiatric Epidemiology 31:137–148, 1996
[CrossRef] | [PubMed]
 
Joyce  JL;  Leese  M;  Szmukler  G:  The Experience of Caregiving Inventory: further evidence.  Social Psychiatry and Psychiatric Epidemiology 35:185–189, 2000
[CrossRef] | [PubMed]
 
Dixon  L;  Lucksted  A;  Stewart  B  et al:  Outcomes of the peer-taught 12-week family-to-family education program for severe mental illness.  Acta Psychiatrica Scandinavica 109:207–215, 2004
[CrossRef] | [PubMed]
 
Ware  J  Jr;  Kosinski  M;  Keller  SD:  A 12-Item Short-Form Health Survey: construction of scales and preliminary tests of reliability and validity.  Medical Care 34:220–233, 1996
[CrossRef] | [PubMed]
 
Derogatis  LR;  Melisaratos  N:  The Brief Symptom Inventory: an introductory report.  Psychological Medicine 13:595–605, 1983
[CrossRef] | [PubMed]
 
Derogatis  LR:  BSI-18: Administration, Scoring and Procedures Manual .  New York,  Pearson, 2001
 
Sawin  KJ;  Harrigan  MP:  Measures of Family Functioning for Research and Practice .  New York,  Springer, 1995
 
Carver  CS;  Scheier  MF;  Weintraub  JK:  Assessing coping strategies: a theoretically based approach.  Journal of Personality and Social Psychology 56:267–283, 1989
[CrossRef] | [PubMed]
 
Marcus  SM;  Stuart  EA;  Wang  P  et al:  Estimating the causal effect of randomization versus treatment preference in a doubly randomized preference trial.  Psychological Methods 17:244–254, 2012
[CrossRef] | [PubMed]
 
Rosenbaum  PR;  Rubin  DB:  The central role of the propensity score in observational studies for causal effects.  Biometrika 70:41–55, 1983
[CrossRef]
 
Stuart  EA;  Marcus  SM;  Horvitz-Lennon  MV  et al:  Using non-experimental data to estimate treatment effects.  Psychiatric Annals 39:41451, 2009
[CrossRef] | [PubMed]
 
Haviland  AM;  Nagin  DS;  Rosenbaum  PR:  Combining propensity score matching and group-based trajectory analysis in an observational study.  Psychological Methods 12:247–267, 2007
[CrossRef] | [PubMed]
 
Rosenbaum  PR;  Rubin  DB:  Reducing bias in observational studies using subclassification on the propensity score.  Journal of the American Statistical Association 79:516–524, 1984
[CrossRef]
 
Segal  SP;  Silverman  CJ;  Temkin  TL:  Self-help and community mental health agency outcomes: a recovery-focused randomized controlled trial.  Psychiatric Services 61:905–910, 2010
[CrossRef] | [PubMed]
 
Brown  LD:  Consumer-Run Mental Health: Framework for Recovery .  New York,  Springer, 2012
 
References Container
+
+

CME Activity

There is currently no quiz available for this resource. Please click here to go to the CME page to find another.
Submit a Comments
Please read the other comments before you post yours. Contributors must reveal any conflict of interest.
Comments are moderated and will appear on the site at the discertion of APA editorial staff.

* = Required Field
(if multiple authors, separate names by comma)
Example: John Doe



Related Content
Articles
Books
The American Psychiatric Publishing Textbook of Geriatric Psychiatry, 4th Edition > Chapter 29.  >
Textbook of Psychotherapeutic Treatments > Chapter 8.  >
Dulcan's Textbook of Child and Adolescent Psychiatry > Chapter 22.  >
The American Psychiatric Publishing Textbook of Psychopharmacology, 4th Edition > Chapter 11.  >
Dulcan's Textbook of Child and Adolescent Psychiatry > Chapter 58.  >
Topic Collections
Psychiatric News
PubMed Articles