0
Get Alert
Please Wait... Processing your request... Please Wait.
You must sign in to sign-up for alerts.

Please confirm that your email address is correct, so you can successfully receive this alert.

1
Article   |    
Use of Clients' Self-Reports to Monitor Project Liberty Clinicians' Fidelity to a Cognitive-Behavioral Intervention
Susan M. Essock, Ph.D.; Nancy H. Covell, Ph.D.; Katherine M. Shear, M.D.; Sheila A. Donahue, M.A.; Chip J. Felton, M.S.W.
Psychiatric Services 2006; doi: 10.1176/appi.ps.57.9.1320
Abstract

Objective: This study examined outcomes associated with clinicians' fidelity to key elements of a cognitive-behavioral treatment intervention developed for Project Liberty's enhanced services counseling program. Methods: In telephone interviews 60 individuals reported how often their clinicians provided six components considered central to the intervention by the intervention developers. Respondents received services at sites where some (25 to 50 percent) or all clinicians had received training in the intervention. Results: Compared with respondents who received services where only some clinicians had received training (N=19), those who received services where all clinicians had received training (N=41) were significantly more likely to report that their clinicians applied techniques central to the intervention (p<.01). Additionally, those who received services from sites where all clinicians were trained were significantly more likely to report that they had been given homework (p<.05). Conclusions: Brief questions to service recipients are a useful and cost-effective way to monitor intervention fidelity. (Psychiatric Services 57:1320-1323, 2006)

Abstract Teaser
Figures in this Article

Dr. Essock and Dr. Covell are affiliated with the Department of Psychiatry, Division of Health Services Research, Mount Sinai School of Medicine, Box 1230, 1 Gustave L. Levy Place, New York, NY 10029-6574 (e-mail: susan.essock@mssm.edu). Dr. Essock is also with the Mental Illness Research, Education, and Clinical Center, James J. Peters Veterans Affairs Medical System, the Bronx. Dr. Shear is with the Columbia University School of Social Work and the Department of Psychiatry, Columbia College of Physicians and Surgeons, New York City. Ms. Donahue and Mr. Felton are with the Center for Information Technology and Evaluation Research, New York State Office of Mental Health, Albany. This report from Project Liberty is part of a special issue of Psychiatric Services commemorating the five-year anniversary of the September 11, 2001, attacks. Susan M. Essock, Ph.D., served as guest editor.

In response to the World Trade Center attacks on September 11, 2001, the New York Office of Mental Health (NYOMH) received funds from the Federal Emergency Management Agency (FEMA) to implement Project Liberty, a crisis counseling program aimed at alleviating psychological distress and helping people regain predisaster functioning. Project Liberty offered short-term outreach, crisis counseling, and public education services to affected individuals and groups at no cost to them. Project Liberty counselors also made referrals to longer-term mental health services when necessary (1). The project provided face-to-face disaster-related services through outreach in community settings, such as homes, businesses, and schools. The assumptions underlying this broad-based response strategy are that most people's stress reactions, although personally disturbing, are normal responses to a traumatic event and will be short-term in duration (2).

Crisis counseling services sufficiently aided a majority of individuals in recovering from distress related to September 11, 2001 (3). Nevertheless, a sizeable group experienced persistent and disabling trauma symptoms for a prolonged period. Encounter data indicated that about 40 percent of all service recipients showed evidence of reactions suggestive of posttraumatic stress disorder, depression, or both and that the proportion of persons who entered services with such reactions was quite consistent over the 27 months of program operation (4). This finding is consistent with previous reports indicating that a substantial proportion of persons may exhibit posttraumatic symptoms that persist after they experience a disaster (2).

Using this information, NYOMH initiated additional counseling services for individuals more severely affected by September 11, 2001. The enhanced services program included a screening methodology, use of evidence-informed counseling interventions, and training and technical assistance for a select group of clinicians. Approved by FEMA and the Center for Mental Health Services, the enhanced services program represented the first time that crisis counseling services under a FEMA grant had been expanded to include clinical services. Separate interventions were developed for adults and children who exhibited severe or persistent trauma symptoms (1).

In order to ensure the use of evidence-informed interventions for traumatic (5), depressive (6), and grief (7) symptomatology, NYOMH contracted with the National Center for Post-Traumatic Stress Disorder (NCPTSD) and the Pittsburgh Bereavement and Grief Program to develop two interventions lasting ten to 12 sessions and to provide training, training manuals, and ongoing technical assistance to the clinicians selected to provide the interventions. Additionally, NCPTSD and NYOMH collaborated on the creation of a tool to make referrals to enhanced services (8). Providing training in an intervention, however, did not guarantee its systematic implementation; therefore, it was important to evaluate the degree to which interventions were delivered with fidelity to the manual.

Although the gold standard for evaluating fidelity of treatment intervention relies upon multiple sources of assessment (9), there is increasing emphasis on users' reports (10). Essock and colleagues (11) have suggested that individuals' reports about the presence of core components of empirically supported treatments may provide cost-effective evaluations of treatment fidelity. This study examined whether participants who received services at sites where all clinicians received training in one of the enhanced services interventions—the Brief Intervention for Continuing Postdisaster Distress, which used cognitive-behavioral techniques—were more likely to report receiving core components of that intervention than participants at sites where only some clinicians received the training. This construct was employed because the data set used does not permit direct linkage between the recipient and the clinician. However, data indicating the site where service was received were available for each recipient. Therefore, recipients from sites where all clinicians were trained in the intervention were contrasted with those where only some staff was trained. Although training and the manual for the grief intervention developed by Shear and colleagues was also provided as part of enhanced services implementation, too few recipients whose clinicians received the training were included in the evaluation survey to assess fidelity to this intervention.

Following a protocol approved by Mount Sinai School of Medicine's and NYOMH's institutional review boards, Project Liberty clinicians invited English- and Spanish-speaking recipients of enhanced services aged 18 or older to participate in an evaluation of Project Liberty services via telephone interview. Because clinicians who worked with the Fire Department of New York City (FDNY) did not offer the intervention that was developed by NCPTSD as part of its enhanced service program, interviews with FDNY service recipients were not included in the analyses reported here.

Beginning in October 2003, all enhanced services recipients (excluding those served from the FDNY) were invited to participate in a telephone interview. Clinicians providing enhanced services were instructed to distribute a permission-to-contact form to all adults using services. Recruitment continued until mid-December 2003, when enrollment in enhanced services ended. Approximately 225 individuals were served during this period. Of the 102 recipients who signed the permission-to-contact forms, 74 (73 percent) completed the first telephone interview and 60 (59 percent) completed follow-up interviews an average of seven weeks later. Because we were interested in how service recipients characterized the intervention after some exposure to treatment, we focused our analyses on responses during the second interview. The analyses examined the specific items developed in collaboration with the NCPTSD to capture core features of the intervention.

The 60 participants who completed second interviews had a mean±SD age of 46±10.5 years, 19 (32 percent) were male, 44 (73 percent) were Caucasian, four (7 percent) were African American, one (2 percent) was Asian American, nine (15 percent) were from another racial or ethnic group or more than one race, and two (3 percent) declined to provide racial identity. Comparison of demographic characteristics as well as site and event exposures showed no significant differences between those who completed both interviews and those who were lost to follow-up.

Participants were grouped according to whether they received services from one of nine sites where all clinicians completed training (41 participants) or whether they received services at one of the six sites where only some clinicians (25 to 50 percent of clinicians) completed such training (19 participants). Respondents rated how often their clinicians provided each of five components of the intervention using Likert scales (0, not at all; 1, a little; 2, a moderate amount; or 3, a lot). A sixth question was also asked regarding whether their clinicians assigned homework; this question was scored as 0, not at all; 1, at least once; 2, after most sessions; or 3, after each session. We created summary scores by computing means across the five intervention-specific items. We applied independent samples t tests to examine group differences on outcomes measures.

Across all five intervention-specific items, individuals who received services from sites where all clinicians were trained reported that their clinicians, on average, applied techniques specific to the intervention more often than those who received services from sites where only some clinicians had been trained (Table 1). Compared with respondents who received services where only some clinicians had received training, those who received services where all clinicians had received training were significantly more likely to report that clinicians identified pleasant activities that they could do every day (p<.01) and were more likely to report that clinicians provided all five key components of the intervention, although these results did not reach statistical significance (p=.053). Additionally, individuals who received services from sites where all clinicians were trained reported that their clinicians gave them homework significantly more often than those who received services from sites where only some clinicians were trained (p<.05) (Table 1).

We have presented a novel evaluation strategy to determine clinician adherence to a planned short-term intervention. Our results are informative with regard to the intervention we studied and as a possible new methodological approach to quality assurance monitoring. However, there are many limitations to this study, and results should be considered preliminary. For example, only a small percentage of service recipients agreed to participate in the interviews, so our results may not be generalizable. The sample was small, so failure to detect differences between the two groups may reflect a lack of power rather than a lack of between-group differences. The second interviews were conducted at different periods during the intervention, which could have influenced ratings. A person who had attended only a few sessions may have rated items differently than one who had completed the intervention. Also, there may be confounds with clinician skill and attendance at the training sessions. Perhaps administrators from the sites where all clinicians were trained were more supportive of using the manualized interventions and thus have developed a more skilled provider group. Thus the differences we observed should be considered suggestions of training effectiveness, not confirmation. Nevertheless, as the first (to our knowledge) systematic effort to evaluate treatment fidelity in disaster mental health services, we developed a method that was able to provide reliable distinctions between sites.

There is considerable interest in developing ways to optimize community response to disasters. In the aftermath of an unprecedented terrorist disaster, FEMA and NYOMH marshaled resources to provide an enhanced disaster mental health services program using a multimodal intervention strategy and evaluated this response.

The subset of findings from the evaluation study presented here, although preliminary, provides useful information related to intervention quality. The data suggest that clinicians generally implemented core features of the program. The generally high ratings on fidelity items may reflect the selection process for enhanced services providers. In order to compete for a contract, agencies were required to provide a description of the brief, manualized intervention they would implement if selected. Most planned to provide cognitive-behavioral interventions that were not so different from the intervention developed for Project Liberty. Nevertheless, our simple evaluation strategy identified significant training effects. This suggests that such a strategy has good sensitivity.

This study also suggests that evaluations of treatment fidelity can be conducted in a cost-efficient manner. Using individuals' responses to questions as simple as, "Did your enhanced services counselor give you homework?" may help administrators determine whether training produces desired changes in practice, determine the extent of treatment fidelity across sites, identify compliance outliers, develop remedial procedures, and make other programmatic adjustments.

Assessment of clinicians' adherence to planned intervention elements is an important component of assessment of therapeutic outcomes. The evaluation method used here is a cost-effective approach to rating such intervention fidelity. At the very least, service recipients' assessments can be used as a first step in identifying sites to be the focus of more time-intensive training, supervision, and monitoring.

This evaluation was funded by grant FEMA-1391-DR-NY (titled "Project Liberty: Crisis Counseling Assistance and Training Program") to New York State from the Federal Emergency Management Agency. The Center for Mental Health Services of the Substance Abuse and Mental Health Services Administration administered the grant. This study was also funded by grants MH-60783, MH-30915, and MH-52247 from the National Institute of Mental Health to the University of Pittsburgh. The authors thank the Department of Veterans Affairs National Center for Post-Traumatic Stress Disorder and Schulman, Ronca, and Bucuvalas, Inc.

Felton C: Project Liberty: a public health response to New Yorkers' mental health needs arising from the World Trade Center terrorist attacks. Journal of Urban Health: Bulletin of the New York Academy of Medicine 79:429-433, 2002
 
North CS, Nixon SJ, Shariat S, et al: Psychiatric disorders among survivors of the Oklahoma City bombing. JAMA 282:755-762, 1999
 
Jackson CT, Covell NH, Shear KM, et al: The road back: predictors of regaining pre-attack functioning among Project Liberty clients. Psychiatric Services 57:1283-1290, 2006
 
Jackson CT, Allen G, Essock SM, et al: Clusters of event reactions among recipients of Project Liberty mental health counseling. Psychiatric Services 57:1271-1276, 2006
 
Ruggiero K, Morris TL, Scotti JR: Treatment of children with posttraumatic stress disorder: current status and future directions. Clinical Psychology: Science and Practice 8:210-227, 2001
 
Hopko DR, Lejuez CW, Ruggiero KJ, et al: Contemporary behavioral activation treatments for depression: procedures, principles, and progress. Clinical Psychology Review 23:699-717, 2003
 
Shear MK, Frank E, Foa EB, et al: Traumatic grief treatment: a pilot study. American Journal of Psychiatry 158:1506-1508, 2001
 
Norris FH, Donahue SA, Felton CJ, et al: A psychometric analysis of Project Liberty's Adult Enhanced Services Referral Tool. Psychiatric Services 57:1328-1334, 2006
 
Oagles BM, Lambert MJ, Fields SA: Essentials of Outcome Assessment. New York, Wiley, 2002
 
Lambert MJ, Hawkins EJ: Measuring outcome in professional practice: considerations in selecting and using brief outcome instruments. Professional Psychology: Research and Practice 35:492-499, 2004
 
Essock SM, Covell NH, Weissman EM: Inside the black box: the importance of monitoring treatment implementation. Schizophrenia Bulletin 30:613-615, 2004
 
 
Table 1  Use of essential techniques incorporated in Project Liberty's enhanced services intervention, by respondents from sites where all versus some clinicians had been trained in this specific intervention
Table 1  Use of essential techniques incorporated in Project Liberty's enhanced services intervention, by respondents from sites where all versus some clinicians had been trained in this specific intervention
+

References

Felton C: Project Liberty: a public health response to New Yorkers' mental health needs arising from the World Trade Center terrorist attacks. Journal of Urban Health: Bulletin of the New York Academy of Medicine 79:429-433, 2002
 
North CS, Nixon SJ, Shariat S, et al: Psychiatric disorders among survivors of the Oklahoma City bombing. JAMA 282:755-762, 1999
 
Jackson CT, Covell NH, Shear KM, et al: The road back: predictors of regaining pre-attack functioning among Project Liberty clients. Psychiatric Services 57:1283-1290, 2006
 
Jackson CT, Allen G, Essock SM, et al: Clusters of event reactions among recipients of Project Liberty mental health counseling. Psychiatric Services 57:1271-1276, 2006
 
Ruggiero K, Morris TL, Scotti JR: Treatment of children with posttraumatic stress disorder: current status and future directions. Clinical Psychology: Science and Practice 8:210-227, 2001
 
Hopko DR, Lejuez CW, Ruggiero KJ, et al: Contemporary behavioral activation treatments for depression: procedures, principles, and progress. Clinical Psychology Review 23:699-717, 2003
 
Shear MK, Frank E, Foa EB, et al: Traumatic grief treatment: a pilot study. American Journal of Psychiatry 158:1506-1508, 2001
 
Norris FH, Donahue SA, Felton CJ, et al: A psychometric analysis of Project Liberty's Adult Enhanced Services Referral Tool. Psychiatric Services 57:1328-1334, 2006
 
Oagles BM, Lambert MJ, Fields SA: Essentials of Outcome Assessment. New York, Wiley, 2002
 
Lambert MJ, Hawkins EJ: Measuring outcome in professional practice: considerations in selecting and using brief outcome instruments. Professional Psychology: Research and Practice 35:492-499, 2004
 
Essock SM, Covell NH, Weissman EM: Inside the black box: the importance of monitoring treatment implementation. Schizophrenia Bulletin 30:613-615, 2004
 
+
+

CME Activity

There is currently no quiz available for this resource. Please click here to go to the CME page to find another.
Submit a Comments
Please read the other comments before you post yours. Contributors must reveal any conflict of interest.
Comments are moderated and will appear on the site at the discertion of APA editorial staff.

* = Required Field
(if multiple authors, separate names by comma)
Example: John Doe



Web of Science® Times Cited: 8

Related Content
Articles
Books
The American Psychiatric Publishing Textbook of Substance Abuse Treatment, 4th Edition > Chapter 33.  >
Gabbard's Treatments of Psychiatric Disorders, 4th Edition > Chapter 22.  >
Textbook of Psychotherapeutic Treatments > Chapter 27.  >
Textbook of Psychotherapeutic Treatments > Chapter 27.  >
Textbook of Traumatic Brain Injury, 2nd Edition > Chapter 25.  >
Topic Collections
Psychiatric News
PubMed Articles