0
Get Alert
Please Wait... Processing your request... Please Wait.
You must sign in to sign-up for alerts.

Please confirm that your email address is correct, so you can successfully receive this alert.

1
Brief Reports   |    
Community Mental Health Provider Modifications to Cognitive Therapy: Implications for Sustainability
Shannon Wiltsey Stirman, Ph.D.; Amber Calloway, B.A.; Katherine Toder, B.A.; Christopher J. Miller, Ph.D.; Andrea K. DeVito, B.A.; Samuel N. Meisel, B.A.; Regina Xhezo, M.S.; Arthur C. Evans, Ph.D.; Aaron T. Beck, M.D.; Paul Crits-Christoph, Ph.D.
Psychiatric Services 2013; doi: 10.1176/appi.ps.201200456
View Author and Article Information

Dr. Wiltsey Stirman is affiliated with the Women’s Health Sciences Division, Veterans Affairs (VA) National Center for PTSD at the VA Boston Healthcare System, and with the Department of Psychiatry, Boston University, 150 S. Huntington Ave. (116B3), Boston, MA 02130 (e-mail: sws@bu.edu). Ms. Calloway is with the Department of Psychology, University of Massachusetts, Boston. Ms. Toder, Dr. Beck, and Dr. Crits-Christoph are with the Department of Psychiatry, University of Pennsylvania, Philadelphia. Dr. Miller is with the Center for Healthcare, Organization, and Implementation Research, VA Boston Healthcare System. Ms. DeVito and Mr. Meisel were with the Department of Psychiatry, Boston University, when this research was conducted. Ms. Xhezo and Dr. Evans are with the Philadelphia Department of Behavioral Health and Intellectual disAbility Services.

Copyright © 2013 by the American Psychiatric Association

Objective  This study identified modifications to an evidence-based psychosocial treatment (cognitive therapy) within a community mental health system after clinicians had received intensive training and consultation.

Methods  A coding system, consisting of four types of contextual modifications, 12 types of content-related modifications, seven levels at which modifications can occur, and a code for changes to training or evaluation processes, was applied to data from interviews with 27 clinicians who treat adult consumers within a mental health system.

Results  Nine of 12 content modifications were endorsed, and four (tailoring, integration into other therapeutic approaches, loosening structure, and drift) accounted for 65% of all modifications identified. Contextual modifications were rarely endorsed by clinicians in this sample. Modifications typically occurred at the client or clinician level.

Conclusions  Clinicians in community mental health settings made several modifications to an evidence-based practice (EBP), often in an effort to improve the fit of the intervention to the client’s needs or to the clinician’s therapeutic style. These findings have implications for implementation and sustainability of EBPs in community settings, client-level outcomes, and training and consultation.

Abstract Teaser
Figures in this Article

When evidence-based psychosocial interventions (EBPs) are implemented and disseminated in practice settings, modifications to the EBP models are common, and perhaps even necessary. However, the frequency and impact of EBP modifications in mental health service settings remain poorly understood. The few studies that have investigated modifications to EBPs have focused on group interventions and have found that modifications are intended to promote a better fit between the intervention and the setting or recipients (1,2). However, understanding the impact of such modifications on key outcomes, including consumer-level outcomes such as symptoms and functioning, and on long-term implementation, is also critical (3). Yet the variety of possible modifications to EBPs has created difficulty in classifying such changes rigorously and reliably and in identifying their impact.

Better characterization of commonly occurring modifications is an important first step to understanding their impact on key outcomes of interest. In addition, data regarding common modifications in a particular context can guide professionals tasked with training and implementation in the numerous initiatives to implement psychosocial EBPs throughout public mental health settings (46). Thus, in this study, we applied a framework for classifying modifications to a range of modifications described by clinicians who had been trained in cognitive therapy, an established EBP.

Cognitive therapy teaches individuals to develop and use skills for modifying problematic beliefs and behaviors through a variety of interventions (7). Sessions are structured to include brief summaries of previous and current sessions, agenda and goal setting, skill development, feedback from clients, and practice assignments (7). Cognitive therapy has an extensive evidence base for a variety of psychiatric disorders (8).

This study was conducted in the context of a cognitive therapy training program within a provider network in a large urban mental health system between 2007 and 2012. Clinicians were trained to use cognitive therapy to address depression, suicidal behaviors, and conditions that commonly co-occur with depression among adults in community mental health settings (9). Training consisted of 24 workshop hours, followed by six months of weekly consultation. Twenty-seven clinicians agreed to participate in interviews regarding their experience using the therapy after the training. Twenty of them had demonstrated the ability to conduct cognitive therapy sessions at an acceptable level of skill and fidelity at the end of the training, and all indicated that they used cognitive therapy interventions in their routine practice. Sixty-seven percent (N=18) of the participants were female, and participants had a mean±SD of 5.7±1.6 years of experience working in mental health treatment settings. Eighty-one percent (N=21) had a master’s degree, 7% (N=2) had completed some graduate work (such as toward a master’s degree), and 12% (N=4) had a bachelor’s degree. Seventy percent (N=19) of participants were Caucasian, 19% (N=5) were black, and 4% (N=1) were Asian; 7% (N=2) were also Latino, and 7% (N=2) were multiracial or endorsed a different race or ethnicity.

The study was approved by university and municipal institutional review boards, and clinicians provided informed consent. Hour-long, semistructured interviews included specific questions that assessed participants’ implementation of cognitive therapy, including any modifications they made to its delivery or content and the rationale for those modifications.

We used a coding scheme to identify modifications to cognitive therapy that was previously developed on the basis of articles describing modifications to EBPs across various interventions and settings (10). The coding scheme consists of four contextual modifications (for example, change in setting or format), 12 content modifications (such as adding or omitting components), seven levels at which modifications could occur (such as consistently throughout a system or organization or only for particular clients), and a code for modifications to training or evaluation processes. Content modifications were rated for the level at which they occurred and for the type and nature of the modification. Modification codes were applied to interview responses related to cognitive therapy modification. The two primary study raters (one bachelor’s level and one doctorate level) overlapped on 23% (N=41) of the identified interview segments, thus allowing calculation of rater agreement. The primary rater’s data are reported, but where disagreement existed in the ratings that overlapped, the two raters arrived at consensus through discussion.

Cohen’s kappa statistic was .80 for the nature of the modification and .95 for the level at which the modifications occurred, suggesting substantial to almost perfect interrater agreement (11).

Across the 27 interviews with clinicians, 175 modifications were identified. All clinicians described at least one modification. Table 1 lists modifications, their relative frequencies, and the number of clinicians endorsing each. Contextual modifications were rarely endorsed by this sample of clinicians, although delivery of cognitive therapy in different settings (such as at a client’s home) or through a different format (in a group or by telephone, for example) were occasionally noted.

 
Anchor for Jump
Table 1Modifications and adaptations to cognitive therapy by community mental health providersa
Table Footer Note

a A total of 175 modifications were identified by 27 therapists. Thirteen modifications that were identified in the interviews did not contain sufficient detail to be rated.

The most common content modification was tailoring the intervention to the needs of the client. Most frequently, clinicians endorsed changing terminology or language. Loosening the structure of cognitive therapy, such as not following an agenda for therapy sessions, was also common. Clinicians acknowledged that they integrated elements of cognitive therapy into a different therapeutic approach when they believed that a full cognitive therapy protocol was not an ideal fit either for their client or, more commonly, for themselves as clinicians.

More than half of the clinicians acknowledged that they sometimes drifted or departed from a cognitive therapy approach. In contrast to most other modifications in the coding scheme, which often appeared to be strategically determined when considering the needs of individual clients, drift was described as occurring less strategically and more in reaction to challenging situations. In approximately half of the interview segments for which drift was identified, it appeared to signal a discontinuation of cognitive therapy with individual clients. In the remaining half, clinicians indicated that they would return to a cognitive therapy approach in a subsequent session if the client seemed more able or willing to engage in the treatment.

Other content modifications that were endorsed somewhat less frequently included integrating another intervention (such as motivational interviewing or spiritual counseling) into cognitive therapy, adding specific interventions or modules (for example, a relapse prevention module), lengthening or shortening sessions or protocol length, and skipping particular interventions or strategies. On a few occasions, clinicians described what they believed to be modifications that raters judged to be consistent with cognitive therapy principles (problem solving, for example).

Content modifications most commonly occurred to address the needs or clinical presentations of individual clients. However, one-third of the modifications occurred at the clinician level, meaning that clinicians indicated that there were certain modifications that they routinely applied to cognitive therapy to increase their own comfort level or sense of fit between cognitive therapy and their personal style. Clinicians also stated that they adapted cognitive therapy for clients in particular diagnostic or demographic groups. Modifications across an entire clinic or organization were rarely endorsed.

In this study, we classified the types of modifications that clinicians made to cognitive therapy in community mental health settings. Four of the modifications identified in the coding scheme—tailoring, integrating cognitive therapy into other approaches, loosening the structured elements of the session, and drift—accounted for 65% of the changes that were identified. These changes have potentially important implications for several key implementation outcomes. Perhaps with the exception of drift, these modifications appear to improve clinician satisfaction with cognitive therapy, and clinicians believed that some modifications increased the intervention’s acceptability to clients. However, whether these changes affect other client-level outcomes, such as symptoms and functioning, remains to be seen.

Although no studies have examined the impact of adaptations that increase acceptability of cognitive therapy or other psychotherapies, some research findings imply that lower fidelity may have a negative impact on client-level outcomes (12). Additional research is necessary to fully understand the implications of such changes. More than half of the clinicians acknowledged that they at least occasionally drifted away from a cognitive therapy approach, and others indicated that they borrow elements of cognitive therapy but work primarily from another approach, which suggests that cognitive therapy may not be fully sustained in its original form throughout the system over the long term. Thus, when training clinicians in EBPs, it may be beneficial to proactively address the potential for modifications. Discussion and consultation about adaptation that upholds core elements of the intervention, as well as when and how another therapeutic approach might be appropriate, may provide useful guidance to clinicians who are likely to be confronted with complex client presentations.

Some limitations are important to note. First, ratings were based on clinician descriptions of modifications, rather than on observations of sessions. Other modifications may have occurred, or clinician and observer perspectives may have differed concerning what constituted a modification to cognitive therapy. Interviews provided valuable insight into why clinicians made changes to cognitive therapy, but the addition of observation may have led to greater precision. Also, we did not investigate the relationship between the modifications that were identified and implementation- or client-level outcomes. These are important topics for future research. Finally, our sample comprised a small number of clinicians from agencies in one mental health system and focused on a single EBP. Additional research on other EBPs, within other settings, is under way and will be necessary to better understand how, when, and with what impact EBPs are modified during implementation efforts.

This study provides further evidence that raters can reliably use a coding system to classify modifications to EBPs. In contrast to a previous study that applied this classification system to descriptions of adaptations that were mostly planned by treatment developers and researchers in advance of implementation (10), this report describes the variety of modifications that were made by individual clinicians when implementing an EBP in their everyday practice. Thus these findings have important implications for training and consultation as well as for future research on implementation and sustainability of EBPs. Further research on the impact of EBP modifications on implementation and client-level outcomes is necessary.

The preparation of this report was supported through grant R00 MH01800 from the National Institute of Mental Health (NIMH). While the study was being conducted, Dr. Wiltsey Stirman was receiving training from the Implementation Research Institute at the George Warren Brown School of Social Work, Washington University in St. Louis, which is funded through NIMH award R25 MH080916-01A2 and the Department of Veterans Affairs, Health Services Research and Development Service, Quality Enhancement Research Initiative. The authors are grateful to the providers who participated in the interviews and the agencies that participated in the adult outpatient training program. J. Bryce McLaulin, M.D., and Matthew Hurford, M.D., and the program’s mentors and consultants also played important roles in facilitating this research. The content is solely the responsibility of the authors and does not necessarily represent the official views of NIMH, the National Institutes of Health, or the Department of Veterans Affairs.

The authors report no competing interests.

Lundgren  L;  Amodeo  M;  Cohen  A  et al:  Modifications of evidence-based practices in community-based addiction treatment organizations: a qualitative research study.  Addictive Behaviors 36:630–635, 2011
[CrossRef] | [PubMed]
 
Aarons  GA;  Miller  EA;  Green  AE  et al:  Adaptation happens: a qualitative case study of implementation of The Incredible Years evidence-based parent training programme in a residential substance abuse treatment programme.  Journal of Children’s Services 7:233–245, 2012
[CrossRef]
 
Wiltsey Stirman  S;  Kimberly  J;  Cook  N  et al:  The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research.  Implementation Science 7:17, 2012
[CrossRef] | [PubMed]
 
McHugh  RK;  Barlow  DH:  The dissemination and implementation of evidence-based psychological treatments: a review of current efforts.  American Psychologist 65:73–84, 2010
[CrossRef] | [PubMed]
 
Karlin  BE;  Brown  GK;  Trockel  M  et al:  National dissemination of cognitive behavioral therapy for depression in the Department of Veterans Affairs health care system: therapist and patient-level outcomes.  Journal of Consulting and Clinical Psychology 80:707–718, 2012
[CrossRef] | [PubMed]
 
Karlin  BE;  Ruzek  JI;  Chard  KM  et al:  Dissemination of evidence-based psychological treatments for posttraumatic stress disorder in the Veterans Health Administration.  Journal of Traumatic Stress 23:663–673, 2010
[CrossRef] | [PubMed]
 
Beck  AT;  Rush  AJ;  Shaw  BF  et al:  Cognitive Therapy of Depression .  New York,  Guilford, 1979
 
Butler  AC;  Chapman  JE;  Forman  EM  et al:  The empirical status of cognitive-behavioral therapy: a review of meta-analyses.  Clinical Psychology Review 26:17–31, 2006
[CrossRef] | [PubMed]
 
Stirman  SW;  Miller  CJ;  Toder  K  et al:  Perspectives on cognitive therapy training within community mental health treatment settings: implications for clinician satisfaction and skill development.  Depression Research and Treatment  (Epub Sept 29, 2012)
 
Stirman  SW;  Miller  C;  Toder  K  et al:  Development of a framework and coding system for modifications made to evidence-based programs and interventions.  Implementation Science 8:65, 2013
[CrossRef] | [PubMed]
 
Landis  JR;  Koch  GG:  The measurement of observer agreement for categorical data.  Biometrics 33:159–174, 1977
[CrossRef] | [PubMed]
 
References Container
Anchor for Jump
Table 1Modifications and adaptations to cognitive therapy by community mental health providersa
Table Footer Note

a A total of 175 modifications were identified by 27 therapists. Thirteen modifications that were identified in the interviews did not contain sufficient detail to be rated.

+

References

Lundgren  L;  Amodeo  M;  Cohen  A  et al:  Modifications of evidence-based practices in community-based addiction treatment organizations: a qualitative research study.  Addictive Behaviors 36:630–635, 2011
[CrossRef] | [PubMed]
 
Aarons  GA;  Miller  EA;  Green  AE  et al:  Adaptation happens: a qualitative case study of implementation of The Incredible Years evidence-based parent training programme in a residential substance abuse treatment programme.  Journal of Children’s Services 7:233–245, 2012
[CrossRef]
 
Wiltsey Stirman  S;  Kimberly  J;  Cook  N  et al:  The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research.  Implementation Science 7:17, 2012
[CrossRef] | [PubMed]
 
McHugh  RK;  Barlow  DH:  The dissemination and implementation of evidence-based psychological treatments: a review of current efforts.  American Psychologist 65:73–84, 2010
[CrossRef] | [PubMed]
 
Karlin  BE;  Brown  GK;  Trockel  M  et al:  National dissemination of cognitive behavioral therapy for depression in the Department of Veterans Affairs health care system: therapist and patient-level outcomes.  Journal of Consulting and Clinical Psychology 80:707–718, 2012
[CrossRef] | [PubMed]
 
Karlin  BE;  Ruzek  JI;  Chard  KM  et al:  Dissemination of evidence-based psychological treatments for posttraumatic stress disorder in the Veterans Health Administration.  Journal of Traumatic Stress 23:663–673, 2010
[CrossRef] | [PubMed]
 
Beck  AT;  Rush  AJ;  Shaw  BF  et al:  Cognitive Therapy of Depression .  New York,  Guilford, 1979
 
Butler  AC;  Chapman  JE;  Forman  EM  et al:  The empirical status of cognitive-behavioral therapy: a review of meta-analyses.  Clinical Psychology Review 26:17–31, 2006
[CrossRef] | [PubMed]
 
Stirman  SW;  Miller  CJ;  Toder  K  et al:  Perspectives on cognitive therapy training within community mental health treatment settings: implications for clinician satisfaction and skill development.  Depression Research and Treatment  (Epub Sept 29, 2012)
 
Stirman  SW;  Miller  C;  Toder  K  et al:  Development of a framework and coding system for modifications made to evidence-based programs and interventions.  Implementation Science 8:65, 2013
[CrossRef] | [PubMed]
 
Landis  JR;  Koch  GG:  The measurement of observer agreement for categorical data.  Biometrics 33:159–174, 1977
[CrossRef] | [PubMed]
 
References Container
+
+

CME Activity

There is currently no quiz available for this resource. Please click here to go to the CME page to find another.
Submit a Comments
Please read the other comments before you post yours. Contributors must reveal any conflict of interest.
Comments are moderated and will appear on the site at the discertion of APA editorial staff.

* = Required Field
(if multiple authors, separate names by comma)
Example: John Doe



Web of Science® Times Cited: 1

Related Content
Articles
Books
Gabbard's Treatments of Psychiatric Disorders, 4th Edition > Chapter 22.  >
Dulcan's Textbook of Child and Adolescent Psychiatry > Chapter 30.  >
Dulcan's Textbook of Child and Adolescent Psychiatry > Chapter 30.  >
Gabbard's Treatments of Psychiatric Disorders, 4th Edition > Chapter 48.  >
Gabbard's Treatments of Psychiatric Disorders, 4th Edition > Chapter 48.  >
Topic Collections
Psychiatric News
Read more at Psychiatric News >>
PubMed Articles