System-Level Influences on the Sustainability of a Cognitive Therapy Program in a Community Behavioral Health Network

As programs to implement evidence-based psychotherapies (1–5) in mental health systems have proliferated and matured, the need for research to understand the conditions and processes necessary to sustain them also has increased. Sustainability is a multidimensional construct that has been conceptualized as being embedded within a system and involving the application of a program or an approach with ongoing fidelity, ongoing benefits to consumers, the continuation of program activities, continuous or increased capacity, and the extent to which partnerships are maintained (6,7). Recent research suggests mixed results for some aspects of sustainability, such as the degree of fidelity and penetration into the system (8,9). However, other reports of statewide implementations have documented sustained program activities, capacity, and partnerships. Consistent with implementation models (10), these reports suggest the importance of factors in the inner and outer context, such as favorable political climate for policy reform and mandated change (2,11), which are coupled with a multifaceted strategy, including collaboration with stakeholders and sustainable funding (5,12). Much of the literature has focused on national or statewide initiatives, and little guidance exists for local mental health systems and large behavioral health organizations that are implementing these treatments (13) in the absence of a broader mandate or policy. Furthermore, despite an increasing emphasis on dynamic relationships between these factors in the implementation literature (6,10), many studies to date have described individual contributing factors but without much discussion of how their interrelationships may influence implementation outcomes. An exploration of mutual influence can facilitate the development of a comprehensive and successful plan to promote sustainability in other local systems.

This study focused on a seven-year-old community-based partnership to implement cognitive therapy (CT), with the purpose of understanding factors that may influence the continuation of program activities to promote and support implementation. Evidence that this program has been sustained includes continued funding, ongoing involvement among the majority of agencies that completed training, enrollment of new agencies, and expansions to offer CT to additional populations and settings (14). Also, although employee turnover affected sustainability at the clinic level, a high proportion of the clinicians who remained at their clinics two years after CT training in a community-based implementation program demonstrated sustained CT fidelity (Stirman SW, Pontoski K, Creed T, et al., unpublished manuscript, 2014).

Methods

Setting and Context

In 2007, the Beck Initiative (BI) was formed as a partnership between the Philadelphia Department of Behavioral Health and Intellectual disAbilities Services (DBHIDS) and the University of Pennsylvania (Penn) to implement, within the city’s behavioral health provider agencies, a CT program for depression and suicide prevention and for disorders that commonly co-occur with depression (13,14). DBHIDS is an urban mental health system with over 300 provider agencies that provide mental health and substance abuse services to the city’s 470,000 Medicaid recipients and thousands of uninsured and underinsured individuals (13,14).

Data Sources

Analysis included interviews with key informants at the administrative level at DBHIDS, Penn, and participating agencies, as well as a document review. In-depth, semistructured interviews were conducted with clinic administrators as training began and again 12–24 months postconsultation. Interviews with system administrators and Penn BI personnel occurred four years after the program began. The primary interviewer was not affiliated with the BI. Interview guides (available by request) were developed to assess interrelated aspects of sustainability related to the outer context (such as in the service environment) and the inner context (including intraorganizational characteristics) and to assess intervention characteristics in the implementation and “sustainment” phases of Aarons and colleagues’ (10) implementation framework and other research and theory on sustainability (6,10,12,15). Study investigators developed targeted questions and probes by mapping questions onto each key construct (16), with a framing question (17) included at the beginning to orient participants and allow them to express opinions about key facilitators and barriers to the program’s sustainability.

A document review was conducted with 100% of the existing BI documentation to complement the interviews and provide additional insight into policies and processes that may have influenced sustainability. Available documents included publicity and Web-based materials, notes, minutes, and all available summaries from BI workshops and consultation sessions (N=16), internal meetings (N=193), BI policy documentation and reports between 2007 and 2012 (N=10), and reports and publications (N=6).

Analysis

We used a directed content analysis (16) approach to validate and extend existing theory on sustainability. Digitally recorded interviews were transcribed and checked for accuracy. We developed a preliminary codebook of operationalized, a priori codes based on existing theory and research. Using NVIVO 10 software, three authors (SWS, AM, and JG) coded transcripts from different stakeholders and identified additional codes through consensus to finalize the codebook and specify links between key concepts. Two coders (AM and JG) who were not affiliated with the BI then coded all transcripts. A subsample was coded by all coders, who also audited each other’s work and resolved disagreements via consensus in weekly meetings. Policies, internal documentation, and summary memos based on reviews of the meeting minutes were coded with the same codebook and process. Queries were then run to identify codes and relationships related to program sustainability. Central themes were determined by diversity and triangulation across data sources, by the frequency with which potential influences were identified, and by attention to key decision points and interactions described in data sources.

Results

Out of 28 administrators and training consultants available at the time of the interviews, 24 participated in interviews specifically about their experiences with the BI, yielding an 86% response rate. Three additional training consultants and one DBHIDS administrator left the system after the first year of the initiative; because no additional themes were identified by the end of our interviews with the existing sample (17), we did not solicit their participation. Thus 75% of all administrators and training consultants who had been involved with the BI were interviewed. The sample comprised DBHIDS (N=3) and Penn (N=4) administrators, training consultants (N=8), and agency administrators and supervisors (N=9). Participants identified as Caucasian (75%, N=18), African American (17%, N=4), and Asian (8%, N=2). Sixty-two percent of participants were female (N=15), and 38% were male (N=9); 54% (N=13) had a master’s degree, and 46% (N=11) had an M.D. or Ph.D. degree. Data on participant ages were not collected.

Analyses revealed key interconnections between constructs identified in theories of sustainability, which are described below and further illustrated in Figure 1 and Table 1. Table 2 describes the number of stakeholders in each category who endorsed each theme.

Collaboration and Communication

The strong collaborative relationship between DBHIDS, Penn, and participating agencies was noted to be a fundamental facilitator of program sustainability by the majority of stakeholders (Table 2). As a Penn administrator stated, “With DBHIDS administration, we’ve had a very positive, very reciprocal, and very collaborative, very respectful relationship. I don’t think we could have done this without that very strong relationship with them.”

Also noted in interviews, and apparent in policy documentation and meeting minutes, were processes and policies that clarified the roles and responsibilities of BI personnel. For example, the DBHIDS program director was a key liaison between the agencies, Penn, and DBHIDS and was tasked with ensuring that appropriate expectations were set and that training and posttraining activities were running smoothly. Challenges were generally resolved by program directors and agency administrators through established procedures, without escalation to higher levels of administration. Meeting minutes and policy documents clearly indicated who would collect and maintain different types of evaluation data, which was to be shared between Penn and DBHIDs for quality improvement. This clarity helped reduce the potential problem of “too many cooks in the kitchen.” Collaborators also demonstrated trust in the abilities of all of the individuals within the partnership to fulfill their unique responsibilities. Transitions in leadership went smoothly, according to all Penn and DBHIDS administrators; for example, “If anything [after a leadership transition], we’re meeting more frequently now . . . to make sure that we’re laying down the proper rails for communication.”

The collaborative relationship extended to multiple levels within and across agencies. Meeting minutes and administrator interviews indicated that the commissioner of DBHIDS routinely discussed BI activities at networkwide meetings to share information and publicize successes. Operational meetings to assess agency needs, clarify what participation in the program entailed, and troubleshoot throughout the training phase further fostered the collaboration. Agency directors and clinical supervisors also participated in quarterly BI meetings to facilitate cross-agency communication (Table 1). Nearly all key informants noted that these strategies to promote communication and collaboration were critical to the initiative’s success. As described below and in Table 1, the collaborative relationships had a positive impact on other facilitators by promoting further planning and support, which contributed to perceived benefits and program evolution.

Sociopolitical Influences and Alignment With System and Agency Priorities

Document review, supported by interviews with all three DBHIDS administrators and other stakeholders (Table 2), indicated that the initiative was viewed as an important early component of a broader effort within the system. A DBHIDS administrator stated, “The department has made an explicit priority of identifying, implementing, disseminating, and evaluating evidence-based and emerging best practices. The Beck Initiative for a number of years has been the crown jewel of the project in this regard.” Positioning the program within these broader efforts ensured that as priorities emerged from sociopolitical influences (Table 1), the system looked for ways to expand the BI to support these goals.

Intervention Fit and Flexibility

BI training consultants, Penn, and DBHIDS administrators noted that CT’s applicability to a broad range of presenting problems and populations allowed the program to evolve and change in response to sociopolitical influences and system- and agency-level needs. Administrators at four agencies noted the different programs and populations that CT reached through the BI at their agencies (Table 1). As a DBHIDS administrator explained, “There’s something about the adaptability and also the universality of cognitive therapy that has blended itself extremely well and has also expanded across the system.”

Program Adaptation and Evolution

As Figure 1 indicates, each of the aforementioned influences played a role in the ways in which the program evolved and the model was adapted. The collaborative relationship and confidence in the flexibility of CT led DBHIDS to ask the BI to address emerging system priorities and to meet the needs of the populations served by DBHIDS. Multiple data sources indicate that the program evolved primarily to offer CT to additional populations, and at times the expansion necessitated additional context-level adaptations, such as the delivery of CT outside of an outpatient setting (including inpatient stays and home visits) and the training of multiple disciplines on treatment teams (for example, psychiatric assistants and vocational specialists) to support CT delivery by clinicians. As noted by a BI training consultant (Table 1), collaboration between participating agencies, Penn, and DBHIDS ensured that these expansions went smoothly and increased perceived benefits.

Evaluation and Perceived Benefit

Meeting minutes and program evaluation data documented consistent tracking of clinicians’ satisfaction with training, fidelity scores during training and every two years thereafter, and information on the continuation of CT activities and consultation within the clinics. This evaluation indicated that training and program-level goals were largely met (Table 1). However, there was agreement that consumer-level evaluation was a challenge. Table 1 outlines efforts and challenges related to individual-level outcomes. Administrators acknowledged that a standard system for evaluation at the consumer level had not been developed and expressed a commitment to developing evaluation procedures. However, DBHIDS administrators and seven agency administrators noted important benefits to the BI, including the development of a trained workforce and benefits to consumers. These perceived benefits appeared to positively affect the collaboration and to result in ongoing support and further program expansion (Figure 1).

Planning and Support for Sustainability

The supports and policies put into place by DBHIDS and Penn leadership within the first year, and documented throughout the years in internal meeting minutes, indicate that sustaining the program was not an afterthought and that considerable effort was made to support and sustain the overall program and the use of CT within the agencies.

Ongoing technical support and consultation.

A requirement was put into place early in the initiative that participating agencies commit to ongoing internal CT consultation and periodic fidelity assessment after the training phase had ended, with ongoing support from Penn, which was perceived to be essential (Tables 1 and 2). Agencies also needed support to expand or maintain their cohort of CT providers because of turnover (Table 1). All Penn and DBHIDS administrators and four agency administrators discussed the impact of the Web-based training developed for this purpose. All DBHIDS administrators also noted plans to require both sustainability and evaluation plans for requests for proposals (RFPs) for participation in the BI or other programs, as well as a new Evidence-based Practice and Innovation Center developed to support implementation of evidence-based practices throughout the system.

Financial support.

Financial support was primarily a facilitator of sustainability, although some aspects of the current funding strategy were noted to be potential and actual challenges (Table 1). Funding for program administration, training, and reimbursement to agencies for lost productivity during training and consultation activities has been provided by DBHIDS through annual reinvestment dollars accrued through DBHIDS’ administrative services organization’s effective management of the Medicaid behavioral health contract. Medicaid billing covered the delivery of CT by trained providers in most cases. Challenges that were noted included uncertainty related to availability of reinvestment funds, constraints of state Medicaid budgets, and the impact of lost revenue at the agency level associated with staff’s participation in BI activities after training ended. This uncertainty limited the scope and scale of new initiative activities at times (Table 1).

Penn and DBHIDS administrators recognized that without a financing model independent of reinvestment funds, the initiative would be in jeopardy in the absence of an annual surplus. Meeting minutes and early memos indicated plans since the inception of the program, and in the last year of the study, DBHIDS began to experiment with alternative funding models (Table 1). New models could also help agencies absorb costs related to training and ongoing consultation, given that meeting minutes and interviews with three agency administrators noted that these costs were a challenge to sustainability. In addition, because the use of evidence-based treatments is now a core expectation, a DBHIDS administrator noted that evaluation of responses to RFPs includes prioritization of providers who demonstrate that they use them, stating, “We’ve actually changed the incentives within our system such that being trained . . . puts you at a competitive advantage within the system.”

Discussion

This study identified a number of interrelated contextual factors that emerged as potential influences on sustainability of an evidence-based treatment in a city’s behavioral health system. Many findings, particularly those related to the outer context (sociopolitical influences, system needs, funding, and collaboration), as well as intervention fit and the need for ongoing support, were consistent with Aarons and colleagues’ model (10), and our findings also support Pluye and colleagues’ (18) emphasis on planning for sustainability. Some aspects, such as perceived benefits, are captured in other implementation models (19), and the concepts of evaluation and program adaptation map most closely onto Chambers and colleagues’ dynamic sustainability framework (20). That the findings do not map exactly onto a single model of implementation or sustainability suggests that different aspects of individual implementation models may be more and less relevant, depending on the context and nature of the intervention.

It is noteworthy that sociopolitical influences could have shifted priorities away from CT as new or broader initiatives occurred (21), particularly because CT implementation was not a state or national mandate. In this case, CT was not supplanted by new system-level priorities, but its flexibility allowed it to be applied to differing contexts to meet emerging needs. The need to broaden the application of CT to ensure sustainability is consistent with some of the recommendations regarding the use of practice-based efforts to adapt and optimize interventions (20,22), although evaluation of program impact on consumer outcomes has been a considerable challenge for the BI as well as other programs (23). Such evaluation remains an important future direction, especially because policy makers may require evidence of consumer-level benefits before supporting ongoing implementation. In addition, our findings suggest that to be sustained, implementation programs may require openness to mutual adaptation, which is closely linked with quality management strategies, such as the development of processes and tools for evaluation, small cycles of change and evaluation when adapting interventions, and provision of ongoing consultation (24). Although a majority of themes that we identified appeared to be positive influences on program sustainability, funding appeared to be both a facilitator and a barrier. Our findings suggest that several finance strategies are indicated for sustainability, including incentives for evidence-based practice delivery, increasing referrals, and diversifying funding sources (24). Development of a workable funding model can take time, so early planning and piloting of funding strategies can reduce the likelihood that programs will end when the initial funding source is no longer available.

Finally, our findings support the recent literature that suggests that understanding interconnections between individual influences is critical to advancing understanding of sustainability in dynamic systems (6,20). When selecting implementation strategies to promote sustainability, considering how each strategy may ultimately interact with and affect other key facilitators of sustainability is important. For example, during planning, it may be important to anticipate shifts in priorities and emerging needs to select treatments that may lend themselves to adaptation or flexible application. Similarly, when selecting strategies for quality assurance or financing, understanding how they will meet immediate needs and understanding how they may fit over the long term and affect collaborative relationships at all levels are important. Relationship development strategies, such as coalition building, formal agreements regarding roles and activities, and community-academic partnerships (24), may facilitate this understanding.

This article makes a unique contribution to the literature in its focus on interrelated influences on the sustainability of a program to implement evidence-based mental health treatments at the local system level in the absence of a broader mandate. However, some limitations are to be noted. First, this study focused on a single program within a single mental health system, and some potential key informants were not interviewed. The perspectives of clinicians and consumers of mental health services were not included, because preliminary coding of interviews with them through a larger program of research identified little material relevant to sustainability at the system level, as opposed to the organization level or individual level. The qualitative nature of the study precludes conclusions about causal relationships between factors identified here and sustainability. Further, this study did not investigate all aspects of sustainability. It is possible that despite fairly robust activity at the system level, variation will be present at agency or provider levels and that different influences will be more salient. Investigation of the extent to which the factors identified in the interviews are associated with quantitative measures of sustainability (such as long-term CT fidelity and sustained improvement in client outcomes) requires other types of research designs.

Conclusions

Through an analysis of program documentation and stakeholder attributions, this study investigated potential facilitators of sustained activities in a program to implement evidence-based psychotherapy in a city’s behavioral health program. How the successes at this level, and the decisions to expand and broaden the program, influence activities at organization and provider levels is an important topic of ongoing research, particularly as established implementation efforts mature and seek to sustain their achieved capacity. Findings from this program of research may inform efforts of other local systems to sustain implementation programs and may contribute to the refinement of sustainability models that are applicable to local and regional implementation.