Costs to Community Mental Health Agencies to Sustain an Evidence-Based Practice

The growing interest in improving patient outcomes and the passage of the Affordable Care Act have mobilized efforts to increase access to high-quality, cost-efficient evidence-based practices (EBPs). Although research on strategies and efforts to disseminate and implement EBPs is increasing (1–3), research suggests that rates of long-term sustainment of EBPs are relatively low (4). Cost is one of the most significant barriers to sustainment of EBPs in community mental health agencies (CMHAs), yet EBPs are rarely reimbursed at higher rates than usual care despite evidence that they save money over the long term (5). This study was designed to describe the incremental implementation costs of providing an EBP compared with usual care.

Implementation costs have been cited as the most significant and least modifiable barrier to EBP implementation and sustainment (6). Costs are one of eight implementation outcomes relevant for evaluating EBP dissemination, yet there has been less research of implementation costs than of any other EBP implementation outcome (7). For CMHAs, EBP implementation costs include the time, effort, and monetary expenses associated with activities necessary to implement an EBP during all phases of implementation (7). Staff may need to attend training sessions (which may involve travel), engage in model-specific consultation and supervision, report and interpret data, modify data systems, provide quality assurance, review model materials, administer and score measures, modify agency practices and policies, and educate other agency staff or external partners about the EBP. Many of these implementation costs fall on the shoulders of the CMHA tasked with delivering the EBP (8). Implementation activities require additional staff time that may negatively affect the CMHA, possibly by reducing revenue.

Several studies have explored implementation costs during early stages of the implementation process (9,10). However, little research has been published on the sustainment phase of EBP implementation (11,12). Further, virtually no studies have examined implementation costs associated with the sustainment phase. The costs borne by CMHAs during sustainment may differ from costs associated with initial implementation, and understanding these sustainment costs is critical to fully integrate and maintain EBPs. For instance, one study reported that only 23 (47%) of 49 CMHAs sustained an EBP over a six-year period after initial implementation (4). Among the sites that discontinued the EBP, 95% cited costs as the primary reason. Thus sustainment costs are especially important to understand because initial implementation costs, although often quite high (11), are typically funded through a time-limited grant that does not support the sustainment costs of the EBP indefinitely.

Implementation costs are defined in this study as the additional costs in hours, effort, and dollars of implementing an EBP compared with usual care. These costs are critical to agencies’ decision to adopt, implement, and sustain an EBP. This study is the first to our knowledge that provides a method for quantifying the incremental costs of an EBP, and it provides preliminary data to answer the following research questions: First, what is the incremental cost to an agency to sustain an EBP? Second, what is the average annual incremental cost to an agency per child served and staff member trained of sustaining the EBP? Third, what activities account for the largest costs of sustaining an EBP? Fourth, to what extent do agencies provide productivity reductions to clinicians providing the EBP? Last, are the characteristics of an agency related to incremental costs?

Methods

Background

The Connecticut Department of Children and Families (DCF) contracted with the Child Health and Development Institute, an independent nonprofit organization with expertise in EBP dissemination, to disseminate trauma-focused cognitive-behavioral therapy (TF-CBT) to 16 community-based agencies in Connecticut from 2007 to 2010 (13). Agencies selected a subset of staff clinicians to be trained through the statewide dissemination efforts. TF-CBT is an outpatient treatment for traumatic stress reactions among children and adolescents ages three to 17 who have experienced a traumatic event (14). TF-CBT has demonstrated efficacy in randomized controlled trials (15) and has been successfully disseminated in community mental health settings (9). TF-CBT was the first widely implemented EBP at these CMHAs. The initial Connecticut dissemination was conducted from 2007 to 2010 by using the Breakthrough Series Collaborative model (16), after which all agencies continued providing TF-CBT for three to six years and received external training and consultation support through a state-funded TF-CBT Coordinating Center (17). Data were collected for quality improvement purposes, and the initiative was deemed exempt by the DCF Institutional Review Board.

Participants

Agency leaders from the 16 CMHAs that participated in the initial TF-CBT dissemination were invited to participate, and all agreed. Two agencies submitted incomplete data and were excluded from the analysis, resulting in valid data from 14 agencies. Each CMHA included at least one clinician, supervisor, and senior leader (for example, an agency administrator with oversight of children’s services) who provided or supported TF-CBT directly.

Measures

Survey development.

The Sustainment Cost Survey (SCS) is a 49-item measure developed for this study to collect administrative data on the costs to CMHAs—both in time and money—associated with sustaining an EBP following initial implementation. Items were initially generated through focus groups with CMHA senior leaders and were developed by the second author through an iterative process of revision across all participating agencies. Each item was generated to capture activities that pertained specifically to TF-CBT and that otherwise would not be conducted for usual care (such as TF-CBT training sessions). The survey was designed to be completed by the agency administrator with input from team members as needed.

The SCS includes information about agency characteristics (such as total staff size, EBP staff size, and agency budget), average salaries, various activities related specifically to TF-CBT sustainment, and questions about agency processes. These activities included training; consultation and supervision; data systems and data reporting requirements; clinical preparation, such as reviewing the TF-CBT treatment manual; and additional costs, such as costs for site coordinator managerial duties, not required for usual care. For example, a survey item related to training asked, “How many staff attended training by a TF-CBT trainer in the past 12 months?” A consultation and supervision item was used to note the amount of “staff time to attend TF-CBT consultation calls/Webinars with TF-CBT trainers.” Respondents reported the amount of time spent by staff on these activities on a weekly, monthly, or annual basis (all hours were converted to annual estimates). Each agency also reported the weekly productivity (billable hours) requirement for clinicians and noted any recurring (for example, weekly) productivity reductions related specifically to TF-CBT. For instance, agencies were asked, “Describe any standard productivity reductions related to TF-CBT.”

Incremental lost revenue (LR) and additional staff time (AST).

In the SCS, agency costs are categorized by three different types of costs. First, LR hours were defined as time spent by staff that resulted in a reduction in a billable session and that would not have been spent for usual care. LR time differed from the “standard productivity reductions” described above because LR hours did not represent ongoing reductions in productivity requirements for clinicians. Instead, they represented revenue lost for other reasons, such as revenue lost for several days while a clinician attended in-person training. Second, AST was defined as time spent on TF-CBT activities that would otherwise not be required for usual care and for which there was no loss of revenue (for example, time spent participating in consultation calls for which clinicians did not receive a productivity reduction). Finally, expenses were defined as direct monetary costs to the agency that were required or recommended to provide TF-CBT but not usual care; these included travel expenses, supplies, and other materials.

Data Collection Procedures

Surveys were distributed to 16 agencies in June 2013, and data collection ended in November 2013, by which time each agency had provided TF-CBT for four to five years. Agencies were requested to retrospectively report on the past year’s TF-CBT activities as part of the TF-CBT Coordinating Center’s quality improvement activities. To ensure accuracy of data collection, agency leaders were asked to work collaboratively with administrators and review records at their agency to complete the survey. Most agencies returned surveys within two to three months (mean±SD=2.8±1.4) of initial receipt.

Once the survey was completed, a rigorous review process was conducted to ensure accurate data. First, authors reviewed the surveys for missing data, discrepancies, and miscalculations. Common issues included whether an activity was categorized as LR or AST, duplication of costs in multiple categories, and miscalculation of hours. Next, agencies were contacted via e-mail if there were questions about their responses, and the agencies provided clarification; the review process was repeated as necessary. On average, surveys were sent back twice to agencies for further clarification; agency surveys were finalized through correspondence with the authors in approximately four months (3.9±.7). Throughout the review process, the authors created and used a decision tree to standardize guidance and decisions.

Conversion of Hours to Dollars

All time spent on TF-CBT activities was reported in hours. Agency reports of hours were standardized to obtain an annual figure assuming 48 working weeks per year, accounting for vacation and sick leave. These hours were summed and averaged across and within agencies. All monetary amounts were reported in U.S. dollars and were adjusted for the rate of inflation from 2013 to 2014.

Conversion of LR.

To convert hours of LR to a monetary cost, the sum of LR hours per agency was multiplied by the cost per hour for child outpatient psychotherapy. The standard rate for 13 of the CMHAs was $73.99 per session; one CMHA received a standard rate of $59.19, per state regulations. We then adjusted the LR by using an indicator of average rate of attendance (86%±6%), which was based on reported no-show rates for TF-CBT appointments at each agency.

Cost of AST.

The total monetary cost of AST was determined by multiplying the average number of hours of AST for each staff position. Because the SCS did not specifically query for hours by each staff position, we used the proportion of staff in each position to estimate how many hours that staff position spent on TF-CBT activities within each agency. Next, we multiplied AST hours for each position by the hourly agency salary for senior leaders, supervisors, and clinicians, respectively, and then summed the total costs for each agency. These costs represented the monetary cost of AST spent on sustaining TF-CBT.

Estimated full-time-equivalent staff.

Total time associated with LR and AST was converted into estimates of the full-time-equivalent (FTE) staff associated with providing TF-CBT. Total hours were divided equally across staff members to estimate the average FTE per staff member associated with providing TF-CBT. For example, agency 1 reported providing 2,975.75 total hours for TF-CBT activities annually across 19 TF-CBT staff members. Thus we estimated each TF-CBT staff member at agency 1 spent an average of 156.62 hours per year, or .08 FTE, on TF-CBT activities.

Results

Agency Characteristics

The 14 CMHAs varied in size and staffing. Agencies reported annual outpatient child mental health services budgets ranging from $756,350 to $3 million ($1,818,503±$652,753). A summary of agency characteristics, including number of staff trained in TF-CBT and number of children who received TF-CBT, is included in Table 1.

Clinician salaries ranged from $53,799 to $78,042 ($70,968±$11,959). Supervisor salaries ranged from $72,421 to $125,448 ($92,519±$18,495). Senior administrator salaries ranged from $81,900 to $193,536 ($130,893±$31,939).

Incremental Implementation Costs

CMHAs varied widely on which activities they reported as LR or AST, meaning CMHAs varied in the extent to which they made occasional reductions in billable session requirements for staff (Table 2). The amount of time spent by the average staff member on TF-CBT activities was the equivalent of .04 FTE staff. Table 3 summarizes all monetary costs of LR, AST, and direct expenses and provides estimates of costs by child served and staff member trained. These incremental costs to sustain TF-CBT accounted for an average of 3.81%±1.44% of each CMHA’s annual budget.

Implementation Costs by Activity

Table 4 summarizes the LR and AST hours and agency expenses by sustainment activity. The most time (sum of LR and AST) was spent on consultation (for example, internal TF-CBT coaching and supervision), followed by “other” activities, training, practice, and data requirements. Agencies reported the greatest expenses for “other” activities, primarily related to the time spent by site coordinators to manage the TF-CBT team. Overall, AST hours were greater than LR hours, suggesting that most of the additional time required for implementation did not result in associated productivity reductions for staff.

Agency Productivity Reductions

Only three of 14 CMHAs reported any formal reduction in productivity requirements for TF-CBT staff. That is, although most agencies reported LR hours, this loss was limited to specific one-time activities and was not associated with a regular reduction in billable requirements. Of the three agencies that provided an ongoing reduction in productivity requirements, the reduction was equal to .5 to 1.0 billable hours, or sessions, per week (.83±.29).

Agency Characteristics and Implementation Costs

Exploratory analyses were conducted to examine how agency characteristics (budget, total staff size, and TF-CBT staff size) were related to costs. Table 5 presents bivariate two-tailed Pearson correlation analyses, which revealed that the agency’s budget for outpatient child services was positively correlated with total sustainment costs (sum of AST, LR, and expenses) (r=.61, p<.05). Agencies that provided a standard productivity reduction demonstrated greater total sustainment costs (t=5.87, df=10, p<.01) compared with agencies that did not provide a standard productivity reduction.

Discussion

This study outlined a novel approach to quantifying incremental costs incurred by CMHAs to sustain an EBP and provided preliminary data to estimate costs. Our results suggest that there was great heterogeneity across agencies in implementation costs ($25,641–$102,976), costs per client ($513 to $4,646), and costs per staff ($1,709 to $8,581). Given the high variability in costs, findings should be interpreted cautiously pending further replication.

Nevertheless, the wide variability in costs provides insight into how agencies absorb costs and into agency efficiency. The positive correlation between agency budget and total costs suggests that larger agencies were better able to absorb costs and may be more likely to invest in sustaining TF-CBT, for example, by training more staff. Nonsignificant correlations between costs and variables such as clients served may be more reflective of agency efficiencies and inefficiencies. For example, agency 13 reported training 23 TF-CBT staff and providing TF-CBT to 83 children, whereas agency 2 reported training 26 staff and providing TF-CBT to far fewer children (N=16). Given the high staff turnover rates in many community mental health settings, it is possible that efforts to prevent having too few clinicians trained in TF-CBT resulted in agencies training too many staff at once or not identifying enough children to receive TF-CBT.

Our preliminary findings highlight another issue related to agency efficiency—variation in LR and AST costs. There was great variation in LR and AST costs within and across agencies, but there was less variability across agencies in total (combined LR and AST) costs. This indicates that there was some consistency in the amount of total time staff spent on the EBP across agencies but there was less consistency in how agencies accounted for this time. Indeed, some CMHAs provided regular or occasional reductions in productivity requirements for activities, whereas others required clinicians to absorb the additional time without such accommodations. The negative correlation between LR and AST is likely explained by this distinction.

Despite wide variability across agencies, all agencies reported significant costs to sustain the EBP, with few agencies reporting any formal reductions in productivity to reduce staff burden. The costliest activities included training, consultation, and “other” activities (primarily for a site coordinator). Although TF-CBT is just one EBP, it is a fairly representative EBP for outpatient children’s mental health treatment in terms of the general activities needed to sustain it. The implications of these variations in how agencies sustain EBPs and what activities are costliest are ripe for future research.

There were several important limitations inherent in the cost estimates in this study. First, retrospective estimates may be less reliable than prospective studies and may be subject to greater measurement error. Prospective cost studies requiring regular logging of activities may provide more accurate estimates, albeit with greater burden on CMHA staff. Second, all estimates were self-reported by staff, who used agency data when possible. Future studies might seek to validate the survey instrument against agency billing data. It is also important to note that agencies in this study received external sustainment support from the statewide TF-CBT Coordinating Center, including the external costs of TF-CBT trainers. Thus, sustaining an EBP without this support could create additional costs for CMHAs, but it could also reduce the number of additional responsibilities related to the EBP, for example, by requiring less use of standardized measures or data entry. Finally, it is possible that demand characteristics of this study led agency leaders to overestimate costs.

An important area for future research is the incorporation of implementation costs into comprehensive cost-effectiveness models. These future studies should incorporate potential cost savings not included in this study, such as reduced no-show rates, less staff turnover, and reduced length of treatment compared with usual care. Furthermore, more research is needed in the area of understanding usual care practices in order to help interpret incremental costs of the EBP. One of the challenges with community-based psychotherapy research is that usual care varies greatly and is difficult to define (18). Future replication studies could employ mixed-methods designs to quantitatively account for incremental costs and savings of sustaining an EBP and qualitatively explore usual care practices and cost variations across agencies.

Conclusions

Results suggest that reimbursement models for supporting EBPs must consider the costs of sustainment. When this study was conducted, there was no difference between the reimbursement rate for TF-CBT and usual care. The lack of commensurate staff productivity reductions to alleviate the burden of sustaining the EBP is concerning because it places agencies in a difficult position of choosing to implement and sustain an EBP while incurring staff and agency burden.

Policies to incentivize EBPs over usual care may increase the proliferation of EBPs and high-quality care, potentially saving money in the long term (17). Although costs to sustain an EBP may be substantial, there may be savings that accrue to other actors in the system, for example, fewer emergency department visits, that could be allocated to support the EBP. Another implication of these findings is the need to develop innovations in EBPs and implementation strategies that can reduce costs. For example, development of components-based EBPs that are appropriate for a range of clinical concerns is likely to create efficiencies (19). Alternatively, development of less complex EBPs that are simpler to implement and require less training and consultation time would reduce costs. Ultimately, a public health approach to improving the quality of care will require balancing the efficacy of EBPs with the costs required to implement and sustain them across a system of care.