Independent Contractors in Public Mental Health Clinics: Implications for Use of Evidence-Based Practices
Abstract
Objectives:
Community mental health clinics are increasingly utilizing independent contractors to provide clinical services. At the same time, many organizations are participating in initiatives intended to increase implementation of evidence-based practices (EBPs). The primary aim of this study was to understand the associations of utilizing independent contractors with clinician knowledge and attitudes toward EBPs and organizational culture and climate. The study also sought to understand the potential impact of using independent contractors on mental health services delivery from the perspective of organizational leadership.
Methods:
Quantitative data were collected from 130 therapists in 23 organizations; qualitative data were collected from executive administrators in nine of the 16 organizations participating in EBP initiatives sponsored by the City of Philadelphia. Regression with random effects was used to estimate the associations between worker status (contractor or employee) and clinician attitudes toward EBPs, knowledge of EBPs, and organizational culture and climate. Qualitative inquiry was used to understand the impact of reliance on independent contractors on organizational participation in EBP initiatives.
Results:
Independent contractors endorsed less positive attitudes toward EBPs and scored lower on knowledge of EBPs. Interviews revealed four main themes: reasons for using independent contractors, general consequences of using independent contractors, specific impact of independent contractors on participation in EBP initiatives, and suggestions for alternatives.
Conclusions:
A growing number of community mental health clinics rely on independent contractors. There may be consequences of this shift that deserve exploration.
Over the past 15 years, there has been an increased emphasis on the crisis facing the behavioral health workforce, including lack of providers and concerns about effectiveness of services (1). An effort to address these issues was undertaken by the Annapolis Coalition on the Behavioral Health Workforce, culminating in a document outlining strategic goals (2), including broadening the concept of workforce, strengthening the workforce, and building structures to support the workforce (3). Relatedly, the last 15 years have also seen a growing interest in the implementation of evidence-based practices (EBPs) by community mental health clinics, largely in response to policy mandates (4). This is consistent with the recommendations made by the Annapolis Coalition to buttress the competencies in EBPs of the behavioral workforce (5).
One important matter related to the behavioral health workforce that has received little attention is the growing number of independent contractors in specialty community mental health settings (personal communication, Manderscheid R, 2015). Salaried employees receive employee benefits and have productivity requirements, whereas independent contractors typically neither receive benefits nor have productivity requirements. The organizational management literature suggests that utilizing independent contractors destabilizes the workforce because it may result in conflict between salaried employees and independent contractors (6) and may increase turnover for both groups (7). In community mental health settings, the increasing reliance on independent contractors may have sequelae for both mental health services delivery and implementation of EBPs. Given that independent contractors are not employees per se, organizations may be less likely to invest in their professional development, resulting in potentially poorer knowledge of and attitudes toward EBPs. Reliance on independent contractors may also affect organizational culture and climate by increasing stress within an organization.
Conversely, the use of independent contractors may have a positive impact on services and EBP implementation because it may allow organizations to be more nimble in hiring individuals with the skill sets deemed necessary for the setting.
The first aim of this study was to explore the associations between independent contractors and therapist and organizational characteristics. These factors have been identified as potentially important predictors of use of EBP strategies (8,9). No a priori hypotheses were specified. Using quantitative methods, we compared attitudes and knowledge of EBPs among independent contractors and salaried clinicians. Next, we examined the relationship between organizational culture and climate and the ratio of independent contractors within each organization. The second aim of the study was to use qualitative methods to explore stakeholder perspectives on the impact of using independent contractors in a subset of organizations participating in EBP initiatives.
Methods
Setting
The Philadelphia Department of Behavioral Health and Intellectual disAbility Services (DBHIDS) has supported EBPs in the public mental health system through four initiatives: cognitive therapy (10), prolonged exposure, trauma-focused cognitive-behavioral therapy, and dialectical behavior therapy. These initiatives include training and consultation in line with treatment developer recommendations and entail a significant allocation of resources on the part of clinicians, organizations, and the system.
Agencies
Estimates provided by Community Behavioral Health suggest that more than 100 community public mental health agencies in Philadelphia provide outpatient services to youths. We used purposive sampling to recruit outpatient public mental health agencies that served the largest percentage of youths. Out of the total number of agencies in 2013, there were 29 that serve approximately 80% (N=23,354) of youths receiving publicly funded mental health outpatient care, and thus we targeted those agencies. Of these 29 agencies, 18 (62%) agreed to participate. Representatives of another organization participating in an EBP initiative asked to participate, resulting in a final sample of 19 agencies with 23 sites, 130 clinicians, and 36 supervisors. Sixteen of the 23 organizations reported participating in EBP initiatives. There were no exclusion criteria for staff.
Procedures
This research was approved by the City of Philadelphia and University of Pennsylvania institutional review boards. The leader of each organization was approached to solicit organizational participation. A two-hour meeting was held to explain the study to clinicians, during which we obtained informed consent and collected data. All participants were compensated $50 and provided lunch. Quantitative data were collected in 2013.
The qualitative data were collected as part of a follow-up study in which we conducted interviews with agency leadership to understand the process of participating in EBP initiatives sponsored by the City of Philadelphia (11). We used purposive sampling to specifically target the agencies in our sample that reported participating in EBP initiatives (N=16). When approaching agencies, we asked leaders to identify individuals who would have an understanding of operations, workforce issues, and agency and system factors needed to support EBP initiatives. Data were collected in 2013–2014. Respondents received an additional $50 for participation.
Measures
Clinician and organizational characteristics.
Demographic characteristics of the clinicians were assessed with the Therapist Background Questionnaire (12). We also asked participants to report on worker status (independent contractor or salaried), hours worked weekly, and participation in DBHIDS EBP initiatives. Supervisors provided information on the number of clinicians in their unit and the clinicians’ worker status. From that information, we determined the percentage of staff who were independent contractors. DBHIDS provided us with a list of organizations that had participated in EBP initiatives, the year they began participation, and completion year of participation in each EBP initiative (if applicable). From that information, we calculated the cumulative years of participation in DBHIDS EBP initiatives for each organization.
Dependent variables.
Clinician attitudes were assessed with the Evidence-Based Practice Attitude Scale (EBPAS) (13), a self-report questionnaire that assesses appeal of EBPs, requirements for using EBPs, general openness to new practices, and divergence between EBPs and usual practice. The EBPAS demonstrates good internal consistency and validity (14).
Clinician knowledge about EBPs was measured with the Knowledge of Evidence-Based Services Questionnaire (KEBSQ), a 40-item self-report instrument that measures general knowledge of EBPs (15). Knowledge is measured on a continuum from 0 to 160, with higher scores indicative of more knowledge. Psychometric data suggest temporal stability, discriminative validity, and sensitivity to training (15).
Organizational culture and climate were measured with the Organizational Social Context Measurement System (OSC) (16), a 105-item measure of the social context of mental health organizations. Organizational culture includes proficiency, rigidity, and resistance, and climate includes engagement, functionality, and stress. Proficient cultures are those in which clinicians prioritize the well-being of clients and are expected to be competent. Rigid cultures are those in which clinicians have little autonomy. Resistant cultures refer to environments in which clinicians are expected to be apathetic. Engaged climates refer to settings in which clinicians feel they can accomplish worthwhile goals and remain invested. Functional climates are those in which clinicians can get their job done effectively. Stressful climates refer to settings in which clinicians feel emotionally exhausted. Organizational culture and climate were measured with T scores (mean±SD=50±10) based on a normed sample of 100 community mental health clinics (16). The OSC has strong psychometric properties (17).
Semistructured interviews.
We developed a semistructured interview guide to collect information about stakeholders’ experiences with the four DBHIDS EBP initiatives, particularly agency characteristics and system characteristics that affected implementation (11). We included follow-up questions specifically oriented toward understanding the relationships between utilizing independent contractors and participation in EBP initiatives.
Data Analytic Plan
Quantitative analysis.
Analyses were conducted by using PROC MIXED and PROC REG in SAS, version 9.0. Less than 10% of data were missing for predictor variables; series means were imputed for missing predictor variables. Minimal data (3%) were missing for dependent variables.
We used five mixed-effects linear regression models to evaluate the associations between clinician factors (worker status, age, hours worked per week, and participation in EBP initiatives [yes or no]) and knowledge and attitudes about EBPs. These mixed-effects models included random intercepts for organization and fixed effects for staff factors.
We also used six fixed-effects regression models in which the organization was the unit of observation to evaluate the associations among the ratio of independent contractors and cumulative years of participation in city-sponsored EBP initiatives and organizational culture and climate. Organizational culture and climate were included in analyses by aggregating responses of individual therapists within the organization (18,19), given that rwg, a commonly used statistic for aggregation of organization constructs, was substantially above the suggested .60 level (19,20).
Qualitative analysis.
Transcripts were analyzed in an iterative process on the basis of modified grounded theory (21–24). Through a close reading of a subset of transcripts, the investigators developed a set of codes that were applied to the data. A codebook was developed that included codes emerging from the transcripts, such as workforce, as well as a priori codes, such as barriers, that were derived from the original research questions that pertained to barriers to and facilitators of participating in EBP initiatives (for that study, 56 interviews were completed [11]). A subset of 12 transcripts (21%) was coded by two investigators, and interrater reliability was excellent (κ=.98) (25). Through an inductive process, the first author independently read through a subset of data from agencies participating in EBP initiatives to examine workforce themes related to independent contractors, producing memos including examples and commentary to reach consensus regarding newly derived, emergent themes (22,23). The second author read through and coded 22% of the same data (two of nine interviews) for reliability purposes.
Results
Quantitative findings
Table 1 provides demographic information about the clinicians who participated in the quantitative portion of the study. Approximately 49% (N=130) of 268 therapists providing services in the 23 organizations participated in the study; 57% (N=74) of the 130 therapists were independent contractors.
| Variable | N | % |
|---|---|---|
| Gendera | ||
| Male | 30 | 23 |
| Female | 99 | 76 |
| Race-ethnicitya | ||
| Asian | 6 | 5 |
| Black or African American | 27 | 22 |
| White | 67 | 55 |
| Hispanic or Latino | 13 | 11 |
| Multiracial | 5 | 4 |
| Other | 5 | 4 |
| Academic backgrounda | ||
| Bachelor’s degree | 5 | 4 |
| Master’s degree | 107 | 82 |
| Doctoral degree | 12 | 9 |
| Licensure statusa | ||
| Yes | 32 | 25 |
| No | 51 | 39 |
| In process | 42 | 32 |
| Age (M±SD) | 38.09±11.63 | |
| Years at current organization (M±SD) | 3.35±4.65 |
TABLE 1. Characteristics of 130 clinicians at public mental health clinics in Philadelphia
Table 2 presents the results of the analyses predicting clinician attitudes and knowledge. Table 3 presents the results of the analyses predicting organizational culture and climate.
| Attitudesa | |||||
|---|---|---|---|---|---|
| Variable | Requirements | Appeal | Openness | Divergence | Knowledgeb |
| M±SD score | 2.77±1.07 | 3.16±.68 | 3.07±.70 | 1.35±.71 | 94.50±9.90 |
| Predictor estimate | |||||
| Independent contractor (reference: salaried) | .09 | –.28* | .02 | –.16 | –5.12* |
| Age | –.01 | –.01* | –.01 | .01 | –.18* |
| Hours worked weekly | –.02* | –.01 | .00 | –.00 | –.16* |
| Participated in EBP initiative (reference: did not participate) | .22 | .02 | .18 | –.09 | 4.94* |
TABLE 2. Predictors of attitudes and knowledge related to evidence-based practices (EBPs) among 126 clinicians at community mental health clinics
| Variable | Proficiency | Rigidity | Resistance | Engagement | Functionality | Stress |
|---|---|---|---|---|---|---|
| M±SD scorea | 52.59±11.55 | 58.30±6.94 | 64.97±9.60 | 53.77±7.04 | 62.97±13.06 | 56.05±8.56 |
| Predictor estimate | ||||||
| Ratio of independent contractors to salaried workers | 7.54 | –2.57 | –4.75 | 1.87 | 9.40 | –11.17† |
| Cumulative years of EBP participationb | 1.77† | .56 | .56 | .19 | 1.85† | .04 |
TABLE 3. Predictors of organizational culture and climate at 23 public mental health clinics
Participation in EBP initiatives.
An ad hoc analysis using a logistic mixed-effects model for organizations participating in EBP initiatives found no association between worker status and participation in an EBP initiative. Of the 130 study participants, 41 therapists at the 16 organizations participated in the organizations’ EBP initiatives, and 43 did not.
Attitudes.
Independent contractors had less favorable attitudes toward adopting EBPs. Specifically, they endorsed being less willing to adopt EBPs even if they found them appealing. In addition, older clinicians were also less willing to adopt EBPs that they found appealing. Clinicians who worked fewer hours per week had more open attitudes toward adopting EBPs more generally.
Knowledge.
Independent contractors, younger clinicians, clinicians who worked fewer hours per week, and clinicians who did not participate in EBP initiatives scored lower on knowledge of EBPs. Given that both worker status and participation in EBP initiatives were significant predictors of knowledge, we created an interaction term that combined the two variables (worker status × participation in EBP initiatives) to our model. The interaction term was not significant, and therefore we removed it from the model.
Organizational culture and climate.
The ratio of independent contractors and cumulative years of participation in EBP initiatives did not significantly predict organizational proficiency, rigidity, resistance, engagement, functionality, or stress. Marginally significant associations between cumulative years of participation in EBP initiatives and proficiency (p=.06) and functionality (p=.10) were observed. A marginally significant negative association between proportion of independent contractors and stress was observed (p=.10).
Qualitative Findings
Nine executive administrators from nine organizations were interviewed. Six participants were female, and three were male. Six participants were white, two were Hispanic or Latino, and one was American Indian/Alaska Native. The mean±SD age was 45.0±8.4. Four participants came from salaried organizations (>50% of outpatient therapists were salaried), and four came from independent contractor organizations (>50% of outpatient therapists were contractors). Of the nine organizations, two had transitioned within the past year to using only independent contractors.
A majority of individuals noted the financial difficulties of maintaining salaried employees in the current fiscal environment for mental health services and that these difficulties led to a need for independent contractors (Table 4). A minority of participants noted that using independent contractors removes financial risk from the organization and places it on the therapist because therapists are not paid if a client does not attend a session. Participants also noted the general ramifications of using independent contractors, including high staff turnover, employment instability, lack of connection to the organization, and potentially reduced quality of services.
| Theme | N | % | Example |
|---|---|---|---|
| Reasons for switching to the independent-contractor model | |||
| Financial difficulties of the salaried model | 6 | 67 | “The outpatient department was so horribly underpaid so we made that transition [to independent contractors], and so now the outpatient department is not the huge loss leader. I ran the outpatient department when it was salaried and I cannot tell you the financial stress of running that because your financial goals were literally impossible, and you knew no matter what [you] would do, you were going to be extremely [over] budget and looking at the faces of folks that are working so very hard.” |
| Contractor model transfers financial risk to the clinician | 2 | 22 | “If a patient doesn’t come, the agency doesn’t pay the clinician.” “The organization wants to pass that fee on to a contractor and say, ‘You know, we’ll support you going, but we can’t pay for it.’ So, you’re talking about travel expenses and all that, plus the not being paid while they’re going to the training.” |
| General consequences of the independent contractor model | |||
| Turnover | 3 | 33 | “[The contractor model] does lend itself to that kind of instability, kind of wondering. And, you know, I have yet to have a contractor give me proper notice.” |
| Time = money | 3 | 33 | “Contractor therapists are paid a piece rate, which means that they see somebody for an hour, and their rate is about $25.50 an hour. And, if the person doesn’t show up, they get nothing. Understood. They know that coming in. They work full-time hours, and they are making a lot of money, but in the back of their mind they are thinking about the other stuff they are doing that they are not getting paid for, such as note writing and [attending] team meetings. I recruited people who I can sense have a desire to learn things because I figured that would work best. But, it’s difficult, even if you have a desire to learn things, you still have to eat.” |
| Lack of recognition and connection to the organization | 2 | 22 | “The [agency] sometimes forgets about contractors in a sense that they do a lot of nice things for employees. Employees just got a raise or are getting ready to get a raise. We have a day that we all get together and celebrate and when we have that day, contractors go home generally. You know, so they are kind of left out, but at this point, you’re looking at that being almost half the agency.” “People that are contractors are not connected to the agency. So if you have a mission- and vision-driven work, you’re going to have a hard time making that connection. They don’t go to all-staff meetings, [and] they don’t go to group meetings; they come do the work, [and] they go.” |
| Uncertainty | 1 | 11 | “[T]hat also limits the kind of people that I can recruit because I have to find someone who is able to maintain themselves with no insurance and be able to consider that they might not get all the money every week.” |
| Reduced quality of services | 2 | 22 | “You lose a quality element as well. I’m not saying that independent contractors do poor work—what I’m saying is that my ability to develop and enhance and train individuals is also very limited.” |
| Consequences of the independent contractor model related to use of evidence-based practices (EBPs) | |||
| Impacts staff selection | 3 | 33 | “We only have contractors. So, the agency, even if they do have money, because we do have money for training, is not willing to invest money to send a contractor to an expensive training . . . that they may or may not get a return on.” |
| Contractors cannot meet initiative requirements | 4 | 44 | “But where the challenge has been is that we have people who are contractors, and therefore, their ability to commit time to the project is limited.” |
| Contractors have to be committed to learning EBPs because they are not paid for extra time | 1 | 11 | “I have to find contractors who are willing to do [EBPs], which I am not paying them for, including training and consultation, which means ultimately they would just have to be committed to the fact that they want to learn a skill.” |
| Incentive for contractors to use EBPs | 1 | 11 | “Per diem staff [independent contractors] are only paid if they’re doing a session. So, they are going to do what works with the people they see. There is an incentive there to do that.” |
| Alternatives to or ways to improve the independent contractor model | |||
| Shared-risk model | 1 | 11 | “For employees we have tuition reimbursement. When you sign up for tuition reimbursement, you also sign off that you will be at the agency for 2 years or you have to pay the money back. So similarly, I think they could include this in the contract of independent contractor clinicians.” |
| Primary care integration | 1 | 11 | “I think we’re going to keep the [salaried] model because we believe in it and in addition to that, it is going to be integrated in primary care and behavioral health. We’re going to try to go as far as we can with the health home models.” |
TABLE 4. Qualitative themes related to use of the independent-contractor model derived from interviews with nine executive administrators of community mental health clinics
Participants also noted the specific impact of using independent contractors on participation in EBP initiatives. Respondents stated that they do not send contractors to EBP training sessions because of the potential loss of investment due to contractors’ perceived transiency. Furthermore, the additional requirements of EBP initiatives, such as training and ongoing consultation, make it difficult to include contractors in these efforts.
A majority of participants shared their opinions about how to alter or improve the independent contractor model. Several participants noted that it is important to treat independent contractors with respect and provide them with professional development opportunities to increase their connection to the organization and reduce turnover. Examples included increasing the hourly rate of independent contractors with training in EBPs, paying for parking when independent contractors go to training sessions, and providing clinical support for independent contractors. Some participants advocated paying independent contractors to attend clinical support meetings by using a training rate (which is typically lower than their clinical hourly rate).
Discussion
Nationally, a growing number of community mental health clinics are relying on independent contractors. We used mixed methods to begin to understand the associations between worker status and factors associated with self-reported use of EBP strategies (9). Our quantitative results suggest that independent contractors reported less positive attitudes toward EBPs and less knowledge compared with salaried clinicians. Our qualitative results provide preliminary insights into why some organizations use independent contractors and the consequences of that choice.
Our quantitative data suggest that independent contractors were not less likely than salaried therapists to participate in EBP initiatives; however, this finding was not corroborated by the qualitative interviews. Furthermore, independent contractors demonstrated poorer knowledge of and attitudes toward EBPs, suggesting that contractors potentially have less access to professional development opportunities. Administrators stated that their organizations would be less likely to include independent contractors in EBP initiatives because of the greater likelihood of turnover among independent contractors (26) and because independent contractors are less able to participate in initiative requirements. These findings suggest that the increase in the number of organizations using independent contractors may pose a threat to implementation of EBPs.
Although these findings are preliminary, they have important implications for efforts to better train the behavioral health workforce (1–3,5,27,28). There may be a segment of the specialty provider workforce receiving less access to training opportunities. Raising awareness about this possibility is an imperative for future efforts to support the behavioral health workforce. It is important to note, however, that the clinician and organizational variables studied have not been identified as indicators of fidelity to EBPs; rather, they are factors that have been associated with self-reported use of EBPs. Future research should investigate the impact of using independent contractors on actual implementation of EBPs.
The proportion of independent contractors in an organization did not affect organizational culture and climate. However, the study was underpowered to detect small effects at the organizational level. For this reason, we included results significant at p<.1. These results suggest that organizations with more independent contractors may have less stressful climates. This finding may be explained by productivity requirements, given that salaried staff are expected to bill a certain number of weekly patient hours whereas independent contractors decide how many hours they would like to work each week, according to our informal conversations with stakeholders. Given the high no-show rate in community mental health settings (29), paid staff may feel increased pressure to bill or carry a high caseload, which may be reflected in a stressful climate. Although the results do not indicate causation, it is important to attend to the impact of worker status on climate because organizational climate is predictive of job performance (30,31) and client outcomes (32,33).
A number of important preliminary insights were gleaned from the qualitative interviews. Participants discussed the rationale for using independent contractors and their perceptions of the impact of such a model on their organization and staff. Overwhelmingly, participants discussed the dire financial environment in community mental health settings (26,34,35) that they believed contributed to the decision to use independent contractors rather than salaried employees. There may be alternatives for improving organizational fiscal health, such as teaching organizations better business practices (36) or changing federal and state policies to support these organizations. The general consequences of relying on independent contractors were perceived as potentially negative for both the organization and the staff. Stakeholders suggested that independent contractors were more likely to quit, leading to high turnover; were less likely to feel connected to their organization; and were more likely to provide poorer clinical services.
A number of study limitations should be noted. First, and most important, we did not have information about therapist behavior, thus precluding us from drawing conclusions about the actual implementation of EBPs. Second, not all therapists in the organizations participated in the quantitative portion and we did not directly talk with therapists in the qualitative portion. Third, our interviews were conducted with a small number of organizations in Philadelphia, so the generalizability is unknown. Fourth, administrators may be biased toward their fellow employees; for example, salary increases are tied to one another’s performance. Finally, we did not collect information regarding whether cases are assigned differentially to clinicians on the basis of their employment status. Assigning cases according to the unique strengths of contractors and employees may resolve some of the concerns identified in this study.
Conclusions
The findings from this study have important implications. Very little has been written about the increasing number of independent contractors in the behavioral health workforce; in a report to Congress on mental health workforce issues, no mention was made of this phenomenon (34). Nonscholarly discourse around this topic has been plentiful, given that many individuals, organizations, and systems working within and with specialty mental health outpatient services are affected by this shift. It is our hope that this initial investigation launches a systematic research agenda that delineates the number and characteristics of independent contractors in the behavioral health workforce, the characteristics of organizations utilizing independent contractors, and the associations with using independent contractors and EBP implementation efforts. Although the effect of independent contractors on EBP implemention is an important topic to examine from an implementation science lens, it has a much broader potential impact on mental health services delivery—an impact in critical need of study.
1 : Mental health and addiction workforce development: federal leadership is needed to address the growing crisis. Health Affairs 32:2005–2012, 2013Crossref, Medline, Google Scholar
2 : An Action Plan on Behavioral Health Workforce Development. Rockville, Md, Substance Abuse and Mental Health Services Administration, 2007Google Scholar
3 : A national action plan for workforce development in behavioral health. Psychiatric Services 60:883–887, 2009Link, Google Scholar
4 : The status of states’ policies to support evidence-based practices in children’s mental health. Psychiatric Services 60:1672–1675, 2009Link, Google Scholar
5 : Implementing best practices in behavioral health workforce education—building a change agenda. Administration and Policy in Mental Health and Mental Health Services Research 32:85–89, 2004Crossref, Medline, Google Scholar
6 : Determinants of employment externalization—a study of temporary workers and independent contractors. Administrative Science Quarterly 38:195–223, 1993Crossref, Google Scholar
7 : Happy together? How using nonstandard workers affects exit, voice, and loyalty among standard employees. Academy of Management Journal 46:475–485, 2003Crossref, Google Scholar
8 : Randomized trial of MST and ARC in a two-level evidence-based treatment implementation strategy. Journal of Consulting and Clinical Psychology 78:537–550, 2010Crossref, Medline, Google Scholar
9 : Predictors of community therapists’ use of therapy techniques in a large public mental health system. JAMA Pediatrics 169:374–382, 2015Crossref, Medline, Google Scholar
10 : Training and consultation in evidence-based psychosocial treatments in public mental health settings: the ACCESS model. Professional Psychology, Research and Practice 41:48–56, 2010Crossref, Medline, Google Scholar
11 : A multi-level examination of stakeholder perspectives of implementation of evidence-based practices in a large urban publicly-funded mental health system. Administration and Policy in Mental Health and Mental Health Services Research (Epub ahead of print, Dec 12, 2015)Google Scholar
12 : Therapist Background Questionnaire. Los Angeles, University of California, 1997Google Scholar
13 : Mental health provider attitudes toward adoption of evidence-based practice: the Evidence-Based Practice Attitude Scale (EBPAS). Mental Health Services Research 6:61–74, 2004Crossref, Medline, Google Scholar
14 : Psychometric properties and US national norms of the Evidence-Based Practice Attitude Scale (EBPAS). Psychological Assessment 22:356–365, 2010Crossref, Medline, Google Scholar
15 : Implementation of evidence-based services for youth: assessing provider knowledge. Behavior Modification 33:48–65, 2009Crossref, Medline, Google Scholar
16 : Assessing the organizational social context (OSC) of mental health services: implications for research and practice. Administration and Policy in Mental Health and Mental Health Services Research 35:98–113, 2008Crossref, Medline, Google Scholar
17 : Assessing the organizational social context (OSC) of child welfare systems: implications for research and practice. Child Abuse and Neglect 36:621–632, 2012Crossref, Medline, Google Scholar
18 : Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology 69:85–98, 1984Crossref, Google Scholar
19 : Interrater agreement reconsidered: an alternative to the r(wg) indices. Organizational Research Methods 8:165–184, 2005Crossref, Google Scholar
20 Bliese P: Within-group agreement, non-independence, and reliability: implications for data aggregation and analysis; in Multilevel Theory, Research, and Methods in Organizations: Foundations, Extensions, and New Directions. Edited by Klein KJ, Kozlowski SWJ. San Francisco, Jossey-Bass, 2000Google Scholar
21 : Therapist’ perspectives on the effective elements of consultation following training. Administration and Policy in Mental Health and Mental Health Services Research 40:507–517, 2013Crossref, Medline, Google Scholar
22 : Consensual qualitative research: an update. Journal of Counseling Psychology 52:196–205, 2005Crossref, Google Scholar
23 : A guide to conducting consensual qualitative research. Counseling Psychologist 25:517–572, 1997Crossref, Google Scholar
24 : Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implementation Science 8:65, 2013Crossref, Medline, Google Scholar
25 : The measurement of observer agreement for categorical data. Biometrics 33:159–174, 1977Crossref, Medline, Google Scholar
26 : A prospective examination of clinician and supervisor turnover within the context of implementation of evidence-based practices in a publicly-funded mental health system. Administration and Policy in Mental Health (Epub ahead of print, July 16, 2015)Google Scholar
27 : A Synthesis of Direct Service Workforce Demographics and Challenges Across Intellectual/Developmental Disabilities, Aging, Physical Disabilities, and Behavioral Health. Washington, DC, National Direct Service Workforce Resource Center, 2008Google Scholar
28 : Workforce competencies in behavioral health: an overview. Administration and Policy in Mental Health and Mental Health Services Research 32:593–631, 2005Crossref, Medline, Google Scholar
29 : Psychotherapy appointment no-shows: rates and reasons. Psychotherapy 47:413–417, 2010Crossref, Medline, Google Scholar
30 : The job satisfaction–job performance relationship: a qualitative and quantitative review. Psychological Bulletin 127:376–407, 2001Crossref, Medline, Google Scholar
31 : Organizational and psychological climate: a review of theory and research. European Journal of Work and Organizational Psychology 17:5–32, 2008Crossref, Google Scholar
32 : Organizational climate, services, and outcomes in child welfare systems. Child Abuse and Neglect 35:582–591, 2011Crossref, Medline, Google Scholar
33 : Testing a theory of organizational culture, climate and youth outcomes in child welfare systems: a United States national study. Child Abuse and Neglect 38:757–767, 2014Crossref, Medline, Google Scholar
34 : Report to Congress on the Nation’s Substance Abuse and Mental Health Workforce Issues. Rockville, Md, Substance Abuse and Mental Health Services Administration, 2013Google Scholar
35 : The perfect storm: collision of the business of mental health and the implementation of evidence-based practices. Psychiatric Services 67:159–161, 2016 Link, Google Scholar
36 : How to Maximize Service Capacity: Nuts and Bolts Solutions for Implementing Change in Behavioral Healthcare Organizations Below the Senior Management Level. Rockville, Md, National Council for Community Behavioral Healthcare, 1998Google Scholar
