More money is spent on treatment for mental illness among children in the United States than for any other childhood medical condition (1). Unfortunately, outcome data on the effectiveness of community-based psychotherapeutic care are discouraging (2,3,4), and virtually nothing is known about what types of psychotherapeutic care are delivered in usual care settings, making it difficult to know how to target care improvement (5,6,7). National research and policy initiatives call for dissemination and implementation of evidence-based practices in usual care (8). These efforts could be more efficient and sustainable if informed by a better understanding of the current care context (9,10)—that is, "it is difficult and perhaps foolhardy to try to improve what you do not understand" (6).

Studies of children's psychotherapeutic usual care have focused primarily on examining outcomes as opposed to treatment processes. On average, findings regarding the effectiveness of usual care youth psychotherapy reflect minimal impact on children's symptom severity or functional status (2,3,4). However, none of these studies included detailed descriptive data about the nature of the usual care treatment; therefore, attributions regarding the links between usual care treatment processes and outcomes are largely speculative.

In one of the only studies of the nature of usual care for children receiving publicly funded mental health services, Zima and colleagues (11) used a chart review method to assess quality of care. Approximately half of the 813 cases studied met broad quality-of-care indicators for psychosocial treatment. However, the authors acknowledged that chart review was not a good data source for details about psychotherapy practice. Characterizing psychotherapeutic treatment processes is methodologically complex (12). Treatment can be characterized across a continuum ranging from broad molar classifications (for example, theoretical orientations) to molecular-level detailed verbal and nonverbal behaviors (13). Assessment of clinical strategies represents an intermediate level of analysis and includes assessment of therapeutic techniques (for example, modeling skills and interpreting the meaning of behavior) and content (for example, problem-solving skills and family member roles) (14). The most objective method to assess clinical strategies in psychotherapy is direct observation, but it is labor intensive (12).

This study utilized observational assessment to examine clinical strategies delivered in publicly funded usual care for children with disruptive behavior problems and their parents. This patient population was selected because the vast majority of youths in publicly funded care are referred for disruptive problems, including oppositional, defiant, aggressive, and delinquent behavior (15). Effective treatment is essential because children with disruptive behavior problems are at significantly elevated risk of multiple maladaptive outcomes in adolescence and adulthood, including criminal behavior and psychopathology (16,17). Further, families served in the public sector are at particularly high risk because of multiple risk factors and life stressors (11).

Many psychotherapy treatment models have demonstrated efficacy for children with disruptive behavior problems (18,19,20,21), and several common core elements across individual evidence-based treatment models have been identified (22,23). Common elements of evidence-based treatment for this patient population include clinical strategies directed toward children (for example, affect or anger management) and toward parents or caregivers (hereafter referred to as parents) (for example, principles of positive reinforcement and limit setting). It is assumed that delivery of complete evidence-based treatment models is rare in usual care (21); however, no studies have assessed the extent to which usual care includes treatment strategies generally consistent with common elements of evidence-based practice. Empirical data that identify specific convergence and divergence between elements of evidence-based practice and usual care are needed to provide a road map for targeted efforts to improve care.

The goals of this study were to provide a detailed description of usual care psychotherapy for a representative sample of children with disruptive behavior problems treated in publicly funded outpatient clinics and to examine the extent to which therapists employ strategies conceptually consistent with common elements of evidence-based practices in order to inform efforts to implement evidence-based practices in usual care.

Participants

Clinics. The six participating clinics were selected because they represented the largest contractors for publicly funded, clinic-based outpatient care for children in one of the largest counties in the United States. These clinics serve a racially, ethnically, and diagnostically diverse group of children and their families. None of the clinics specialized or provided therapist training in delivery or incorporation of evidence-based treatments before or during the study period.

Therapists. In late 2003, all therapists from participating clinics were assigned random numbers and recruited sequentially until a sample was gathered that reflected the county's therapist distribution by discipline, training, and licensure status. From 2004 to 2006, all new therapists working at least half-time were recruited. The sample included a large representation of trainees and early career therapists because of greater turnover rates for these groups. Of the 163 therapists recruited, 131 (80%) agreed to participate; only 96 had a patient in the study who met criteria, agreed to participate, and attended sessions for video data collection. Therapists who declined to participate did not differ significantly from those who participated on age, gender, or racial or ethnic distribution; licensed staff had a slightly lower participation rate (N=17 of 24, 72%) compared with unlicensed staff or trainees (N=114 of 139, 86%).

Children and parents. Children were eligible to participate if they presented with a disruptive behavior problem (aggression, defiance, delinquency, or oppositional behavior by parent report), if they were aged four to 13 years, if the primary language for them and their parent was English or Spanish, and if they were entering a new episode of psychotherapy (defined as no therapy for the previous three months) with a participating therapist. Clinic administrative staff asked parents who contacted the clinic seeking an appointment for permission to share contact information with the researchers for recruitment; 10% declined to be contacted. Of the 550 who agreed to be contacted and met inclusion criteria, 292 (53%) did not attend a clinic appointment, leaving 258 potential participants; 218 (84%) agreed to participate. Videotaped sessions were collected for 191 children (27 participants did not continue treatment after the baseline research interview). Health Insurance Portability and Accountability Act restrictions prohibited a comparison of participants and nonparticipants.

Informed written consent was provided by therapists and parents, and assent was provided by children aged eight to 13 years. Participants received a modest cash honorarium for research participation, and all protocols were approved by affiliated human subjects review committees.

Procedures

Descriptive data on children, parents, and therapists were collected during baseline interviews. Number of treatment sessions attended and clinician-assigned child psychiatric diagnoses were collected from billing and administrative records. Data on other treatment services received were obtained from telephone interviews with parents conducted every four months for 16 months.

All psychotherapy sessions occurring between the baseline research interview and the 16-month follow-up were videotaped; 3,241 videotaped sessions were collected. A random sample of up to ten sessions per child was selected for coding (four tapes from baseline to four months, three tapes from five to eight months, two tapes from nine to 12 months, and one tape from 13 to 16 months; this pattern was chosen in order to provide the most data during the intervals in which most patients were attending), resulting in 1,215 total coded sessions.

Measures

Adaptation of TPOCS-S. The Therapeutic Process Observational Coding System for Child Psychotherapy—Strategies scale (TPOCS-S) (24,25) assesses for a wide array of intervention strategies that are theoretically and nontheoretically, or cross-theoretically, derived. The content was adapted from the Therapy Procedures Checklist (TPC) (26), and the format is based on the Therapist Behavior Rating Scale (TBRS) (27). The TPOCS-S was adapted for the current study (known locally as Practice and Research: Advancing Collaboration [PRAC]) in collaboration with the PRAC Therapist Advisory Group (TAG) (12,28) to maximize relevance to community practice. The PRAC TAG included one therapist from each participating clinic and is described in the study by Garland and colleagues (28). The final revised PRAC-TPOCS-S includes 27 clinical strategies, divided into 15 therapeutic techniques (for example, modeling and addressing the client-therapist relationship), and 12 therapeutic content areas (for example, affect management and principles of positive reinforcement) (29). Coders looked at the occurrence and intensity of each strategy directed to children and directed to adults separately. Occurrence indicates whether the strategy was observed at all. Intensity reflects both the time spent on the strategy and the thoroughness with which it was pursued (27). The combined occurrence and intensity scale was measured on a Likert scale ranging from 0 to 6 (0, clinical strategy was not observed; 1 or 2, low intensity; 3 or 4, medium intensity; and 5 or 6, high intensity). For example, a low-intensity rating on the therapeutic content strategy of problem-solving skills would reflect addressing one aspect of problem-solving skills, such as generating alternative solutions, but only for one particular experience the child or parent faced and in a somewhat fleeting or cursory manner. A high-intensity rating would reflect a thorough approach addressing multiple steps in problem solving and generalization to other problems.

Among the 27 clinical strategies of PRAC-TPOCS-S, a subset has been identified that comprises common elements of evidence-based treatment for children with disruptive behavior problems (23). This subset was identified through an iterative process of culling out elements of established empirically supported youth treatment models, parent-training models, and treatment models targeting youths and parents, identifying elements common across treatments, and validating these elements through expert consensus.

Coders and coder training. Seventeen research assistants served as coders (three were fluent in Spanish and coded sessions were conducted in Spanish). Coder training was conducted by three of the authors and included didactics, manual review, practice sessions, and training to an established criterion (defined as reaching 80% agreement on training tapes) (12).

Interrater reliability for the PRAC-TPOCS-S. Of the 1,215 total coded sessions, a random selection of 379 (31%) were coded by two coders to test interrater reliability. The intraclass correlation (ICC) assessing reliability on the combined occurrence and intensity scale across all codes at the session level was .78, representing strong reliability. ICCs were also calculated at the individual code level and ranged from .21 to .91, with a mean of .61, which is within an acceptable range (30). The eight codes with very low occurrence (observed <10% of sessions) had the lowest reliability.

Data analyses

Descriptive data on frequencies of observed clinical strategies across all 1,215 coded sessions were calculated. Therapists varied in the number of cases (mean±SD=2.5±1.8, range=1–7) and sessions (mean=12.7±11.6, range=1–44) coded; however, the observed frequencies did not vary significantly (all were within 3 points, and the majority were within 2 points) when just one case per therapist was randomly selected. Thus analyses presented rely on the full sample of 1,215 sessions.

Participants

Characteristics of the 96 therapists are presented in Table 1, and characteristics of the child and parent participants are presented in Table 2.

Treatment received

Patients attended an average of 22.4±15.6 sessions (range=0–70) during the 16-month study period, and 54 (28%) were still attending sessions at 16 months. Across the 1,215 coded sessions, children participated in 1,184 sessions (97%) and parents participated in at least part of 851 sessions (70%). By parent report, children received a variety of services in addition to outpatient psychotherapy. Specifically, during the 16-month study period, 134 (63%) of the children received some type of psychoactive medication, 30 (15%) presented to emergency rooms for mental health or behavioral reasons, 19 (9%) were placed in a psychiatric hospital or residential treatment center, and 177 (88%) received school-based mental health services.

Psychotherapeutic clinical strategies observed

Table 3 lists the 27 clinical strategies assessed in the PRAC-TPOCS-S. The average number of clinical strategies observed per session directed to children was 10.9±3.7 (range=1–20), and the average number directed to parents was 8.1±3.9 (range=1–22 of 27). Table 3 presents data on the observed occurrence and intensity for each of the assessed strategies in rank order of occurrence frequency. Any occurrence represents the percentage of sessions in which the strategy was observed relative to the total number of sessions in which each target (child or parent) participated (for example, affect education was observed in 964 (81%) of the 1,184 sessions in which a child participated). High-intensity occurrence represents the percentage of sessions in which the strategy was observed at an intensity level of 5 or 6 on the 6-point scale. Mean intensity is the average intensity of the strategy when it was observed. Mean intensity when observed was 2.3±.3 across strategies directed to children and 2.4±.5 for parent strategies. Of all observed sessions, 724 (60%) included at least one strategy delivered to either the child or parent at high intensity.

Clinical strategies that have previously been identified as common to evidence-based treatment for children with behavior problems and their parents (23) are noted in Table 3. Some of these evidence-based strategies were observed frequently (for example, affect education and use of positive reinforcement), whereas others were observed relatively infrequently (for example, assigning or reviewing homework and role-playing with parents). On average, all elements were observed at a low intensity.

This study provides the first detailed data on the type and variability of psychotherapeutic treatment strategies observed in mental health care delivered to children with disruptive behavior problems, the most common presenting problems in child mental health services. Results reflect heterogeneity in amount and type of treatment, including treatment duration (0–70 sessions in 16 months), additional service use, and within-session psychotherapeutic strategies observed.

Although treatment duration varied widely, the mean number of sessions attended (22.4) is somewhat consistent with the limited data available on duration of community-based care, with reported total treatment session averages ranging from 17 sessions (31) to 23 sessions (32). Most evidence-based treatment models for children with disruptive behavior problems require a minimum of 12 weekly visits (23). In addition to having outpatient sessions, a majority of patients reportedly received other types of mental health services. Almost two-thirds of these four- to 13-year-olds received some psychoactive medication, which is relatively consistent with 58% of a somewhat similar sample of youths with disruptive behavior problems who received medication as part of their community-based care in the Netherlands (33). A majority of children in our sample also received some type of school-based mental health services (88%), and a minority received more intensive services, including hospitalization for psychiatric reasons (9%), during the 16-month study period. Combined with the fact that 28% of children were still in treatment after 16 months, these data reflect a substantial volume of mental health services for families who engaged in usual care. The extent to which these services were coordinated across treatment modalities is not known.

Within and across treatment sessions, therapists were observed delivering a wide array of clinical strategies, which is consistent with the limited body of research indicating that usual care therapists prefer an eclectic approach to psychotherapy (10,14). Although only 25% of the therapists in this study endorsed "eclectic" as their primary theoretical orientation, the observational data indicate that eclecticism is the norm rather than the exception. The findings that almost half of sessions did not include a single therapeutic strategy delivered at high intensity and that the mean intensity rating across all observed strategies was relatively low reflect a cursory or incomplete application of the treatment strategy with limited follow-through. Explanations for low intensity are largely speculative, but the low intensity may reflect variability in therapist training, interference of patient crises and case management challenges, assumptions or perceptions of patients' responsiveness, or pragmatic constraints (for example, not enough time). Overall, the usual care psychotherapy we observed could be characterized as reflecting great breadth but not depth in therapeutic approaches.

Parents participated in at least part of most treatment sessions, and similar therapeutic techniques were observed frequently being delivered to both children and parents (for example, information gathering, psychoeducation, and establishing or reviewing goals). However, therapists were observed addressing different therapeutic content areas with parents than they did with children. Specifically, two of the most common content areas addressed with parents were "child's external care" (that is, case management or coordination of services received outside of the therapy session) and parent's own issues (for example, psychosocial issues and care), but these topics were not addressed as often with children. These findings are consistent with anecdotal reports from usual care therapists who indicate that the complex, multidetermined needs of families in public-sector care require significant case management, which can interfere with delivery of evidence-based psychotherapeutic approaches such as child skill building or parent management training. Although our data confirm that therapists are spending a great deal of time with parents on case management, the data cannot address whether this use of time is necessary or whether it has an impact on child outcomes.

The second aim of this study was to examine the extent to which usual care therapists employed strategies conceptually consistent with predetermined common elements of evidence-based treatment for this patient population. Results indicate that although several strategies common to evidence-based practices were observed in a majority of sessions (for example, affect education, problem-solving skills, use of positive reinforcement, and psychoeducation), other strategies common in evidence-based treatment were observed relatively infrequently. Even when observed, several evidence-based strategies were usually rated at a low-intensity level, thus not consistent with expectations in evidence-based treatment models. Similar observations were reported in other work comparing observational and therapist self-report methods for characterizing psychotherapy practice (34).

Many of the infrequently observed evidence-based treatment strategies, such as assigning or reviewing homework, role-play or behavioral rehearsal, and modeling, are characterized as more directive psychotherapeutic techniques (35), and these strategies are at the core of virtually all evidence-based treatment models for children with disruptive behavior problems. We have found that therapists in usual care clinics generally have positive attitudes about many of these psychotherapeutic techniques (36), yet they do not employ them often. Research on adult psychotherapy similarly finds that directive therapeutic approaches are not observed as frequently in usual care as they are in evidence-based treatment models (35). Because more directive treatment approaches have been associated with greater improvement in specific behavioral outcomes (37), more attention to this discrepancy is warranted.

Our findings raise natural questions about factors associated with therapist variability in practice. In related research, we have tested for therapist characteristics that may be associated with delivery of treatment relatively more or less consistent with elements of evidence-based practice; few significant effects were found (38). For example, therapist experience (months practiced), discipline, and staff versus trainee status were not significantly associated with observed intensity of delivery of evidence-based practice elements in our sample of therapists, which includes many therapists with few years of accumulated experience (38). Thus explanations for variability in practice likely require more complex investigation of training and supervision experiences. Accordingly, our results have implications for therapist training (as discussed below).

The study findings need to be interpreted in the context of some additional strengths and limitations of the study. Representativeness of the therapist and patient sample is critical for the generalizability of these findings. Our patient sample was comparable to several other clinical samples of children in publicly funded care. Specifically, the 2:1 male-to-female ratio is consistent across many studies (11,20), as is the overrepresentation of youths in some racial or ethnic minority groups relative to general population estimates (11,39) and the distribution of the most common diagnoses (conduct disorder or oppositional defiant disorder, attention-deficit hyperactivity disorder, and mood disorders) (39,40). Our therapist sample was also highly similar to a recent national sample of 1,200 providers in children's mental health care in terms of distribution by education level, gender, and race-ethnicity (41). Trainees with limited experience were somewhat overrepresented in our study, but other studies of community-based usual care also report high representation of trainees (42). Our sample also includes a large representation of therapists trained in the marriage and family therapy discipline, which represents a rapidly growing segment of the workforce across the United States but is overrepresented in California (43), where this study was conducted. Of course, the extent to which the participating therapists represent therapists in other types of service sectors (for example, private practice) or geographic regions is unknown.

The strengths of the study methods reflect a balance of relevance and rigor. We achieved adequate interrater reliability on our observational measure (PRAC-TPOCS-S), which also benefited from collaborative input from practicing therapists to ensure ecological validity and comprehensiveness (28). Use of such qualitative methodology to support the validity of practice measures has been strongly reinforced in previous studies (10). Despite these strengths, the resulting measure did not capture all possible therapeutic interventions. The data reflect only the observable behavior of the therapist, and they do not capture therapists' intentions, goals, or decision-making processes; patients' responses to different intervention strategies; or additional therapeutic contacts, such as communication outside the office. Finally, we do not know how observing practice may have affected practice itself, although we attempted to minimize this effect by establishing videotaping as routine in the clinics and by using small, unobtrusive cameras.

In the context of recent reports of the extraordinarily high cost of mental health treatment for children and discouraging data on effectiveness, this study offers timely detail regarding the types of care being delivered in one large public system. The study provides the first glimpse into usual care psychotherapy practice offices, and it thus provides essential contextual data for the development of tailored efforts to improve care, as well as providing baseline data for change efforts. Our results highlight some areas of relative convergence between evidence-based practice and usual care, as well as significant discrepancies between evidence-based treatment elements and usual care. Areas of convergence may represent a common ground upon which to build, and discrepancies represent potentially potent targets for improving care. For example, training efforts designed to increase the delivery of infrequently observed directive evidence-based treatment strategies for children and parents, such as role-playing, modeling, and assigning or reviewing homework, are needed. These efforts must also address needed improvements in intensity of treatment. Mental health clinician training is most effective when training interventions are tailored to address the existing service context (44). Further, it may be important to infuse evidence-based training into graduate programs preparing master's-level therapists because they represent a significant proportion of the providers of community-based mental health services.

Critical next steps in this practice-based research program include examination of child and family outcome trajectories to determine how specific practice patterns may be associated with different clinical outcomes. More detailed analyses can examine potential moderating effects of patient characteristics (demographic or clinical factors), as well as therapist characteristics. Analyses of the mediating role of perceived therapeutic alliance are also needed because previous research supports the role of alliance in outcomes. Richer information about usual care practice will provide essential contextual data for ongoing efforts to improve the translation of evidence-based interventions into practice, balancing the emphasis on evidence-based practice with attention to practice-based evidence (45).

This work was supported by grants R01-MH66070 (AFG), K23-MH077584 (LBF), and K01-MH064079 (MSH) from the National Institutes of Health. The authors thank Scott Roesch, Ph.D., for data analysis assistance, as well as Deb Dupuis, M.P.H., and Robin Taylor for project management. In addition, this project could not have been completed without significant contributions from the Therapist Advisory Group and all participating therapists and families.

The authors report no competing interests.