Evidence-Based Assessment in Routine Mental Health Services for Youths

Evidence-based assessment (EBA) includes use of reliable, valid, and standardized measures throughout treatment and is fundamental to evidence-based treatment (1, 2). Pretreatment assessment informs treatment selection and planning. Ongoing assessments are associated with faster improvement across settings, treatments, client ages, and presentations (3). Posttreatment assessment assists with evaluation of outcomes and need for additional services. EBA training initiatives to support EBA use are increasing, and a recent report suggests these initiatives may improve EBA use (4).

Unfortunately, we know little about EBA use in usual care to inform these initiatives. Some surveys indicate whether clinicians assess at all and how often they use standardized measures in EBAs; few clinicians (e.g., 29%−37%) conduct assessments (5, 6), and among those who do, few use standardized measures in the assessment process (12%−40%) (5, 7–9). Very little is known about when clinicians conduct assessments, except for three surveys of usual care: 39% of U.S. clinicians (7) and 12% of Canadian clinicians (8) reported using ongoing assessment, and 65% of U.K. clinicians reported using pretreatment, ongoing, and posttreatment assessments, but only 29% did so systematically (10). More studies of assessment throughout the treatment process (i.e., pretreatment, ongoing treatment, and posttreatment) are needed to understand how the usual practices of mental health clinicians who treat youths align with EBA.

Further, evaluation of EBA facilitators and barriers may suggest which important features to integrate into EBA initiatives. Individual attitudes are hypothesized to predict behavioral intentions, which, in turn, predict behavior (11). Among clinicians, more positive attitudes toward standardized measures (particularly perceptions about practicality, or time and cost of accessing, administering, and scoring standardized measures) are associated with greater use of these measures (7, 10). Among psychologists, assessment training also has been associated with use of standardized measures (5). This may explain why psychologists, with their discipline’s traditional focus on assessment, are more likely to use rating scales than other clinicians (9). Similarly, cognitive-behavioral therapy (CBT) clinicians report measuring outcomes more frequently than insight-oriented clinicians, perhaps because of greater emphasis on symptom monitoring within CBT (5). Having fewer years of experience is also associated with use of standardized measures (5, 7); this too may relate to training, in that graduate programs have enhanced assessment training to meet demand for demonstrated outcomes.

Practice context is also important. Clinicians working in agencies or group settings (5, 7) and those who are reimbursed by Medicaid (5) report use of more assessments. These clinicians may have more resources than other clinicians have or may be required to conduct assessments. Among psychologists, those working primarily with adults are more likely to use standardized assessments (9). Finally, within one EBA initiative, White youths were more likely than those with unknown race and ethnicity to receive ongoing assessments (12).

Although informative, surveys have often focused on clinicians serving adults, may be limited to one discipline, may examine only ongoing assessment (not pre- or posttreatment assessment), or may follow EBA initiatives. To extend our understanding of EBA to clinicians treating youths in usual care, we surveyed two multidisciplinary samples: a national survey of professional guild members and a statewide survey of Medicaid-billing clinicians. We present information on when, how, and under what conditions clinicians conduct assessments. We hypothesized that a psychology discipline (9), a learning theory orientation (5), positive attitudes toward standardized measures (2, 7), and more adult (5, 8) and Medicaid or low-income (5) clientele would predict greater standardized measure use. We also hypothesized that a private practice setting (7), more clinical experience (7), and more clientele from racial-ethnic minority groups (12) would predict less standardized measure use.

Methods

All procedures were approved by the University of Missouri’s institutional review board. We provided consent information in a cover letter. Participants consented by returning the survey. For the national survey (2007–2008), we randomly selected 1,000 clinicians each from the American Psychological Association, American Academy of Child and Adolescent Psychiatry, American Counseling Association, National Association of Social Workers, and American Association of Marriage and Family Therapy membership rosters. For the statewide survey (2008), we identified 3,084 clinicians from a public listing of health service clinicians who had billed Medicaid the previous year. Clinicians received up to five mailings. The national survey’s adjusted response rate (undeliverable N=347) was 62% (N=2,863); 1,520 who provided youth assessments were included. The statewide survey’s adjusted response rate (undeliverable N=364) was 50% (N=1,348), and 1,055 were included.

We developed both surveys by using the tailored design method (13). Clinicians reported their professional characteristics: discipline, year of highest degree, percentage of youths and clients with low income and from racial-ethnic minority groups, employment setting, and percentage of Medicaid-reimbursed cases (statewide survey only). Clinicians reported their primary theoretical orientation with a recent, representative case of anxiety, depression, or disruptive behavior among youths. For assessment practices, in both surveys we asked, “How often do you conduct an assessment of problems or strengths during the following points of treatment? At intake or the beginning of treatment, ongoing throughout treatment (e.g., weekly, monthly), at or near the last session, or well after the last session (e.g., weeks, months later)?” In the national survey, we asked, “How often do you use the following specific assessment procedures when treating children and adolescents? Standardized checklists for child/family symptoms or functioning (paper-and-pencil measure) completed by child, parent, teacher, or others with a standardized scoring (e.g., Child Behavior Checklist [CBCL])?” In the statewide survey, we asked, “How often do you use standardized assessment measures with the children/families you treat (e.g., Child Behavior Checklist [CBCL], Connors)?” All responses used a 5-point scale: 1, never or almost never; 2, rarely; 3, sometimes; 4, often; and 5, all or most of the time. The national survey also included the Attitudes Toward Standardized Assessment (ASA [2]), a 22-item measure of attitudes toward standardized measures with three subscales (i.e., benefit over clinical judgment, psychometric quality, or practicality); higher scores on the ASA indicate more positive attitudes.

We first present descriptive information (mean±SD) for assessments at four time points (pretreatment, ongoing treatment, posttreatment, and follow-up). We conducted one-way repeated-measures analysis of variance with a Greenhouse-Geisser correction to compare means across the four time points and a Bonferroni adjustment of p=0.008 for pairwise comparisons. We also present the percentage of clinicians who conducted assessments “often” or “all or most of the time” at each time point and throughout the treatment process (i.e., pretreatment, ongoing, and posttreatment). Second, we present descriptive statistics and the percentage of clinicians who used standardized measures “often” or “all or most of the time” with youths. Third, we conducted a multiple regression analysis predicting standardized measure use by professional discipline (0, not psychology; 1, psychology); percentage of clients from racial-ethnic minorities, with low income, or with Medicaid (statewide survey only); case-mix age (0, primarily [≥50%] youths; 1, primarily adults); private practice setting (0, not private practice; 1, private practice); learning theory orientation (0, other; 1, CBT); and ASA subscale scores (national survey only).

Results

In the national survey (N=1,520), the mean±SD age of clinicians was 52.7±10.0 years; they had 20.4±10.3 years of clinical experience, and most were women (N=958, 63%). The clinicians were Caucasian (N=1,379, 91%), African American (N=39, 5%), Asian/Pacific Islander (N=39, 5%), Latinx (N=43, 3%), other (N=19, 1%), and Native American (N=10, 1%). Most were in private practice (N=922, 61%). A third (N=482, 32%) endorsed learning theory orientation, and more than half had doctorate degrees (N=831, 55%). Their disciplines were psychology (N=482, 32%), counseling (N=357, 23%), marriage and family therapy (N=329, 22%), social work (N=323, 21%), and psychiatry (N=294, 19%). On average, clinicians reported that 32%±28% of their clients belonged to racial-ethnic minority groups and that 35%±32% of the clients had low income. Clinicians reported often conducting assessments pretreatment (4.8±0.6), during treatment (4.2±0.9), and posttreatment (4.0±1.1) and were rarely doing follow-up (2.0±1.2) (F=2,311.44, df=2.60, 3,025.57, p<0.001; all pairwise comparisons p<0.001). Most clinicians conducted pretreatment (N=1,418, 93%), ongoing treatment (N=1,165, 77%), and posttreatment (N=1,032, 68%) assessments often or all or most of the time, and 11% (N=171) used follow-up assessments often or all or most of the time. About half (N=749, 49%) conducted assessments often or all or most of the time throughout the entire treatment process. Clinicians used standardized measures sometimes (3.2±1.4), and 48% (N=726) used them often or all or most of the time. Having more clients from racial-ethnic minority groups (β=0.13, p=0.002) and higher ASA practicality scores (β=0.37, p<0.001) predicted more frequent use of standardized measures, and serving primarily adult clients (β=−0.11, p=0.003) predicted less frequent use of these measures (Table 1).

In the statewide survey (N=1,055), clinician age was 47.5±11.6 years; they had 13.9±9.4 years of clinical experience, and most were women (N=749, 71%). Clinicians were Caucasian (N=944, 90%), African American (N=71, 7%), other (N=11, 1%), Latinx (N=10, 1%), Native American (N=9, 1%), and Asian/Pacific Islander (N=6, 1%). More than a third (N=410, 39%) endorsed learning theory orientation. Most had a master’s degree (N=763, 72%), and most were in private practice (N=651, 59%). Disciplines included counseling (N=533, 51%), social work (N=374, 35%), psychology (N=282, 27%), marriage and family therapy (N=156, 15%), and psychiatry (N=8, 1%). On average, the clinicians reported that of their clients 28%±30% belonged to racial-ethnic minority groups, 32%±24% had low income, and 53%±35% were Medicaid insured. Clinicians conducted pretreatment (4.80±0.55), ongoing treatment (4.1±1.0), and posttreatment (4.1±1.2) assessments often and follow-up (1.9±1.0) never or almost never (F=2,507.99, df=2.75, 2,776.31, p<0.001). Pretreatment assessment was more frequent and follow-up assessment less frequent than assessments at the other two time points (p<0.001). Most clinicians conducted pretreatment (N=1,005, 95%), ongoing treatment (N=811, 77%), and posttreatment (N=768, 73%) assessments “often” or all or most of the time; 7% (N=78) conducted assessments at follow-up. More than half (N=582, 55%) conducted assessments often or all or most the time throughout the treatment period. On average, clinicians used standardized measures sometimes (3.1±1.3), and 38% (N=404) used them often or all or most of the time. Psychology discipline (β=0.27, p<0.001) and having more clients from racial-ethnic minority groups (β=0.12, p<0.001) predicted more frequent standardized measure use, and serving primarily adults (β=−0.11, p=0.001), being in a private practice setting (β=−0.10, p=0.002), and serving more Medicaid clients (β=−0.11, p=0.002) predicted less frequent use of these measures (see Table 1).

Discussion and Conclusions

Using two large multidisciplinary surveys, we examined when, how, and under what conditions clinicians involved in usual care conduct EBAs with youths. On average, clinicians reported often assessing before, during, and after treatment. Consistent with EBA, approximately half of the clinicians routinely conducted assessments throughout the three treatment stages. However, consistent with previous studies (5, 7–9), the clinicians used standardized measures only sometimes, and less than half routinely used standardized measures. This discrepancy suggests that assessment most often involves informal unstandardized methods, and EBA training initiatives are needed to support the routine use of standardized measures among clinicians.

These results point to two drivers of standardized measure use that may inform future EBA initiatives: practical measures and clinician training. In line with previous studies in which clinicians cited lack of practicality as a primary barrier to standardized measure use (7, 10), ASA practicality was the greatest predictor of standardized measure use in the national survey, suggesting that practical measures may support routine use of standardized measures. In the statewide survey, a greater percentage of Medicaid cases was associated with less frequent standardized measure use. The state survey had no mandated assessment requirements (this finding may not hold if EBA is required for Medicaid billing), and our finding may largely reflect that many clinicians in this survey may work in low-resource settings. As such, EBA initiatives should consider free or low-cost and brief measures that can be easily integrated into universal care.

Training to support standardized measure use throughout the treatment process and with specific populations is also needed. Consistent with previous surveys (9), psychology discipline predicted increased use of standardized measures. If greater assessment training in psychology graduate programs is driving these results, EBA training for all disciplines may yield improved attitudes, comfort with, and use of EBA measures. Private practice setting was also associated with less frequent standardized measure use. Given that clinicians in private practice may have less favorable views of standardized measures (7), training that increases positive attitudes toward standardized measures may increase their use in private practice. Consistent with another survey (8), clinicians serving primarily adult clients used standardized measures less frequently with youths. Having a higher proportion of Medicaid clients was also associated with less frequent use of standardized measures; this was contrary to our hypothesis but consistent with recent findings of a quality improvement project (14). Hence, addressing issues specific to assessments (e.g., multi-informant reports) of youths and increasing familiarity with standardized measures for youths from diverse backgrounds both seem important. Of note, contrary to past research (12), having a greater proportion of youths from racial-ethnic minority groups predicted greater standardized measure use in both samples; perhaps clinicians serving more racial-ethnic minority youths are using EBA more often because of the smaller body of literature on treatment of youths from racial-ethnic minority groups (15).

This study had some limitations. Clinicians may have overreported use of EBAs because of social desirability or recall bias. Clinicians who conduct more EBAs may have also been more likely to respond, although our response rates were comparable to or higher than those in previous surveys (5–9), and we included two samples to increase representativeness (i.e., the Medicaid sample was recruited because guild members may have more resources or be more current on standards than nonguild members). Given the timing of these surveys and the push for greater EBA use in the past decade, standardized measure use may now be higher. A multidisciplinary survey is needed to evaluate current assessment practices. Nevertheless, our results provide a useful data point for understanding assessment practices as they evolve. Our surveys were administered as routine progress monitoring was being recognized as an evidence-based practice and before any large-scale EBA initiatives. As such, our findings may provide a benchmark for gauging the success of EBA initiatives.