Testing DSM-5 in Routine Clinical Practice Settings: Feasibility and Clinical Utility

The Diagnostic and Statistical Manual of Mental Disorders (DSM) was primarily developed to enhance communication and provide reliable diagnostic guidelines in clinical practice. Since the first edition of DSM in 1952, each of the four revisions over the past 45 years introduced changes in the description of mental disorders or diagnostic criteria that reflected contemporary advances in the science and understanding of mental disorders (1). Improved diagnostic reliability has been achieved by adding explicit criteria for mental disorders that provide descriptive information about signs, symptom expression, symptom duration, and consequences of these syndromes (2).

Similarly, proposed changes to DSM-5 were developed to capture the latest scientific advances and to enhance the reliability of diagnostic criteria while also addressing the need for improved clinical utility. The process of revising DSM-IV provided an opportunity to better integrate relevant mental health research findings that have emerged over the past two decades. DSM-5 introduces a more dimensional spectrum approach to several disorders, including autism spectrum disorder (ASD), learning disorders, substance use disorders, and somatic symptom disorders. Substantial modifications were also made to criteria for such disorders as schizophrenia, schizoaffective disorder, bipolar disorder, posttraumatic stress disorder (PTSD), and obsessive-compulsive disorder. The major revisions incorporated in DSM-5 have been summarized elsewhere (3–5) and are described in greater detail on the American Psychiatric Association’s (APA’s) online resource page for DSM-5 (www.psychiatry.org/dsm5). [A table listing key changes in the diagnostic criteria for several disorders is available in an online supplement to this article.]

DSM-5’s introduction of patient-reported symptom and disability measures (5) and changes to select diagnostic criteria underscored the need for field testing, which has been a regular part of the DSM revision process since the third edition (6–10). One set of field trials for DSM-5, conducted at high-volume academic and research centers (3–5,11), tested the reliability of diagnoses, or the degree to which two clinicians could agree on diagnoses in separate examinations of the same patient when using DSM-5 diagnostic criteria and clinical assessment measures. Most of the diagnoses that were adequately tested had good to very good test-retest reliability with representative clinical populations (4), and there was generally good to excellent reliability for patient-reported symptom measures (5). However, because most clinical practice takes place outside academic and research centers (12,13), it is important to know whether the criteria that are used in such settings are feasible and clinically useful in routine clinical practice settings. A second set of DSM-5 field trials was planned specifically for this purpose. This article presents results from the DSM-5 Field Trials in Routine Clinical Practice Settings (RCP), which addressed the feasibility and clinical utility of the proposed criteria revisions in these settings.

Methods

The DSM-5 RCP Field Trials were conducted in a wide range of practice settings in the United States, including Alaska, Hawaii, and Puerto Rico, and in Canada, Australia, and the United Kingdom. Data collection took place over a six-month period from the fall of 2011 through early 2012.

Study participants

Study participants included mental health professionals and their patients. Clinicians included psychiatrists, licensed doctoral-level clinical psychologists, clinical social workers, advanced practice psychiatric–mental health nurses, licensed counselors, and licensed marriage and family therapists. Clinicians were recruited from two groups, a random sample of 9,460 psychiatrists drawn from the American Medical Association (AMA) Physician Masterfile (14) and a volunteer sample of more than 5,600 mental health clinicians, recruited primarily on the DSM-5 Web site (www.dsm5.org).

Clinicians were eligible to participate if they were licensed to independently make DSM diagnoses; treated at least one new patient per month; had access to a computer with an Internet connection in the setting where field trial patients would be seen; were willing to provide computer access for both patient and clinician use during the study visit; could complete several hours of formal online training; and were able to enroll at least one patient in the study, conduct a diagnostic assessment of the patient, and provide patient data using the proposed DSM-5 criteria and related measures. Clinicians who were DSM-5 Task Force or DSM-5 Work Group members or who were otherwise involved with the DSM-5 development process were excluded from RCP Field Trials participation to reduce the chance of real or perceived bias.

After exclusion of residents, nonpsychiatrists, psychiatrists not involved in direct patient care, and those with invalid addresses, 2,679 of 8,953 psychiatrists (30%) completed the eligibility survey. Of these, 1,299 (48%) met eligibility criteria, and of this group, 472 submitted written consent forms, and 110 completed all online training and provided patient data. Therefore, about one-fourth of those submitting consent forms completed the study.

A total of 12,797 individuals volunteered for the DSM-5 RCP Field Trials through the DSM-5 Web site. Of these volunteers, 5,647 completed the eligibility survey, and 3,841 mental health clinicians (68%) met eligibility criteria. Of this group, 2,034 submitted written consent forms, and 511 completed all online training and provided patient data. Again, about one-fourth of those submitting consent forms completed the study.

Because the participation rate of the random sample was low, this sample was considered a sample of convenience and was combined for analyses with the volunteer sample of psychiatrists. The full sample of 621 participants included 154 psychiatrists, 77 licensed doctoral-level clinical psychologists, 111 clinical social workers, 33 advanced practice psychiatric–mental health nurses, 126 licensed counselors, 113 licensed marriage and family therapists, and seven licensed master’s-level psychologists.

Each clinician was asked to report on at least one patient randomly sampled from his or her active caseload. Eligible patients or their parents or legal representatives had to be able to read and communicate in English. Patients had to be symptomatic at the time of enrollment and had to be at least six years of age, but otherwise no patient was excluded on the basis of age or cognitive impairment. The 621 participating clinicians provided data on 1,269 patients. Among these clinicians, 536 reported on the patient enrollment process, from which we estimated a patient participation rate of 32% (3,317 patients invited to participate, of whom 1,071 agreed). The remaining 85 clinicians enrolled 198 patients.

Study measures

Both patients and clinicians completed computer-assisted assessments as part of the study. The measures have been described in detail elsewhere (3,5). Briefly, they included patient-completed assessments of 13 psychological “cross-cutting” symptom domains and disability (5) and clinician-completed diagnostic checklists, severity measures, relevant clinical and family histories, and patient and clinical utility questionnaires.

Procedures

With rare exceptions—for example, in correctional settings where patient computer access was forbidden—study measures were electronically administered. Data were collected using the Research Electronic Data Capture (REDCap) system, an encrypted, HIPAA-compliant electronic database system developed by investigators at Vanderbilt University and supported by the National Institutes of Health (NIH). REDCap is a suite of Web-based tools designed to support data capture and dissemination (15), which was modified to meet the specific needs of the DSM-5 field trials (3).

Eligible clinicians submitted written informed consent and completed a rigorous, staged, online training program that included an overview of the RCP Field Trials, a self-directed review of the proposed DSM-5 criteria that had been posted on the DSM-5 Web site (www.dsm5.org), an introduction to REDCap, practice using the REDCap system with mock patients, and instructions on how to enroll patients and obtain written informed consent.

On the basis of a modified Kish procedure (16,17) that randomly assigned a start day and time, clinicians invited consecutive patients from their active caseload for field trial participation and entered all subsequent patients on a log until consent was obtained from one or more patients. Clinicians were asked to report on at least one existing or one new patient; the number of patients seen by individual clinicians ranged from one to eight, with a median of two patients per clinician. For the study visit, the patient or proxy completed the patient measures and a brief set of questions about his or her impression of the measures before meeting with the clinician. The results and interpretation of the patient-reported measures were transmitted to a database for clinician review before patient evaluation. The clinician then conducted a diagnostic interview and completed one or more relevant DSM-5 diagnostic checklists and severity measures, any additional indicated measures, and the clinical utility questionnaire. All required clinician forms were completed in REDCap.

Protection of human subjects

Human subjects protection training was provided for clinicians who did not have current certification. All RCP Field Trials procedures and forms were reviewed and approved by the institutional review board (IRB) of the American Psychiatric Institute for Research and Education. Additional IRB review and approval were obtained by participating RCP Field Trials clinicians who were affiliated with institutions that required local IRB approval. A Certificate of Confidentiality was obtained from NIH to protect sensitive patient information.

Incentives

Remuneration was provided to clinicians in the form of continuing medical education (CME) units. Clinicians who completed a study visit and provided patient diagnostic data received recognition as Collaborating Investigators and were offered a complimentary DSM-5.

Analytic approach

Responses to specific questions in the clinical utility questionnaire and the patient feedback questionnaire were used to examine feasibility, clinical utility, clinician comparison of DSM-5 with DSM-IV, and patient impressions of the helpfulness of the patient measures. All analyses were conducted with SAS, version 9.3, statistical software. Descriptive statistics were generated for study population patients and clinicians; clinician and patient responses were examined separately for diagnostic categories that included 20 or more patients.

Results

The findings presented here reflect data reported by 621 clinicians who completed a study visit and provided diagnostic data for 1,269 patients.

DSM-5 diagnoses

Table 1 shows the frequency of primary DSM-5 diagnoses among patients randomly selected for a study visit. The most frequent primary diagnoses among the 164 pediatric patients were attention-deficit hyperactivity disorder (ADHD) (27% of patients age six to 17 years), followed by major depressive disorder (8%) and ASD (7%). The most frequent primary diagnoses among 1,105 adult patients were major depressive disorder (19%), generalized anxiety disorder (10%), and bipolar I disorder (8%).

Feasibility and utility: DSM-5 criteria and cross-cutting measures

Clinicians reported that the overall DSM-5 approach was generally easy to use and that the proposed criteria and cross-cutting measures were clinically useful. Responses to the specific questions about overall feasibility and clinical utility of the proposed diagnostic criteria and cross-cutting measures are summarized for all disorders in Table 2. For more than 80% of patients, clinicians indicated that the overall DSM-5 approach to diagnostic assessment, including use of the patient-reported cross-cutting assessment measures, diagnostic criteria, and severity measures, was moderately to extremely feasible to apply in their clinical practice. Similarly, for approximately 80% of patients, clinicians reported that the DSM-5 criteria were moderately to extremely useful for the primary diagnosis compared with DSM-IV criteria. Thus the proportion of physicians giving ratings of “very” or “extremely” greatly exceeded the proportion giving ratings of “not at all” and “slightly,” which discounted any central tendency for reporting. Large proportions of clinicians also reported that the patient-reported, cross-cutting measures were very or extremely useful.

DSM-5 approach compared with DSM-IV

Figures 1 and 2 present more detailed data—by specific diagnoses and patient age group—on clinicians’ impressions of the DSM-5 approach to patient assessment compared with DSM-IV. Of the clinicians with pediatric patients, more than 75% responded that the DSM-5 approach for all disorders was better (60%) or much better (16%) than the DSM-IV approach to the diagnosis of a mental disorder. Nearly 85% of those treating patients with ADHD rated the approach to ADHD better (71%) or much better (13%) (Figure 1). Similarly, approximately 75% of the clinicians with adult patients reported that the DSM-5 approach for all disorders was better (57%) or much better (18%) than DSM-IV for assessment of adult patients (Figure 2). The most favorable ratings were reported for PTSD, personality disorders, bipolar II disorder, substance use disorders, and somatic symptom disorder.

Clinician assessment by discipline

Figure 3 shows clinician responses by discipline, summarized for all disorders. Regardless of discipline, most clinicians participating in the DSM-5 RCP Field Trials considered the DSM-5 assessment approach to be better or much better than DSM-IV. Ratings ranged from a cumulative 69% for psychiatrists (better, 55%; much better, 14%) to a cumulative 81% for licensed counselors (better, 60%; much better, 21%).

Patients' impression of cross-cutting measures

Fifty-one pediatric patients age 11 to 17 years and 111 parents of patients age six to 17 years responded to the questions about the helpfulness of the patient cross-cutting measures (Figure 4). In general (for all disorders), among patients and parents who provided feedback, a higher proportion of parents than patients reported that they thought the questions would be helpful (response of “yes”) to the clinician in understanding their child’s symptoms (parents of children age six to ten, 64%; parents of children age 11 to 17, 72%; children age 11 to 17, 47%).

Among adult patients, consistently large proportions indicated that they thought the questionnaires would help the clinician better understand their symptoms (Figure 5). The two diagnostic categories with the highest proportion of “yes” responses were substance use disorders (73%) and bipolar I disorder (66%). Somatic symptom disorder (41%) and PTSD (42%) had the lowest proportion of “yes” responses; for both disorders, however, a large proportion of patients endorsed the “maybe” category (35% and 41%, respectively). Few patients indicated they thought the cross-cutting measures would not be helpful to the clinician; “no” responses ranged from 0% for substance use disorders and ADHD to 10% (N=2) for any personality disorder.

Discussion

The DSM-5 RCP Field Trials examined the feasibility and clinical utility of the proposed diagnostic criteria in routine clinical practice settings. Clinicians from six mental health disciplines reported that the proposed criteria were generally easy to use in their practices and useful in patient assessment and that the DSM-5 approach was better than the DSM-IV approach. Most patients reported that they thought the patient measures would be helpful to their clinicians in understanding their symptoms.

The development of DSM-5 involved a multidisciplinary and international advisory and work group process, with three postings of draft criteria on the DSM-5 Web site (www.dsm5.org) to obtain public critiques and input. In both arms of the DSM-5 field trials, these criteria were implemented in a diagnostic assessment approach that involved computer-assisted, patient-completed assessments of commonly occurring “cross-cutting” symptoms and disabilities, a routine clinical interview rather than a structured research interview; and completion of computer-assisted diagnostic checklists and severity measures. The substantial changes in assessment procedures and in selected diagnostic criteria proposed for DSM-5 necessitated reliability testing in large academic and research settings (3–5), as well as feasibility and clinical utility testing in both the academic and RCP Field Trials. Conducting a field trial of this scope in routine practice settings required a considerable degree of training in a new approach to diagnosis, using moderately revised content and unfamiliar technology. We recognized that participation in the RCP Field Trials required clinicians’ interest, time, and willingness to adjust routine clinical schedules to the field trial protocol and electronic data collection, and that many clinicians might not be able to complete all stages of the study. Despite these challenges, more than 600 clinicians successfully evaluated more than 1,200 patients and reported useful data.

Clinicians reported that they generally found the proposed DSM-5 approach to diagnosis easy to apply and clinically useful. Diagnostic criteria for PTSD, substance use disorders, bipolar disorders, and personality disorders received the most favorable clinical utility ratings from clinicians with adult patients. ADHD criteria received the highest ratings from clinicians with pediatric patients. These were among the most significantly revised disorders in DSM-5, yet the revised criteria received some of the highest ratings for their feasibility and clinical utility. This finding suggests that these DSM-5 criteria will facilitate better care for patients who suffer from these disorders.

Among the new manual’s most noticeable differences from previous editions is the inclusion of patient-reported measures to identify initial clinical status and longitudinal outcome status in the DSM-5 approach to diagnostic assessment. In clinical practice, a mental disorder diagnosis is only one aspect of a clinical formulation that also includes the presenting complaint; past medical history; a mental status examination; a review of physical and mental systems; and an assessment of the patient’s family, social, and cultural context, ability to perform various life activities, environmental exposures, and other risk factors. A diagnosis made in this context can benefit from the type of patient-generated information introduced in DSM-5—which is supported by the levels of patient approval reported here—and will become increasingly standardized as part of federally mandated “meaningful use criteria” for electronic health records (18). Likewise, measurement-based care models for monitoring response to treatment for both mental disorders (19) and other medical conditions have already been incorporated as “quality indicator” measures in some health systems (20). As a result, patient-generated symptom and disability measures that can be incorporated into routine clinical use without additional costs, such as those that are made available for further study with the DSM-5, are an increasingly important component of clinical practice. This underscores the value of these study findings and indicates that a measurement-based approach to mental health care may be preferable to clinicians and patients alike.

It is particularly noteworthy that the version of the criteria for personality disorders that was studied in the DSM-5 RCP Field Trials involved a trait-based system that was well received by study clinicians. Likewise, the use of cross-cutting symptom measures for patient completion before a clinical interview was rated highly by both clinicians and patients, who saw these instruments as a useful means of communicating their reasons for coming in for clinical evaluation or treatment. These measures were recommended for further study in the final APA review process and are currently placed in Section III of DSM-5 (21).

Clinicians from the various mental health disciplines exhibited similar patterns in their evaluations of the overall assessment and the proposed changes in diagnostic criteria. Regardless of any overall professional differences among disciplines in training and assessment techniques, a substantial majority, more than 75%, of the volunteer clinicians from all disciplines gave a rating of “better” or “much better” for the overall DSM-5 approach compared with the approach provided by DSM-IV.

This study had a number of limitations. First, the data came from a highly motivated and self-selected, rather than a representative, sample of clinicians. On the basis of our previous experience with sampling from the AMA Physician Masterfile and the likelihood that a substantial proportion of psychiatrists in solo or small practices might not meet the eligibility criterion of providing computer access for patients, we drew a relatively large random sample of psychiatrists in anticipation of a potentially low response rate. It is possible that clinicians who originally responded did not fully realize the challenges of the online training and the time commitment required to complete the study, which may have contributed to the final participation rates that we observed. After the randomly selected psychiatrists and the other volunteer clinicians submitted consent forms, about one-fourth of each group completed all training and contributed patient data. The findings reported here, therefore, should not be generalized to all mental health clinicians and patients. Similarly, the distributions of proposed DSM-5 diagnoses reported by the RCP Field Trials clinicians do not represent treated prevalence rates and should not be generalized.

Second, a separate evaluation of clinicians’ experience with the online data collection system was not provided in REDCap. Although clinicians received an opportunity to comment on the online data collection experience when they claimed their CME credit, it is possible that some clinicians included their reactions to the online data collection rather than just to the proposed assessment instruments and diagnostic criteria. Further evaluation is needed of the implementation of more practice-based and clinician-friendly data collection technologies, as opposed to research-based technologies, for DSM-5. Third, unlike the DSM-5 Field Trials in Large Academic Settings (3–5), the RCP arm of the field trials was intended to examine the experience of mental health clinicians in a wide range of typical practice settings. However, because of small numbers of patients in some diagnostic categories and because patients were not necessarily a representative sample of all patients in the population, it was not possible to ascertain the feasibility and clinical utility of the proposed DSM-5 criteria for some diagnostic categories, especially among pediatric patients. The findings for diagnostic categories with small numbers of patients should be interpreted with caution. Fourth, our analyses focused on primary diagnoses; co-occurring diagnoses were not addressed in this study. Finally, although clinicians had the option of enrolling multiple patients, many provided data on only one patient.

Conclusions

The DSM-5 RCP Field Trials introduced a new component of testing that extended well beyond the previous focus on tests of reliability and disorder prevalence (8–10). Unique to both arms of the DSM-5 field trials was the inclusion of an assessment strategy that involved patient input, which could inform a diagnostic evaluation based on DSM-5 criteria by using routine diagnostic interviews. Because research interviews with highly structured instruments are not the norm for clinical practice, it was important to evaluate DSM-5 in the manner in which it would be used. Although assessing the reliability of the new diagnostic criteria is clearly important, it is equally vital to evaluate the convenience and clinical relevance of DSM-5 for mental health clinicians in the range of disciplines who will depend on it. The RCP Field Trials suggest that the DSM-5 approach is feasible and clinically useful in routine care settings and acceptable to both clinicians and their patients.