ICD-11 Guidelines for Mental and Behavioral Disorders of Children and Adolescents: Reliability and Clinical Utility

Expert groups developed the World Health Organization diagnostic guidelines for mental and behavioral disorders among children and adolescents (1). After publication, all diagnostic guidelines for mental and behavioral disorders in ICD-11 must be evaluated in terms of their ability to achieve consistent clinical diagnoses, or interrater reliability, and in terms of their clinical utility (2). Clinical descriptions and diagnostic guidelines for mental and behavioral disorders were published in 2015 (1). The interrater reliability and clinical utility of ICD-11 diagnostic guidelines for psychotic, mood, and anxiety disorders among adults were published in 2019. In a Mexican sample, all kappa interrater values were above 0.41, and a high proportion of clinicians considered the guidelines to be extremely useful (3).

In the developing world, the use of diagnostic guidelines is important for improving disorder recognition and reducing the large treatment gap (>70%) (4). Results of the Mexican Adolescent Mental Health Survey showed that although 9% had a serious mental disorder, 20% had a moderately severe disorder, and 10% had a mild disorder, most did not receive treatment (5). Therefore, the use of reliable, effective diagnostic guidelines could increase the provision of evidence-based care for patients (6).

Diagnostic guidelines for mental and behavioral disorders in ICD-11 include a section on developmental presentations, which describes the variability of the symptomatic manifestations of mental disorders according to an individual’s stage of development. The incorporation of expressions of psychopathology across the lifespan provides a longitudinal view of the manifestations, and the chapter on children and adolescents in which certain clinical characteristics were repeated has been eliminated. Disorders traditionally conceptualized as being present only in childhood that may persist and manifest differently in adult life can be codified at any stage of life (1).

Changes in ICD-11 Classification of Child and Adolescent Disorders

Some high-priority changes in the conceptualization of the ICD-11 classification of child and adolescent disorders in terms of their prevalence, service use, and care costs, are described below (7).

Depressive Disorders

The ICD-11 classification has not been significantly modified. The description of depression in the section on developmental presentations highlights the fact that among children, depressive mood may occur in the form of somatic complaints, increased anxiety, or excessive crying; among children, depressive mood may present as general irritability; among children and adolescents, reduction in the ability to concentrate and sustain attention may be manifested in a decline in academic performance; and decreased appetite may be expressed in a lack of weight gain expected for a given age and level of development, rather than weight loss.

Anxiety and Fear-Related Disorders

Generalized anxiety disorder (GAD), specific phobia, separation anxiety disorder, social anxiety, and selective mutism are the most frequent anxiety disorders among children and adolescents. The GAD description in the section on developmental presentations highlights the fact that among children, expressions of concern may include a particular interest in and adherence to rules, with a strong desire to please others, or seeking others’ approval excessively to reassure themselves; and among young people, excessive irritability and somatic and depressive symptoms are common (8).

Disorders Specifically Associated With Stress-Related Disorders

Four disorders are included: posttraumatic stress disorder (PTSD), complex PTSD, prolonged grieving disorder, and adjustment disorder. The PTSD description underlines the following: among children, both the presence of reexperiencing and active avoidance of internal states and the perception of an intensified current threat are usually expressed through behavioral manifestations; adolescents may be more reluctant to report their reactions to traumatic events, and when reexperiencing occurs, adolescents communicate a lack of affection or other emotions. As for adjustment disorders among children, these tend to be expressed through behavior such as tantrums, bedwetting, and sleep disorders. Somatic symptoms, excessive irritability, or psychoactive substance use may be reported, and adolescents may not explicitly verbalize a connection between stressful events and their own symptoms, meaning that the clinician must consider the relationship between the time of stressor and the attendant symptoms.

Attention-Deficit Hyperactivity Disorder

Hyperkinetic disorder was removed from ICD-11 and renamed attention-deficit hyperactivity disorder (ADHD). ADHD may present with or without impulsivity and hyperactivity. Inattentive presentation is manifested in a diversity of contexts, with a combination of difficulty concentrating, tendency toward distraction, and organizational problems; losing things frequently; and inattention to details of the task at hand. ADHD combined presentation is diagnosed when both inattention and hyperactivity or impulsivity are present. The detection of ADHD is complicated by the fact that it is often not identified or is attributed to other causes, even though it is a common cause of academic problems (9).

Disruptive Behavioral Disorders

Oppositional defiant disorder and conduct disorder (CD) were incorporated into disruptive behavioral disorders (DBD). Oppositional defiant disorder may be present with or without chronic irritability-anger; the subtype with chronic irritability-anger was incorporated as the ICD-11 equivalent of disruptive mood dysregulation disorder (DMDD) in DSM-5 (10, 11). Field studies and secondary analysis of their diagnostic criteria showed that DMDD has limited interrater and temporal reliability, a lack of consensus between psychiatrists, and high comorbidity rates, especially with ADHD and oppositional defiant disorder (12–14). Both oppositional defiant disorder and CD could present as “limited prosocial emotions.” This specifier is linked to more severe, persistent violent behavior (15) and could be identified in internalizing and externalizing disorders (16).

Given the need to evaluate whether the diagnostic guidelines work in Mexico, the aims of this research were to establish the interrater reliability and clinical utility of ICD-11 guidelines for mood, anxiety, and fear-related disorders and ADHD and DBD among children and adolescents.

Methods

This was a cross-sectional study drawing on a sample of children and adolescents seeking mental health services at two specialized psychiatric care facilities in Mexico City. The research was carried out from March to December 2017. Although the two recruitment centers are specialized hospitals, they accept both first-contact patients and patients referred by other centers. A naturalistic design was used whereby the raters conducted conventional diagnostic interviews based on ICD-11 diagnostic guidelines to avoid the artificial increase in diagnostic reliability that usually occurs with the use of structured interviews. Details of ICD-11 field studies have been described elsewhere (17).

Participants

Children and adolescents ages 6–17 participated. In the first consultation, the clinician explained the symptoms of all the disorders to the participant and the family in the two venues that participated. Youngsters with one of their parents received a comprehensive explanation of the nature and aims of the research and agreed to sign the assent/consent forms. Exclusion criteria included the presence of communication difficulties, clinically evident cognitive dysfunctions, and physical disabilities that could interfere with the participation of patients in the diagnostic interview. Patients at imminent risk of self-harm or injury to others were excluded, and these patients were managed in the psychiatric emergency units available at both institutions. All participants, both children and adolescents, were interviewed in the presence of one of the parents.

Clinical Raters

All clinical raters at both institutions were psychiatrists; some were also child and adolescent psychiatrists and others were 6th-year residents in the specialty. All clinical raters received a 4-hour training session on the most important changes in ICD-11 and the use of diagnostic guidelines. The research thereby emulated what will happen in specialized scenarios once ICD-11 is implemented. Rater training was provided by a clinical expert in child and adolescent psychiatry (FRP) with over 20 years’ experience in the field, who was a member of the ICD-11 guideline development team. As part of the training, clinician raters practiced applying the diagnostic guidelines to case vignettes and discussed the issues that arose during this process.

Two clinicians participated in each interview, one as an observer and the other as an interviewer, without communicating with each other. In addition to establishing a diagnosis for each participant (a main diagnosis and up to two secondary diagnoses when appropriate), they evaluated the clinical utility of the guidelines as used with each patient. Clinical rater pairs were assigned according to a systematic sampling procedure by using a list of clinicians available daily and considering their most recent participation as an observer or interviewer to maximize the variability of dyads and roles.

Guidelines and Measure of Clinical Utility

The instrument for assessing the clinical utility of the ICD-11 diagnostic guidelines for mental and behavioral disorders was specifically developed for this purpose (18). It comprises 15 questions on the different domains of the construct and is answered by using a 4-point Likert scale. According to an exploratory and confirmatory factor analysis of data from a study of Mexican clinicians who evaluated the guidelines for common mental and behavioral disorders in adulthood (3), all the items were grouped congruently in two general dimensions: implementation characteristics, and identification and management. Cronbach’s alpha coefficients for the subtotals and total scale were 0.90, 0.90, and 0.93, respectively.

Procedures

The ethics review boards at both institutions approved all the procedures used in the research, including the assent/consent forms for the children and adolescents and their parents. Clinicians responsible for the initial evaluations were asked to refer all participants who met the selection criteria to a research assistant, who oversaw the informed assent/consent process. Only participants and parents who had agreed to participate voluntarily in the research signed the assent/consent form, and all were invited at their convenience to a 2-hour interview with a dyad of clinical raters. After the interviewer had finished the questions, the observer had an opportunity to ask any questions that the observer considered necessary.

At the very least, each clinical interview covered the entire diagnostic evaluation, including symptoms present, history of the current episode, and an examination of mental status and personal, family, medical, and psychiatric history. If studies or other information existed, it was agreed that both raters would have access to it. The main objective of these measures was for clinicians to establish a diagnosis on the basis of exposure to the same information.

After the interview, both clinicians recorded their diagnostic impression of the patient without consulting each other, completed a step-by-step evaluation of the essential ICD-11 characteristics of the selected diagnoses for each patient, and answered the clinical utility evaluation questionnaire of the ICD-11 diagnostic guidelines for each case. In addition to the main diagnosis, each clinician could include up to two secondary diagnoses. It was also possible to register a “nondiagnosis,” if the clinician considered that the participant failed to present the diagnostic characteristics required to determine the presence of a mental disorder (none of the patients in the sample received a nondiagnosis).

Clinicians were not allowed to discuss the case with each other until they had independently entered their diagnostic evaluation into the database. In addition, the electronic data collection system allowed the two clinicians to complete the same patient’s information independently to ensure that they were blind to their peer’s opinions. Data were collected by using the Qualtrics online survey platform, making it possible to ensure the confidentiality and security of information.

Statistical Analyses

A statistical analysis was conducted with SPSS, version 20. General characteristics were described based on means and standard deviations or frequencies and percentages and were compared by using Student’s t tests of independent samples or chi-square tests. The frequencies and percentages of each of the diagnoses assigned by both the rater and the observer were also calculated. Each specific diagnosis was subsequently grouped into one of five general categories: mood disorders, anxiety and fear-related disorders, disorders specifically related to stress, ADHD, and DBD. The corresponding kappa values were calculated to determine the level of general diagnostic agreement between both clinicians. In all cases, the preestablished level of significance was p≤0.05. Finally, to summarize information on the clinical utility of the guidelines, the frequencies and percentages of responses to each item in the corresponding questionnaire were calculated for both the interviewers and the observers.

Results

Twenty-five clinicians (ten interviewers and 15 observers) participated in the research. Their demographic characteristics and professional experience are summarized in Table 1. The clinical sample comprised 52 participants, 21 (40%) of whom were children (ages 6–11) and 31 (60%) of whom were adolescents (ages 12–17). In the clinical sample, 19 (37%) were females (four children and 15 adolescents) and 33 (63%) were males (17 children and 16 adolescents). The mean±SD age of the clinical sample was 11.9±3.2.

Assigned Individual Diagnoses

Eight diagnoses were the most frequently assigned main diagnoses (Table 2). ADHD combined presentation and oppositional defiant disorder with chronic irritability-anger were the most frequent disorders identified by interviewers and observers.

Assigned Group Diagnoses and Kappa Values

Table 3 shows the frequencies and percentages of participants included in each diagnostic group. No significant differences between the interviewer and the observer were found in the diagnostic assignments for any group. Kappa values between raters were subsequently calculated, taking into account the number of interviewees whom they both assigned to the same diagnostic group (Table 4).

Clinical Utility of Guidelines

Clinicians’ responses to the questions on the instrument assessing clinical utility of the guidelines are shown in Table 5. Over 80% of clinicians reported that the guidelines were clear, easy to use, and accurate; that the guidelines and qualifiers were useful in helping to communicate with patients, select treatment, and determine patients’ prognosis; and that descriptions of developmental presentations were useful as applied to the patient.

Discussion

ICD-11 diagnostic guidelines for depressive disorders, anxiety and fear-related disorders, disorders specifically associated with stress, and ADHD demonstrated moderate interrater reliability, and those for DBD showed a strong level of agreement between clinicians (19). Additionally, most clinicians considered the diagnostic guidelines useful. These ICD-11 guidelines cover diagnoses that are among those with the highest prevalence among children and adolescents in Mexico City and the surrounding metropolitan area (20); mood, impulsive, and anxiety disorders are the most frequent and thus the main reasons for seeking some form of mental health treatment (21).

ADHD combined presentation and oppositional defiant disorder were the most frequent specific diagnoses, and interviewers and observers recognized both. This is a frequent pediatric mental health comorbidity, not only in clinical populations but also in general populations (22–26). Thus it was not surprising that these diagnoses were frequently applied by clinicians to our clinical sample, and the fact that the percentages of both disorders were very similar between interviewers and observers is consistent with the adequate levels of interrater reliability observed for the diagnostic groups of which each disorder is part (i.e., ADHDs and DBDs, respectively). Because research on comorbid and noncomorbid ADHD has been a priority for clinicians in Mexico and Latin America (24), ICD-11 guidelines may serve as an important tool for identifying these diagnoses and should thus be incorporated into clinical practice in the region.

Interrater Reliability Values

More specifically, interrater reliability based on ICD-11 guidelines was strong for DBD, particularly for oppositional defiant disorder with chronic irritability-anger. This finding is especially relevant, because one of the most important debates regarding the classification of mental disorders in childhood and adolescence focuses precisely on the evaluation, diagnosis, and treatment of children with severe chronic irritability and anger. In the past, classification systems have found it difficult to identify this phenomenon, which has delayed its diagnosis and treatment. Fortunately, the formulation of irritability and oppositionality put forth in ICD-11 may more accurately identify chronic irritability than do the ICD-10 or DSM-5 proposals (11, 27).

Conversely, when contrasting our kappa values with those of ICD-11 adult guidelines for internalizing disorders (3), we found similar moderate values, perhaps because of the high comorbidity of the anxiety-depression dimension. The high comorbidity of anxiety and depressive disorders may also account for the fact that not all disorders or groups of disorders obtained strong interrater values.

Clinical Utility

In general terms, good clinical utility was the most frequently chosen answer by both interviewers and observers for almost all the questions. Moreover, the clinical utility of qualifiers, as well as the usefulness of the description of developmental presentations in the guidelines (developmental presentations section), was good to extremely good, which could facilitate the dissemination of guidelines among child and adolescent psychiatrists.

In other words, clinicians perceived that these guidelines improve communication with patients, are detailed enough, and constitute a useful tool for clinical management decisions (2), suggesting that they might be helpful in reducing the global burden of such diseases through early identification and treatment. This could be particularly relevant for countries with a shortage of economic and specialized human resources.

Limitations

The study had some limitations. The group of raters was diverse and included clinicians with several years’ experience and others with fewer years of practice. Moreover, it should be emphasized that kappa values and clinical utility perceptions were obtained from clinicians who had received a 4-hour training session that sought to harmonize criteria and decision making when disorders were diagnosed, highlighting the changes proposed in the diagnostic guidelines for each category. This process may have sensitized clinicians to the diagnostic criteria and rationale of the guidelines, specifically in the developmental presentation sections, which constitutes an innovative way to identify mental disorders across the lifespan (28).

Additionally, the sample was relatively small in that we did not have enough children (ages 6–11) to determine whether interrater reliability indices for clinicians evaluating children are different from those for evaluating adolescents. Moreover, the use of a convenience sampling strategy at specialized centers limited the generalization of results, particularly to patients in other health care settings (such as primary care centers).

Conclusions

Use of ICD-11–based diagnostic guidelines for assessments of mood disorders, anxiety and fear-related disorders, ADHD, and DBD among children and adolescents had moderate interrater reliability in this study. Definitions of these disorders were perceived as useful for selecting treatment and determining patients’ prognosis when applied to children and adolescents.