Reliability of the Term “Serious Mental Illness”: A Systematic Review

The constructs of “serious mental illness” and “severe mental illness” (the abbreviation SMI is used for both) have been priorities in public policy, mental health, and research funding initiatives in the United States and elsewhere (1–4). However, there is a lack of clarity and standardization regarding how SMI is operationally defined across contexts, including within legal standards, clinical settings, and scientific research literature (5). In addition to the arguments for empirical precision, there is the argument that by labeling already heavily stigmatized mental health diagnoses, such as schizophrenia (6), as “SMI,” we may be exacerbating the well-established consequences of stigma for individuals with these diagnoses, including social rejection and perceived incapacity (7). Considering the substantial treatment and policy efforts currently targeted toward SMI, it is important to evaluate the term’s reliability and validity.

Operational Definitions for “SMI”

The construct “severe and persistent mental illness” (SPMI) was utilized for decades prior to any attempts to formalize the nature and degree of diagnosis, disability, or illness duration (8). In their 1990 review, Schinnar and colleagues (8) identified SPMI criteria from 17 scientific publications. Researchers applied the separate criteria to an urban community mental health center sample (36% diagnosed as having schizophrenia or affective disorder) to identify the number of individuals meeting SPMI criteria. They found that depending on the criteria applied, anywhere from 4% to 88% of individuals met SPMI criteria.

“SMI” is not an official diagnostic term defined within existing classification systems, such as the DSM (9) or the ICD ( 10). The American Psychiatric Association (APA) is variable in how it defines SMI. For example, on the APA-hosted “What is mental illness?” Web page (11), SMI is defined as “a mental, behavioral, or emotional disorder (excluding developmental and substance use disorders) resulting in serious functional impairment that substantially interferes with or limits one or more life activities.” On its Web page specific to SMI (“What is serious mental illness?”) ( 12), APA’s SMI Adviser indicates that an individual who meets SMI criteria is “over the age of 18 who has (or had within the past year) a diagnosable mental, behavioral, or emotional disorder that causes serious functional impairment that substantially interferes with or limits one or more major life activities.” The National Institute of Mental Health definition of SMI is most like the latter criteria but without any reference to timing of illness (13).

Although quite similar, these criteria hold nuanced yet meaningful differences. The first set of criteria specifically exclude developmental disorders, such as intellectual disability, which cannot be treated in the traditional sense; rather, individuals with intellectual disability are provided support to compensate for low intellectual and adaptive functioning capacities (9). The second set of criteria specify age and potential recency of symptoms (12). By these latter criteria, an individual with a developmental disability or a substance use disorder could meet criteria for SMI. Across resources, APA indicates that diagnostic examples of SMI include schizophrenia, severe major depressive and bipolar disorders, and “a few other disorders” (11, 12). However, it does not provide a comprehensive list of conditions that meet SMI criteria or any assessment approaches or clear benchmarks that would indicate that an individual has SMI. Thus, the lack of consistency first delineated by Schinnar et al. (8) remains prevalent.

It is important to acknowledge that motivations for flexible terminology that can be applied across various behavioral health settings are understandable. As stated by Wing (14):

Although Wing referenced severe mental illness in particular, the broader literature demonstrates that both severe mental illness and serious mental illness are nebulous diagnostic constructs.

Wing (14) observed in 2004 that research had yet to reveal which mental illnesses were consistently considered SMI. Until that point, various SMI criteria demonstrated sensitivity to which individuals could be classified as having SMI (15, 16). Since then, researchers have sought to resolve SMI’s “fuzziness.” In an Italian study, Parabiaghi et al. (17) concluded that SPMI terminology had clinical utility because it was associated with marked psychopathology and disability, high mental health service utilization, and unemployment. However, in their concept analysis of SPMI terminology, Zumstein and Riese (18) found that scholars use SPMI and SMI interchangeably, that the constructs lack definition, and that further clarification is needed prior to determining its utility. Of note, the authors opined that SPMI terminology should be retained if it was refined to be “context dependent” to reflect “local” conventions of health and health care.

Implications of “SMI” in Research, Practice, and Policy

Although the desire to use “SMI” as a catchall term is well intentioned and rooted in practicality, its inconsistency can have meaningful practice, research, and policy implications. Without a consistent definition, researchers who consider their study populations to have SMI have difficulty building a generalizable and evidence-based practice literature, and practitioners seeking evidence-based SMI practice may be limited in their ability to find a solid, reliable research base to inform their work. Further, the term “SMI” has been used to advocate for prioritization of services and benefits among individuals who have been determined to meet criteria (12). Without a consistent empirically based definition, the decision-making process becomes less clear regarding which populations to prioritize and under which circumstances.

Further, mental illness diagnostic labels are associated with self-stigma (i.e., the internalization of negative stereotypes) for individuals labeled as “mentally ill” and can act as a barrier to recovery (19). Theoretical models for the process by which negative social conceptualizations of mental illness become internalized have suggested that the act of psychiatric diagnosis is a key component; once an individual is given an official diagnostic label, preexisting conceptualizations of mental illness become personally relevant (20). This is not to say that psychiatric diagnosis is without benefits, because it may provide a shared language for navigating distressing experiences, a guide for treatment development and planning, and a starting point for service access (21). However, theorists have argued that such benefits also illustrate the overly functional nature of psychiatric diagnosis as a status-based tool (22).

A growing body of research has attended to the nature of psychiatric diagnostic categories themselves, including heterogeneity regarding classification of disorders (23) and negative consequences of continued use for heavily stigmatized diagnostic labels, such as schizophrenia (24). Importantly, this literature has solely focused on official diagnostic labels found in the DSM. “SMI” is often used as an umbrella term encompassing several diagnostic categories and implies a disorder of a severe and potentially unremitting nature. Considering the literature demonstrating an association between stigma and diagnostic psychiatric labels more generally, use of an additional negative specifier that notes a more “severe” nature of a psychiatric diagnosis may exacerbate attitudinal consequences of diagnostic labeling, including stigma and pessimism regarding treatment prognosis and recovery (25–29).

At the policy level, poor reliability of the term “SMI” can detract from programs that allot social services and government assistance for people with “SMI.” An individual who meets SMI criteria in one context may not meet it in another, which, at the person-level, could be disorienting and a hardship when that person seeks needed services. Hundreds of millions of U.S. federal dollars are allocated specifically for SMI programming each year (30), and without a strong evidence base to guide policy, the fiscal implications could be exceptional.

Objectives of this Systematic Review

Empirical investigation is lacking in regard to the reliability and utility of “SMI” as a psychological construct. Given the lack of standardization and potentially detrimental impact of using the “SMI” label in addition to more empirically established diagnostic terms, our objectives were to conduct a systematic review of its utility and incremental validity, understand how researchers use the term and examine consistencies and differences in meaning, and provide recommendations for research and practice. This systematic review was guided by two research questions: How is “SMI” operationally defined within the scientific literature? and What is the reliability and validity of “SMI” within the scientific literature?

Methods

Defining “Serious Mental Illness”

To review potential SMI search terms, we invited input from three research experts identified as having extensive publication records (more than 50 empirical papers) regarding mental illness treatment and policy and serving on relevant peer-reviewed editorial boards. In addition, we chose experts from varied geographic regions of the United States. Finalized terms included serious or severe mental illness, serious or severe and persistent mental illness, and serious or severe emotional disturbance. We also included non–person-centered versions (e.g., seriously or severely mentally ill). The term “chronic mental illness” was also considered as an SMI search term; however, this was regarded by experts as used primarily in the 1980s and 1990s and not relevant for a review of more recent literature and was ultimately discarded.

Systematic Review

The systematic review followed reporting guidelines as established by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) (31). In January 2020, the first author conducted an initial search (details are available in an online supplement to this article) of PsycINFO, PsycArticles, and PubMed for empirical papers containing SMI terminology published online first in peer-reviewed journals between January 1, 2015, and December 31, 2019. After removing duplicates, the first and second authors initially screened studies for the following eligibility criteria: English language, peer reviewed, original empirical study, human participants, and “SMI” used to characterize the participant sample.

Three reviewers (C.C., L.L-A., B.M.) then conducted full-text reviews of the remaining articles, including additional eligibility review that may have been missed in the initial screen. Each article was assigned to two of the reviewers, and all three reviewers met weekly to discuss any coding discrepancies and further clarify coding guidelines if needed. Reviewers coded article characteristics as follows: the sample’s country or countries, whether the study was a randomized controlled trial (RCT), age group (<18, ≥18, or mixed-age sample), SMI terminology (e.g., serious mental illness and severe mental illness), whether and how researchers defined SMI, and the sample’s diagnostic makeup (coding variable levels are presented in accompanying tables). We did not collect additional demographic or outcome data, given that we were interested in study methodology rather than products. The meta-analytic coding process is iterative, meaning that a priori coding schemes may be amended as coders encounter new variables that warrant coding additions (32). The first author and the three reviewer-coders discussed new and amended codes as applicable. Weekly meetings were joined by the first author for the first 100 records; for the remainder, discrepancies with no immediate consensus among the three reviewers were resolved with input from the first author (33 ).

Interrater Reliability

Interrater reliability for coding was calculated by using Cohen’s kappa. Most kappas ranged from 0.70 to 1.00 across coding categories. However, several categories presented more difficulty than anticipated upon review, including whether a paper provided an operational definition for SMI (κ=0.65), provided diagnoses for participants considered to have SMI (κ=0.68), and reported participant age group (κ=0.51). Low interrater reliability for these categories can be partially explained by coding updates made mid-review to reflect the nature of included studies; for example, “participant age group” was updated to include an additional coding category accounting for a significant number of studies that included both adult and youth samples. However, whether studies provided an operational definition of SMI was an especially complicated category that required multiple rounds of discussion. This need for extended discussion was due to a tendency of many studies to provide diagnostic examples or allude to “SMI” without providing a formal operational definition used for study purposes; as a result, we created an additional code of “no operational definition, but diagnostic examples provided.” The first and second authors coded SMI operational definitions and sample diagnoses into additional distinct categories. Most Cohen’s kappas for these categories ranged from 0.77 to 1.00, except for two categories with low base rates (presence of other trauma-related disorders, κ=0.53; presence of eating disorders, κ=0.60). We met and resolved all coding discrepancies prior to analysis.

Results

A total of 788 papers met eligibility criteria and were coded for inclusion in descriptive content analysis. (The online supplement provides a flow diagram reporting total records screened and excluded across the stages of review.)

Qualitative Synthesis

Only univariate statistics (frequencies) were conducted for this systematic review, given that we were interested only in study and sample characteristics rather than study outcomes. Table 1 provides characteristics of studies included in analyses. Researchers conducted studies in approximately 47 countries, and approximately half the studies (47%) recruited participants from the United States. RCTs accounted for 23% of study designs. Participants were most often described as individuals with SMI (81% of studies), but studies also included SMI treatment providers (4%), family members of individuals with SMI (2%), and SMI caregivers (1%). Participants typically included adults (88% of studies), with youth-adolescent and combined youth-adolescent and adult samples comprising 4% of studies.

Defining SMI.

In the bulk of studies, researchers did not operationalize “SMI” (85%). Only 15% of studies explicitly defined SMI, and 26% provided diagnostic examples of SMI without formal definitions (e.g., severe mental illness encompasses schizophrenia, schizoaffective disorder, and bipolar affective disorder). When referencing the general SMI construct, most researchers referenced severe mental illness (41%) or serious mental illness (37%). Fewer studies used the terms severe and persistent mental illness (2%), serious emotional disturbance (1%), or serious and persistent mental illness (<1%). Several studies varied their language when referencing SMI, e.g., using severe mental illness and serious mental illness in the same report (19%).

Operational definitions.

Researchers most often (37% of studies) operationalized “SMI” as a handful of specific diagnoses (Table 2), and relatively few indicated that the construct of SMI could be as broad as “any” mental illness (4%). Authors described SMI as involving functional impairment in 9% of studies. Several studies (5%) referenced timing (e.g., duration or chronicity) or specific services received (2%) as central to the SMI concept. Less often, researchers used Global Assessment of Functioning scores or other psychometric measures (1% each) to identify prospective participants with SMI. Very few studies (1%) adopted a formal legal or organizational standard to define SMI.

Participant diagnoses.

Participants’ diagnoses were specified in 82% (N=645) of the 788 studies. Given diagnostic differences across youth and adult populations, we split diagnostic frequencies according to whether researchers referenced serious emotional disturbance (SED) or experiences of psychiatric disorders that cause substantial functional impairment, typically used in reference to conditions among youths (34), or whether they used an SMI variant, typically used in reference to adult conditions. We included diagnoses across terminology (Table 3).

In general, studies referencing SED included diagnoses or symptoms similar to those used in studies referencing SMI. Overall, the most prevalent diagnoses were schizophrenia and schizoaffective disorder (62%), bipolar disorder (52%), and depression (34%). In a third of studies (33%), participant samples consisted of individuals with other psychotic disorders or, more broadly, “psychosis” as a symptom. Researchers incorporated other mood disorders (14%), anxiety (14%), personality disorders (14%), substance use disorders (11%), and posttraumatic stress disorder (9%) with some regularity. All neurodevelopmental disorders, which included attention-deficit hyperactivity disorder (ADHD), autism spectrum disorder, and intellectual disability, were included in 6% (N=48) of studies. Neurocognitive disorders, which included dementias, accounted for only 2% of diagnostic mentions. Obsessive-compulsive disorder (1%) and eating disorders (1%) were cited much less frequently. Of note, the only diagnosis more characteristic of SED research than of SMI research was ADHD, which was cited by 2% of SMI researchers and 9% of SED researchers. Further, no SED researchers cited disruptive behavior disorders among their diagnostic criteria.

Eleven percent of studies noted additional criteria that we collapsed into an “other” catchall category. Most often, we coded “other” when authors explicitly stated that “other” diagnoses fit their inclusion criteria. Three studies (<1%) included epilepsy. Only one study included “suicidality” among its criteria.

Risk of Bias

Traditionally, the utility of a meta-synthesis is largely dependent on the validity of its incorporated studies, and researchers assess risk of bias according to specified criteria (35). This project was unique, given that it sought to understand “back-end” methodological issues, specifically construct reliability and validity, rather than participant-focused study outcomes, as do most meta-syntheses. Thus, the typical study features that could increase risk of bias—including selective reporting of outcomes, random sequence generation, allocation concealment, blinding, and incomplete outcome data (36)—were irrelevant here. Our goal was to identify the prevalence of specific SMI terminology and meaning. According to an authoritative systematic review source (37), “The procedures for publication bias should only be applied when the likelihood that a study will be published is affected by a finding of statistical significance.” It is unlikely that the validity of this synthesis would be affected by issues related to statistical significance. It is possible that loose and inconsistent operational definitions of SMI could reduce statistical power, thereby reducing the likelihood that a study produces significant results (38) and decreasing its chance of publication. However, the “file drawer” phenomenon under these circumstances would result in more, not fewer, inconsistent SMI operational definitions. Our findings already indicate that SMI is an unreliable construct. In the aforementioned methodological scenario, should unpublished research be revealed, it would likely only bolster the finding of this synthesis.

Discussion

Although the large number of studies (N=788) included in this review illustrates widespread prevalence and use of “SMI” throughout the empirical literature, our findings suggest that the term is often insufficiently and inconsistently defined, demonstrating poor reliability. Our review found substantial variability across the literature regarding how SMI is operationally defined, with 85% of studies failing to define the construct at all. Significant variability was noted regarding the criteria used to define SMI, including whether the SMI label was inferred depending on a diagnostic label or mental health services received (e.g., schizophrenia spectrum disorder or psychiatric hospitalization), whether functional impairment was required, whether a durational component was required, and whether SMI included specific symptoms or behaviors (e.g., psychosis or suicidality). Additional variability was noted regarding which, if any, diagnostic classifications fall under the SMI umbrella; this suggests that attempts to broadly distinguish individuals as having SMI or not having SMI based on diagnosis alone is without empirical basis. Overall, the lack of reliability for SMI definitions across the empirical literature suggests that the term has poor validity and questionable utility as an empirical or diagnostic construct.

What makes a mental illness “serious”? From our review, very few studies based this classification on any indicators of symptom severity or mortality; the most prevalent criteria among studies that provided an operational definition of SMI included specified mental health diagnoses and “functional impairment.” As discussed, the wide variability of diagnoses suggests that basing an SMI classification on one or several diagnostic categories is empirically questionable. The additional “functional impairment” criterion is similarly nonspecific, considering that clinically significant distress or functional impairment is required for any psychiatric diagnosis (9).

The inference of “SMI” on the basis of participants’ diagnoses, without further operationalization (e.g., schizophrenia is a severe mental illness), is also dubious. If individuals are considered as having “SMI” simply because of their diagnosis, this calls into question the utility of the term to provide additional clinically relevant information beyond that of a psychiatric diagnosis. Further, the diagnoses highlighted in the literature commonly included what researchers have proposed to be some of the most stigmatized, such as schizophrenia (39). For such individuals, being labeled with SMI in addition to their already stigmatized diagnoses may result in internalized stigmatization and label-avoidance behaviors (40).

In regard to practice and policy-related implications, if some individuals are indicated as having SMI, we must also consider the consequences for those individuals not given an SMI label—in other words, if we deem some mental illnesses “serious,” what does this say about the others? Specifically, SMI labeling may act to further stratify mental illnesses. Such processes can have social and material impacts on the care and support administered to people not labeled as having SMI. The SMI label may also be harmful because it may limit the help that those labeled “high functioning” (i.e., those without SMI) have access to and how seriously their experiences are taken by others (41). Importantly, “high-functioning” and “low-functioning” labels have also demonstrated questionable validity and utility for autistic communities (42). SMI labels may similarly divide and prioritize certain types of psychiatric experiences and “productivity levels.”

The inconsistencies of the SMI label are further illustrated by the fact that eating disorders are known to have the highest mortality rate of any diagnostic category (43) but appeared in a small minority (1%) of studies. Suicidality, currently the 10th leading cause of death in the United States (44) and the fourth leading cause for individuals ages 15–29 globally (45), was explicitly included among only one study’s eligibility criteria. The designation of specific diagnostic categories as SMI has led to targeted policy and treatment efforts worldwide (1–4); however, if such campaigns are directed at a diagnostic classification that is not reliable, valid, or empirically sound, any large-scale benefits of such programs will be unclear, rather sporadic, and potentially even iatrogenic.

Study limitations included a literature search limited to three databases; however, considering the vast literature found in the search and the emphasis on a specifically psychiatric-psychological construct, use of PsycInfo, PsycArticles, and PubMed can be considered sufficient for coverage of the SMI literature. This study was also limited by its inclusion of articles for which an English-language version was available; future research may evaluate whether the construct of SMI remains consistent across languages.

Conclusions

Overall, the findings from this review have relevance for use of the terms “serious mental illness” or “severe mental illness” across research, practice, training, and policy settings. First, findings highlight a critical need for establishing clear operational definitions of SMI in the empirical literature. This is indeed a tall order, because variations in the definition of SMI are clearly prevalent, with widespread use of varying definitions. However, the psychiatric literature must strive for precision regarding the populations described in order to promote scientifically sound research and empirically based practice. We encourage professional work groups to generate possible solutions to resolve ambiguity in the term “SMI.” In the absence of a collective term, we encourage researchers, practitioners, and policy makers to refer to specific diagnostic categories, impairment “benchmarks,” or some other agreed-upon approach to standardizing the meaning of SMI. Second, potential negative and stigmatizing consequences of use of “SMI” in practice settings should be evaluated further, considering the findings of this review. Additional research should evaluate the reliability of the term “SMI” in clinical practice, because use of a term with poor reliability suggests that two providers may use SMI in their practice with different meanings. Finally, training competencies and policies targeting SMI should include clear operational definitions and reference empirically validated classifications of disorders for which treatments and resources can be tailored and directed.