Informatics Research on Mental Health Functioning: Decision Support for the Social Security Administration Disability Program

The Social Security Administration (SSA) directs two disability benefits programs, which are consequential for millions of Americans, including those who experience a severe mental disorder. The two programs, Social Security Disability Insurance and Supplemental Security Income, pay benefits to millions of individuals who have been found to be disabled under a set of regulations operationalizing a statutory definition of disability. The definition requires that an individual must be “unable to perform any substantial gainful activity due to a medically determinable physical or mental impairment that has lasted or can be expected to last for a continuous period of at least 12 months or result in death” (1).

To begin the process, an individual claimant files an application for disability with the SSA and usually is evaluated by a state Disability Determination Service (DDS). The DDS assembles a dossier on each claimant, including evidence on medical and nonmedical issues, which can include information from multiple sources concerning the claimant’s health conditions, impairments, and functional limitations. Sources may include the self-reported allegations of the claimant and reports from family, previous employers, schools, treating sources, and consultative examinations. Often, the Medical Evidence of Record (MER) comprises hundreds of pages of documentation, and it can be challenging to organize the MER information to fairly and completely evaluate each claimant’s application for disability benefits. At times, it is difficult to determine whether the claimant has a severe impairment, such as schizophrenia or bipolar disorder, and whether the information in the MER supports a claim of functional limitations that preclude performance of substantial gainful activity (SGA). The disability examiner must be able to extract relevant information from the lengthy MER to evaluate a claim. Yet, the MER often lacks information on functional limitations, such as in areas of mental functioning, including cognition and social interaction. It would be advantageous for the disability examiner to have more information on functional abilities and limitations, such as from a functional assessment tool.

Because the disability determination process is so challenging, deliberation often takes considerable time, which can delay decisions and the payment of benefits. Furthermore, the process is subject to appeals, hearings, and even court cases. Over the past decade, SSA has embarked on a program to develop a variety of decision support tools to improve the disability determination process. This article describes a collaboration between SSA and the National Institutes of Health Clinical Center (NIH-CC), Rehabilitation Medicine Department (RMD), in which a team of epidemiologists, biostatisticians, disability subject matter experts, and data scientists have spent more than a decade developing several decision support tools designed to improve the disability determination process. Some are tools to extract existing information from the MER, and others are new tools for collecting needed information on work-related functioning. We particularly focus on the disability claims made on the basis of mental impairments, which pose a challenge to the disability determination process. This collaboration is likely unknown to the readers of Psychiatric Services because the collaboration’s scholarship is published mostly outside of the mental health field. However, the work of this collaboration is relevant to research and practice in psychiatry.

This article outlines the specific issues related to mental impairments and evaluation of mental functioning. The work presented here is both conceptual and practical. It explores how a new paradigm for understanding disability affects research and how new techniques in information science affect the design of assessment tools to support disability determination. The paradigm shift is consistent with contemporary concepts in disability and functioning, viewing the relationship between disability and work-related functioning as a multifactorial phenomenon resulting from the interaction between people and environments (2–4). As such, assessment of disability due to mental impairment is challenging, and novel approaches are needed. We explore the implications of these concepts, describe a set of tools under development on the cutting edge of data science, and discuss the set's use in the evaluation of functional limitations and disability. The tools were designed to fit into the disability determination process at SSA, but no specific assessment profile is yet under consideration. Although the use case for tools focuses on the SSA process, the tools may have broader applicability beyond SSA.

The Special Challenge of Evaluating Disability Due to Mental Impairments

Mental impairments pose a special challenge in the disability determination process, which depends on objective medical evidence to inform the decision-making process. Accurate diagnosis depends on observation of signs of mental impairment consistent with symptoms self-reported by the claimant. It is challenging to infer an individual’s ability to perform SGA on the basis of the severity of any impairment in a body system, but it is particularly challenging for mental impairments. The special difficulty arises because the determination process depends heavily on reconciling self-reported symptoms with observed signs from mental status examinations or clinical settings and from the observations of family and others, when available. To address this challenge, SSA uses assessments of functional limitations and capacities related to work, such as the ability to apply knowledge, interact with others, concentrate, or adapt to new situations. These criteria are set forth in regulations called Listings of Impairment and in criteria for assessing residual functional capacity (RFC). These functional measures are the principal criteria for assessment of disability due to mental impairments. A recent report on functional assessment from a consensus committee of the National Academies of Sciences, Engineering, and Medicine reviews these issues in detail (5).

SSA and its partners at NIH have addressed these issues by developing a new decision support tool designed to elicit information on functional limitations and capabilities directly from claimants and beneficiaries. SSA and NIH also have been developing other decision support tools to improve the assessment of physical and mental functioning by using natural language processing (NLP) to extract data on functioning from the large MER assembled for each claimant. Developing these tools has necessitated some conceptual work on the relationship between impairment and functioning and on the ability to perform SGA. The remainder of this article describes the products and processes of the collaboration between SSA and NIH, with a special focus on assessing mental functioning.

History of the SSA and NIH-CC Collaboration

The SSA approached NIH in 2007 while searching for ways to support the disability determination process. The NIH-CC’s RMD had experience with disability assessment and large data analytics, which had a direct value for SSA. This collaboration led to an interagency agreement signed in 2008, which has been renewed annually and will continue throughout fiscal year 2022. The RMD team supporting this effort has grown over the years and, in order to address the complex challenges of disability assessment, has deliberately included professionals with a wide range of expertise, such as in computer science, epidemiology, medicine, occupational therapy, physical therapy, psychiatry, public health, and statistics.

The main goal of the interagency agreement was to address the accuracy, consistency, and timeliness of SSA’s disability determination processes. This goal is supported by two broad objectives: analysis of SSA data to develop data-driven approaches that inform SSA’s decision-making processes and collection of more systematic and comprehensive data on functioning through the development of the Work Disability–Functional Assessment Battery (WD-FAB). The WD-FAB is a self-report instrument to assess functioning relevant to work across physical and mental spheres of activity. The WD-FAB scales were developed by using item response theory, which enables administration through computer-adapted testing methods in which the relevant questions are selected and administered on the basis of previous responses. The WD-FAB assesses work-related functional limitations in four areas of physical activity and four areas of mental functioning (6–11). The development of and ongoing research on the WD-FAB are discussed in more detail below.

Conceptual Foundations for Assessing Mental Functioning

Throughout the tenure of the SSA-NIH collaboration, NIH’s focus has been on physical and mental function of disability claimants. The statutory definition of disability is rooted in a medical model of disability, but more modern models conceptualize disability as the gap between an individual’s functional abilities and their environmental demands, which in the SSA context means work demands. Therefore, we have used the World Health Organization’s International Classification of Functioning, Disability and Health (ICF) as a framework for much of the work supporting this collaboration (2). In particular, we focus on the activity component of the ICF, which describes functioning at the level of the individual, rather than at the cellular, organ, or body system level. Our team is working on an ontology to organize our thinking and terminology concerning mental functioning to make it easier to communicate with other informatics scientists who might want to conduct studies or develop tools for assessing mental functioning.

Conclusions from the scientific literature (2–6, 11) and the practical experience of disability evaluators agree that the relationships among symptoms, mental functioning, and work performance are not always clear. Establishing this relationship is particularly challenging for determining whether a claimant’s limitations in mental functioning preclude engaging in SGA-level employment. Numerous personal and contextual factors, in addition to health status, are interconnected and can influence an individual’s overall ability to work. This complex relationship has been increasingly recognized as one of the fundamental challenges in work disability assessment (3–5). For example, someone experiencing a psychotic disorder who may display maladaptive social behavior patterns may function well in a job that is relatively solitary and requires little interaction with others but may not be able to engage in work where collaboration and feedback are essential to the job functions. Additionally, with the emergence of telework and remote-work options, it is unknown how workplace changes will affect the evaluation of work-related mental functioning.

To that end, we have proposed grounding our work in disability assessment in a multidimensional conceptual foundation that, beyond symptoms and impairments, includes aspects of the work environment, functional abilities, and behaviors. Therefore, the activity component of the ICF was key in guiding the development of the WD-FAB, focusing on an individual’s functioning related to work. The WD-FAB assesses function in eight areas across the two domains of physical function and mental function. It produces a profile of scores in its four scales of mental functioning relevant to work: communication and cognition, self-regulation, resilience and sociability, and mood and emotions (6). WD-FAB scale scores are standardized to the scores from a large general U.S. working-age sample, with a mean±SD score=50±10. (Scores range from 0 to 100, with higher scores indicating higher functioning.) This means, for example, that a score of 40 in communication and cognition is one SD below the population mean. Extensive research was conducted on the measurement properties, such as validity, and on reliability and related statistics, such as minimal detectable difference (6–10). The mental functioning measured with these scales also tracks closely with the functional criteria in use at SSA for determining disability: understanding, remembering, and applying knowledge; social interaction; concentration and task persistence; and adaptation and self-management.

In addition to assisting a disability examiner in making a disability determination, the WD-FAB potentially offers a way to assess work-related mental functioning comprehensively. The WD-FAB yields important information on an individual’s abilities and limitations, which can help assess the individual’s capacity to work or to perform other roles. The results of a WD-FAB assessment might be useful to a clinician in identifying the areas of functional abilities and limitations of a service user in a rehabilitation program. WD-FAB results also might inform a job search by an employment specialist on behalf of a client seeking supported employment.

Current Work on the WD-FAB

After the WD-FAB was developed and its psychometric properties were validated (6–11), research focused on supporting interpretation of the WD-FAB scores and aligning WD-FAB data with work-related outcomes. Our focus in supporting SSA’s disability determination process is to help understand the alignment of a person’s functional abilities with the functional capacity required to meet work demands. The WD-FAB helps measure functional abilities, and we are now exploring the relationship between WD-FAB scores and work demands. We are in the process of collecting and analyzing data from three separate efforts to start to characterize this alignment.

The first effort is a pilot study comparing WD-FAB scores for individuals with self-reported work disability and for individuals who are employed. We are collecting data on three key job duties for the individuals’ last held position or current occupation. These data allow us to explore whether threshold scores on the WD-FAB can indicate when individuals can work and to establish functional profiles for the occupations represented in the study.

We are also collecting WD-FAB data as part of SSA’s Supported Employment Demonstration (SED). This is a longitudinal study to examine whether providing “evidence-based interventions of integrated vocational, medical, and behavioral health services to individuals with behavioral health challenges can significantly reduce the demand for disability benefits and help individuals remain in the labor force” (12). The WD-FAB data allow us to characterize the mental functioning of study participants, study how workplace function changes over time, and evaluate whether levels or changes in function, as reflected in WD-FAB scores, are associated with work outcomes. Assessing mental functioning in this way is a critical element in psychiatric services research, such as for evaluating outcomes in intervention studies.

SSA is also conducting a pilot study to assess the potential inclusion of the WD-FAB in its continuing disability review (CDR) process. The CDR is a periodic review of current beneficiaries’ cases to ensure that they continue to meet SSA’s definition of disability. Like the SED, a key component of the CDR pilot study is understanding changes in functioning over time as measured by the WD-FAB. New tools such as the WD-FAB should find their place in rehabilitation practice and in evaluation research that focuses on mental health and related impairments (5, 6, 11). We hope that the WD-FAB will yield individual profiles of abilities and limitations in both physical and mental functioning as reported by claimants. These profiles would support the decision about the ability of a claimant to perform substantial gainful employment.

Using NLP to Support Disability Decision Making

Function at the individual level is another principal focus of our analytic work. As part of SSA’s five-step determination process, an SSA adjudicator must determine whether a claimant can work, either at a past job or at another position. To decide whether a person can work, SSA needs information on the claimant’s functional abilities. SSA generally captures this type of information through the RFC assessment, which has separate assessments for physical and mental functions (1). However, the RFC assessment is not based on a direct assessment of the claimant, such as with a tool like the WD-FAB, but rather on a summary compiled from the available medical evidence (i.e., the MER). For example, RFC assessment for functional limitations due to mental impairments involves evaluating information in the MER provided by psychiatrists and other mental health clinicians as well as by the claimant, family members, and former employers, when available. Information on function is often documented in the free-text portions of the medical record, and it can therefore be time-consuming to locate all relevant evidence. Informatics methods, such as NLP, offer ways to scan medical records more efficiently and to automatically identify, extract, and organize information on function that is relevant to SSA’s disability determination process.

Because of the large number of applications for disability benefits the SSA receives, the adjudicators may benefit from tools to efficiently locate relevant medical and functioning information that can support the applicant’s claims and assist in making more accurate decisions. Given the time-consuming process for reviewing applications, the adjudicators may greatly benefit from automated solutions that can extract and highlight the relevant information they are searching for (13). NLP models have proven to be very effective and show high performance in extraction tasks if they are provided with sufficient training data. Compared with extraction of functional information, extraction of general medical information has been shown to be an easier task, given the ICD codes available in the applicant’s records and the availability of predefined terminologies to extract diagnoses and symptoms (14–16). Functioning information, on the other hand, is more challenging to extract because of the lack of codes and because interpretation of the free-text portion of the medical record depends highly on context (17, 18). This challenge is exacerbated in areas such as mental health, where information about functioning is found in the nuances of the language used in the free-text notes (e.g., “The patient is able to concentrate and follow instructions”) (19). Our team is conducting work in mental health functioning by using NLP techniques, and NLP research in the broader mental health domain is abundant and focused on data set generation (mainly from social media posts and electronic health record reports), automatic risk assessment, and symptom and diagnosis extraction (e.g., involving use of coded information such as concept-unique identifiers [20], dictionaries [21, 22], or ontologies [23, 24]).

To collect information on mental health functioning for disability determination, we have proposed an information extraction framework (19) derived from a careful examination of the key elements of the SSA statutory definition of disability. To guide the extraction, we identified four dimensions of the extracted information that are crucial for disability determination: temporal information, including sequence and duration of observations of mental functioning; level of performance or degree of difficulty in extracted mentions of functioning; the context of mentions about mental functioning with respect to work and work-related information; and the source of the information. All four dimensions are used to identify and characterize a “span of text” attributed to mental health functioning in a claimant’s MER. The NLP technology extracts information from the MER that specifically mentions a claimant’s mental functioning and characterizes information related to when the limitation in functioning occurred and its duration, the degree of the functional limitation, whether it affected work or emerged in some other context, and who reported the limitation.

The following statements illustrate the kind of extracts that NLP might produce from the computer review of an outpatient clinic note: “The patient reports that he has been unable to complete simple tasks at home, such as watering his plants, for the past 6 months.” “The patient’s spouse said that he was very withdrawn at work, and he speaks little with her or with friends.” NLP also can predict or classify information in the MER or other records concerning various aspects of mental functioning, such as severity of a limitation. The approach in the use case of the SSA disability determination utilizes a variety of NLP technical solutions used in other NLP applications, such as temporal reasoning (25–27), sentiment analysis and risk assessment (28–33), incorporation of the use of environmental characteristics and social determinants of health (34, 35), and author attribution (36). These NLP techniques have been used to evaluate suicide risk in social media posts and to assess the need for treatment. They also have been used to identify text in electronic medical records that characterizes the severity of symptoms, when they occurred, and who noted them. There is a risk that extracting small bits of narrative text can decontextualize the information identified, potentially missing cultural or regional contexts, but the NLP extracts are not the sole sources of information about a service user or disability claimant. In addition to NLP extracts, disability examiners and other NLP users should also attend to contextual information available in the MER or other source material.

In 2017, the NIH team began a dedicated effort to develop NLP models and resources to extract information related to individuals’ function, again by using the activity component of the ICF to guide the framing and definition of the project. Recognizing that identifying and extracting information related to mental functioning would be particularly challenging, we formed a working group in fall 2018 to support NLP efforts around mental functioning. The first area of mental functioning we chose to model was the interpersonal interactions and relationships (IPIR) chapter of the ICF.

The computer modeling work of IPIR aligned well with one of SSA’s key mental-functioning criteria for determining disability for a claimant: interacting with others. The work required developing an annotation schema, an approach that allows the computer to capture (i.e., to identify and extract, as well as search and find) IPIR information in clinical documents. The schema is based on a terminology built by extracting IPIR-related terms from SSA’s mental RFC assessment, all related to interacting with others. On the basis of this list of IPIR-related terms, an IPIR schema for annotation and an IPIR annotation guideline were developed. The data set used for annotation and computer modeling came from clinical and medical records from the NIH-CC, and disability applications came from SSA. Using the annotated data for training and validation, the team built NLP models to automatically detect and extract sentences containing IPIR information from the medical record. Interannotator agreement was a challenge before training, but a study of agreement among trained raters achieved a Cohen’s κ=0.81 (J. Porcino, personal communication, May 17, 2022). Examples of IPIR extracts include the following: a note reporting on an occupational therapy group saying, “The patient did not interact with anyone in the group session today, similar to behavior, off and on, during the past year,” and a progress report from an inpatient psychiatric unit stating, “The patient engaged in a heated argument with family members during a family therapy session.”

NLP tools for extracting content about mental functioning from narrative text may have practical benefits beyond supporting SSA disability determination. Such tools might be useful in any review of the health records of a service user, such as in a second opinion review of someone with an extensive clinical history and a voluminous chart. NLP tools might also be used in managed care reviews to search for specific categories of information such as mental functioning and types of impairment. These tools could be used in forensic evaluations and legal depositions to extract all relevant information on a specific topic from testimony and written evidence. Furthermore, NLP tools have been used to extract data to determine clinically significant measures, such as the duration of untreated psychosis (26).

Potential Impact of the Collaboration Between NIH-CC and SSA

The collaboration between SSA and NIH has yielded numerous benefits and promises more in the future. It has advanced conceptual thinking on disability as well as on the relationship between impairments and functional limitations and between those limitations and work-related activities. These are thorny issues, but the collaboration has encouraged SSA to emphasize functioning and work activity along with its focus on impairment severity. Conceptual work has also contributed to thinking about approaches to using informatics techniques such as NLP to support disability decision making. Applying those concepts to tool development has guided NIH and its collaborators in academia in developing the WD-FAB and in piloting its use in various aspects of SSA’s disability determination processes.

Developing decision support tools for use in disability determinations holds promise for more complete evaluations that take less time and perhaps result in fewer appeals. People who use psychiatric services often depend on SSA disability benefits as the sole source of income support. Beneficiary status also confers eligibility for Medicaid and Medicare benefits, which give them access to behavioral health services. Eligibility status for these benefits may also confer eligibility for other social services and subsidies for housing or food. Greater accuracy and faster disability determinations are better for applicants and for the broader society, building confidence in the important disability programs at SSA. Beyond their potential use in SSA disability determination, these tools could also provide insights into approaches to rehabilitation that might lead some individuals to employment or to other forms of social participation.

Conclusions

This article describes how informatics can benefit the field of psychiatric services research and practice in disability assessment and rehabilitation. The valuable and ongoing collaboration between two venerable U.S. institutions—SSA and the NIH-CC—has important implications for the field and for persons with mental impairments submitting claims to the SSA.