Development and Validation of a Computerized-Adaptive Test for PTSD (P-CAT)

Computerized-adaptive tests (CATs) based on item response theory (IRT) provide brief but sensitive and accurate assessments of health status (1–4). IRT models estimate how precise test items are in assessing symptom levels at any point along a latent dimension (θ). The result is an efficient algorithm using precisely calibrated items to estimate level of distress, with a stopping point based on uncertainty of the estimate. A CAT first selects a highly discriminating item from an item bank. Given the person’s previous response(s), the algorithm then selects the next item with the highest possible information that contributes to a score on the underlying dimension. This adaptive item presentation continues until a predefined measurement precision is reached or a specified number of items have been presented.

CATs have reduced the number of items needed for reliable and valid assessment to 14% of the original number, requiring only 20% of the time needed for completion of the original assessment (5). Median time to complete a 12-item CAT for anxiety was 2.48 minutes (6). This reduced assessment time can be particularly valuable in fast-paced clinics, where efficiency is a high priority. Additional benefits of CATs include reduced scoring and processing time, increased measurement precision, and more individualized assessment (3,4,7). These benefits can reduce errors and increase accuracy of assessment in both clinical decision making and research (7).

CATs have been developed for several conditions, including anxiety, depression, physical functioning, community integration, and personality disorder (5,6,8–12). However, to date no CAT has been developed for posttraumatic stress disorder (PTSD). A recent Institute of Medicine report on PTSD treatment for service members and veterans noted that lack of routine outcomes measurement made it nearly impossible to determine whether treatment in naturalistic settings was effective (13). The report recommended that the “DoD [Department of Defense] and VA [Department of Veterans Affairs] should develop, coordinate, and implement a measurement-based PTSD management system that documents patients’ progress over the course of treatment . . . with standardized and validated instruments.” The VA uses the PTSD Checklist (PCL) and a four-item screener (in primary care). A CAT could increase efficiency and utility over paper-and-pencil measures and facilitate evaluation of treatment outcomes at the provider, facility, and organization levels. Thus the goal of the work reported here was to develop a CAT to facilitate PTSD screening, assessment, and outcomes monitoring. Veterans are an appropriate population for a PTSD CAT (P-CAT) because they are at greater risk of PTSD than the general population (14), although a P-CAT could also be useful in non-VA treatment settings and for the general population.

Methods

Overview

The study was conducted in two phases. In phase 1, we administered a previously developed item bank to 1,085 veterans to calibrate the items and develop the P-CAT (15). In phase 2, we administered the P-CAT to a separate sample of 203 veterans to test its validity. Data collection began in January 2011 and was completed in September 2013.

Participants

Calibration sample.

The calibration sample included 1,085 veterans: 908 were recruited from a national panel maintained by an Internet survey company that provides its members with a computer and Internet access. To ensure inclusion of individuals on the higher end of the PTSD symptom spectrum, this sample was supplemented with 177 veterans who had a diagnosis of and were being treated for PTSD at a VA hospital. Eligibility criteria for both groups included at least one traumatic event (DSM-IV-TR PTSD criterion A1 [16]) and consent to complete a one-time survey assessing demographic characteristics, stressful life events, and 89 PTSD symptoms (the item bank). To include veterans from various wartime eras, the sample was stratified by age (50% less than age 45 and 50% age 45 or above). To ensure representation of women, they were oversampled to constitute 20% of the sample. Oversampling segments of the population has been used in previous CAT development work (17,18).

Sample for field test validation.

The field test validation enrolled a convenience sample of 203 veterans, similarly stratified by age and gender. Three groups were enrolled. Group 1 (N=91) included veterans who had received a diagnosis of and treatment for PTSD within the past year (at least two visits with a PTSD diagnosis and two visits for PTSD specialty care). Group 2 (N=60) included veterans treated for a different mental health condition (no PTSD but at least two non-PTSD mental health visits in the past year). Group 3 (N=52) included veterans treated for a general medical condition but not a mental health condition (no psychiatric diagnoses or mental health visits but at least two general medical visits in the past year). Because of clinician concerns about symptom exacerbation, veterans with a suicide “flag” in the medical record were excluded (<1.3% of mental health service users at participating sites).

Calibration Measures

Traumatic exposure screen.

The screening question from the PTSD module of the Structured Clinical Interview for DSM-IV (SCID) (19) was used to screen for traumatic exposure: “Sometimes things happen . . . that are extremely upsetting . . . like being in a life threatening situation . . . major disaster, . . . serious accident, . . . fire, . . . physically assaulted or raped, seeing another person killed, dead, or badly hurt. . . . Have any of these . . . things happened to you?” An affirmative answer was followed by the question, “At that time, did you feel intense fear, helplessness or horror?” A second affirmative answer was required for inclusion in the calibration sample.

PTSD item bank.

The item bank consisted of 89 items described previously (15). Briefly, we used the DSM-IV PTSD diagnostic criteria as a conceptual framework, conducted a systematic review of PTSD instruments, and created a database of items from existing instruments. Although DSM-5 was not completed when we began the study, we followed its progress and included items from new domains that were likely to emerge. The time frame for all items was the past month.

Validation Measures

Three self-report measures were used to validate the P-CAT: the civilian version of the PCL (PCL-C), Mississippi Scale for Combat-Related PTSD (M-PTSD), and Primary Care PTSD Screen (PC-PTSD) (20–22). To minimize burden and distress for participants, we used the SCID PTSD module (19) to obtain a standardized clinical diagnosis rather than the Clinician-Administered PTSD Scale. The PCL-C is a 17-item PTSD symptom scale with high reliability (.96) and validity (23). The M-PTSD is a 35-item instrument with high sensitivity (.93) and specificity (.89) (21). The PC-PTSD is a four-item screener with high sensitivity (>.75) and specificity (>.86) (22) that has been implemented in VA primary care. Validation measures were based on DSM-IV because no DSM-5 measures were available at the time.

Procedures

Item calibration.

All calibration sample participants (N=1,085) completed demographic questions and the 89-item PTSD item bank. The Internet sample was contacted and screened for eligibility by the Internet survey company; participants then completed the survey via the Internet. The VA sample completed a mailed survey. Written consent for this phase of the study was waived by the institutional review boards.

Field test item validation.

All validation sample participants (N=203) provided written informed consent. They were then administered the P-CAT on a tablet computer and also completed a paper-and-pencil packet containing the validation measures. A trained interviewer administered the SCID PTSD module. All procedures were approved by the institutional review boards of the participating hospitals.

Data Analysis

Using the calibration sample, we conducted a factor analysis to assess dimensionality of the P-CAT items. We first examined a unidimensional one-factor model by using confirmatory factor analysis. Anticipating that PTSD symptoms might encompass multiple domains paralleling the diagnostic criteria, we also explored a bifactor model, which allowed for a primary dimension (PTSD) and multiple subdomains (6,8). Model fit was examined by fit indices; model comparison was based on the likelihood ratio chi-square test. On the basis of the fitted model, we applied a multidimensional graded-response IRT model to estimate the item parameters and examine item fit on the basis of the Z score (standardized difference between the observed and expected log likelihood of response patterns). We used a one-sided test; under the null hypothesis, the z scores were normally distributed. A value less than −1.645 indicated misfit (24,25). Analyses were conducted using Mplus, IRTPRO, and SAS, version 9.2 (26,27).

P-CAT Development and Psychometric Properties

To create the P-CAT, we used CAT software developed by the Health and Disability Research Institute (11,28,29). Using a maximum a posteriori estimation procedure (30), the software selects successive items that maximize precision of the respondent’s score estimate, based on responses to each previous item. We used Newton-Raphson iterative methods (31) to compute initial estimates of PTSD severity; the score was updated after each successive response. The first P-CAT item presented was selected on the basis of the highest information function across the score range and on the basis of item content considered central to PTSD. We set 12 items as the program stopping rule to allow scores for all four DSM-5 domains. To ensure content balancing, we required the first four items to come from each of the four PTSD domains, after which the program selected subsequent items from those remaining in the item bank. To optimize item selection, we selected items with the first-, second-, and third-highest determinants of the posteriori information at the respondent’s score level (32). If the second or third item’s general PTSD factor discrimination parameter was greater than 2.7 (median value of the general PTSD factor discrimination parameter), we randomly selected one of them; otherwise we selected the first item as the next administered.

To examine P-CAT score accuracy and precision, we generated the score and standard error (SE) estimates for simulated ten- and 20-item P-CATs and compared them through real-data simulation with those generated from the full item bank, a method used to reduce the length of a conventional test and determine the minimum number of items needed to achieve acceptable accuracy and precision (32). To evaluate accuracy, we calculated the correlation between scores generated by the P-CATs and those for the full item bank. Precision (reliability) was assessed by calculating the SEs across the range of scores for the P-CATs and for the full item bank. Reliability was defined as ρ=1–SE(θ)². Better accuracy and precision mean higher reliability and lower SEs.

Validation

Using the validation sample, we assessed concurrent validity by examining correlations between the P-CAT, the M-PTSD, and the PCL-C. We assessed discriminant validity by examining differences in P-CAT scores on the basis of clinically diagnosed and treated PTSD, a SCID PTSD diagnosis, and a positive PTSD screen. Finally, we examined sensitivity and specificity of the P-CAT against the SCID PTSD diagnosis by using receiver operating characteristics (ROCs).

Results

Sample Characteristics

Both the calibration and validation samples were predominantly white (76% and 72%, respectively) and male (78% and 79%) (Table 1). Most had graduated from high school, and about a third had graduated from college (31% for the calibration sample and 29% for the validation sample). In the calibration sample, 22% reported a PTSD diagnosis, and 37% reported another mental health condition; 60% reported no psychiatric diagnosis. Reported PTSD and other mental health conditions were substantially higher in the validation sample (51% and 67%, respectively).

Factor Analysis and Item Calibration

A bifactor model (33,34) with one general PTSD factor and four subfactors, consistent with the four DSM-5 (35) domains (reexperiencing, avoidance, negative mood-cognitions, and arousal), yielded much better fit than a unidimensional model (comparative fit index=.958, Tucker-Lewis index=.956, root mean square error of approximation=.058; χ²=4,821.30, df=89, p<.001, for the likelihood ratio test between models). All items fitted the model with z greater than −1.645.

P-CAT Scoring and CAT Simulations

P-CAT scores were standardized (mean=0 and SD=1). To ease interpretation, we converted the z scores to T scores, with a mean of 50 and an SD of 10. Higher scores indicate greater PTSD symptom severity. The score correlations of the full item bank (89 items), with simulated ten- and 20-item CATs were .94 and .98, respectively. Figure 1 presents the score SE estimate distributions, which show that P-CAT precision was higher at moderate and severe PTSD symptom levels; precision generally decreased at mild or minimal PTSD symptom levels. Regarding score reliability >.90, the 20-item CAT worked about as well as the full item bank (36). When the score exceeded the mean of 50, the ten-item CAT reliability was >.90, indicating that the ten-item worked better in cases of moderate or severe PTSD symptom severity.

Concurrent and Discriminant Validity

The Pearson correlation of the P-CAT with the PCL-C was r=.88, and with the M-PTSD, it was r=.85, indicating strong concurrent validity. P-CAT scores also showed highly significant differences among the three clinical groups in the validation sample (PTSD, other mental health condition, and no mental health condition) (Table 2), with Cohen’s d effect sizes >.90. Differences were also significant between participants who did or did not meet SCID criteria for PTSD and between those who did or did not screen positive for PTSD (Table 3).

Sensitivity and Specificity

The ROC analysis indicated that the area under the curve (which quantifies the P-CAT’s ability to discriminate between participants meeting SCID criteria for PTSD from those without PTSD) was .88. A cutoff score of 58 provided sensitivity of 82% and specificity of 80%. For the PCL, a cutoff score ≥50 yielded sensitivity of .70 and specificity of .87. For the PC-PTSD screener, a cutoff score ≥3 yielded sensitivity of .90 and specificity of .71.

Example of P-CAT Administration

The first P-CAT item is “I felt upset when I was reminded of the trauma.” Table 4 presents response profiles, P-CAT scores, and SEs for two individuals—one with low and one with high PTSD symptom severity. As shown in Table 4, different responses to the same item resulted in markedly different P-CAT scores and different subsequent items.

Discussion

Accurate assessment of mental health is critical for referring patients to appropriate services and monitoring outcomes. CATs can facilitate this process and reduce the burden of assessment on patients and health systems. The P-CAT adds another psychiatric condition for which a CAT is now available, in addition to depression and anxiety. Although PTSD is not as prevalent in the general population as depression or anxiety, PTSD can be highly disabling and expensive to treat (37–39). Thus a briefer, more efficient, and cost-effective measure of PTSD can be valuable to patients, clinicians, and researchers. Average completion time for the P-CAT was 116 seconds, with 89% of participants completing it in less than three minutes (data not shown). This completion time is similar to that reported for a seven-item screener and less than the five to ten minutes for the PCL-C and other instruments of similar length (40).

The P-CAT showed strong concurrent and discriminant validity, and sensitivity (.82) and specificity (.80) were comparable with those of other measures based on the same cut points (sensitivity ranging from .70 to .78 and specificity from .85 to .92 for the PC-PTSD screen, and sensitivity from .21 to .82 and specificity from .84 to .99 for the PCL-C) (14,41). The fact that a simulated ten-item P-CAT was not as reliable at low levels of PTSD suggests that some patients who do not have PTSD may test as false positives. Further clinical evaluation would be needed to make this determination. Test-retest reliability, predictive validity, and responsiveness to change were not assessed, which is a limitation of this study.

The stopping rule used for the P-CAT was 12 items to allow scoring of the four PTSD domains and an overall score. However, the two sample administrations shown in Table 4 indicate that for low PTSD symptom severity, only eight items were needed to reach an SE of .30 (which implies reliability >.90 [42]); for high PTSD symptom severity, only six items were needed to reach this SE. Thus it is likely that the P-CAT could be substantially shorter than 12 items without significant loss of precision while still allowing for assessment of the four domains. However, even the 12-item P-CAT is shorter, less burdensome, more individualized, and more efficiently scored and processed than the 35-item M-PTSD or the new 20-item PCL-5 (43). A P-CAT shorter than four items would not allow for reliable domain scores.

Outcome evaluation of evidence-based treatments may be possible with the P-CAT, although further work to assess the instrument’s responsiveness to change is needed. Two evidence-based psychotherapies for PTSD, cognitive processing (44) and prolonged exposure (45), require weekly monitoring of PTSD symptoms. The P-CAT allows items to vary from one administration to the next, which can reduce both practice and memory effects that may occur with paper-and-pencil measures, but does not allow for monitoring change in the same items over time, which may be a limitation of adaptive testing. However, even with varying items, the P-CAT provides scores for all four PTSD symptom domains, allowing assessment of change in each domain. Further testing of the P-CAT at multiple time points is needed to assess responsiveness to change, which has been shown for other mental health CATs used in clinical settings (46,47).

The P-CAT was developed among U.S. military veterans, for whom trauma is likely related to military service, although other types of trauma are also possible. Among civilians, military service is not a source of trauma, and the proportion of women is much higher in the civilian population. Consequently, future work should test the P-CAT in broader samples to understand how it can best be used in clinical practice and research. Calibration of items in other samples (for example, women and college students) may identify different items with high information value.

This study excluded veterans at high risk of suicide because of clinical concerns about potential symptom exacerbation; however, <2% of mental health service users were excluded. The study included veterans with comorbid diagnoses (for example, depression and substance abuse), which commonly co-occur with PTSD.

During the course of this study, DSM-5 (43) replaced DSM-IV, and each has different diagnostic criteria for PTSD. With DSM-5, criterion A1, exposure to trauma, now must be a more direct experience; criterion A2 (reaction with intense fear, helplessness, or horror) is eliminated; and symptoms have been added to cover revamped symptom clusters. Recent research suggests that the prevalence of PTSD among nonveterans is somewhat lower when measured by DSM-5 than when measured by DSM-IV (4.2% and 5.1%, respectively), but prevalence rates are almost identical among veterans with high PTSD prevalence (38.8% when measured by DSM-5, and 39.9% by DSM-IV) (48,49). Because revised diagnostic criteria may alter the comparison groups used to validate the P-CAT, future work is warranted to test our results.

Conclusions

The P-CAT adds to the growing collection of CATs available for screening and symptom monitoring of general medical and mental health conditions. In 2008, the question was raised, “Are we ready for computerized adaptive testing?” (3). Concerns were expressed in regard to technology and infrastructure limitations in health care facilities. Now, eight years later, these technologies have proliferated, including tablet computers and smart phones. Many health care systems offer patients secure Internet portals for communication of sensitive health information (including VA through its electronic patient portal, MyHealtheVet). The P-CAT can be installed on any of these media, thus enabling implementation in many mental health care settings.