Patient-Clinician Agreement on Treatment Type and Helpfulness: Results From a WTC Rescue and Recovery Worker Cohort

In the years since the destruction of the World Trade Center (WTC) on September 11, 2001, it has become clear that the estimated 50,000 or more individuals involved in rescue and recovery at the WTC site were exposed to increased risk of mental illness, such as major depressive disorder, posttraumatic stress disorder (PTSD), panic disorder (1), other anxiety disorders, and possibly substance use disorders (2).

Several studies have investigated demographic and exposure variables and their roles in mental illness in the rescue and recovery population (3). We were not able to identify any studies investigating rescue and recovery workers’ understanding of or attitudes toward mental health treatment. Yet knowledge of and beliefs about mental illness have been shown to have an impact on treatment and interactions with clinicians for a variety of conditions (4,5). The relationship between patients’ beliefs about the nature or appropriate treatment of mental illness and their response to the treatments recommended by their clinicians is complex, but, in general, agreement between patients and their providers has been shown to have a positive impact on treatment outcome (6). Investigators have demonstrated that beliefs expressed about treatment and illness etiology can correlate with treatment preference (7) and that agreement between patients and providers about the nature of their primary problem can correlate with some measures of clinical improvement (8). Others have shown that insight into depression (9) and explanatory models about PTSD (10) have an impact on treatment adherence.

The aim of this study was to determine rescue and recovery workers’ and clinicians’ appraisal of the nature and helpfulness of the care provided. We hypothesized that survey responses would reflect better patient-clinician agreement for medication management than for various forms of psychotherapy. We further hypothesized that agreement between patient and clinician would be correlated with greater reported treatment benefit.

Methods

The WTC Volunteer Mental Health Treatment Program (MHTP) at Mount Sinai has operated since 2003 and is a clinical service that treats patients with mental health problems associated with their work in their rescue and recovery efforts at and around the WTC. Study participants were drawn from the approximately 700 patients served by the MHTP. Individuals were eligible for inclusion if they had been admitted to the MHTP at least three months before the time of survey administration. The Mount Sinai School of Medicine Institutional Review Board (IRB) approved the study protocol and waived the need for separate consent forms.

To gather information about the types of treatment provided to study participants, rescue workers’ awareness and perception of treatment, and clinicians’ assessments, we performed a search for previously validated survey instruments. Although survey instruments have been created to assess prevalence of trauma-associated diagnoses (3) and to assess patients’ ability to understand, appreciate, and logically reason about relevant health information in expressing a choice of treatment (11), we did not find an instrument designed specifically to assess patients’ or clinicians’ knowledge of or attitudes toward treatment in a trauma-exposed population. Therefore, the study team devised a pair of surveys—one for patients and one for clinicians. Surveys were translated into both Spanish and Polish and then back-translated to confirm the accuracy of the translation.

Patients were recruited sequentially from the waiting room of the clinic during MHTP operating hours between January 4, 2010, and March 12, 2010, and between April 26, 2010, and May 7, 2010. For survey administration, MHTP reception staff members who were fluent in the primary language of each patient provided a brief letter describing the survey before the patient’s treatment appointment. A research coordinator then approached consenting patients and asked for the name of their MHTP mental health clinician (some patients had more than one MHTP clinician). Each identified MHTP clinician was given a survey.

Given the high profile of the WTC-related clinical population, IRB restrictions required a high degree of care in maintaining confidentiality. To limit potential breaches of participant confidentiality, the clinicians reported demographic information about their patients so that protected health information would not be available to nonclinical research staff. The patient and clinician surveys were otherwise identical.

Both surveys asked respondents to report all treatments currently being provided from a list of 13 treatments, with two additional options —“other” or “I don’t know.” The surveys then asked clinicians and patients to report treatment helpfulness across six domains: improving family relationships, finding or keeping a job, feeling better about one’s job, gaining a positive outlook toward the future, feeling better about oneself, and reducing symptoms. We divided the six domains into two clusters by using exploratory factor analysis with varimax rotation: the first three were assigned to the cluster “external benefits,” and the others to the cluster “internal benefits.” For each domain, patients and clinicians were asked to choose “not applicable” or rate their perceived helpfulness on a 4-point Likert scale: very helpful, somewhat helpful, not very helpful, or unhelpful. Responses were then converted to a numerical score from 1 to 4 for analysis, with 1 corresponding to very helpful and 4 to unhelpful.

Because there is no true reference or gold standard to compare two reports, we calculated agreement using the kappa statistic (12) and using sensitivity and specificity analyses. In calculating sensitivity and specificity, we chose to prioritize the clinician’s report as the reference standard because clinicians make decisions about treatment types and had access to patient records when completing the survey. We considered a treatment to be reported if any of a patient’s clinicians reported it.

For ratings of treatment helpfulness, we calculated mean scores and used Wilcoxon’s signed-rank test for determining the statistical significance of the difference between patient and clinician mean ratings. The Mann-Whitney U test was performed to test the null hypotheses that internal and external benefit scores are independent of agreement status by reported treatment type. For these calculations, we again defined a treatment to be reported by the clinician if any clinician reported it. If the patient and the clinician agreed that a treatment was not given, then there would be no reason to expect that it would be associated with corresponding differences in benefit in the same direction as would be evident if they had agreed that a treatment was given. Therefore, we looked only at cases in which a clinician reported that the treatment was provided. Because of the small sample size for each item for individual psychotherapy, we also tested a combined individual psychotherapy item that merged the cognitive behavioral, exposure, supportive, exploratory, and psychoeducation items. We also used Student’s t test to determine whether mean scores for reported benefit differed between patients with a single provider and those with multiple providers.

Results

A total of 234 patients received surveys and identified their treatment providers. Of these, patients, 196 (83%) returned completed surveys. Nine surveys were excluded: five patients were identified by their providers as having less than three months in the clinic, and four surveys were identified by providers as duplicates. Of the 187 included patient surveys, 89 were in English, 77 in Spanish, and 21 in Polish. Seventy-two percent of patient respondents (N=135) were male. The mean±SD age of patient respondents was 50.3±8.1 years, 52% (N=98) were married, 18% (N=33) were single, 21% (N=40) were divorced, 4% (N=7) were widowed, and for 5% (N=9) marital status was not reported. Only 21% (N=39) were currently employed; 41% (N=76) were on sick leave or receiving disability benefits, 20% (N=37) were unemployed or had been laid off, 6% (N=11) were retired, and for 13% (N=24) no employment data were reported. A total of 109 respondents included information about occupation as follows: 23% (N=25), “other”; 13% (N=14), asbestos handler; 13% (N=14), cleaning-maintenance, 13% (N=14), technical and utilities; 10% (N=11), law enforcement; 9% (N=10), construction; 6% (N=7), civilian public sector; 6% (N=6), office, administrative, or professional; 5% (N=5), transportation; 2% (N=2), health care; and 1% (N=1), firefighter.

Fifty-eight percent (N=135) of the 234 recruited patients identified one clinician as providing treatment, 40% (N=94) identified two, and 2% (N=5) identified three. In all, 338 clinician surveys were distributed. For those with more than one clinician, the agreement between clinicians about which diagnosis was primary was low (κ<.22). However, the diagnoses reported for the 234 recruited patients were as follows (includes nonprimary diagnoses): 52% (N=122), PTSD; 45% (N=106), major depressive disorder; <1% (N=1), alcohol abuse or dependence; and 20% (N=46), “other.” Patients were reported to have been in treatment for a mean±SD of 28.8±18.5 months.

Of the 338 clinicians given a survey, 48 did not return it. Of the surveys returned, four were excluded because the providers identified them as duplicates, and six more (corresponding to five unique patients) were excluded because the patients were reported to have fewer than three months in the clinic. Most clinicians identified multiple treatment modalities. Only 19% (N=54) of the 280 included clinician surveys clinicians identified a single modality. Less than 1% (N=1) did not respond to the item; 18% (N=50) identified two modalities, and 63% (N=175) identified three or more. Results of kappa and sensitivity and specificity analyses are reported in Table 1.

Regarding the helpfulness of treatment, mean scores for both patients and clinicians ranged between somewhat and very helpful for all items except clinicians’ ratings on finding or keeping a job, which fell between somewhat helpful and not very helpful. Clinicians’ ratings of treatment helpfulness were higher than patients’ ratings to a small but statistically significant degree. [A table presenting mean helpfulness scores is available in an online data supplement to this brief report.] The Cronbach's alpha for the external benefit cluster was .87, and for the internal cluster it was .84.

Three items achieved a statistically significant rejection of the null hypothesis (that the reported benefit from treatment and agreement between clinician and patient are independent): medication management with internal benefit (p=.050), the combined individual psychotherapy item with internal benefit (p=.037), and workers’ compensation evaluation with external benefit (p=.050). Having multiple providers was also associated with a lower mean score for internal benefit (p=.02), indicating that patients with multiple providers rated their treatment as more helpful for the items clustered into internal benefit (gaining a positive outlook toward the future, feeling better about yourself, and reducing symptoms), compared with patients who identified only a single provider.

Discussion

Kappa scores were modest, and sensitivities and specificities varied in our sample. However, a cluster of treatment modalities stood out as being somewhat more strongly related to patient-clinician agreement: medication management and eye movement desensitization and reprocessing. In addition, group psychotherapy had both reasonable sensitivity and specificity. Group therapy can be defined simply on the basis of its taking place in a group rather than defining it on the basis of a particular theoretical school. In contrast, patients may have more difficulty than clinicians differentiating between styles of individual psychotherapy. Two-thirds of the clinicians surveyed (67%) reported providing two or more treatment modalities to a patient. A recent survey of 2,200 North American psychologists found that most endorsed using multiple theoretical orientations in practice (13). There is evidence that adoption of an eclectic-integrative approach may be necessary for psychotherapy to be effective in a culturally diverse population (14) and that stylistic flexibility leads to a stronger therapeutic alliance when there is cultural dissimilarity between therapist and patient (15), although this requires further study.

The relationships between patient-clinician agreement and reported treatment benefit are consistent with our hypotheses and congruent with prior studies finding that agreement between study participants and clinicians about treatment selection is associated with better outcomes. Our results extend this literature in that they apply to ethnically and linguistically diverse populations exposed to disaster-related trauma and to agreement regarding ongoing treatment in a naturalistic, clinical setting. No single psychotherapy item was significantly correlated with a difference in reported outcomes, although these analyses were limited by sample size. Previous studies examining one or two therapy modalities with larger samples have supported such an association (6–10). Our combined individual psychotherapy item was correlated with internal benefit.

The study had several limitations. The primary language of each patient was known; however, we could determine patients’ race-ethnicity only by clinicians’ report because of the need to preserve patient confidentiality. Also, we were unable to identify a suitable validated survey, and we did not have access to direct measures of clinical outcome. Therefore, survey design cannot be ruled out as a factor in our results. However, the strong clustering of results by external benefit and internal benefit and the concordance with results of previous studies in other populations support the reliability of the findings. Third, although our overall sample was fairly large, the number receiving a particular treatment modality were small, thereby limiting the power and precision to estimate agreement and benefits from those modalities.

Conclusions

Our results have implications for investigators and clinicians treating diverse populations of first responders to disaster. The results add to the body of literature on the relationship between treatment beliefs and outcomes and generally support previous findings that patient-clinician agreement about treatment is associated with improved outcomes. The findings also highlight the need for better understanding of the eclectic therapies offered in real-world clinical practice.