Best Practices: The Feasibility of Standardized Patient Assessments as a Best Practice in an Academic Training Program

In the United States, a tremendous gap exists between high-quality care and current practices. To address this “quality chasm” in psychiatry, the Institute of Medicine recommended that clinicians use standardized patient assessments (SPAs) to assess outcomes of treatment and to continuously improve quality of care (1).

There is growing evidence that the use of depression severity measures can influence treatment decisions (2) and improve patient outcomes (3). Detection of various psychiatric conditions is increased with structured interviews (4), and use of measures can reduce psychiatric inpatient days and service costs (5). Use of SPAs can also facilitate patient participation, which is associated with increased satisfaction and treatment adherence (6). Furthermore, partnering with patients to monitor symptoms allows for shared decision making (7) and is consistent with a patient-centered care model.

Validated SPAs exist for a wide variety of psychiatric conditions to assist with diagnosis and monitor symptoms. These measures are widely used in research, yet few psychiatrists routinely use them in clinical practice (6,8,9). To facilitate uptake of SPAs within our institution, we felt that it was important to first understand local practice patterns, resident and faculty attitudes, and perceived barriers to incorporating SPAs into clinical practice. We recognize the limitations of self-report surveys; for example, faculty may overestimate their use of SPAs if they perceive that using SPAs is the standard of care, and supervisors who are more interested in SPAs may be more likely to respond to a survey on this topic. However, such methods can be valuable when exploring potential barriers and facilitators to implementing evidence-based practices.

Toward this end, we sent all second-, third-, and fourth-year residents (N=33) and faculty supervisors (N=170) at the New York State Psychiatric Institute (NYSPI) and the Department of Psychiatry at Columbia University a confidential, Web-based survey regarding their opinions and use of SPAs. The survey was conducted between September and December 2009, with multiple e-mail reminders to residents and an additional reminder to supervisors after two weeks. Residents and faculty were informed that participation was part of a research study approved by the NYSPI Institutional Review Board.

Thirty-two residents (97%) and 67 faculty members (39%) responded to the survey. Because some faculty responses contained only demographic information, we limited the analysis to the 54 faculty members who responded to questions about their personal use of and opinions about SPAs. [Information about faculty demographic and other characteristics is available in an online data supplement to this column.]

SPAs in supervision and clinical practice

Residents reported whether their supervisors encouraged them to use SPAs to screen or monitor patients for affective, anxiety, psychotic, and cognitive (for example, dementia) disorders or chemical abuse and dependency. Faculty supervisors indicated whether they used such assessments in their own clinical work. Twenty-two residents (69%) reported that they were encouraged to routinely use SPAs as screening tools for at least one disorder. The proportion of residents reporting supervisor encouragement ranged between 9% for chemical abuse and dependency to 53% for affective disorders. Half of the residents (N=17, 53%) reported that supervisors encouraged them to routinely use SPAs to monitor treatment response for at least one disorder and to watch for relapse. The proportion of residents reporting supervisor encouragement ranged from 3% for chemical abuse and dependency to 47% for affective disorders.

Fifty-seven percent of supervisors reported that they used SPAs routinely for patient screening for at least one disorder; responses ranged from 6% for psychotic disorders to 38% for affective disorders. Twenty-eight percent of supervisors reported routinely using SPAs to monitor changes in patient symptoms, and 78% reported encouraging a clinical trainee to use a SPA to screen, diagnose, or monitor his or her patients. [Figures summarizing resident and faculty responses in regard to each type of disorder are included in the online data supplement. Note that variations in rates across disorders may reflect the predominant patient population on the resident and faculty caseloads as opposed to particular biases in screening and monitoring patterns.]

Barriers to using SPAs

Among a list of ten potential options, residents and supervisors ranked the top four barriers to using SPAs in clinical practice. Participants could specify additional barriers not included in the list. Residents identified limited formal training as the top barrier to using SPAs (31%), along with lack of time (19%), limited access to SPAs (16%), and lack of consensus as to which instrument to use (9%). Faculty members most frequently cited the fact that completion of SPAs is not a requirement in their clinical work as their top barrier (25%); they also cited limited formal training (17%) and the belief that administering SPAs would be too time consuming (11%). Among the top four barriers, nine faculty members (17%) added under “other” that they did not think that structured assessments were useful or offered an advantage over their own clinical assessments (a comment also added by one resident). [A table summarizing resident and faculty responses is included in the online data supplement.]

Attitudes toward SPAs

Residents and faculty rated 12 statements about their attitudes toward SPAs on a 5-point Likert scale, ranging from strongly agree (coded 2) to strongly disagree (coded –2). We created a mean score across the 12 items. Agreement with most items indicated a positive opinion, and we reverse-coded items that reflected a negative opinion. On the basis of mean scores, residents and faculty generally agreed that SPAs could help them monitor a patient’s symptoms, formulate a diagnosis or treatment plan, communicate with a patient about his or her symptoms, and engage patients in monitoring their own symptoms. Residents and faculty were fairly neutral on whether SPAs would put a strain on their overall workload or whether SPAs would help them use their time more efficiently (responses trended toward more negative attitudes). They agreed that administering the Structured Clinical Interview for DSM-IV (but not brief SPAs) would be too time consuming. Residents and faculty tended to disagree that SPAs could inhibit building rapport with a patient and detract from the therapeutic relationship. The overall mean attitude score for both residents and faculty was positive. [A figure summarizing the responses is included in the online data supplement.]

We applied independent t tests to compare faculty and resident responses to attitude statements. We accounted for multiple comparisons with a Bonferroni correction, using a p value of ≤.0038 to identify significant differences. Residents were less likely than faculty members to agree that a thorough clinical assessment is as good as or better than a SPA (mean scores of .06 and .77; p<.001). There was an insignificant trend for residents to be more optimistic than faculty members about the ability of SPAs to provide meaningful information about functional outcomes in their patients (mean scores of .81 and .55; p=.048) and to help them remember their patients’ exact symptoms and the severity of those symptoms (mean scores of .84 and .45; p=.024).

Factors associated with faculty attitudes and use of SPAs

We applied logistic regression to examine the relationship between mean faculty scores on attitudes, specific characteristics (age, other advanced degrees, appointment status at an academic medical center, and prior experience in clinical research), and self-reported use of SPAs for patient screening and monitoring. Older faculty members were less likely than younger faculty members to report using SPAs to screen patients (odds ratio [OR]=.84, 95% confidence interval [CI]=.75–.94, p=.003) or to monitor symptoms (OR=.84, CI=.73–.96, p=.016) for any clinical disorder. Older faculty members were also more likely to have a negative attitude toward SPAs (OR=.93, CI=.88–.99, p=.009). Faculty who had been part of a research team conducting a clinical study that included systematically tracking patients’ symptoms over time were more likely than those who had not had this experience to report routine use of SPAs for patient screening for at least one disorder (OR=10.08, CI=1.51–67.08, p=.017) but not for patient monitoring. [A table presenting the results of this analysis is available in the online data supplement.]

SPAs as an emerging best practice

The academic medical center, with its culture of scientific advancement and synergy of researchers, clinicians, and students, should be uniquely poised to promote the highest-quality clinical care. Unfortunately this is not always the case. In this study, many faculty supervisors (72%) reported that they did not routinely use SPAs to monitor patients over time. In addition, a substantial proportion of residents (47%) reported that they were not encouraged to use SPAs for routine monitoring of outcomes. However, residents and faculty were generally optimistic regarding the potential value of SPAs in clinical practice, suggesting that practical barriers may impede the use of SPAs.

Consistent with prior findings (8), both residents and faculty endorsed a lack of formal training as one of the top barriers. Residents also indicated that they had limited access to scales and were unsure which scales to use, suggesting that a standard measurement tool kit for clinicians would be helpful (10). Given concerns regarding time constraints, it would be important to educate faculty and residents on how to integrate SPAs into clinical practice. For example, patients can complete self-report SPAs before appointments, allowing time with their doctor to be spent more efficiently. Interventions geared toward early hands-on experience may be particularly influential. Data from this study seem to support this hypothesis, because faculty with experience using SPAs in research studies were more likely to report using them in clinical practice.

In addition to addressing practical barriers, educational efforts should address physician attitudes. Although SPAs are not intended to replace a clinical evaluation, some faculty questioned their “added value.” In another study, psychiatrists questioned the validity, reliability, and sensitivity of SPAs to detect change, despite the fact that virtually all clinical research is based on such tools (9). Others have suggested that physicians are offended at the thought of using a checklist and believe that the complex practice of medicine requires a kind of “expert audacity,” even though data suggest that checklists can directly improve outcomes (11), engage patients as collaborators, and promote patient-centered care (6).

Education initiatives to promote the use of SPAs must be linked to policy-level changes at both an institutional and a national level that support the training, development, and use of SPAs in clinical care. As expressed by the faculty in this study, completing SPAs is currently not required as part of clinical work. Until SPAs are supported systemwide and practicing psychiatrists find that the potential gains outweigh the effort required, they will remain unpopular.

Policy changes at the national level are already under way, with links between payment for services and performance reporting on the horizon. At an institutional level, we have an electronic medical record with access to SPAs. We are also employing quality improvement approaches to improve the use of SPAs within the resident outpatient clinic and resident supervision. In addition, departmental leadership has invested in developing an infrastructure to facilitate the use of SPAs in clinical practice. For example, the day treatment program in the Columbia Faculty Practice is rolling out a Web-based collection system for self-administered SPAs. Through online modules, all patients will complete SPAs as part of an initial assessment and ongoing monitoring. As others have reported, these types of programs can be powerful tools in facilitating shared decision making (7).

Conclusions

Although residents and faculty were generally optimistic regarding the potential value of SPAs, use of SPAs does not currently represent the standard of care. Residency programs in psychiatry need to develop comprehensive training programs on SPAs for both residents and faculty. Understanding local attitudes and barriers can be an important first step toward instituting change. Although residents may be exposed to SPAs in traditional didactic courses, a vast majority of their learning occurs in clinical settings. As described by Agrawal and colleagues (12), new evidence-based practices such as SPAs must be “adopted as an expectation of clinical care, modeled and valued by supervisors and regularly evaluated” if residents and faculty are to incorporate them into their clinical practice.