A Multimethod Study of Mental Health Care Patients’ Attitudes Toward Clinician-Level Performance Information

Research shows statistically significant and clinically meaningful differences among mental health care (MHC) clinicians in their patients’ outcomes (1). As assessed by standardized, patient-reported outcome (PRO) measures, some clinicians are consistently more than twice as effective as others, even after controlling for case-mix variables that influence outcomes (2). Clinician-level effectiveness also appears to be stable; clinicians who achieve better than average outcomes generally continue to achieve better than average outcomes with subsequent patient cohorts (3).

It is important for MHC stakeholders to consider how to make outcome information actionable to improve care quality. Implications of outcome variability already affect MHC and reimbursement decisions. The Centers for Medicare and Medicaid Services have instituted data-informed, value-based payment programs, recently extending requirements to psychologists. In addition to requiring collection and report of quality data, the payment program has established reimbursement levels on the basis of a clinic’s or individual provider’s performance relative to predetermined benchmarks (4).

The Institute of Medicine (IOM) (5) recommends that patients be granted access to clinician performance data to guide treatment decisions. However, patient perspectives on clinician performance track records remain conspicuously absent in MHC discussions. For example, do patients view this information as trustworthy and useful? Care decisions and preferences are multidimensional and complex (6). Ostensibly, helping patients become informed and empowered in their decision making is a laudable goal; however, it is conceivable that patients may struggle to interpret and use clinical-level performance information. Moreover, patient preferences for how to use clinician outcomes data (e.g., direct to consumer, through third-party referral sources) have received little attention.

This exploratory, multimethod study examined MHC patients’ attitudes toward, valuation of, and potential concerns about clinician selection and use of track record information about clinician outcomes in treatment decision making. For this study, a team of MHC stakeholders developed a quantitative patient survey and a semistructured interview protocol. Such efforts can inform the development and implementation of measurement-based care strategies to improve MHC decision making and outcomes.

Methods

All procedures were approved by a university institutional review board. Survey recruitment was based on rolling, voluntary self-selection between November 9, 2015, and November 1, 2016. Survey completers could then volunteer to participate in an interview to gather more in-depth and personally relevant information and recommendations. All participants were eligible for gift card compensation.

Survey participants were patients receiving treatment at one of 12 community mental health centers in the northeast United States (N=403). Inclusion criteria were any individual who was seeking or receiving MHC services and was responsible for MHC decision making for him- or herself, a family member, or an important other who was unable (e.g., because of cognitive impairments) to participate on his or her own behalf.

The study survey was developed in collaboration with a stakeholder advisory board. (A description of the stakeholder advisory board and the survey items are available as an online supplement to this report.) The survey assessed participants’ demographic information, MHC service history and provider experience, and attitudes and preferences regarding provider performance information, both in general (yes/no items) and in comparison with alternative treatment factors (Kuder Richardson-20 internal consistency = 0.74).

Through convenience sampling, 37 participants volunteered to complete follow-up interviews via conference call. Recorded interviews were transcribed. Using standards for consensual qualitative research (CQR) (7), we randomly selected 15 interviews for analysis. The final set of questions (see online supplement) covered the following domains: experiences finding an MHC provider, information that would have been helpful in making a more informed provider selection, experiences with PRO measures, interest in and reasons for (or against) accessing clinician outcome information, factors that are important when finding a provider, willingness to use and comfort level using a list of “matched” providers based on PRO information, and perceived costs or pitfalls of using provider track record information. Interview transcripts and raw audio recordings were analyzed with CQR (7) methods (see online supplement).

Results

The survey sample was mostly female (N=268, 67%) and White (N=206, 51%); the mean±SD age was 41.20±12.58 years. (A table with additional sample characteristics can be found in the online supplement.) The most common problem areas were anxiety, depression, and trauma. The most common current treatment was individual psychotherapy, followed by medication, group psychotherapy, and community-based support. The most common referral source was self, followed by another MHC provider and primary care physician. (A table with additional treatment and referral details is available in the online supplement.)

Survey results, as a percentage of “yes” endorsements for each yes/no item, are reported in Table 1. A majority of participants endorsed experiencing difficulty finding an MHC provider in their lifetime; a minority endorsed receiving a recommendation to see a specific MHC provider. When a specific referral had been provided, little explanation was given regarding its basis. Overall, participants endorsed struggling to navigate the clinician selection process and feeling as though they had received little systematic guidance. Additionally, approximately 21% of participants endorsed the belief that all MHC providers are capable of helping.

Remaining survey questions posed the hypothetical scenario of patients being able to access clinician track record information in the context of PRO monitoring. A majority of participants endorsed interest in accessing clinician performance track record information and the belief that such information would lead to more effective referrals and care outcomes. Additionally, over 90% endorsed the belief that matching informed by providers’ track records would increase the likelihood of being helped.

Interview coding domains and tables showing complete results are available in the online supplement, and themes are summarized here. Regarding experience with MHC, several participants remarked about struggling with clinician turnover, which retriggered the difficult process of finding a provider. Several patients reported experiencing limited clinician choice and becoming savvy in advocating for themselves as patients only after years in the MHC system. Regarding experience with PROs, most participants reported completing self-report evaluations of some kind; however, few were able to articulate the purpose of completing such evaluations. Concerning the use of provider performance information, participants expressed interest in accessing a list of well-matched providers. Several noted that they would be willing to remain on a waitlist for a longer period to see an empirically better-matched provider.

Regarding potential pitfalls of using clinician performance track records, participants expressed concern about placing too much weight on them in decision making, because patient perspectives of patient outcomes (which drive the track records) are subjective. Concerning preferences, a majority of participants identified interpersonal and relational factors as being similarly important (in relation to outcome track records) when finding a good-fitting clinician. Access to track record information was perceived as valuable, yet relational fit remained important. Participants reported that outcome track records would be valued and used to some degree, if made available. Ideally, other preference-related information would also be taken into consideration (e.g., being assigned to a clinician with a track record of positive outcomes who also matches a gender preference).

Discussion

Clinicians are an important source of outcome variability in MHC. Agencies and policy makers have advocated for quality monitoring and the use of quality data in MHC stakeholder decision making. Little is known about patients’ attitudes regarding the use of clinician outcome track records in decision making (8). These results demonstrate that patients are given little information or direction regarding clinician selection. When patients engage in MHC, unsystematic and nonpersonalized referrals may result in a poor patient-provider match and poor outcomes (9). Clinician track record information would be a valued resource in the decision-making process. Few participants endorsed the belief that all MHC providers are capable of helping. This finding may indicate that most participants do not view care providers as interchangeable or that patients lack faith in MHC in general.

Consistent with IOM recommendations, results indicate that access to and use of clinician outcome track record information would be a patient-centered MHC practice. This practice does not necessarily require directly providing track record information to patients; rather, such information could be provided to referral sources, such as primary care, to guide decisions. Many people with psychiatric problems have them treated solely in primary care (10). Yet, primary care providers often feel underequipped to meet patients’ MHC needs (11), reinforcing the importance of effective referral from primary care to MHC. Regardless of indirect or direct access, a majority of patients endorsed the belief that such information would enhance the referral process and treatment outcomes.

Results from the interviews provide more nuance. Specifically, track record information based on PRO data should not be the sole factor in referral and assignment decisions. Rather, track record information should be integrated into a broader assessment of and responsiveness to care preferences when possible. Notably, patients were somewhat protective of clinicians with regard to the implications of track record information. Both the survey and interview distinguished between patient satisfaction ratings (e.g., number of stars out of five on a website) and outcome data based on psychometrically valid self-report measures of symptoms and functioning. Participants viewed satisfaction ratings and testimonials as somewhat useful yet also expressed skepticism regarding their validity. Some expressed concern about overreliance on outcome information for personnel selection and reimbursement policies; potential negative effects (e.g., inequitable reimbursement practices due to failure to account for contextual factors) of overreliance on PROs to inform health service decisions cannot be ignored (12). However, overall, PRO-derived performance information was perceived as more valid than satisfaction ratings. Rather than rely on clinicians’ self-identified competence domains and patient testimonials, patients may value the inclusion of outcome-based track record information in provider directory platforms.

The results of this study must be considered in light of several limitations. These results come from a single study of community patients from one region of the United States. Although survey items were developed on the basis of stakeholder input with an emphasis on content validity and internal consistency was adequate, structural and predictive validity were not established. In addition, both aspects of the study were based on self-selecting, volunteer samples. Estimates of sampling error could not be calculated; thus, results may be subject to multiple sources of error.

We also recognize that care decisions and provider selection are not solely within the control of the individual. Numerous factors may limit choice, and barriers often hinder access to any service. Additionally, the framing of this study and both the survey and interview questions implied somewhat hypothetical scenarios in which patients have more freedom to make choices about their care than they might have in real-world situations, although the results clearly indicate that patients would value being more empowered in their treatment decisions. Relatedly, our sample comprised participants with relatively fewer financial resources. Although we acknowledge the implications of this characteristic for generalizability, it could be argued that this subpopulation is in particular need of decision support tools to avoid perpetuating health disparities and further marginalization.

Conclusions

Historically, MHC stakeholders have been unaware of clinicians’ outcome track records, which represents a gap in knowledge transfer. Suboptimal improvement rates in MHC may partly be due to the provider and a lack of personalized referrals or assignments. Because such treatment decisions are often not personalized or are based on convenience or the provider’s self-defined expertise (which research shows is often overestimated or inaccurate) (13), patients are equally as likely to see a provider who is stably ineffective at treating their condition as they are to see a more effective provider. Conversely, there is potential in harnessing performance information to match patients to clinicians on the basis of scientific outcomes data. Overall, patients appear to be interested in such an approach. It will be important to replicate these findings and to continue to explore potential implementation strategies.