Association Between Quality Measures and Perceptions of Care Among Patients With Substance Use Disorders

Quality measures can be a valuable tool to assess and improve quality of care for substance use disorders, yet there are few validated substance use disorder quality measures (1–4). Furthermore, very few measures endorsed by the National Quality Forum (NQF) assess care for substance use disorders (4). Nonetheless, there is increased emphasis on assessing quality and using quality measures in new payment and incentive models, and greater access to substance use disorder treatment could follow from health care reform legislation. Together these developments may stimulate interest in validated substance use disorder quality measures. Recently, in fact, quality measures for substance use disorder treatment were highlighted as a key priority for new measure development (5–9).

When process-based behavioral health quality measures are developed and tested, they are typically evaluated in terms of their ability to predict clinical outcomes (4,10). Demonstrating this process-outcome link helps to ensure that resources targeted toward improving quality of care are also likely to result in improved patient outcomes. Yet, despite several important efforts, few process-based quality measures of substance use disorder treatment have demonstrated a process-outcome link (11–21). There remains a need for validated quality measures to assess care for substance use disorder.

An alternative approach to examining the association between process-based quality measures and clinical outcomes is to assess whether quality measures are associated with patient perceptions of the care they received, given that patient perceptions of care are also an important outcome. Measures of patient perceptions of care provide critical information for understanding whether care delivered is patient centered (22), a dimension of health care quality highlighted over a decade ago (23). It has been argued that delivering patient-centered care is an important goal, even independent of its relationship with health outcomes (24). Beyond simple patient satisfaction, patient perceptions of care include a variety of dimensions, including timely access to care, communication (for example, feeling respected and listened to by providers), shared decision making, and the helpfulness of treatment (25,26). Additionally, prior work suggests that perceptions of care may be related to treatment adherence and outcomes. Specifically, better patient care experiences are associated with higher levels of adherence to recommended prevention and treatment processes, better clinical outcomes, better patient safety within hospitals, and less health care utilization (27). Furthermore, data suggest that patients with substance use disorders may have lower perceptions of care compared with patients receiving care for psychiatric diagnoses (28,29). Taken together, improvement in patient perceptions of care may be an important outcome on its own.

There have been few attempts to evaluate quality measures in terms of their associations with patient perceptions of care, but preliminary work has been promising (30). To address the paucity of literature in this area, we sought to evaluate whether eight quality measures that assess care for patients with a substance use disorder were associated with patient perceptions of care, including perceived improvement and global rating of behavioral health care.

Methods

We conducted secondary data analyses of administrative and patient telephone survey data collected as part of a national evaluation of mental health and substance use services provided by the Veterans Health Administration (VHA) (31,32), which examined care for substance use disorders, schizophrenia, bipolar disorder, posttraumatic stress disorder (PTSD), and major depressive disorder. Administrative data were obtained from the VHA Medical SAS data sets and included demographic information, claims, diagnoses, dates and types of services, admissions, and discharges. The study population included patients who received care from or paid for by the VHA in fiscal year (FY) 2007 and who had at least one inpatient episode or two outpatient encounters of which at least one was for a primary or secondary diagnosis of schizophrenia, bipolar disorder, PTSD, or depression. Patients were assigned to one of the four psychiatric diagnostic cohorts based on the modal frequency of episodes of care, with ties resolved by using the following rank order: schizophrenia, bipolar, PTSD, and depression. In addition, patients with substance use disorder diagnoses were identified as a separate cohort. Patients with co-occurring substance use and psychiatric disorders were assigned to both the substance use disorder cohort and a psychiatric diagnostic cohort.

We drew a random sample, stratifying the population by geographic region, psychiatric diagnostic cohort, and substance use disorder cohort. Given concerns with recall bias in asking respondents to recall the care they received in FY 2007, survey questions focused on mental health care during the 12 months prior to the interview date. Therefore, we restricted the study population to patients with any VHA health care utilization in FY 2008 to minimize the number of survey respondents who would report having received no care from the VHA in the prior 12 months at the time they were contacted for the survey. A total of 6,190 patients participated in the telephone interview (response rate of 67% for all patients and 62% for patients in the substance use disorder cohort), which was fielded from November 2008 to August 2009.

These analyses focus on telephone survey respondents who had a substance use disorder diagnosis when selected for a cohort and had available patient perceptions data (N=2,074). Specifically, approximately one-third (N=736) of the sample had only a substance use disorder, and two-thirds (N=1,338) had a substance use disorder and a co-occurring psychiatric diagnosis (depression, bipolar, PTSD, or schizophrenia). Additional detail on the survey methods is available elsewhere (29,33). The RAND Human Subjects Protection Committee and institutional review boards of the Central Arkansas Veterans Healthcare Center and the University of Arkansas for Medical Sciences approved all procedures. Informed consent was obtained prior to the telephone interview.

Measures

Patient Characteristics

Age, gender, and disability service connection were drawn from administrative data, and race-ethnicity, marital status, education, employment status, income, mental and physical health functioning, and service utilization (inpatient nights and outpatient visits) were drawn from survey data. Functioning was assessed by using the Veterans RAND 12-Item Health Survey (34). The two composite scores reflecting mental and physical health functioning range from 0 to 100, with higher scores indicating better functioning. Both scores are norm-based and can be interpreted in relation to the distribution of scores in the 1990 adult U.S. population (mean±SD=50±10).

Process-of-Care Measures

We examined eight quality measures that are based on administrative data, including initiation and engagement in substance use disorder care (35), receipt of any psychotherapy or any psychosocial treatment, and follow-up after hospitalization. Four of these measures are endorsed by the NQF (initiation, engagement, and follow-up after hospitalization within seven days and 30 days). The remaining four measures were developed in a prior evaluation of VHA care (31). Some measures assess care received in the four months following initiation of a new treatment episode for a substance use disorder, and some assess care for the entire FY 2007 observation period.

A new treatment episode was defined as any inpatient hospitalization for or an outpatient visit with a substance use disorder diagnosis after a break in care of 150 days or more. A break in care was defined as no outpatient visits with a substance use disorder diagnosis or substance use disorder–related medication, which differs from the 60 days used in the NQF specifications for initiation and engagement, with the goal of increasing the likelihood of including only patients who were not in treatment. The psychotherapy measure was restricted to receiving various types of psychotherapy, whereas the psychosocial treatment indicator included both psychotherapy and other psychosocial interventions (mental health intensive case management, family psychoeducation, and supported employment). Detailed technical specifications of each quality measure are available elsewhere (32).

Outcome Measures

We assessed patients’ perceived improvement by computing a mean of four items from the Experience of Care and Health Outcomes (ECHO) survey (36,37), which assesses improvement in ability to deal with daily problems and social situations, ability to accomplish goals, and symptoms in the past 12 months. Respondents are asked to rate each target area on a 5-point scale, from 1 (much worse) to 5 (much better), compared with 12 months ago. Internal consistency reliability of these items was .88, providing support for computing a mean score. Respondents were required to have responses for at least three of the four items. We used person-mean imputation for 21 respondents who were missing one item. For patients who endorsed receiving care from the VHA in the past year (N=1,575), we assessed their overall perception of their VHA behavioral health care by using one item from the ECHO. Patients were asked to rate all their “counseling or treatment” in the past 12 months from 0 to 10, where 0 was the “worst counseling or treatment possible” and 10 was the “best counseling or treatment possible.” The format of this item maps to the Consumer Assessment of Healthcare Providers and Systems (CAHPS) global rating item, which has been implemented across a range of health care settings (25). Because responses to this item tend to be highly skewed, we analyzed this item as a dichotomous measure (ratings of 0–8 versus 9–10). The correlation between perceived improvement and global rating of care was .34, suggesting that these measures reflect related but still unique aspects of patient perceptions.

Statistical Analyses

We present demographic and other treatment characteristics and performance on each quality measure, weighted to reflect the population of patients who received care associated with a substance use disorder diagnosis. We then present results from multiple regression models of the association between perceived improvement and each quality measure and logistic regression models of the association between the global rating of care and each quality measure. These models adjusted for patient characteristics (age, race-ethnicity, gender, marital status, service connection, and rural/urban residence status) and mental and physical health functioning. Sampling weights were used to adjust for the stratified sampling design (38). A nonresponse weight reflecting the inverse probability of each sampled patient completing the survey was derived by using logistic regression of an indicator of survey completion on veteran characteristics (39). To reflect the population of patients who received care associated with a substance use disorder diagnosis, all estimates were weighted by a final analysis weight that was the product of the sampling and nonresponse weights. The sandwich estimator was used to obtain robust standard error estimates for regression coefficients (40). Given that multiple statistical tests were conducted, we accounted for multiple comparisons by holding the false discovery rate to 5% (41).

Results

Sample Characteristics

Weighted data on the demographic and treatment characteristics of the sample are provided in Table 1. The average age was approximately 54 years, over 90% were men, and about 60% were non-Hispanic white. A majority of patients completed at least some college (53%), were out of the labor force (68%), and reported annual incomes of $30,000 or less (74%). Only about a third of patients (37%) had a service-connected disability for any condition, meaning a mental or general medical condition that was incurred or aggravated during active duty. Respondents’ mental and physical functioning scores were approximately 1.0 and 1.5 standard deviations below the general population, respectively. Among patients with at least one outpatient visit or one inpatient night in the past year, patients reported a median of six VHA outpatient visits and 14 VHA inpatient nights, respectively.

Quality Measure Performance

The performance across the eight quality measures varied widely, ranging from 11% for treatment engagement to 82% for outpatient follow-up within 30 days after an inpatient discharge (Table 2).

Multivariate Models

Of the eight quality measures, four were significantly associated with perceived improvement (Table 3). Specifically, high performance on engagement in substance use disorder treatment, receiving any psychotherapy within four months of beginning a new treatment episode, receiving any psychotherapy, and receiving any psychosocial treatment were each positively associated with perceived improvement. An additional quality measure, receiving any psychosocial treatment within four months of beginning a new treatment episode, was significantly associated with perceived improvement, but only before the analyses were adjusted for multiple comparisons.

Of the eight quality measures, three measures were significantly associated with the global rating of care, although they varied from those that were significantly associated with perceived improvement (Table 3). Specifically, receiving any psychotherapy within four months of beginning a new treatment episode, receiving a follow-up outpatient visit within seven days of psychiatric inpatient discharge, and receiving a follow-up outpatient visit within 30 days of psychiatric inpatient discharge each were significantly associated with higher global ratings of care.

Discussion

Our results suggest that several process-based quality measures were associated with higher perceptions of care, including higher perceived improvement and higher global rating of care, among patients with substance use disorders, although the two perceptions-of-care measures were not associated with the same quality measures. This relationship suggests that both these process-based quality measures and patient perception of care are valid measures of quality. Global rating of care has been assessed more frequently across health care, including ratings of health plans, providers, and hospitals, given that it is included as an item in all CAHPS measures (25). Given that assessments of patient perceptions have emphasized a global rating of care, we anticipated that quality measures would be associated with the global rating in this population. Yet quality measures were more frequently associated with patient perceptions of improvement. These results suggest that perceived improvement may be an additional patient perception of care outcome that could be useful in assessing and improving the quality of care for patients with substance use disorders.

Four of the eight process-based quality measures evaluated are endorsed by the NQF (an endorsement that signals the rigor of the measures and increases the likelihood of broad implementation), and of those, three were significantly associated with one of the two patient perception measures. Treatment engagement was significantly associated with perceived improvement, but not global rating of care. Although treatment initiation was not associated with either outcome, initiation is often considered to be best understood when assessed alongside treatment engagement because initiating treatment would not be expected to indicate sufficient treatment. Both measures of timely follow-up after inpatient discharge were significantly associated with global rating of care but not with perceived improvement. These results provide additional support for the use of these NQF-endorsed measures and suggest that adherence to these measures may be associated with increased delivery of patient-centered care.

Our study had limitations. Identification of the cohort of patients with substance use disorders included in these analyses and operationalization of the quality measures relied on administrative claims data. Thus substance use disorder diagnoses recorded in the medical record may not accurately represent diagnoses assigned through standardized clinical assessments. Furthermore, the quality measures predominantly focus on the number and timing of visits rather than on more detailed information about care that would be documented in patient medical records.

There were limitations associated with patient perceptions of care, given that these measures have limitations similar to those of other data gathered through patient self-report (42). Patients may have difficulty recalling the care they received, differentially weight the value of positive and negative experiences, differentially weight recent versus distal experiences, or base their perceptions on varying expectations for their care. Patients were asked to report on their behavioral health care, likely leading some patients to include care received for co-occurring psychiatric diagnoses. Furthermore, perceived improvement is a retrospective measure, so it likely differs from clinician-rated improvement measures or patient self-report measures collected during the course of treatment. Patient perspectives of care were assessed after the process of care assessed by the quality measure occurred, and this time lag was lengthy for some patients. This delay may have hampered recall of their care, or other care received more recently could have affected their perceptions. Therefore, it would be important to assess these relationships in a new sample that assesses patient perceptions more proximal to the care being assessed. It is notable, however, that several significant associations were found between quality of care and patient perceptions despite this gap.

Finally, these correlational analyses found significant associations between quality measures and patient perceptions of care, but evidence of this relationship should not be interpreted as a unidimensional causal link (in other words, that increased quality causes increased patient perceptions of care). For example, it is also possible that patients who were satisfied with their care or who perceived their treatment as more helpful were more conscientious about their care, continued their care more consistently, and accessed further resources. Simply put, if patients perceive a benefit of treatment, they may be more likely to receive additional treatment. It is possible that there is a bidirectional relationship between treatment quality and patient perceptions of care.

Conclusions

Some quality measures assessing care for patients with substance use disorders were associated with patient perceptions of care. Results provide additional support for these quality measures and suggest that patient perceptions of care are an important outcome in assessing care.