Development, Testing, and Dissemination of a Public-Facing Tool to Help Consumers Find Higher-Quality Addiction Treatment

Individuals with substance use disorders and their families are often unsure how to find high-quality addiction treatment, resulting in patients receiving inappropriate, low-quality care or giving up trying to find addiction treatment altogether (1–3). The federal government hosts online search engines to help consumers find high-quality treatment for most provider types, including hospitals, nursing homes, psychiatric hospitals, inpatient rehabilitation facilities, home health care, hospice care, long-term-care hospitals, dialysis centers, and physicians (4). For example, consumers can use the Centers for Medicare and Medicaid Services (CMS) Medicare Compare website (at https://www.medicare.gov/care-compare) to identify nursing homes in their region and to compare their quality along various dimensions, such as staff hours per resident, rates of health citations and complaints, and rehospitalization rates. The federal government has not implemented a similar system for addiction treatment.

Although the Substance Abuse and Mental Health Services Administration (SAMHSA) and the National Institute on Alcohol Abuse and Alcoholism (NIAAA) make available online treatment locators, their websites give consumers only general criteria for identifying higher-quality addiction treatment (5, 6). They do not indicate whether providers included in the search engine meet these criteria. This is a missed opportunity. Studies have shown that public reporting of provider quality measures stimulates higher quality by allowing consumers to shop for quality dimensions, encouraging providers to compete on quality, and stimulating providers to learn from each other about improving quality (7–10).

In this article, we describe the development and testing of quality metrics for a public-facing search engine for addiction treatment. The system was developed and launched in six U.S. states (Massachusetts, New York, Delaware, West Virginia, North Carolina, and Louisiana).

Methods

We developed addiction quality measures by using three data sources: treatment facility surveys, insurance claims, and patient experience-of-care surveys. The project began in November 2018. The quality measures were made available to participating states, health plans, providers, and the public in July 2020. The project was reviewed by the RTI Institutional Review Board and determined not to be human subjects research.

Treatment Facility Survey

An addiction treatment facility survey was fielded that queried 2,441 providers in the six states about whether they offered the services and used processes identified by the National Institute on Drug Abuse, NIAAA, SAMHSA, and nonprofit organizations as signs of higher-quality treatment (11–15). These signs include rapid access to treatment when needed, use of evidence-based behavioral health therapies, access to medications for addiction, accreditation and appropriately credentialed staff, attention to mental health and general medical health needs, a personalized treatment plan for each patient, engagement in treatment for a period long enough to be effective, and the provision of recovery support services.

Addiction providers were identified with SAMHSA’s Inventory of Behavioral Health Services and input from state substance use disorder agencies (16). Data were collected from October 14, 2019, through January 31, 2020, via a Web-based survey. Across the six pilot states, 1,245 (out of 2,441; 51% participation rate) specialty substance use disorder treatment facilities completed the survey. Participation rates varied from 74% in New York to 28% in North Carolina.

We implemented validity checks to enhance the accuracy of survey responses. Facilities were required to upload documentation supporting their responses to specific questions, such as a copy of the facility’s license and a screenshot of an electronic medical record. Overall, 85% of providers uploaded a copy of the facility license, and 84% provided screenshots of their electronic medical record. A senior leader at the facility was required to sign off on the accuracy of the survey responses (96% attested to their accuracy). Inconsistent results triggered additional verification processes, including direct outreach to a treatment facility to confirm data accuracy and cross-checking against other data sources. For a random 5% of facilities, we compared responses to our survey with those on the SAMHSA National Survey of Specialty Substance Abuse Treatment Systems. Specifically, we compared responses to the questions about providing medications and recovery support services. Responses matched 94% of the time for methadone, 77% for buprenorphine, and 72% for naltrexone. The SAMHSA treatment locator data were collected approximately 1 year before our survey. Facilities may have added buprenorphine and naltrexone to their treatment array over the time between the two surveys.

Insurance Claims Data

As of 2018, the National Quality Forum (NQF) had endorsed the following quality measures for substance use disorder treatment for use in Medicaid programs and health plans: patients with opioid use disorder receiving medications for opioid use disorder (NQF 3400), continuity of medication use for opioid use disorder among patients with opioid use disorder (NQF 3175), and follow-up within 7 and 30 days after treatment for substance use disorder in inpatient or residential settings (NQF 3453). We respecified these measures to apply at the provider level rather than the Medicaid or health plan level. We also created a new provider-level claims-based measure that captured the percentage of a facility’s patients with a substance use disorder diagnosis who had a substance use disorder–related hospitalization or emergency department (ED) visit within 14, 30, 90, or 180 days from their second encounter over a 14-day period with the provider or after discharge from a 14-day inpatient stay.

We worked with one state Medicaid program and one commercial health plan to test and refine the methods for calculating the claims-based measures. We then provided final programming specifications to the other participating state Medicaid programs and commercial insurers. Four state Medicaid programs (Massachusetts, New York, Delaware, and West Virginia) and four private health insurance plans submitted the provider-level claims-based measures.

After states and commercial insurers submitted their metrics, we performed quality checks, such as ensuring that exclusion criteria were applied correctly and that final counts of beneficiaries, discharges, and medication episodes were consistent with the programming procedures. We then conducted reliability and validity testing by using the approaches recommended by NQF and CMS (17).

Patient Experience-of-Care Survey

The patient experience-of-care survey consisted of seven close-ended questions drawn from previously validated surveys. Most questions were sourced from the Consumer Assessment of Healthcare Providers and Systems Experience of Care and Health Outcomes Survey. We also included one open-ended question that asked for feedback on what the facility was doing right and where it could improve.

Because of heightened concerns about patient privacy in the context of addiction treatment, we developed a specific patient survey approach that involved no exchange of patient identifiers. We disseminated a letter unique to each addiction treatment facility and asked each provider to distribute it to all patients. The letter invited patients to go to a secure website and fill out the patient experience-of-care survey. Each letter had a personal identification number specific to that facility, which ensured that patient responses were linked to the correct facility.

Between October 14, 2019, and February 28, 2020, a total of 7,970 patients responded to the survey. Of the facilities invited (N=2,441), 16% (N=389) had at least one patient who completed an experience-of-care survey, and the number of facilities with at least one response to each item on the survey ranged from 370 to 386. Five percent (N=122) had ≥20 patient survey responses. Reliability testing of pilot data indicated that a minimum of 20 patient survey responses was needed for reliable comparison of facilities on the basis of patient experience-of-care survey responses.

Results

The addiction treatment quality metrics described above were made available in a public-facing search engine (https://treatmentatlas.org). Table 1 shows the variation in some treatment facility characteristics identified as signaling higher-quality treatment. We measured whether facilities offered rapid treatment access when needed by asking them to report on their hours of operation and whether they offered same-day access. Research has shown that same-day access can substantially reduce wait times (18–21). Across the six states, 77% of facilities said that they offered same-day access. Another sign of higher quality is a personalized treatment plan for each patient. We captured this in the facility survey by asking whether the facility conducted a comprehensive biopsychosocial assessment at intake as indicated by whether they asked patients about the six biopsychosocial dimensions of care defined by the American Society of Addiction Medicine (ASAM). Among the facilities responding, 81% said they conduct comprehensive biopsychosocial assessments.

Other indicators of higher care quality are offering evidence-based behavioral health therapy and addiction medications. Overall, 70% of facilities said they offered opioid use disorder medications, and 60% said they offered alcohol use disorder medication (Table 1). NIAAA lists several evidence-based, effective behavioral health therapies, and we asked whether the facilities offered any of these (22). Among the facilities surveyed, 94% offered at least one evidence-based behavioral intervention. Cognitive-behavioral therapy was offered by 89% of the facilities, and contingency management was offered by 32%.

The federal agencies highlight that higher-quality facilities also attend to patients’ general medical and mental health needs. Most facilities reported being able to provide mental health care onsite (68%) and to offer psychiatric medications onsite (70%). In contrast, only 20% of facilities offered primary care onsite, which might have limited these facilities’ abilities to screen for and manage medical comorbid conditions that co-occur with substance misuse. Finally, federal agencies emphasize the importance of providing recovery supports. The survey asked whether the facility offered different types of recovery supports. The most frequently offered recovery supports were assistance obtaining social services (77%) and case management (76%). The least common were legal aid and child care.

Claims-Based Quality Measures

We conducted reliability tests to determine whether the claims measures could distinguish well-performing providers from poorly performing providers—that is, whether the differences in the quality measures among providers were due to provider-specific differences and not due to random noise. We conducted three types of reliability tests: signal-to-noise ratio and effect size tests, equality-of-distribution tests, and Adams’s rho. All tests indicated that the measures were reliable. The face validity of the measures was determined by a literature review (e.g., of research on the association between postdischarge follow-up and positive outcomes). We also tested the measures’ convergent validity by calculating the correlation between related quality concepts. We found that higher provider rates of opioid use disorder prescriptions among patients with opioid use disorder were associated with lower provider-level rates of substance use disorder hospitalizations, fewer ED visits, and higher follow-up visit rates.

Figure 1 shows the median and range for the four claims-based measures for the Medicaid population. Only providers with at least 10 patients who met the measures’ denominator requirements were included to ensure patient privacy and measure reliability. In the median (“typical”) facility, 29% (N=408 facilities) of patients with an opioid use disorder diagnosis filled a prescription for an opioid use disorder medication within 30 days of visiting that provider. The bottom and top quartile providers had rates of ≤18% and ≥67%, respectively. The 6-month (180-day) adherence rate for the median provider was 23% (N=283 facilities), meaning that about a quarter of that provider’s patients stayed on the opioid use disorder medication for at least 6 months. This metric ranged from ≤18% in the lowest quartile to ≥67% in the highest quartile.

Among the inpatient or residential specialty substance use disorder providers, the median facility follow-up rate was 40% (N=129 facilities), with a rate of ≤30% in the lowest quartile and ≥47% in the highest quartile. In terms of hospitalizations and ED visits for substance use disorders or overdoses, the median facility had 8% of patients admitted within 30 days of their first encounter (N=136 facilities). This metric ranged from ≤5% to ≥15% in the lowest and highest quartiles, respectively.

Patient Experience-of-Care Measures

Figure 2 shows the median and range for the patient experience-of-care measures. The survey responses that allowed for four response possibilities (i.e., never, sometimes, usually, always) were top coded, indicating the percentage of patients at the facility who gave the best possible answer (i.e., always).

The ratings showed significant variation among facilities. In the median facility, 100% (N=371 facilities) of patients said they were given information about different treatment options (interquartile range [IQR]=11%). In the median facility, 90% (N=374 facilities) of patients reported that staff at the program discussed including family or friends in treatment (IQR=27%), 89% (N=386 facilities) said that they had been helped a lot by the facility (IQR=25%), 86% (N=372 facilities) reported that the staff always showed respect for what they had to say (IQR=35%), and 64% (N=370 facilities) indicated that they always saw staff from the treatment program as soon as they wanted (IQR=39%).

In terms of day-to-day functioning, in the median facility, 70% (N=376 facilities) of patients said their ability to deal with their daily problems was much better than before treatment (IQR=43%). In the median facility, 60% (N=382 facilities) of patients rated the facility as 9 or 10 on a scale from 1 to 10 (with 1 indicating worst and 10 indicating best; IQR=50%).

Discussion

The results of this quality measurement development project indicate that it is feasible to collect and publicly report reliable and valid addiction treatment provider quality measures. Over 5 months, survey data were received from 1,245 addiction treatment facilities and 7,970 patients across six states. Four state Medicaid programs submitted the claims-based quality measures. The quality measures varied significantly among providers, indicating that these measures could help consumers select providers.

Our results revealed several lessons for future efforts aimed at measuring addiction treatment quality. A limitation of the project was that the rate of participation of facilities in the facility survey varied from 74% to 28% among the six states. An adequate number of patient experience-of-care surveys (i.e., 20 surveys) were received for 5% of facilities. Participation in quality measurement systems usually grows over time as providers become more comfortable with the reporting requirements and realize the importance of participation to their reputation.

Federal quality reporting systems have the advantage over privately sponsored systems in that they can compel provider participation as a condition for receiving Medicare or Medicaid payments or contracts. The federal government may also have more resources to keep the information collected up to date. Moreover, in contrast to this project, federal and state governments have greater latitude to disseminate patient experience-of-care surveys directly to patients by using contact information collected as part of Medicaid and Medicare, which can facilitate more responses from patients.

This study established the feasibility of creating valid and reliable provider-level claims measures. When we began the project, the only NQF-endorsed, claims-based addiction quality measures were for use at the health plan or state Medicaid program levels. Our provider-level measures could help Medicaid programs uncover the root causes of performance lags. For example, hospitals and residential providers with low discharge follow-up rates may need training in discharge planning and assistance in identifying appropriate and available outpatient programs. The claims-based measures were limited to only four domains of quality of care. Ideally, the claims measures would expand to capture additional evidence-based interventions such as the delivery of contingency management and cognitive-behavioral therapy (assuming procedure codes exist to capture these interventions). Moreover, another limitation was that the measure of substance use disorder–related hospitalizations and ED visits was not risk adjusted. Risk adjustment could be applied in future iterations.

The federal government’s sponsorship and support of public-facing quality measures began more than two decades ago with Nursing Home Compare in 1998. Over time, CMS has added new providers and measures, refined existing measures, and improved how the data are displayed and communicated. Their effort has resulted in meaningful improvements in the quality of care.

Conclusions

This article describes a successful effort in six U.S. states to develop and disseminate addiction treatment provider quality metrics. Despite this success, we remain concerned by the lack of federal leadership in the dissemination of addiction treatment quality metrics. The federal government invests tens of millions of dollars each year in quality measurement targeted to most health care settings. It disseminates public-facing, consumer-friendly quality metrics on thousands of providers. The quality of addiction treatment would benefit from being incorporated into federal quality measurement programs.