0
Get Alert
Please Wait... Processing your request... Please Wait.
You must sign in to sign-up for alerts.

Please confirm that your email address is correct, so you can successfully receive this alert.

Articles   |    
When Quality Indicators Undermine Quality: Bias in a Quality Indicator of Follow-Up for Alcohol Misuse
Katharine A. Bradley, M.D., M.P.H.; Laura J. Chavez, M.P.H.; Gwendolyn T. Lapham, Ph.D., M.P.H.; Emily C. Williams, Ph.D., M.P.H.; Carol E. Achtmeyer, M.N.; Anna D. Rubinsky, Ph.D., M.S.; Eric J. Hawkins, Ph.D.; Richard Saitz, M.D., M.P.H.; Daniel R. Kivlahan, Ph.D.
Psychiatric Services 2013; doi: 10.1176/appi.ps.201200449
View Author and Article Information

Dr. Bradley and Dr. Lapham are with the Group Health Research Institute, 1730 Minor Ave., Suite 1600, Seattle, WA 98101 (e-mail: bradley.k@ghc.org). Ms. Chavez, Dr. Williams, Ms. Achtmeyer, Dr. Rubinsky, and Dr. Kivlahan are with the Northwest Center of Excellence, Veterans Affairs (VA) Health Services Research and Development, Seattle. Ms. Chavez and Dr. Williams are also with the Department of Health Services, School of Public Health, University of Washington, Seattle. Dr. Hawkins is with the Center of Excellence in Substance Abuse Treatment and Education, VA Puget Sound Health Care System, Seattle. Dr. Saitz is with the Department of General Internal Medicine, Boston University, Boston.

Copyright © 2013 by the American Psychiatric Association

Abstract

Objective  Valid quality indicators are needed to monitor and encourage identification and management of mental health and substance use conditions (behavioral conditions). Because behavioral conditions are frequently underidentified, quality indicators often evaluate the proportion of patients who screen positive for a condition who also have appropriate follow-up care documented. However, these “positive-screen–based” quality indicators of follow-up for behavioral conditions could be biased by differences in the denominator due to differential screening quality (“denominator bias”) and could reward identification of fewer patients with the behavioral conditions of interest. This study evaluated denominator bias in the performance of Veterans Health Administration (VHA) networks on a quality indicator of follow-up for alcohol misuse that used the number of patients with positive alcohol screens as the denominator.

Methods  Two quality indicators of follow-up for alcohol misuse—a positive-screen–based quality indicator and a population-based quality indicator—were compared among 21 VHA networks by review of 219,119 medical records.

Results  Results for the two quality indicators were inconsistent. For example, two networks performed similarly on the quality indicators (64.7% and 65.4% follow-up) even though one network identified and documented follow-up for almost twice as many patients (5,411 and 2,899 per 100,000 eligible, respectively). Networks that performed better on the positive-screen–based quality indicator identified fewer patients with alcohol misuse than networks that performed better on the population-based quality indicator (mean 4.1% versus 7.4%, respectively).

Conclusions  A positive-screen–based quality indicator of follow-up for alcohol misuse preferentially rewarded networks that identified fewer patients with alcohol misuse.

Abstract Teaser
Figures in this Article

Your Session has timed out. Please sign back in to continue.
Sign In Your Session has timed out. Please sign back in to continue.
Sign In to Access Full Content
 
Username
Password
Sign in via Athens (What is this?)
Athens is a service for single sign-on which enables access to all of an institution's subscriptions on- or off-site.
Not a subscriber?

Subscribe Now/Learn More

PsychiatryOnline subscription options offer access to the DSM-5 library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.

Need more help? PsychiatryOnline Customer Service may be reached by emailing PsychiatryOnline@psych.org or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).

Figure 1 Comparison of VHA network ranks on two quality indicators of follow-up for alcohol misusea

aLower-numbered ranks reflect higher Veterans Health Administration (VHA) network performance, with 1 indicating the highest performance.

Anchor for Jump
Table 1Example of denominator bias in quality indicators of follow-up care for a behavioral condition among three hypothetical health systemsa
Table Footer Note

a Assumes three identical health care systems, each with 100,000 identical patients, 13% of whom have a behavioral condition, and variable prevalence of positive screens (“screening prevalence”; 5% and 10%) and variable proportions of patients with appropriate follow-up (50% and 80%) among those with positive screens.

Table Footer Note

b The proportion of patients with positive screens who are offered appropriate follow-up: “positive-screen-based” quality indicator

Table Footer Note

c The proportion of all patients in a health care system who are offered appropriate follow-up: “population-based” quality indicator

Anchor for Jump
Table 2Variation in the screening prevalence of alcohol misuse and follow-up for alcohol misuse based on two types of quality indicators across the 21 VHA networks
Table Footer Note

a The proportion of patients who screened positive (≥5 points) for alcohol misuse on the Alcohol Use Disorders Identification Test–Consumption (AUDIT-C) questionnaire based on Veterans Health Administration (VHA) external peer review program (EPRP) medical record reviews

Table Footer Note

b The proportion of patients who screen positive for alcohol misuse (AUDIT-C score ≥5) who had documented follow-up according to EPRP medical record reviews

Table Footer Note

c The proportion and number per 100,000 of patients eligible for screening who had alcohol misuse identified (AUDIT-C score ≥5) and follow-up documented according to EPRP medical record reviews

Table Footer Note

d Networks are labeled A–U in nonalphabetic order consistent with a previous report (15) and are ordered on the basis of the prevalence of positive screens for alcohol misuse.

Anchor for Jump
Table 3Association between differences in VHA network rank on two quality indicators of follow-up for alcohol misuse and the adjusted screening prevalence of alcohol misuse
Table Footer Note

a Difference in Veterans Health Administration (VHA) network ranks on unadjusted quality indicator of follow-up for alcohol misuse. Positive values of differences in rank indicate higher performance on the positive-screen–based quality indicator; negative values indicate higher performance on the population-based quality indicator.

Table Footer Note

b Proportion of patients who screened positive (≥5 points) for alcohol misuse on the Alcohol Use Disorders Identification Test–Consumption questionnaire based on medical record reviews and adjusted for age, gender, race, and an independent survey measure of the facility prevalence of alcohol misuse.

+

References

Kessler  RC;  Chiu  WT;  Demler  O  et al:  Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the National Comorbidity Survey Replication.  Archives of General Psychiatry 62:617–627, 2005
[CrossRef] | [PubMed]
 
Institute of Medicine:  Improving the Quality of Health Care for Mental and Substance-Use Conditions: Quality Chasm Series .  Washington, DC,  National Academies Press, 2006
 
Table B22: State Estimates of Substance Use From the 2007–2008 National Surveys on Drug Use and Health. Rockville, Md, Substance Abuse and Mental Health Services Administration. Available at www.oas.samhsa.gov/2k8state/AppB.htm#TabB-9. Accessed Oct 3, 2011
 
Demyttenaere  K;  Bruffaerts  R;  Posada-Villa  J  et al:  Prevalence, severity, and unmet need for treatment of mental disorders in the World Health Organization World Mental Health Surveys.  JAMA 291:2581–2590, 2004
[CrossRef] | [PubMed]
 
McGlynn  EA;  Asch  SM;  Adams  JL  et al:  The quality of health care delivered to adults in the United States.  New England Journal of Medicine 348:2635–2645, 2003
[CrossRef] | [PubMed]
 
Pincus  HA;  Spaeth-Rublee  B;  Watkins  KE:  Analysis and commentary: the case for measuring quality in mental health and substance abuse care.  Health Affairs 30:730–736, 2011
[CrossRef] | [PubMed]
 
Chassin  MR;  Loeb  JM;  Schmaltz  SP  et al:  Accountability measures—using measurement to promote quality improvement.  New England Journal of Medicine 363:683–688, 2010
[CrossRef] | [PubMed]
 
Kilbourne  AM;  Greenwald  DE;  Hermann  RC  et al:  Financial incentives and accountability for integrated medical care in Department of Veterans Affairs mental health programs.  Psychiatric Services 61:38–44, 2010
[CrossRef] | [PubMed]
 
Tobacco and Alcohol Measures. Washington, DC, Joint Commission, Jan 21, 2011. Available at www.jointcommission.org/assets/1/6/HIQR_Release_Notes_1.1.13_v.4.2.pdf
 
Humphreys  K;  McLellan  AT:  A policy-oriented review of strategies for improving the outcomes of services for substance use disorder patients.  Addiction 106:2058–2066, 2011
[CrossRef] | [PubMed]
 
Lapham  GT;  Achtmeyer  CE;  Williams  EC  et al:  Increased documented brief alcohol interventions with a performance measure and electronic decision support.  Medical Care 50:179–187, 2012
[CrossRef] | [PubMed]
 
About eValue8. Washington, DC, National Business Coalition on Health, 2009. Available at evalue8.org/eValue8. Accessed Oct 3, 2011
 
Bradley  KA;  Williams  EC;  Achtmeyer  CE  et al:  Measuring performance of brief alcohol counseling in medical settings: a review of the options and lessons from the Veterans Affairs (VA) health care system.  Substance Abuse 28:133–149, 2007
[CrossRef] | [PubMed]
 
Oslin  DW;  Ross  J;  Sayers  S  et al:  Screening, assessment, and management of depression in VA primary care clinics.  Journal of General Internal Medicine 21:46–50, 2006
[CrossRef] | [PubMed]
 
Bradley  KA;  Lapham  GT;  Hawkins  EJ  et al:  Quality concerns with routine alcohol screening in VA clinical settings.  Journal of General Internal Medicine 26:299–306, 2011
[CrossRef] | [PubMed]
 
Williams EC, Achtmeyer CE, Grossbard JR, et al: Barriers and Facilitators to Implementing Alcohol Screening With a Clinical Reminder in the VA: A Qualitative Study. Atlanta, Research Society on Alcoholism, 2011
 
Jha  AK;  Perlin  JB;  Kizer  KW  et al:  Effect of the transformation of the Veterans Affairs Health Care System on the quality of care.  New England Journal of Medicine 348:2218–2227, 2003
[CrossRef] | [PubMed]
 
Bradley  KA;  Kivlahan  DR;  Williams  EC:  Brief approaches to alcohol screening: practical alternatives for primary care.  Journal of General Internal Medicine 24:881–883, 2009
[CrossRef] | [PubMed]
 
Bradley  KA;  DeBenedetti  AF;  Volk  RJ  et al:  AUDIT-C as a brief screen for alcohol misuse in primary care.  Alcoholism, Clinical and Experimental Research 31:1208–1217, 2007
[CrossRef] | [PubMed]
 
Wright  SM;  Craig  T;  Campbell  S  et al:  Patient satisfaction of female and male users of Veterans Health Administration services.  Journal of General Internal Medicine 21(suppl 3):S26–S32, 2006
[CrossRef] | [PubMed]
 
Pronovost  PJ;  Lilford  R:  A road map for improving the performance of performance measures.  Health Affairs 30:569–573, 2011
[CrossRef] | [PubMed]
 
Damman  OC;  Stubbe  JH;  Hendriks  M  et al:  Using multilevel modeling to assess case-mix adjusters in consumer experience surveys in health care.  Medical Care 47:496–503, 2009
[CrossRef] | [PubMed]
 
Buchsbaum  DG;  Buchanan  RG;  Lawton  MJ  et al:  A program of screening and prompting improves short-term physician counseling of dependent and nondependent harmful drinkers.  Archives of Internal Medicine 153:1573–1577, 1993
[CrossRef] | [PubMed]
 
Buchsbaum  DG;  Buchanan  RG;  Poses  RM  et al:  Physician detection of drinking problems in patients attending a general medicine practice.  Journal of General Internal Medicine 7:517–521, 1992
[CrossRef] | [PubMed]
 
Burman  ML;  Kivlahan  DR;  Buchbinder  MB  et al:  Alcohol-related advice for Veterans Affairs primary care patients: who gets it? Who gives it? Journal of Studies on Alcohol 65:621–630, 2004
[PubMed]
 
Volk  RJ;  Steinbauer  JR;  Cantor  SB:  Patient factors influencing variation in the use of preventive interventions for alcohol abuse by primary care physicians.  Journal of Studies on Alcohol 57:203–209, 1996
[PubMed]
 
 Stata Statistical Software: Release 11 .  College Station, Tex,  Stata Corp, 2009
 
Solberg  LI;  Maciosek  MV;  Edwards  NM:  Primary care intervention to reduce alcohol misuse ranking its health impact and cost effectiveness.  American Journal of Preventive Medicine 34:143–152, 2008
[CrossRef] | [PubMed]
 
Maciosek  MV;  Coffield  AB;  Edwards  NM  et al:  Priorities among effective clinical preventive services: results of a systematic review and analysis.  American Journal of Preventive Medicine 31:52–61, 2006
[CrossRef] | [PubMed]
 
Jonas  DE;  Garbutt  JC;  Amick  HR  et al:  Behavioral counseling after screening for alcohol misuse in primary care: a systematic review and meta-analysis for the US Preventive Services Task Force.  Annals of Internal Medicine 157:645–654, 2012
[CrossRef] | [PubMed]
 
Desai  MM;  Rosenheck  RA;  Craig  TJ:  Case-finding for depression among medical outpatients in the Veterans Health Administration.  Medical Care 44:175–181, 2006
[CrossRef] | [PubMed]
 
Kilbourne  AM;  Keyser  D;  Pincus  HA:  Challenges and opportunities in measuring the quality of mental health care.  Canadian Journal of Psychiatry 55:549–557, 2010
 
Walter  LC;  Davidowitz  NP;  Heineken  PA  et al:  Pitfalls of converting practice guidelines into quality measures: lessons learned from a VA performance measure.  JAMA 291:2466–2470, 2004
[CrossRef] | [PubMed]
 
Landon  BE;  O’Malley  AJ;  Keegan  T:  Can choice of the sample population affect perceived performance: implications for performance assessment.  Journal of General Internal Medicine 25:104–109, 2010
[CrossRef] | [PubMed]
 
Garnick  DW;  Horgan  CM;  Chalk  M:  Performance measures for alcohol and other drug services.  Alcohol Research and Health 29:19–26, 2006
[PubMed]
 
HEDIS 2011 Technical Specifications, Vol 2. Washington, DC, National Committee for Quality Assurance, 2011
 
Physician Consortium for Performance Improvement: Preventive Care and Screening: Percentage of Patients Aged 18 Years and Older Who Were Screened for Unhealthy Alcohol Use at Least Once During The Two-Year Measurement Period Using a Systematic Screening Method and Who Received Brief Counseling if Identified as an Unhealthy Alcohol User. Rockville, Md, Agency for Healthcare Research and Quality, National Quality Measures Clearinghouse, 2008. Available at www.qualitymeasures.ahrq.gov/content.aspx?id=13366&search=alcohol. Accessed Oct 3, 2011
 
Bradley  KA;  Johnson  ML;  Williams  EC:  Commentary on Nilsen et al (2011): the importance of asking patients: the potential value of patient report of brief interventions.  Addiction 106:1757–1759, 2011
[CrossRef] | [PubMed]
 
HEDIS CAHPS Health Plan Survey 4.0H Adult Questionnaire (Commercial). Washington, DC, National Committee for Quality Assurance, 2008. Available at www.cchri.org/programs/CAHPS%20PDFs/CAHPS_4_0H_Adult_Commercial_survey.pdf
 
Whitlock  EP;  Polen  MR;  Green  CA  et al:  Behavioral counseling interventions in primary care to reduce risky/harmful alcohol use by adults: a summary of the evidence for the US Preventive Services Task Force.  Annals of Internal Medicine 140:557–568, 2004
[CrossRef] | [PubMed]
 
Bradley  KA;  Williams  EC;  Achtmeyer  CE  et al:  Implementation of evidence-based alcohol screening in the Veterans Health Administration.  American Journal of Managed Care 12:597–606, 2006
[PubMed]
 
Keenan  PS;  Cleary  PD;  O’Malley  AJ  et al:  Geographic area variations in the Medicare health plan era.  Medical Care 48:260–266, 2010
[CrossRef] | [PubMed]
 
Seale  JP;  Shellenberger  S;  Tillery  WK  et al:  Implementing alcohol screening and intervention in a family medicine residency clinic.  Substance Abuse 26:23–31, 2005
[CrossRef] | [PubMed]
 
Seale  JP;  Shellenberger  S;  Boltri  JM  et al:  Effects of screening and brief intervention training on resident and faculty alcohol intervention behaviours: a pre- post-intervention assessment.  BMC Family Practice 6:46, 2005
[CrossRef] | [PubMed]
 
Rose  HL;  Miller  PM;  Nemeth  LS  et al:  Alcohol screening and brief counseling in a primary care hypertensive population: a quality improvement intervention.  Addiction 103:1271–1280, 2008
[CrossRef] | [PubMed]
 
Bradley  KA;  Williams  EC:  Implementation of screening and brief intervention in clinical settings using quality improvement principles; in  Principles of Addiction Medicine . Edited by Fiellin  D;  Miller  S;  Saitz  R  et al.  Baltimore,  Lippincott Williams & Wilkins, 2009
 
Blumenthal  D:  Launching HITECH.  New England Journal of Medicine 362:382–385, 2010
[CrossRef] | [PubMed]
 
Blumenthal  D;  Tavenner  M:  The “meaningful use” regulation for electronic health records.  New England Journal of Medicine 363:501–504, 2010
[CrossRef] | [PubMed]
 
Ahern  DK;  Woods  SS;  Lightowler  MC  et al:  Promise of and potential for patient-facing technologies to enable meaningful use.  American Journal of Preventive Medicine 40(suppl 2):S162–S172, 2011
[CrossRef] | [PubMed]
 
Druss  BG;  Mauer  BJ:  Health care reform and care at the behavioral health–primary care interface.  Psychiatric Services 61:1087–1092, 2010
[CrossRef] | [PubMed]
 
Walker EP: Medicare to cover alcohol, depression screening. MedPage Today, Oct 17, 2011. Available at www.medpagetoday.com/PrimaryCare/PreventiveCare/29085. Accessed Feb 1, 2012
 
References Container
+
+

CME Activity

There is currently no quiz available for this resource. Please click here to go to the CME page to find another.
Submit a Comments
Please read the other comments before you post yours. Contributors must reveal any conflict of interest.
Comments are moderated and will appear on the site at the discertion of APA editorial staff.

* = Required Field
(if multiple authors, separate names by comma)
Example: John Doe



Web of Science® Times Cited: 1

Related Content
Articles
Books
The American Psychiatric Publishing Textbook of Psychiatry, 5th Edition > Chapter 9.  >
Gabbard's Treatments of Psychiatric Disorders, 4th Edition > Chapter 13.  >
The American Psychiatric Publishing Textbook of Geriatric Psychiatry, 4th Edition > Chapter 2.  >
The American Psychiatric Publishing Textbook of Geriatric Psychiatry, 4th Edition > Chapter 8.  >
The American Psychiatric Publishing Textbook of Geriatric Psychiatry, 4th Edition > Chapter 8.  >
Topic Collections
Psychiatric News
Read more at Psychiatric News >>