Impact of Electronic Health Record Transition on Behavioral Health Screening in a Large Pediatric Practice
Abstract
Objective:
The objective was to determine whether transitioning from paper to electronic health records affected behavioral health screening rates in a large Northeastern pediatric practice.
Methods:
The study setting was a pediatric practice with seven pediatricians, serving about 6,000 patients. The patient population was diverse (54% nonwhite, 40% publicly insured or self-paying, and 31% non-English speakers). An interrupted times series design was used to evaluate the impact of electronic record implementation on behavioral health screening rates. The main outcome measure was the rate of such screening 18 months before and 36 months after implementation.
Results:
The rate of behavioral health screening increased from 70% to 91% during the baseline period. The training period—six months before electronic record implementation—was associated with a 28% decline in adjusted screening rates (from 83.3% to 55.5%). Only 50% of eligible youths were screened in the first month after implementation. The screening rate took more than three years to recover to baseline levels, climbing to 82% by April 2008.
Conclusions:
Practice changes resulting from electronic record adoption were highly disruptive of care, and disruptions took several years to resolve completely. When medical assistants rather than physicians were tasked with transferring data from paper screening forms to the electronic record, reporting compliance improved. Compliance with Healthcare Effectiveness Data and Information Set standards and Medicaid performance measures will likely be similarly affected as electronic records are implemented nationwide. Although implementing a fully automated medical record has some benefits, the unintended effects on care after implementation must be acknowledged. (Psychiatric Services 63:256–261, 2012; doi: 10.1176/appi.ps.201100207)
The American Recovery and Reinvestment Act of 2009 calls for the use of an electronic health record for each person in the United States by 2014 (1). This legislation is based on the assumption that electronic health records can improve the safety, timeliness, effectiveness, and efficiency of care (2). Health information technology has been shown to improve the quality of care in some circumstances. For example, embedded alert systems have increased the timeliness of childhood immunization and overall rates of immunization (3,4), and an electronic record add-on template for attention-deficit hyperactivity disorder was reported to have increased follow-up visits and documentation of symptoms (5). However, systematic reviews have concluded that the impact of electronic health records on improving clinical outcomes is unproven (6–8).
The problems and harms associated with electronic health records are also well documented. One report of the effects of computerized physician order entry showed an increase in pediatric mortality after a change in electronic health record software in a Pittsburgh, Pennsylvania, teaching hospital (9). Although a more recent time series analysis reported a decrease in hospitalwide mortality (10), this association may have been confounded by competing interventions (for example, process and workflow changes, adjustments in intensive care unit staffing, and rollout of rapid response teams) (11). Ash and colleagues (12–14) have reported many unintended negative consequences of electronic records, such as increased time necessary to enter data, fragmentation of information across screens or tabs, inflexible processes, and physician overload. Several rigorous studies have reported no improvement in quality of care associated with the adoption of electronic records or even a deterioration in quality of care (9,15–19). Transitioning from a paper to an electronic environment is also well known to be associated with disruptions in workflow (20), increased patient cycle time (21), breakdowns in communication between clinicians (12), the risk of computer system crashes, and lack of communication between different electronic records systems.
Despite the increasing evidence of electronic records' unintended effects (22), no studies have reported how specific pediatric screening or disease management processes may be affected by the transition from a paper to an electronic environment. As electronic records systems are implemented or modified across the country, clinicians and insurers may experience declines in the quality of care (as measured by compliance with mandated services) associated with the disruption of workflow and changes in protocols (17).
The purpose of this study was to determine whether the transition from a paper to an electronic health record affected behavioral health screening in a large Northeastern pediatric practice setting. The increasing prevalence of mental illness among children and the scarcity of child mental health professionals suggest that pediatricians must improve detection and treatment of mental disorders in primary care settings (23). National organizations (24–27) have recommended such behavioral health screening in pediatric primary care to address this need. Behavioral health screens improve identification rates and referral numbers and may affect timeliness of care (28–31). Behavioral health screening and related increases in referrals are also associated with improved behavioral health outcomes measured at subsequent preventive care visits (for example, lower Pediatric Symptom Checklist [PSC] scores) (32,33). Thus, the positive and negative impacts of health information technology on behavioral health screening merit further attention.
Methods
This study received approval from the Cambridge Health Alliance Institutional Review Board.
Setting
In 2002, a class-action lawsuit Rosie D v. Romney (34) was filed in Massachusetts on behalf of eight children with serious psychiatric disabilities who were hospitalized or at risk for hospitalization owing to the state's failure to provide needed services to allow them to live at home and attend local schools. In 2007, the state was found in violation of the Medicaid Act (35). The court's decision required all primary care providers caring for MassHealth-enrolled children younger than age 21 whose benefit package includes primary care service to begin using a clinically appropriate behavioral health screening tool at well-child visits.
At a large pediatric clinic site, behavioral health screening of children at well-child visits began in December 2003, five years before the initiation of the mandatory statewide program. The clinic, which is part of a large public health care system, has seven pediatricians and serves about 6,000 patients, of whom about 33% are younger than age five. On average, 180 children are eligible for behavioral health screening each month. Although this clinic is an academic teaching site, pediatricians are generally scheduled to see approximately 25 patients a day in 15-minute intervals. The population is economically and linguistically diverse (54% nonwhite, 40% publicly insured or self-pay, and 31% non-English speakers).
Screening was conducted using the PSC (for children age four years, 11 months, to age 16) and the PSC-Youth Report (Y-PSC) (for children ages 14–18), both of which are validated tools that have been translated into multiple languages (36,37). On both tools, summary scores above a certain cut point are considered positive screens for behavioral difficulties.
The screening process began with a family member (on behalf of a child patient) or a teenage patient filling out the paper PSC or Y-PSC in the waiting room before the visit. They would then bring the paper form to the physician, who would discuss it with them and score the instrument. Before implementation of the electronic records system, providers inserted the completed paper form into the medical record. Over the course of the first year of screening, nurse management staff and clinical leaders developed processes to enhance screening. Nurse management staff reviewed charts the night before to identify children who were eligible for screening visits. They then attached the appropriate PSC or Y-PSC to the chart for distribution by the receptionist when the patient arrived. In addition, the nurse manager was given regular updates on compliance and worked with front-office staff to improve distribution and with medical assistants to retrieve forms.
Practice changes during electronic record implementation
The go-live date for using the electronic health records system (the EpicCare EMR system) at this practice was March 29, 2005, but planning for implementation began six months earlier. Approximately two months before the go-live date, all staff went to full EpicCare off-site training (40 hours each). Training occurred in a rolling manner with different elements so as not to overwhelm staff. To ease transition, patient visits were reduced by 50% in the first week and by 25% in the second week after electronic records system implementation so providers could adjust to the system. In-house trainers were also on site to help with implementation. To capture screening information electronically, a questionnaire for entering paper-based PSC data was programmed into the electronic records system in June 2005.
The PSC screening process remained the same, with paper forms used throughout; however, once the practice began using electronic records, the results from the paper PSC were entered into the electronic records system via an electronic questionnaire with four required fields. These fields included the total score, responses to two questions—is your child (you) currently seeing a mental health counselor and does your child (you) have any emotional or behavioral problems for which she or he (you) needs help—and indication of disposition (annual visit, return to primary care, or referral to counseling). Initially, the physicians did this themselves; later, the task was transferred to medical assistants, who also scanned the actual PSC form into the system. Reports were developed to provide monthly compliance rates by individual physicians to assist management with process improvement.
Measures
To measure other contextual factors that might have affected screening rates and the transition to electronic records, we conducted a review of past e-mail communication, minutes of implementation meetings, and reports and held discussions with two practice administrators to capture any environmental changes (such as staffing, scheduling, operations, and training information) that took place during the transitional period (October 2004 through March 2005). In addition, these individuals were asked to describe the major challenges to electronic records implementation.
To determine rates of screens, we drew monthly data on preventive care-eligible visits from the MEDITECH system. Until March 2007, copies of PSCs were collected weekly, and data were entered into a database. The crude screening rate was calculated by dividing the number of paper screens by the number of eligible visits. Once the practice began entering data from the paper screens into the electronic questionnaire, a monthly compliance report was also generated from the MEDITECH system to determine screening results in the electronic records and identify the number of screens by provider by month. For the purposes of this study, we used the highest number of screens reported from either paper or electronic data until March 2007. Data were available only from the electronic records after March 2007 because the paper form was discarded after data entry.
Analysis
We used an interrupted times series design to evaluate the impact of electronic records implementation on behavioral health screening rates (38). Interrupted time series is the strongest quasi-experimental design for evaluating the effects of natural experiments (39). Controlling for the baseline rate and trend of behavioral health screening, we estimated changes in the level and slope of the screening rate during the electronic records planning period and after implementation. The regression equation was specified as
We used maximum likelihood estimation to estimate the models and controlled for autocorrelation by including all significant autocorrelation parameters up to 12 months. Backward elimination was used to include covariates in the final model with a type 1 error threshold for inclusion of .05. All statistical analyses were performed using PROC AUTOREG in SAS, version 9.2 (40).
Results
Behavioral health screening rates
Figure 1 shows the baseline, training, and post-electronic records implementation behavioral health screening rates. The rate of screening increased during the baseline period (December 2003 through September 2004), beginning at just below 70% in December 2003 and climbing to 91% in September 2004. The average screening rate during the baseline period was 81%. In the six-month training period (October 2004–March 2005) before electronic records implementation, rates of screening dropped precipitously and significantly from 84% to 64% (p<.001). The switch from paper to electronic records on March 29, 2005, was associated with a further drop in screening rates. Only 50% of eligible youths were screened in the first month of electronic-based records. Rates recovered slowly over three years, climbing to 82% by April 2008, matching baseline rates. The monthly behavioral health screening rate also became much more variable after EHR implementation.
Table 1 shows the results of the segmented regression analysis. Both the time-after-training and the time-after-implementation trend coefficients were significant. The six-month training period was associated with a 5% monthly decrease in the adjusted behavioral health screening rate (p=.002). Behavioral health screening rates increased by 3% monthly (p=.006) between May 2005 and April 2008.
Environmental changes
In our review of e-mail, minutes, and reports and our conversations with practice management, we identified several important challenges that occurred simultaneously with EpicCare implementation. First, two months before implementation, the practice experienced the departure of the nurse manager and several other personnel. The nurse manager had been leading the electronic records roll-out, and with her departure several other people became more involved with the implementation.
Second, according to staff, transitioning from paper to electronic records proved difficult. After the transition, the PSC form was the only paper form that was handed out to parents of children coming in for their well-child visits. Front office staff tended to forget to hand out the PSC forms, particularly on days when there were few eligible visits. In addition, identifying children who met criteria was sometimes confusing. Thus the percentage of hard-copy PSCs collected for eligible visits dropped substantially, as seen in Figure 1. Later, health assistants were asked to back up front-office staff and get a form if the parent did not carry one into the exam room.
On the provider side, many providers lacked computer skills and had trouble keeping up with their workload during the transition. The additional burden of inputting PSC data into the electronic questionnaire proved problematic. As the medical director of the clinic noted, “This computerized questionnaire is new to us here … so I am not too surprised the completion rate is so poor.” Later, it became apparent that providers were entering the information via addendums after the visits were closed, and it was eventually decided that the medical assistants would be responsible for inputting the PSC data into the electronic questionnaire. This process significantly improved screening documentation. With the implementation of the Rosie D mandate (January 2008), additional changes were made for the purpose of billing, including scanning forms and using specific billing codes.
Discussion
Implementing an electronic health records system is a complex process that demands attentive leadership engaged in ongoing solutions as providers adjust to the change (20,41). In addition, the impact of this transition on provider behavior, practice operations, and administrative procedures cannot be underestimated.
In this study, we observed two negative phases in the changeover to electronic records. First, the process of transitioning to an electronic records system, or preimplementation, had an impact on actual screening practices—that is, fewer eligible children were screened in the months leading up to electronic records implementation. We believe that this was due to both the preparation required for the transition (for example, off-site training for all staff and extraction of medical record data) and the changes in nursing management. Preparatory efforts likely distracted staff's attention to electronic records-related issues, and this distraction had a cumulative impact during the roll-out period. However, a leadership void may have contributed to a progressive effect, because providers also lost administrative feedback on screening rates.
The second phase of declines in screening was during the “go-live” implementation phase. Factors affecting this phase included a host of organizational changes necessary to adjust to the electronic records. For example, the electronic records system changed front-desk procedures, which may have affected the patient's receipt of the PSC. Previously, children coming for well-child care were identified by the paper forms attached to the medical record, but once the electronic records went live, the system switched to a paperless system, and the visual cue was lost. Systems had to be developed to identify which children needed a behavioral health screen, and these systems were monitored for improvement. Implementation proceeded in this trial-and-error manner until improvement was seen.
Clinicians should think very carefully about how cues and alerts will be managed and maintained during the planning and training phase of electronic records implementation. Whatever conventions are developed to identify the particular needs of individual patients, they will likely need to be customized to the workflow of each practice. This need for customization challenges the assumption of a national approach to electronic records implementation that uses commercial systems that are difficult to adapt to local practice styles or needs, which was a major reason for the failure and abandonment of the United Kingdom's national health information technology program (42).
Second, the process of shifting data-capture tasks from a paper to an electronic health record resulted in poorer documentation because data from some paper PSCs were not being entered into the electronic records. Only after processes were adapted, including moving documentation responsibilities from physicians to medical assistants, did documentation rates improve slowly over three years. Medical assistants may be particularly important in maintaining patient flow and satisfaction during electronic records implementation (43,44). Our guess is that handoff breakdowns occurred at many levels: between front desk and patient, between patient and physician, and between physician and electronic record. Further research is needed to identify the actual step-by-step influences that this transition has on a practice setting and documentation protocols.
The persistence of reduced screening rates at this practice also appears to be consistent with the findings of previous studies. Impacts of transitioning from a paper to an electronic records environment noted in this study have also been documented by others, including negative impacts on workflow (20), increased patient cycle time (21), and communication breakdowns between clinicians (12). Bloom and Huntington (45) reported that the average time spent documenting during a clinical visit did not decrease between initial electronic records implementation and follow-up 12 months later. Similarly, Samaan and colleagues (21) found that patient cycle time (time spent in the entire visit) did not return to baseline for two-and-a-half years after electronic records implementation. Our findings further support the evidence that switching to an electronic records system has a major impact on a practice, and recovery time may require as many as three years of adaptation.
Transitional issues may have been avoided if the information technology system had been capable of providing information about which children required screens and whether screening had been done within a prescribed time frame. A well-constructed information technology program should be able to prompt patients or patients' parents to enter the screening information directly into an automated system and monitor completion. Information technology systems that require the type of human effort noted in this implementation are doomed to degradation, at least temporarily, until staff learn how to make up for the system's deficiencies. Unfortunately, the costs of and multiple demands for information technology programming during electronic records implementation make the perfect information technology system elusive.
The results of this analysis may serve as a warning about what may happen to health care quality as electronic health records are rolled out across the country. Quality of care (as measured by compliance rates) may decrease as clinicians and staff adapt to new information technology. Reducing clinicians' workload during electronic records implementation may not be enough to avoid lapses in providing mandatory care. In the practice we studied, electronic records changed how patients were identified for particular services, eliminated visual cues to prompt tasks, and altered processes by which information was entered into a medical record.
This study of a single heterogeneous practice made up of only seven pediatricians in the Northeast may not be generalizable to other practices elsewhere. However, we believe that the issues encountered may be generalizable as practices nationwide transition to electronic health records, and few investigators have studied the unintended outcomes of electronic records (22). In addition, no control group was available. It is therefore possible that other factors may have influenced how screening was being conducted during this time; however, practice administrators did not report this.
Conclusions
The declines in behavioral health screening resulting from the study practice's electronic records adoption were highly disruptive and took almost three years to resolve completely. Compliance with primary care Healthcare Effectiveness Data and Information Set standards and Medicaid performance measures will likely be similarly affected. Although implementing a fully automated medical record may provide some benefits, the challenges inherent in implementation must be acknowledged. The effects of electronic health records on patient behavioral health outcomes is still unclear.
1 American Recovery and Reinvestment Act of 2009, Public Law No 111-5, Feb 17, 2009, 123 Stat 115 Google Scholar
2 : Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC, National Academies Press, 2001 Google Scholar
3 : Impact of clinical alerts within an electronic health record on routine childhood immunization in an urban pediatric population. Pediatrics 120:707–714, 2007 Crossref, Medline, Google Scholar
4 : Impact of electronic health record-based alerts on influenza vaccination for children with asthma. Pediatrics 124:159–169, 2009 Crossref, Medline, Google Scholar
5 : Electronic health record decision support and quality of care for children with ADHD. Pediatrics 126:239–246, 2010 Crossref, Medline, Google Scholar
6 : The impact of eHealth on the quality and safety of health care: a systematic overview. PLoS Medicine 8:e1000387, 2011 Crossref, Medline, Google Scholar
7 : Evaluation of outpatient computerized physician medication order entry systems: a systematic review. Journal of the American Medical Informatics Association 14:400–406, 2007 Crossref, Medline, Google Scholar
8 : Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 293:1223–1238, 2005 Crossref, Medline, Google Scholar
9 : Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 116:1506–1512, 2005 Crossref, Medline, Google Scholar
10 : Decrease in hospital-wide mortality rate after implementation of a commercially sold computerized physician order entry system. Pediatrics 126:14–21, 2010 Crossref, Medline, Google Scholar
11 : Study correlates CPOE with decreased mortality rates. HealthcareITNews, May 3, 2010. Available at www.healthcareitnews.com/news/study-correlates-cpoe-decreased-mortality-rates Google Scholar
12 : Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. Journal of the American Medical Informatics Association 11:104–112, 2004 Crossref, Medline, Google Scholar
13 : Some unintended consequences of clinical decision support systems; in American Medical Informatics Association Annual Symposium 2007: Biomedical and Health Informatics: From Foundations to Applications to Policy. Edited by Teich JM. Red Hook, NY, Curran Associates, 2007 Google Scholar
14 : Exploring the unintended consequences of computerized physician order entry. Studies in Health Technology and Informatics 129:198–202, 2007 Medline, Google Scholar
15 : Implementation of computerized physician order entry in seven countries. Health Affairs 28:404–414, 2009 Crossref, Google Scholar
16 : Role of computerized physician order entry systems in facilitating medication errors. JAMA 293:1197–1203, 2005 Crossref, Medline, Google Scholar
17 : The impact of electronic health records on time efficiency of physicians and nurses: a systematic review. Journal of the American Medical Informatics Association 12:505–516, 2005 Crossref, Medline, Google Scholar
18 : Electronic health record use and the quality of ambulatory care in the United States. Archives of Internal Medicine 167:1400–1405, 2007 Crossref, Medline, Google Scholar
19 : Relationship between use of electronic health record features and health care quality: results of a statewide survey. Medical Care 48:203–209, 2010 Crossref, Medline, Google Scholar
20 : Perceived efficiency impacts following electronic health record implementation: an exploratory study of an urban community health center network. International Journal of Medical Informatics 79:807–816, 2010 Crossref, Medline, Google Scholar
21 : The impact of the electronic health record on an academic pediatric primary care center. Journal of Ambulatory Care Management 32:180–187, 2009 Crossref, Medline, Google Scholar
22 : Anticipating and addressing the unintended consequences of health IT and policy: a report from the AMIA 2009 Health Policy Meeting. Journal of the American Medical Informatics Association 18:82–90, 2011 Crossref, Medline, Google Scholar
23 Strategies to Support the Integration of Mental Health Into Pediatric Primary Care. Issue Paper. Washington DC, National Institute for Health Care Management, 2009. Available at nihcm.org/pdf/PediatricMH-FINAL.pdf Google Scholar
24 : Suicide and suicide attempts in adolescents. Pediatrics 105:871–874, 2000 Crossref, Medline, Google Scholar
25 Achieving the Promise: Transforming Mental Health Care in America. Pub no SMA-03-3832. Rockville, Md: Department of Health and Human Services, President's New Freedom Commission on Mental Health, 2003 Google Scholar
26 : Identification and management of psychosocial problems by preventive child health care. Archives of Pediatrics and Adolescent Medicine 155:462–469, 2001 Crossref, Medline, Google Scholar
27 : Screening for psychosocial dysfunction in pediatric practice: a naturalistic study of the Pediatric Symptom Checklist. Clinical Pediatrics 31:660–667, 1992 Crossref, Medline, Google Scholar
28 : Adolescent behavioral risk screening and use of health services. Administration and Policy in Mental Health 36:374–380, 2009 Crossref, Medline, Google Scholar
29 : Screening, triage, and referral of patients who report suicidal thought during a primary care visit. Pediatrics 125:945–952, 2010 Crossref, Medline, Google Scholar
30 : Screening for Child and Adolescent Depression in Primary Care Settings: A Systematic Evidence Review for the US Preventive Services Task Force. Rockville, Md, Agency for Healthcare Research and Quality, 2009 Google Scholar
31 : Mental health screening in pediatric practice: factors related to positive screens and the contribution of parental/personal concern. Pediatrics 118:1896–1906, 2006 Crossref, Medline, Google Scholar
32 : Mental and behavioral health screening at preventive visits: opportunities for follow-up of patients who are nonadherent with the next preventive visit. Journal of Pediatrics 158:666–671, 2011 Crossref, Medline, Google Scholar
33 : Persistence and change in pediatric symptom checklist scores over 10 to 18 months. Academic Pediatrics 9:270–277, 2009 Crossref, Medline, Google Scholar
34 Rosie D v Romney, F Supp 2d 18 (D Mass 2006) Google Scholar
35 Social Security Act Amendments of 1965. Public Law 89–97, July 30, 1965, 79 Stat 286, Section 1900–1946 (1965) Google Scholar
36 : Use of the Pediatric Symptom Checklist in strategies to improve preventive behavioral health care. Psychiatric Services 52:800–804, 2001 Link, Google Scholar
37 : Comparing three short questionnaires to detect psychosocial dysfunction among primary school children: a randomized method. BMC Public Health 9:489, 2009 Crossref, Medline, Google Scholar
38 : Segmented regression analysis of interrupted time series studies in medication use research. Journal of Clinical Pharmacy and Therapeutics 27:299–309, 2002 Crossref, Medline, Google Scholar
39 : Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston, Houghton Mifflin, 2002 Google Scholar
40 SAS OnlineDoc, version 9.2. Cary, NC, SAS Institute Inc, 2006 Google Scholar
41 : A knowledge-based taxonomy of critical factors for adopting electronic health record systems by physicians: a systematic literature review. BMC Medical Informatics and Decision Making 10:60, 2010 Crossref, Medline, Google Scholar
42 : Don't Repeat the UK's Electronic Health Records Failure. Huffington Post, Dec 1, 2010. Available at www.huffingtonpost.com/stephen-soumerai/dont-repeat-the-uks-elect_b_790470.html Google Scholar
43 : Practice profile: transforming the role of medical assistants in chronic disease management. Health Affairs 29:963–965, 2010 Crossref, Google Scholar
44 : Medical assistants: the invisible “glue” of primary health care practices in the United States? Journal of Health Organization and Management 24:288–305, 2010 Crossref, Medline, Google Scholar
45 : Faculty, resident, and clinic staff's evaluation of the effects- of EHR implementation. Family Medicine 42:562–566, 2010 Medline, Google Scholar
Figures and Tables
Figure 1 Trends in the pediatric behavioral health screening rate during transition to electronic health records (EHRs) in a large practice, 2003 to 2008
Table 1 Segmented regression coefficients
