The American Recovery and Reinvestment Act of 2009 calls for the use of an electronic health record for each person in the United States by 2014 (1). This legislation is based on the assumption that electronic health records can improve the safety, timeliness, effectiveness, and efficiency of care (2). Health information technology has been shown to improve the quality of care in some circumstances. For example, embedded alert systems have increased the timeliness of childhood immunization and overall rates of immunization (3,4), and an electronic record add-on template for attention-deficit hyperactivity disorder was reported to have increased follow-up visits and documentation of symptoms (5). However, systematic reviews have concluded that the impact of electronic health records on improving clinical outcomes is unproven (6–8).
The problems and harms associated with electronic health records are also well documented. One report of the effects of computerized physician order entry showed an increase in pediatric mortality after a change in electronic health record software in a Pittsburgh, Pennsylvania, teaching hospital (9). Although a more recent time series analysis reported a decrease in hospitalwide mortality (10), this association may have been confounded by competing interventions (for example, process and workflow changes, adjustments in intensive care unit staffing, and rollout of rapid response teams) (11). Ash and colleagues (12–14) have reported many unintended negative consequences of electronic records, such as increased time necessary to enter data, fragmentation of information across screens or tabs, inflexible processes, and physician overload. Several rigorous studies have reported no improvement in quality of care associated with the adoption of electronic records or even a deterioration in quality of care (9,15–19). Transitioning from a paper to an electronic environment is also well known to be associated with disruptions in workflow (20), increased patient cycle time (21), breakdowns in communication between clinicians (12), the risk of computer system crashes, and lack of communication between different electronic records systems.
Despite the increasing evidence of electronic records' unintended effects (22), no studies have reported how specific pediatric screening or disease management processes may be affected by the transition from a paper to an electronic environment. As electronic records systems are implemented or modified across the country, clinicians and insurers may experience declines in the quality of care (as measured by compliance with mandated services) associated with the disruption of workflow and changes in protocols (17).
The purpose of this study was to determine whether the transition from a paper to an electronic health record affected behavioral health screening in a large Northeastern pediatric practice setting. The increasing prevalence of mental illness among children and the scarcity of child mental health professionals suggest that pediatricians must improve detection and treatment of mental disorders in primary care settings (23). National organizations (24–27) have recommended such behavioral health screening in pediatric primary care to address this need. Behavioral health screens improve identification rates and referral numbers and may affect timeliness of care (28–31). Behavioral health screening and related increases in referrals are also associated with improved behavioral health outcomes measured at subsequent preventive care visits (for example, lower Pediatric Symptom Checklist [PSC] scores) (32,33). Thus, the positive and negative impacts of health information technology on behavioral health screening merit further attention.
This study received approval from the Cambridge Health Alliance Institutional Review Board.
In 2002, a class-action lawsuit Rosie D v. Romney (34) was filed in Massachusetts on behalf of eight children with serious psychiatric disabilities who were hospitalized or at risk for hospitalization owing to the state's failure to provide needed services to allow them to live at home and attend local schools. In 2007, the state was found in violation of the Medicaid Act (35). The court's decision required all primary care providers caring for MassHealth-enrolled children younger than age 21 whose benefit package includes primary care service to begin using a clinically appropriate behavioral health screening tool at well-child visits.
At a large pediatric clinic site, behavioral health screening of children at well-child visits began in December 2003, five years before the initiation of the mandatory statewide program. The clinic, which is part of a large public health care system, has seven pediatricians and serves about 6,000 patients, of whom about 33% are younger than age five. On average, 180 children are eligible for behavioral health screening each month. Although this clinic is an academic teaching site, pediatricians are generally scheduled to see approximately 25 patients a day in 15-minute intervals. The population is economically and linguistically diverse (54% nonwhite, 40% publicly insured or self-pay, and 31% non-English speakers).
Screening was conducted using the PSC (for children age four years, 11 months, to age 16) and the PSC-Youth Report (Y-PSC) (for children ages 14–18), both of which are validated tools that have been translated into multiple languages (36,37). On both tools, summary scores above a certain cut point are considered positive screens for behavioral difficulties.
The screening process began with a family member (on behalf of a child patient) or a teenage patient filling out the paper PSC or Y-PSC in the waiting room before the visit. They would then bring the paper form to the physician, who would discuss it with them and score the instrument. Before implementation of the electronic records system, providers inserted the completed paper form into the medical record. Over the course of the first year of screening, nurse management staff and clinical leaders developed processes to enhance screening. Nurse management staff reviewed charts the night before to identify children who were eligible for screening visits. They then attached the appropriate PSC or Y-PSC to the chart for distribution by the receptionist when the patient arrived. In addition, the nurse manager was given regular updates on compliance and worked with front-office staff to improve distribution and with medical assistants to retrieve forms.
Practice changes during electronic record implementation
The go-live date for using the electronic health records system (the EpicCare EMR system) at this practice was March 29, 2005, but planning for implementation began six months earlier. Approximately two months before the go-live date, all staff went to full EpicCare off-site training (40 hours each). Training occurred in a rolling manner with different elements so as not to overwhelm staff. To ease transition, patient visits were reduced by 50% in the first week and by 25% in the second week after electronic records system implementation so providers could adjust to the system. In-house trainers were also on site to help with implementation. To capture screening information electronically, a questionnaire for entering paper-based PSC data was programmed into the electronic records system in June 2005.
The PSC screening process remained the same, with paper forms used throughout; however, once the practice began using electronic records, the results from the paper PSC were entered into the electronic records system via an electronic questionnaire with four required fields. These fields included the total score, responses to two questions—is your child (you) currently seeing a mental health counselor and does your child (you) have any emotional or behavioral problems for which she or he (you) needs help—and indication of disposition (annual visit, return to primary care, or referral to counseling). Initially, the physicians did this themselves; later, the task was transferred to medical assistants, who also scanned the actual PSC form into the system. Reports were developed to provide monthly compliance rates by individual physicians to assist management with process improvement.
To measure other contextual factors that might have affected screening rates and the transition to electronic records, we conducted a review of past e-mail communication, minutes of implementation meetings, and reports and held discussions with two practice administrators to capture any environmental changes (such as staffing, scheduling, operations, and training information) that took place during the transitional period (October 2004 through March 2005). In addition, these individuals were asked to describe the major challenges to electronic records implementation.
To determine rates of screens, we drew monthly data on preventive care-eligible visits from the MEDITECH system. Until March 2007, copies of PSCs were collected weekly, and data were entered into a database. The crude screening rate was calculated by dividing the number of paper screens by the number of eligible visits. Once the practice began entering data from the paper screens into the electronic questionnaire, a monthly compliance report was also generated from the MEDITECH system to determine screening results in the electronic records and identify the number of screens by provider by month. For the purposes of this study, we used the highest number of screens reported from either paper or electronic data until March 2007. Data were available only from the electronic records after March 2007 because the paper form was discarded after data entry.
We used an interrupted times series design to evaluate the impact of electronic records implementation on behavioral health screening rates (38). Interrupted time series is the strongest quasi-experimental design for evaluating the effects of natural experiments (39). Controlling for the baseline rate and trend of behavioral health screening, we estimated changes in the level and slope of the screening rate during the electronic records planning period and after implementation. The regression equation was specified as
We used maximum likelihood estimation to estimate the models and controlled for autocorrelation by including all significant autocorrelation parameters up to 12 months. Backward elimination was used to include covariates in the final model with a type 1 error threshold for inclusion of .05. All statistical analyses were performed using PROC AUTOREG in SAS, version 9.2 (40).
Behavioral health screening rates
Figure 1 shows the baseline, training, and post-electronic records implementation behavioral health screening rates. The rate of screening increased during the baseline period (December 2003 through September 2004), beginning at just below 70% in December 2003 and climbing to 91% in September 2004. The average screening rate during the baseline period was 81%. In the six-month training period (October 2004–March 2005) before electronic records implementation, rates of screening dropped precipitously and significantly from 84% to 64% (p<.001). The switch from paper to electronic records on March 29, 2005, was associated with a further drop in screening rates. Only 50% of eligible youths were screened in the first month of electronic-based records. Rates recovered slowly over three years, climbing to 82% by April 2008, matching baseline rates. The monthly behavioral health screening rate also became much more variable after EHR implementation.
Table 1 shows the results of the segmented regression analysis. Both the time-after-training and the time-after-implementation trend coefficients were significant. The six-month training period was associated with a 5% monthly decrease in the adjusted behavioral health screening rate (p=.002). Behavioral health screening rates increased by 3% monthly (p=.006) between May 2005 and April 2008.
In our review of e-mail, minutes, and reports and our conversations with practice management, we identified several important challenges that occurred simultaneously with EpicCare implementation. First, two months before implementation, the practice experienced the departure of the nurse manager and several other personnel. The nurse manager had been leading the electronic records roll-out, and with her departure several other people became more involved with the implementation.
Second, according to staff, transitioning from paper to electronic records proved difficult. After the transition, the PSC form was the only paper form that was handed out to parents of children coming in for their well-child visits. Front office staff tended to forget to hand out the PSC forms, particularly on days when there were few eligible visits. In addition, identifying children who met criteria was sometimes confusing. Thus the percentage of hard-copy PSCs collected for eligible visits dropped substantially, as seen in Figure 1. Later, health assistants were asked to back up front-office staff and get a form if the parent did not carry one into the exam room.
On the provider side, many providers lacked computer skills and had trouble keeping up with their workload during the transition. The additional burden of inputting PSC data into the electronic questionnaire proved problematic. As the medical director of the clinic noted, “This computerized questionnaire is new to us here … so I am not too surprised the completion rate is so poor.” Later, it became apparent that providers were entering the information via addendums after the visits were closed, and it was eventually decided that the medical assistants would be responsible for inputting the PSC data into the electronic questionnaire. This process significantly improved screening documentation. With the implementation of the Rosie D mandate (January 2008), additional changes were made for the purpose of billing, including scanning forms and using specific billing codes.
Implementing an electronic health records system is a complex process that demands attentive leadership engaged in ongoing solutions as providers adjust to the change (20,41). In addition, the impact of this transition on provider behavior, practice operations, and administrative procedures cannot be underestimated.
In this study, we observed two negative phases in the changeover to electronic records. First, the process of transitioning to an electronic records system, or preimplementation, had an impact on actual screening practices—that is, fewer eligible children were screened in the months leading up to electronic records implementation. We believe that this was due to both the preparation required for the transition (for example, off-site training for all staff and extraction of medical record data) and the changes in nursing management. Preparatory efforts likely distracted staff's attention to electronic records-related issues, and this distraction had a cumulative impact during the roll-out period. However, a leadership void may have contributed to a progressive effect, because providers also lost administrative feedback on screening rates.
The second phase of declines in screening was during the “go-live” implementation phase. Factors affecting this phase included a host of organizational changes necessary to adjust to the electronic records. For example, the electronic records system changed front-desk procedures, which may have affected the patient's receipt of the PSC. Previously, children coming for well-child care were identified by the paper forms attached to the medical record, but once the electronic records went live, the system switched to a paperless system, and the visual cue was lost. Systems had to be developed to identify which children needed a behavioral health screen, and these systems were monitored for improvement. Implementation proceeded in this trial-and-error manner until improvement was seen.
Clinicians should think very carefully about how cues and alerts will be managed and maintained during the planning and training phase of electronic records implementation. Whatever conventions are developed to identify the particular needs of individual patients, they will likely need to be customized to the workflow of each practice. This need for customization challenges the assumption of a national approach to electronic records implementation that uses commercial systems that are difficult to adapt to local practice styles or needs, which was a major reason for the failure and abandonment of the United Kingdom's national health information technology program (42).
Second, the process of shifting data-capture tasks from a paper to an electronic health record resulted in poorer documentation because data from some paper PSCs were not being entered into the electronic records. Only after processes were adapted, including moving documentation responsibilities from physicians to medical assistants, did documentation rates improve slowly over three years. Medical assistants may be particularly important in maintaining patient flow and satisfaction during electronic records implementation (43,44). Our guess is that handoff breakdowns occurred at many levels: between front desk and patient, between patient and physician, and between physician and electronic record. Further research is needed to identify the actual step-by-step influences that this transition has on a practice setting and documentation protocols.
The persistence of reduced screening rates at this practice also appears to be consistent with the findings of previous studies. Impacts of transitioning from a paper to an electronic records environment noted in this study have also been documented by others, including negative impacts on workflow (20), increased patient cycle time (21), and communication breakdowns between clinicians (12). Bloom and Huntington (45) reported that the average time spent documenting during a clinical visit did not decrease between initial electronic records implementation and follow-up 12 months later. Similarly, Samaan and colleagues (21) found that patient cycle time (time spent in the entire visit) did not return to baseline for two-and-a-half years after electronic records implementation. Our findings further support the evidence that switching to an electronic records system has a major impact on a practice, and recovery time may require as many as three years of adaptation.
Transitional issues may have been avoided if the information technology system had been capable of providing information about which children required screens and whether screening had been done within a prescribed time frame. A well-constructed information technology program should be able to prompt patients or patients' parents to enter the screening information directly into an automated system and monitor completion. Information technology systems that require the type of human effort noted in this implementation are doomed to degradation, at least temporarily, until staff learn how to make up for the system's deficiencies. Unfortunately, the costs of and multiple demands for information technology programming during electronic records implementation make the perfect information technology system elusive.
The results of this analysis may serve as a warning about what may happen to health care quality as electronic health records are rolled out across the country. Quality of care (as measured by compliance rates) may decrease as clinicians and staff adapt to new information technology. Reducing clinicians' workload during electronic records implementation may not be enough to avoid lapses in providing mandatory care. In the practice we studied, electronic records changed how patients were identified for particular services, eliminated visual cues to prompt tasks, and altered processes by which information was entered into a medical record.
This study of a single heterogeneous practice made up of only seven pediatricians in the Northeast may not be generalizable to other practices elsewhere. However, we believe that the issues encountered may be generalizable as practices nationwide transition to electronic health records, and few investigators have studied the unintended outcomes of electronic records (22). In addition, no control group was available. It is therefore possible that other factors may have influenced how screening was being conducted during this time; however, practice administrators did not report this.
The declines in behavioral health screening resulting from the study practice's electronic records adoption were highly disruptive and took almost three years to resolve completely. Compliance with primary care Healthcare Effectiveness Data and Information Set standards and Medicaid performance measures will likely be similarly affected. Although implementing a fully automated medical record may provide some benefits, the challenges inherent in implementation must be acknowledged. The effects of electronic health records on patient behavioral health outcomes is still unclear.
This study was supported in part by a grant through the Integrated Health and Behavioral Health Care for Children, Adolescents and Their Families grant program in the Health Resources and Services Administration's Maternal and Child Health Bureau. Partial support also came from the Harvard Pilgrim Health Care Foundation, the National Institute of Mental Health, and the Health Maintenance Organization Mental Health Research Network (grant 1U19MH092201-01), as well as from the Health Maintenance Organization Research Network Centers for Education and Research on Therapeutics, which is supported by the Agency for Healthcare Research and Quality (grant U18HS010391) and the Harvard Pilgrim Health Care Institute. Additional consultative support came from Harvard Catalyst and the Harvard Clinical and Translational Science Center (National Institutes of Health award UL1 RR 025758). Additional financial contributions were from Harvard University and its affiliated academic health care centers. The content is solely the responsibility of the authors and does not necessarily represent the official views of Harvard Catalyst, Harvard University and its affiliated academic health care centers, the National Center for Research Resources, or the National Institutes of Health. The authors acknowledge David Link, M.D., and the clinic staff who provided detailed information on implementation of EpicCare.
The authors report no competing interests.