Trial of an Electronic Decision Support System to Facilitate Shared Decision Making in Community Mental Health
Abstract
Objectives:
Involvement of community mental health consumers in mental health decision making has been consistently associated with improvements in health outcomes. Electronic decision support systems (EDSSs) that support both consumer and provider decision making may be a sustainable way to improve dyadic communication in a field with approximately 50% workforce turnover per year. This study examined the feasibility of such a system and investigated proximal outcomes of the system's performance.
Methods:
A cluster randomized design was used to evaluate an EDSS at three urban community mental health sites. Case managers (N=20) were randomly assigned to the EDSS-supported planning group or to the usual care planning group. Consumers (N=80) were assigned to the same group as their case managers. User satisfaction with the care planning process was assessed for consumers and case managers (possible scores range from 1 to 5, with higher summary scores indicating more satisfaction). Recall of the care plan was assessed for consumers. Linear regression with adjustment for grouping by worker was used to assess satisfaction scores. A Wilcoxon rank-sum test was used to examine knowledge of the care plan.
Results:
Compared with case managers in the control group, those in the intervention group were significantly more satisfied with the care planning process (mean±SD score=4.0±.5 versus 3.3±.5; adjusted p=.01). Compared with consumers in the control group, those in the intervention group had significantly greater recall of their care plans three days after the planning session (mean proportion of plan goals recalled=75%±28% versus 57%±32%; p=.02). There were no differences between the clients in the intervention and control groups regarding satisfaction.
Conclusions:
This study demonstrated that clients can build their own care plans and negotiate and revise them with their case managers using an EDSS. (Psychiatric Services 62:54–60, 2011)
Shared decision making has been defined as a process by which clinician and client engage in a two-way communication of information, clarify values and influences regarding a treatment decision, and agree on a plan of action (1). Shared decision making has been advocated by influential reports (2,3) and is congruent with broader philosophical and ethical perspectives on the treatment of people diagnosed as having severe mental illness (4–6). Furthermore, a participatory model of decision making is a central element of several evidence-based psychosocial mental health treatments (6–8) and a core component of the Chronic Care Model (9). Two European randomized controlled trials and one recent meta-analysis showed that when client preferences for mental health care are queried and honored, outcomes are better along several dimensions (10–12). Finally, for many mental health care decisions, no one course of action has emerged as the optimal clinical path to recovery from severe mental illness.
At the same time, several decades of research have indicated that mental health clients and their case managers have divergent opinions regarding prioritization of goals, services, and outcomes of care (13–25). Why this discordance exists is uncertain. Case managers may presume that their clients want the same services that they would choose themselves (16). Clients of mental health services may have difficulty expressing differences of opinion with their case managers and may not tell their case managers when they disagree with a course of treatment (26).
Regardless of the reasons for this discordance, information technology that facilitates communication within the dyad may offer clients more involvement in self-management of their mental illness. Considering a mental health workforce that turns over at a rate of approximately 50% per year (27–29) and is made up of 40% of mental health workers without advanced clinical training (30), electronic decision support may be a practical way to facilitate shared decision making and communication of the client perspectives. Electronic decision support can provide a structural support to both client and clinician in the care planning process (4).
This study examined whether use of an electronic decision support system (EDSS) to create a shared-decision-making plan will lead to greater client and case manager satisfaction with the care planning process than the usual agency procedures for creating a care plan. Also investigated was whether clients are more engaged in the process of care planning, as measured by recall of the goals included in the care plan three days after the planning session.
Methods
Design
This study involved a cluster randomized controlled trial of a new EDSS designed to support greater client involvement in goal setting and to assist both clients and case managers in engaging in shared decision making. [A depiction of the study design is available as an online supplement at ps.psychiatryonline.org.] Case managers from three clinics were randomly assigned to the intervention group or treatment as usual. Clients were assigned to the same group to which their case manager was assigned. Because of the cluster randomization, all case managers participated multiple times, but all clients participated only one time. Because one clinic was larger than the others, one clinic recruited approximately one-half the participants and the other two recruited approximately one-quarter each. Two of the clinics provided traditional case management and community support. One clinic provided slightly more intensive services, because it previously provided assertive community treatment and continued to serve the same population. The participating agency the clinics belong to espouses a mission to help mental health clients maintain autonomy over their lives and achieve recovery-oriented goals (Wilkniss S, personal communication, 2009). Data were collected between September 2008 and January 2009.
For the intervention, a three-step EDSS process inverts the usual care planning procedures. First, clients indicate their top priorities and ideas for services at a touchscreen-enabled computer kiosk. Second, the information is electronically sent to the clients' case managers, who then complete a similar process. Finally, the two perspectives are merged electronically and presented graphically in a shared decision-making session with the dyad. [An appendix showing an example of EDSS screens is available as an online supplement at ps.psychiatryonline.org.]
The EDSS was intentionally designed so there would be minimal training burden on both clients and case managers. Clients in the study had not been exposed to the EDSS before. Case managers were given a brief manual and a one-hour didactic and practice session, in which they were able to try out the technology and ask questions. These sessions were conducted by the first author once for each site.
Clients who were assigned to receive usual care planning met with their case managers at the time that their six-month care plans were due. Case managers and clients were told to complete their care plans in the way that they normally would. Care plan discussions were audio recorded, but no research staff member was present. The usual care plan included use of a behavioral health electronic medical record designed to aid with billing and to theoretically help case managers create recovery-oriented care plans. There appeared to be significant heterogeneity in how care plans were completed, as is the case in real-world case management.
Clients who were assigned to the intervention group met with their case managers a minimum of three months before their regularly scheduled six-month care plans were due, so they could create care plans with the EDSS without worry that finishing the six-month care plan would bias the EDSS results. Clients in the intervention group eventually completed their six-month care plan in the usual way for billing purposes; however, they were able to use the intervention care plan as a guide.
Measures
Two questionnaires were developed for this study, one on the case manager satisfaction with the care planning process and the other on client satisfaction with the care planning process. The case manager version contains six statements related to the dimensions of case manager-client communication, time, flow, credibility as a clinical tool, and organization of information. The statements were rated on a 5-point Likert scale, ranging from 1, strongly disagree, to 5, strongly agree; higher summary scores indicate more satisfaction. The Cronbach's alpha for the entire scale was .74.
The case manager satisfaction measure asks case managers to rate their satisfaction with each care planning encounter, instead of their overall satisfaction with the care planning process. Pilot testing indicated that case managers strongly felt that the EDSS would work well for some clients but not others.
The client version contains seven statements rated by the client on a 5-point Likert scale, ranging from 1, strongly disagree, to 5, strongly agree; higher summary scores indicate more satisfaction. Based on the literature regarding what makes clients satisfied with a mental health encounter, the items were designed to assess case manager-client communication, involvement in decision making, being informed about decisions that were made, and having a clear management plan (31). The questionnaire also asks which types of information were communicated for clients who state that the EDSS helped them communicate important information about themselves to their counselor. The Cronbach's alpha for the entire scale was .62.
Clients were asked about their knowledge of the care plan two to four days after the care planning session. The researcher first asked the client to note which goal areas appeared in the care plan. The client was then provided with a list of goal areas not already indicated by the client and asked whether they were in the care plan. For each area correctly identified in either manner, the client was asked to relate what his or her individualized goals or objectives or shared decisions (as applicable) were within each quality-of-life area. All measures except the client knowledge of care plan were completed immediately after participation in the study.
Case managers were randomly assigned to intervention or control conditions within each of three clinics that were part of the same agency. Three clinics were necessary to achieve minimal sample size because clients could participate only if they were scheduled for a regularly occurring six-month care plan. Because case managers were randomly assigned to groups within sites instead of by site, no imbalances due to site characteristics should be present. Twenty case managers at the three clinics participated in the study with an average of four clients each. Clients were clustered by case manager, so all clients assigned to a particular case manager participated in the same condition. Clients participated only one time. Thus 40 dyads were exposed to the prototype EDSS at the time of their biannual case management planning review, and 40 dyads were in the control group and completed their usual care planning sessions. Researchers collected demographic, clinical, and workforce data before the planning sessions.
Both client and case manager questionnaires on satisfaction with the planning process were administered with a self-report computerized format. For the client questionnaire, a research assistant was present to provide assistance as needed. The knowledge of care plan questionnaire was administered by a research assistant over the telephone or in person at the mental health center if the client did not have access to a telephone. Each client was asked to recall which broad areas the care plan or shared-decision-making plan focused on. The proportion of the care plan goals that were recalled by clients was calculated by summing the number of care plan goals appearing in the care plan for the denominator, and summing the number of correctly recalled goals for the numerator. The maximum denominator was 11 goals.
The Dartmouth Committee for the Protection of Human Subjects and the agency's internal institutional review board approved the study. The study was deemed to be of minimal risk to participants. Full informed consent was obtained for research participants by the study research assistant.
Data analysis
Descriptive statistics were used to characterize the case manager and client samples. The differences between the intervention and the control groups' summary scores and individual items for the client and case manager questionnaires on satisfaction with the care planning process were examined via linear regression. Linear regression was also used to control for case manager age in the model that used the questionnaire on case manager satisfaction with the care planning process and to control for years working with the case manager in the model that used the questionnaire on client satisfaction with the care planning process.
Intraclass correlation coefficients (ICCs) were computed within case managers for all questionnaires and to adjust for clustering as appropriate (32,33). The ICCs indicated that case manager responses regarding individual sessions were highly correlated within individual counselors and required clustering of standard errors (ICC=.65) but that client responses did not (knowledge of care planning, ICC=.03; client satisfaction with the care planning process, ICC=.10) (34).
Because of a nonnormal distribution in the variable for client knowledge of the care plan, a nonparametric two-sample Wilcoxon rank-sum (Mann-Whitney U) test was used to assess the hypothesis that clients creating their care plans using the EDSS would have significantly greater knowledge of the goals in their care plans than clients in the usual care group.
Results
Table 1 shows the demographic and workforce characteristics of the case managers (N=19). (Twenty case managers participated; however, demographic data were missing for one of the case managers in the control group.) The workforce and demographic characteristics were diverse. Although case managers were randomly assigned within site, case managers in the control group were significantly younger than case managers in the intervention group (31 years versus 47 years; p<.001). This led to an imbalance in both years of total mental health experience and years worked at the agency.
Table 2 shows the demographic and clinical characteristics of the clients who participated (N=80). Clients in the intervention and control groups were balanced on all variables except for length of time working with the case manager, which is related to the randomization failure on case manager age. The total sample was diagnostically and demographically heterogeneous.
Case managers rated how satisfied they were with the care planning process for each client immediately after the care planning session. Case managers responded for all 40 intervention sessions and 38 control group sessions. The mean summary score for the case manager satisfaction with the care planning process questionnaire was 4.0±.5 for the intervention group and 3.3±.5 for the control group. The multiple linear regression, controlling for counselor age in years, showed that intervention status significantly and positively predicted a higher summary score (intercept=3.29, β=.62, adjusted p=.01). Age of the case manager was not significant in the model. The model explained approximately 30% of the variance in the summary score (p<.001).
Items were also examined with raw means and frequencies, and then each item was analyzed with multiple linear regression that controlled for case manager age, with the standard errors clustered by case manager (Table 3). For all questions except the two that were intended to access information regarding “credibility as a clinical tool,” the mean scores for the intervention group were significantly higher (or lower, if an item was reverse-coded) than the mean scores for the control group.
For the questionnaire on client satisfaction with the care planning process, the intervention and control groups were virtually identical on mean summary score (3.88±.54 in the intervention group versus 3.78±.56 in the control group). In the linear model only client experience with the case manager revealed a clinically insignificant (but statistically significant) negative effect—the longer clients were with their case mangers (in months), the more dissatisfied they were (β=–.003, adjusted p<.001). The fit of the model was not significant and explained little of the variance in score.
There were no statistically significant differences between the intervention and control groups on any individual items on the questionnaire on the client satisfaction with the care planning process. A clear ceiling effect (or floor effect, in the case of the reverse-coded items) was observed for four out of the seven items. Still, a substantial proportion of the clients in both the intervention and control groups said that they wished they had more of an opportunity to discuss something on their minds with their counselors before creating their plans and were not sure what they would be working on with their counselors in the coming months (Table 4).
Sixty-nine (86%) of the 80 clients were successfully contacted two to four days after participation in the care planning sessions. Thirty-six clients (90%) in the control group and 33 clients (83%) in the intervention group were contacted. Clients in the intervention group had significantly greater knowledge of their care plan goals than the clients in the control group (mean proportion of plan goals recalled=75%±28% versus 57%±32%; z=−2.367, p=.02). There were no significant differences in participants' incorrect recall of goals between the intervention and control groups (mean proportion of plan goals that were incorrectly recalled=17%±16% in the intervention group versus 20%±16% in the control group).
Discussion
An EDSS designed to facilitate shared decision making on goal setting improved case managers' awareness of clients' concerns and satisfaction with the process and increased clients' awareness of the goals and services in their plans, compared with the treatment-as-usual group. Client satisfaction with the process did not improve.
Our findings are congruent with the literature on job satisfaction among mental health workers. Perceived organizational, material, and training supports correlate with job satisfaction among case managers (29). In both the usual care and the intervention group, the process of care planning was facilitated by electronic supports. The usual care group used an electronic medical record designed to support documentation and billing of services. In the EDSS group, the care planning process was guided by empirical and practical considerations regarding what is needed for case managers to access needed information and to collaborate with clients to help them make decisions about individual goals.
Our finding that the clients who used the EDSS were no more satisfied with the care planning process than those in the usual care condition could have occurred for several reasons. Measuring satisfaction among receivers of health care in general (35), and mental health care specifically (36), has been controversial for some time. At least one study found that when mental health care users rate quantitative items positively on satisfaction measures, they may be considering a range of factors, including culpability for problems, before they assign a satisfaction rating (37). These raters appear to make allowances for what the user considers extenuating circumstances, such as service system limitations. Mixed-methods studies have found that quantitative measurement of satisfaction also often masks dissatisfaction with services (36–38). It is possible that measuring satisfaction with the care planning process is attempting to measure a process too far removed from receipt of actual services to be highly relevant to clients.
The finding that clients in the intervention group had greater recall of their care plans than clients in the control group could be due to more involvement in the care planning decision making process, to receipt of information in a more accessible format, or to both. The 2005 Institute of Medicine Quality Chasm series on mental health noted that clients are not routinely informed of decisions affecting their care (3). If being informed regarding one's own care decisions can be thought of as a prerequisite for a participatory model of decision making (39), the EDSS at least enhances clients' awareness of their plans for care.
Limitations
This study has some limitations. First, case managers in the intervention group could have rated the EDSS more highly not because of its actual utility but because of its novelty (40). Second, there are many ways to define case manager and client satisfaction (35,36). The satisfaction measures were based on the literature. It is possible that the general literature on satisfaction with mental health services does not apply to satisfaction with the case management care planning process.
Future research
Research on how to maximize the potential benefits of information technology in the public sector mental health arena is in its infancy. Longitudinal studies that test systems that were designed to follow up on shared decisions over time and that test clinically meaningful outcomes are required in order to truly judge the usefulness of EDSSs in supporting shared decision making and improving care for people diagnosed as having severe mental illnesses.
Conclusions
This study demonstrated that clients can build their own care plans and negotiate and revise them with their case managers by using an electronic support. The EDSS brought to light preferences held by clients that were not previously known by case managers. Case managers and clients then appeared to be able to work together to take action on these items. Clients were also more aware of the care decisions that were being made either directly by them or on their behalf.
1 : Decision-making in the physician-patient encounter: revisiting the shared treatment decision-making model. Social Science and Medicine 149:651–661, 1999 Crossref, Google Scholar
2 : The President's New Freedom Commission: recommendations to transform mental health care in America. Psychiatric Services 54:1467–1474, 2003 Link, Google Scholar
3 Supporting patients' decision-making abilities and preferences; in Improving the Quality of Health Care for Mental and Substance-Use Conditions: Quality Chasm Series. Edited by the
4 : A program to support shared decision making in an outpatient psychiatric medication clinic. Psychiatric Services 59:603–605, 2008 Link, Google Scholar
5 : Shared decision-making preferences of people with severe mental illness. Psychiatric Services 58:1219, 2007 Link, Google Scholar
6 : Recovery from schizophrenia and the role of evidence-based psychosocial interventions. Expert Review of Neurotherapeutics 6:993–1004, 2006 Crossref, Medline, Google Scholar
7 : Implementing supported employment as an evidence-based practice. Psychiatric Services 52:313–322, 2001 Link, Google Scholar
8 : Illness management and recovery for severe mental illness: a review of the research. Psychiatric Services 53:1272–1284, 2002 Link, Google Scholar
9 : Chronic disease management: what will it take to improve care for chronic illness? Effective Clinical Practices 1:2–4, 1998 Medline, Google Scholar
10 : The impact of client treatment preferences on outcome: a meta-analysis. Journal of Clinical Psychology65:368–381, 2009 Crossref, Medline, Google Scholar
11 : Structured patient-clinician communication and 1-year outcome in community mental healthcare. British Journal of Psychiatry 191:420–426, 2007 Crossref, Medline, Google Scholar
12 : Evaluation of the two-way communication checklist as a clinical intervention: results of a multinational, randomised controlled trial. British Journal of Psychiatry 184:79–83, 2004 Crossref, Medline, Google Scholar
13 : Impact of services, met needs, and service empowerment on consumer outcomes. Mental Health Services and Research 4:43–56, 2002 Crossref, Medline, Google Scholar
14 : What users and relatives want from mental health services. Current Opinion in Psychiatry 17:289–296, 2004 Crossref, Google Scholar
15 : Conflict in treatment goals between patients and staff. Social Psychiatry and Psychiatric Epidemiology 14:1–4, 1979 Google Scholar
16 : Priorities of consumers, providers, and family members in the treatment of schizophrenia. Psychiatric Services 53:724–729, 2002 Link, Google Scholar
17 : Consumers' and case managers' perceptions of mental health and community support service needs. Community Mental Health Journal 36:161–178, 2000 Crossref, Medline, Google Scholar
18 : Patients versus rehabilitation practitioners: a comparison of assessments of needs for care. Canadian Journal of Psychiatry 43:159–165, 1998 Crossref, Medline, Google Scholar
19 : Urgent need for a need assessment scale in monitoring social support: a validation study of the Swedish version of the Camberwell Assessment of Need. Nordic Journal of Psychiatry 51:173–182, 1997 Crossref, Google Scholar
20 : Measuring disability need and outcome in Australian community mental health services. Australian and New Zealand Journal of Psychiatry 34:850–855, 2000 Crossref, Medline, Google Scholar
21 : Judgments of quality of life of individuals with severe mental disorders: patient self-report versus provider perspectives. American Journal of Psychiatry 153:497–502, 1996 Link, Google Scholar
22 : The Camberwell Assessment of Need (CAN): comparison of assessments by staff and patients of the needs of the severely mentally ill. Social Psychiatry and Psychiatric Epidemiology 31:109–113, 1996 Crossref, Medline, Google Scholar
23 : Perspectives on schizophrenia: a survey of user, family carer and professional views regarding effective care. Journal of Mental Health 4:403–422, 1995 Crossref, Google Scholar
24 : A comparative analysis of psychiatric problems listed by patients and physicians. Hospital and Community Psychiatry 34:848–849, 1983 Abstract, Google Scholar
25 : Needs assessment of the chronically mentally ill: practitioner and client perspectives. Administration and Policy in Mental Health 13:237–248, 1986 Crossref, Google Scholar
26 : Shared decision making in public mental health care: perspectives from consumers living with severe mental illness. Psychiatric Rehabilitation Journal 34:29–36, 2010 Crossref, Medline, Google Scholar
27 : Costing child protective services staff turnover. Child Welfare 79:517–533, 2000 Medline, Google Scholar
28 : Estimates of costs of primary care physician turnover. American Journal of Managed Care 5:1431–1438, 1999 Medline, Google Scholar
29 : Antecedents to retention and turnover among child welfare, social work, and other human service employees: what can we learn from past research? A review and metaanalysis. Social Services Review 75:625–661, 2001 Crossref, Google Scholar
30 Increasing workforce capacity for quality improvement; in Improving the Quality of Health Care for Mental and Substance-Use Conditions: Quality Chasm Series. Edited by the
31 : What users and relatives want from mental health services. Current Opinion in Psychiatry 17:289–296, 2004 Crossref, Google Scholar
32 : Cluster randomised trials in the medical literature: two bibliometric surveys. BMC Medical Research Methodology 4:21, 2004 Crossref, Medline, Google Scholar
33 : CONSORT statement: extension to cluster randomised trials. British Medical Journal 328:702–708, 2004 Crossref, Medline, Google Scholar
34 : Hierarchical Linear Models: Applications and Data Analysis Methods. Thousand Oaks, Calif, Sage, 2002 Google Scholar
35 : Measurement of satisfaction with health care: implications for practice from a systematic review of the literature. Health Technology Assessment 6:232–252, 2003 Google Scholar
36 : Patient satisfaction in mental health care: evaluating an evaluative method. British Journal of Psychiatry 166:559–562, 1995 Crossref, Medline, Google Scholar
37 : The meaning of patient satisfaction: an explanation of high reported levels. Social Science and Medicine 47:1351–1359, 1998 Crossref, Medline, Google Scholar
38 : The continuum of patient satisfaction: from satisfied to very satisfied. Social Science and Medicine57:2465–2470, 2003 Crossref, Google Scholar
39 : Toward the “tipping point”: decision aids and informed patient choice. Health Affairs 26:716–725, 2007 Crossref, Google Scholar
40 : Experimental and Quasi-Experimental Designs for Generalized Causal Inference. New York, Houghton Mifflin, 2002 Google Scholar
Figures and Tables
Table 1 Demographic and workforce characteristics of case managers who participated in shared decision making using an electronic decision support system (EDSS) or usual methods
Table 2 Demographic and clinical characteristics of clients who participated in shared decision making using an electronic decision support system (EDSS) or usual methods
Table 3 Individual items on the case manager satisfaction with the care planning process questionnaire
Table 4 Individual items on the client satisfaction with care planning process questionnaire
