0
Get Alert
Please Wait... Processing your request... Please Wait.
You must sign in to sign-up for alerts.

Please confirm that your email address is correct, so you can successfully receive this alert.

1
Articles   |    
Development of a Core Set of Performance Measures for Evaluating Schizophrenia Treatment Services
Donald Emile Addington, M.B.B.S.; Emily Mckenzie, M.Sc.; JianLi Wang, Ph.D.; Harvey P. Smith, Ph.D.; Beverly Adams, M.D.; Zahinoor Ismail, M.D.
Psychiatric Services 2012; doi: 10.1176/appi.ps.201100453
View Author and Article Information

The authors are affiliated with the Department of Psychiatry, University of Calgary, Second Floor, Special Services Building, Foothills Medical Centre, 1403 29th St. N.W., Calgary, Alberta T2N 2T9, Canada (e-mail: addingto@ucalgary.ca). Dr. Wang is also with the Department of Community Health Sciences at the university.

Copyright © 2012 by the American Psychiatric Association.

Objective:  The purpose of study was to identify a list of performance measures for schizophrenia treatment services and to assemble a multistakeholder group to reach consensus on a core list.

Methods:  The study was conducted in two stages: first, a systematic review of the literature was conducted to identify a comprehensive list of measures; second, a consensus-building technique, the Delphi process, was used with participants from six groups of stakeholders: schizophrenia experts, mental health clinicians, mental health administrators, the payer (the Alberta Ministry of Health and Wellness), patients, and family members. Thirty stakeholders participated in three rounds of self-completed questionnaires. The degree of consensus achieved in the Delphi process was defined as the semi-interquartile range for each measure.

Results:  Ninety-seven measures were identified in the literature review. The Delphi method reduced the list to 36 measures rated as essential. The measures address eight domains of service-level evaluation: acceptability, accessibility, appropriateness, competence, continuity, effectiveness, efficiency, and safety. Despite the diversity in backgrounds of the stakeholder groups, the Delphi technique was effective in moving participants' ratings toward consensus through successive questionnaire rounds. The resulting measures reflected the interests of all stakeholders.

Conclusions:  Several further steps are required before these measures are implemented and include working toward reliability and validity of specific measures, assessing the feasibility and cost-effectiveness of collecting the data, and finally, undertaking risk adjustment for outcome measures. (Psychiatric Services 63:584–591, 2012;doi: 10.1176/appi.ps.201100453)

Abstract Teaser
Figures in this Article

Performance measurement represents a strategy for addressing both quality improvement and accountability in health care. Performance measures for health services should be based on evidence (1) and derived from systematic reviews of the literature and evidence-based guidelines. Studies have found limited adherence to evidence-based care across all health conditions, including addictions and other mental disorders (2,3). It is anticipated that increased use of performance measures and better adherence to guidelines in mental health services will lead to improved outcomes (4,5).

Modeling of health care systems has suggested that increasing adherence to evidence-based models of schizophrenia services could improve the effectiveness and reduce the costs of schizophrenia services (6). In the United States, conformance with evidence-based recommendations for schizophrenia treatment services has been used to assess quality of care (7). Such studies demonstrate greater adherence to pharmacological guidelines than to guidelines on psychosocial services. A review of process measures for assessment and improvement of the quality of care for schizophrenia indicated that 60% of the measures were evidence based and only 29% were fully operationalized (8).

Using conformance with treatment recommendations as a measure of performance presents a number of challenges. First, treatment recommendations reflect processes and not outcomes, and second, data collection is a significant challenge in routine clinical operations. The first Canadian national mental health performance report (9) focused on administratively available outcome measures. In mental health services, rehospitalization rates were found to be the highest for patients with schizophrenia; 13% were readmitted within 30 days of hospital discharge. Further, 15.4% experienced repeat hospitalizations—that is, two or more in the same year (9).

The purpose of this study was to identify a comprehensive set of measures that could be used to address the eight domains identified by the Canadian Institutes of Health Information for service-level evaluation (10). The domains are acceptability, accessibility, appropriateness, competence, continuity, effectiveness, efficiency, and safety. Performance measures have been defined as “the use of statistical evidence to determine progress towards specific defined organizational objectives” (11). Process and outcome information can be used to assess quality only when, and to the extent that, they are causally related (12). Fortunately, there is significant support in the treatment of schizophrenia for a causal relationship between the processes and outcomes measured and the quality of care. Extensive research has demonstrated the effectiveness of pharmacological and psychosocial treatments in schizophrenia (1315).

The study was undertaken in two stages. First, a systematic literature review was conducted to identify, classify, and organize the measures. Second, a consensus technique was used to prioritize and reduce the number of measures to a core set. The study was approved by the local conjoint health research ethics board.

In the first stage, literature databases—MEDLINE, PsycINFO, PubMed, CINAHL, and HealthSTAR—were searched for English language articles on performance measurement published between 1995 and 2004. The following phrases were independently used in the search: performance measure, performance measurement, performance indicator, performance monitoring, quality indicator, quality measure, quality of care, quality of health care, process assessment, outcome assessment, process measure, and outcome measure. In addition, a gray literature search added several online government reports (1619) and reports from professional practice organizations (20,21). Citation lists in articles were reviewed for additional sources, and advice was sought from experts in the field to identify additional measures.

Next, the measures were classified and organized by using a performance measure profile template. The characteristics of measures were grouped into three categories: measure content, data required for construction of the measure, and evaluation of the measure. The content of each measure was captured by ten categories: rationale, operational definition of measure, quality domain, numerator statement, numerator data source, denominator statement, denominator data source, type of measure (outcome, process, or proxy outcome), age groups, and data format (that is, reported as simple dichotomous or proportion).

All the identified measures were rated for the level of supporting evidence by a group of four local schizophrenia experts, including an expert in clinical trials, an epidemiologist, and a health services researcher. The levels of evidence ratings were based on criteria used in the Canadian Task Force on Preventive Health Care and modified for use in the Canadian Schizophrenia Practice Guidelines (20,22). Strong research-based evidence was rated A. For interventions, examples of an A rating include consistent evidence from well-designed randomized control trials, a meta-analysis in which all the studies included in the statistical pooling were classified as randomized controlled trials, and consistent evidence from well-designed cohort and case studies. For evidence relating to prevalence, consistent findings from appropriately designed studies merited an A rating. Moderate research-based evidence was rated B. This is evidence from study types such as well-designed controlled trials without randomization, cohort studies, case-control analytic studies, comparative studies with historical control, and repeated-measures studies with no control group. The B rating was also used when there were well-designed randomized controlled trials favoring effectiveness but the evidence from such trials was not consistent. Weak or reasonable evidence was rated C. Such evidence was from expert opinions or consensus in the field; descriptive, observational, or qualitative studies (case reports, correlation studies, or secondary analyses); formal reviews; and hypothesis-generating or exploratory studies or subanalyses.

In the second stage, a consensus technique, the Delphi process (23), was used to obtain stakeholder ratings of the importance of individual measures (with the associated level of evidence, as assessed by the aforementioned group of experts). Historically, two approaches have been used to summarize health information and to resolve inconsistencies in research studies. Statistical methods such as meta-analysis are appropriate when the questions can be addressed with optimal study methods, such as randomized controlled studies, and relevant published data are available. When the data cannot be managed statistically, consensus methods provide another means of synthesizing information, but they are liable to use a wider range of information than is common in statistical methods. When published information is inadequate or nonexistent, these methods provide a means of harnessing the insights of experts to inform decision making (24).

Several consensus techniques that share the objective of synthesizing judgments when a state of uncertainty exists have been compared. A systematic review of consensus techniques found that the output from such methods may be affected by several factors, such as the way the task is set, the selection of scientific information, the way interaction between members is structured, and the method of synthesizing judgments (25). The authors' conclusion was that adherence to best practice enhances the validity, reliability, and impact of the product (25). The Delphi technique was selected for the study reported here because of four key features. First, its anonymity is seen as an advantage when both patients and clinical experts are participating. Second, multiple rounds allow stakeholders to change their opinions in subsequent rounds. Third, feedback between rounds provides the distribution of the group's response along with the individual's previous response. Finally, the Delphi technique does not require stakeholders to meet in person (26).

The Delphi technique has been previously used in mental health services research, including in the development of a core set of quality measures for mental health and substance-related care (27), identification of key components of schizophrenia care (28), development of quality indicators for primary care mental health services (29), description of service models of community mental health practice (30), characterization of relapse in schizophrenia (31), and identification of a set of quality indicators for first-episode psychosis services (32). Historically, the technique has been used with a panel of experts. The importance of broadening the panel to include clinicians, consumers, and the general public has been emphasized (33,34), although some have argued that such a multistakeholder approach is a departure from the well-researched Delphi methodology, which typically uses only experts (35).

Stakeholders were selected purposefully. Purposive sampling is a nonprobability sampling technique in which participants are not randomly selected. They are deliberately selected to capture a range of specified group characteristics. This form of sampling is based on the assumption that researchers' knowledge of the population can be used to carefully select individuals to be included in the sample (36). For this study, purposive sampling was superior to the alternatives because the stakeholders were selected on the basis of their breadth of experience and knowledge, as well as their willingness and ability to articulate their opinions. Optimal sample size in research using the Delphi technique has not been established, and there is scant empirical evidence on the effect of the number of stakeholders on either the reliability or validity of consensus processes (37). Research has been published based on samples varying from between ten and 50 to much larger numbers (38). We identified 30 stakeholders for participation in the Delphi. The stakeholders were from six groups: schizophrenia experts, mental health clinician providers, mental health administrative providers, the payer, patients, and family members.

At the proposal-writing stage, the first author contacted government representatives from the Alberta Ministry of Health and Wellness (the payer) and administrative representatives from the provider organization to explain the project and details of participation. The agency that funded health services research required that a decision maker be involved in research proposal development as well as project completion in order to support knowledge translation. Potential family and patient participants were identified by staff members (interested clinicians) of a specialized schizophrenia service.

The Delphi questionnaire was developed from the list of performance measures from the systematic review and pilot-tested with local clinicians, patients, and service managers. Pilot testing involved individual, in-person administration of the questionnaire by the study coordinator to stakeholder group representatives. The questionnaire was examined for clarity of the instructions, definitions, and descriptions of the performance measures and for reading level. The Delphi questionnaire was administered in person by the study coordinator to each individual in the patient stakeholder group. All other stakeholders received a written questionnaire, either by e-mail or by post if they did not have computer access. The stakeholders were provided information from the first stage of the research—the systematic review and ratings of supportive evidence.

The Delphi comprised three rounds that occurred between June and November 2005. The first round was an open round in which the stakeholders were invited to provide comments about the indicators. Each round of questionnaires included a qualitative component that offered the opportunity to provide additional feedback in the form of written comments, and each round built upon responses in the former round.

In rounds 2 and 3, the stakeholders were asked to rate the importance of the individual measures on a 5-point Likert scale, from 1 to 5 (1, essential; 2, very important; 3, important; 4, less important; and 5, unimportant). After each round, stakeholders were provided with feedback and a summary of the previous round. The feedback to each participant included the participant's rating of the importance of each performance measure, along with the group's median rating, the percentage of participants with ratings at each point on the Likert scale, and a synopsis of written comments. Participants were then asked to reflect upon the feedback and rate each item again in light of the new information. In the event that their response was more than 2 points away from the group median, they were asked to elaborate with comments.

 
The degree of consensus achieved in the Delphi was assessed by calculating the semi-interquartile range for each measure after each round. The semi-interquartile range is calculated from the following formula: Anchor for JumpAnchor for Jump

The level of consensus was set before data collection began. Consensus was defined as measures for which the final ratings had a semi-interquartile range of ≤.5. Measures with final ratings with a semi-interquartile range of ≤.5 were interpreted as being essential (30). Ratings were analyzed (medians, means, and semi-interquartile ranges) with the Statistical Package for Social Sciences (39).

+

Literature review

The database searches yielded a total of 266 unduplicated references after the inclusion criteria were applied. Appropriateness and eligibility for inclusion in the review were determined by screening the abstract. Inclusion criteria consisted of the following: the focus of the article was performance measurement or evaluation of the quality of care, and either the article represented a review of performance measure work or research evidence was presented that was based on at least one measure with face validity. From these articles 97 performance measures were identified.

Several articles were a rich source of measures. One article identified 42 process measures for the evaluation of schizophrenia services (8). Of the identified measures, 60% were based on research evidence linking conformance to the measure with improved patient outcomes. A systematic review of measures for mental health services identified many measures and reviewed the state of the art of their application (40).

In Australia, a set of measures was developed to monitor the progress of the National Mental Health Strategy (41). All publicly funded mental health services in Australia are required to collect data and report on these measures. In the United Kingdom, the National Service Frameworks set national standards, define service models, and establish performance measures for use in benchmarking (16). The National Centre for Health Outcomes Development (42) recommended a set of 20 outcome measures for severe mental illness.

All performance measures in the effectiveness domain were found in at least two types of sources, and most of the measures within the acceptability and appropriateness domains were found in at least two types of sources, suggesting that our searches had reached a degree of saturation. [Descriptions of the performance measures and their sources are listed in an online data supplement to this article. Most measures are expressed as rates, ratios, or percentages, and those that are marked with an asterisk are categorical measures.]

+

Delphi consensus

The stakeholders reached consensus over three rounds. The number of participants in each round is summarized in Table 1. The Delphi members used the full range of ratings, and agreement improved with each round. The ratings were skewed toward the high end of importance. In round 2, 42% of all of the 97 measures rated by all 30 stakeholders were rated as essential—a rating of 1; 37% were as 2, 16% as 3, 4% as 4, and 1% as 5. The 36 performance measures identified by participants as essential, along with the semi-interquartile range of the median ratings, are listed in Table 2 (16,17,20,4361).

This study used a three-step process to identify and select performance measures deemed essential for the evaluation of schizophrenia services. Each step, from the systematic review to the rating of supportive evidence and the selection of indicators, used a rigorous methodology. The result is a list of 36 measures that encompass the eight performance domains recommended for program evaluation. This list provides a useful starting point for further work to develop operational definitions and data sources for performance measure implementation. The list of measures is more detailed than measures in widely implemented general health system indicator lists, such as the Healthcare Effectiveness Data and Information Set (HEDIS) (61), which have been found to have a narrow focus and limited application to many components of mental health services (62). The National Committee for Quality Assurance, the developer of HEDIS, is in the process of developing a set of schizophrenia measures for state Medicaid programs and U.S. health plans.

The literature review identified 97 performance measures, and the Delphi technique successfully narrowed the list to 36 measures that were identified as essential by a multistakeholder group. The stakeholder consensus process thus established the face validity of these performance measures (63). Several of the measures can be considered evidence based, which is defined as measures supported by clinical trial data that links the process measured to improved outcomes. Examples of evidence-based measures include maintenance antipsychotic medication and family and patient psychoeducation (6365). Only seven outcome measures were identified as essential, and hospitalization rate is the only one in this group that is reliably measured and readily available. It is one of only three mental health performance measures available at a federal level in Canada (9). Hospitalization has been suggested as a good proxy outcome measure for schizophrenia research (30), and its application has been extended in first-episode psychosis by development of a robust risk adjustment model that facilitates the comparison of real-life services (66). Although relapse rates were identified as essential in the Delphi process, in practice these are difficult to measure, and hospitalization has often been used as a more concrete and easily measured proxy for relapse (67).

This study had several limitations. First, the measures rated were based on a literature search up to July 2004. The delay in completing the project and publication was the result of personnel issues. One purpose of using a Delphi process in this study was to reduce the number of items. Although the reduction from 97 to 36 was useful, 36 measures represent a larger group than would be practical in standard program evaluation. There is some redundancy within the list in that there are four items for assessing symptoms and quality of life (in the appropriateness domain) and another four for the percentage of patients showing an improvement in those measures over the course of one year (in the effectiveness domain). In addition, we did not examine differences between the various groups of Delphi participants. Although this might prove interesting, the study was not designed to use the process in this way. We selected a Delphi group that would be large enough to undertake the planned task, but it was not large enough to undertake a secondary analysis involving comparison of the subgroups independently. Programs that plan to use the measures can further reduce the number of measures by carefully examining the cost and feasibility and applicability of the measures in their situation. Mortality rate is an example of a performance measure for which the necessary data are not usually readily available at a program level. Calculation of this rate would require large-scale population-based information.

These findings are encouraging because they demonstrate that there is a useful number of evidence-based performance measures for schizophrenia services. The measures include hospitalization, a proxy outcome that is universally available and reliably measured. Furthermore, the results indicate that a set of measures can satisfy the often competing interests of payers, providers, and consumers. Finally, the measures identified cover a number of domains that are considered important for evaluation of all programs.

This work was supported by grant 17376 from the Alberta Heritage Foundation for Medical Research. The authors thank the stakeholders for their participation in the Delphi process.

The authors report no competing interests.

Hearnshaw  HM;  Harker  RM;  Cheater  FM  et al:  Expert consensus on the desirable characteristics of review criteria for improvement of health care quality.  Quality Health Care 10:173–178, 2001
[CrossRef]
 
McGlynn  EA;  Asch  SM;  Adams  J  et al:  The quality of health care delivered to adults in the United States.  New England Journal of Medicine 348:2635–2645, 2003
[CrossRef] | [PubMed]
 
Druss  BG;  Miller  CL;  Rosenheck  RA  et al:  Mental health care quality under managed care in the United States: a view from the Health Employer Data and Information Set (HEDIS).  American Journal of Psychiatry 159:860–862, 2002
[CrossRef] | [PubMed]
 
Institute of Medicine Committee on Crossing the Quality Chasm:  Adaptation to Mental Health and Addictive Disorders.  Improving the Quality of Health Care for Mental and Substance-Use Conditions   Washington DC,  National Academies Press, 2007
 
Wobrock  T;  Weinmann  S;  Falkai  P  et al:  Quality assurance in psychiatry: quality indicators and guideline implementation.  European Archives of Psychiatry and Clinical Neuroscience 259suppl 2:S219–S226, 2009
[CrossRef] | [PubMed]
 
Andrews  G;  Sanderson  K;  Corry  J  et al:  Cost-effectiveness of current and optimal treatment for schizophrenia.  British Journal of Psychiatry 183:427–435, 2003
[CrossRef] | [PubMed]
 
Buchanan  RW;  Kreyenbuhl  J;  Zito  JM  et al:  The schizophrenia PORT pharmacological treatment recommendations: conformance and implications for symptoms and functional outcome.  Schizophrenia Bulletin 28:63–73, 2002
[CrossRef] | [PubMed]
 
Hermann  RC;  Finnerty  M;  Provost  S  et al:  Process measures for the assessment and improvement of quality of care for schizophrenia.  Schizophrenia Bulletin 28:95–104, 2002
[CrossRef] | [PubMed]
 
 Health Indicators   Ottawa,  Canadian Institute of Health Information, 2011
 
 Roadmap Initiative Launching the Process: The Final Year   Ottawa,  Canadian Institute for Health Information, 2003
 
Measures  TP:  Performance-Based Planning Manual (Preliminary Draft).  Agter NCHRP project 8–32(2)   Washington DC,  Federal Highway Administration Operations Unit, 2002
 
Norquist  G:  Role of outcome measurement in psychiatry.  Outcome Measurement in Psychiatry: A Critical Review IsHak  WW;  Burt  T;  Sederer  LI:  Washington DC,  American Psychiatric Press, 2002
 
Dixon  LB;  Lehman  AF;  Levine  J:  Conventional antipsychotic medications for schizophrenia.  Schizophrenia Bulletin 21:567–577, 1995
[PubMed]
 
Dixon  LB;  Lehman  AF:  Family Interventions for schizophrenia.  Schizophrenia Bulletin 21:631–643, 1995
[PubMed]
 
Tarrier  N;  Yusupoff  L;  Kinney  C  et al:  Randomised controlled trial of intensive cognitive behaviour therapy for patients with chronic schizophrenia.  British Medical Journal 17:303–307, 1998
[CrossRef]
 
 National Service Framework for Mental Health: Modern Standard and Service Models   London, United Kingdom,  Department of Health, 1999
 
 Behavioral Healthcare Performance Measurement System   Alexandria, Va,  National Association of State Mental Health Program Directors Research Institute, 2011
 
Health System Performance Research Network Hospital Report: Mental Health   Toronto, Ontario,  Government of Ontario, Ontario Hospital Association, and the Hospital Report Research Collaborative, 2007
 
McEwan  K;  Goldner  EM:  Accountability and Performance Indicators for Mental Health Services and Supports: A Resource Kit   Ottawa,  Government Services Canada, 2000
 
Canadian Psychiatric Association Working Group:  Clinical practice guidelines: treatment of schizophrenia.  Canadian Journal of Psychiatry 50suppl 1:1S–56S, 2005
 
American Psychiatric Association:  Practice guideline for the treatment of patients with schizophrenia2nd ed.  American Journal of Psychiatry 161suppl2:1–55, 2004
[CrossRef]
 
Canadian Task Force on Preventive Health Care:  New grades for recommendations from the Canadian Task Force on Preventive Health Care.  Canadian Medical Association Journal 169:207–208, 2003
[PubMed]
 
Fitch  K;  Bernstein  SJ;  Aguilar  MS  et al:  The RAND/UCLA Appropriateness Method User's Manual   Santa Monica, Calif,  RAND, 2001
 
Jones  J;  Hunter  D:  Consensus methods for medical and health services research.  British Medical Journal 311:376–380, 1995
[CrossRef] | [PubMed]
 
Black  N;  Murphy  MK;  Lamping  D  et al:  Consensus development methods: a review of best practice in creating clinical guidelines.  Journal of Health Services and Research Policy 4:236–248, 1999
 
Hasson  F;  Keeney  S;  McKenna  H:  Research guidelines for the Delphi survey technique.  Journal of Advanced Nursing 32:1008–1015, 2000
[PubMed]
 
Hermann  RC;  Palmer  H;  Leff  S  et al:  Achieving consensus across diverse stakeholders on quality measures for mental healthcare.  Medical Care 42:1246–1253, 2004
[CrossRef] | [PubMed]
 
Fiander  M;  Burns  T:  Essential components of schizophrenia care: a Delphi approach.  Acta Psychiatrica Scandinavia 98:400–405, 1998
[CrossRef]
 
Shield  T;  Campbell  SM;  Rogers  A  et al:  Quality indicators for primary care mental health services.  Quality and Safety in Health Care 12:100–106, 2003
[CrossRef] | [PubMed]
 
Fiander  M;  Burns  T:  A Delphi approach to describing service models of community mental health practice.  Psychiatric Services 51:656–658, 2000
[CrossRef] | [PubMed]
 
Burns  T;  Fiander  M;  Audini  B:  A Delphi approach to characterising “relapse” as used in UK clinical practice.  International Journal of Social Psychiatry 46:220–230, 2000
[CrossRef] | [PubMed]
 
Addington  D;  Mckenzie  E;  Addington  J  et al:  Performance measures for early psychosis treatment services.  Psychiatric Services 56:1570–1582, 2005
[CrossRef] | [PubMed]
 
Cleary  PD;  Edgman-Levitan  S:  Health care quality: incorporating consumer perspectives.  JAMA 278:1608–1612, 1997
[CrossRef] | [PubMed]
 
Donabedian  A:  The quality of care: how can it be assessed?.  JAMA 260:743–1748, 1988
[CrossRef]
 
Burnam  A:  Selecting performance measures by consensus: an appropriate extension of the Delphi method?.  Psychiatric Services 56:15832005
[CrossRef] | [PubMed]
 
Polit  D;  Hungler  B:  Essentials of Nursing Research   New York,  Lippincott, 1997
 
Murphy  MK;  Black  NA;  Lamping  DL  et al:  Consensus Development Methods and Their Use in Clinical Guideline Development   London,  National Health Services, Research and Development, Health Technology Assessment Programme, 1998
 
Campbell  SM;  Cantrill  JA:  Consensus methods in prescribing research.  Journal of Clinical Pharmacy and Therapeutics 26:5–14, 2001
[CrossRef] | [PubMed]
 
 Statistical Package for Social Sciences, version 11.01   Chicago,  SPSS, 2003
 
Adair  CE;  Simpson  L;  Birdsell  JM  et al:  Performance Measurement Systems in Health and Mental Health Services: Models, Practices and Effectiveness—A State of the Science Review  Alberta, Canada,  Alberta Heritage Foundation for Medical Research, 2003
 
 National Information Priorities and Strategies Under the Second National Mental Health Plan   Canberra, Australia,  Australian Health Ministers' Advisory Council, National Mental Health Working Group, 2003
 
Carlwood  P;  Mason  A;  Goldacre  M:  Health Outcome Indicators: Severe Mental Illness   Oxford, United Kingdom,  National Centre for Outcomes Development, 1999
 
Arns  P;  Rogers  ES;  Cook  J  et al:  The IAPSRS toolkit: development, utility, and relation to other performance measurement systems.  Psychiatric Rehabilitation Journal 25:43–52, 2001
[PubMed]
 
 Report of the American Psychiatric Association Task Force on Quality Indicators   Washington, DC,  American Psychiatric Association, 1999
 
Kessler  RC;  Berglund  PA;  Bruce  ML  et al:  The prevalence and correlates of untreated serious mental illness.  Health Services Research 36:987–1007, 2001
[PubMed]
 
Popkin  MK;  Lurie  N;  Manning  W  et al:  Changes in the process of care for Medicaid patients with schizophrenia in Utah's Prepaid Mental Health Plan.  Psychiatric Services 49:518–523, 1998
[PubMed]
 
Dixon  L:  Dual diagnosis of substance abuse in schizophrenia: prevalence and impact on outcomes.  Schizophrenia Research 35(suppl):S93–100, 1999
[CrossRef] | [PubMed]
 
Young  AS;  Sullivan  G;  Burnam  MA  et al:  Measuring the quality of outpatient treatment for schizophrenia.  Archives of General Psychiatry 55:611–617, 1998
[CrossRef] | [PubMed]
 
Ashton  C;  Del Junco  D;  Soucheck  J  et al:  The association between the quality of inpatient care and early readmission: a meta-analysis of the evidence.  Medical Care 35:1044–1059, 1997
[CrossRef] | [PubMed]
 
Chinman  M;  Young  AS;  Rowe  M  et al:  An instrument to assess competencies of providers treating severe mental illness.  Mental Health Services Research 5:97–108, 2003
[CrossRef] | [PubMed]
 
Marder  SR;  Essock  SM;  Miller  AL  et al:  Physical health monitoring of patients with schizophrenia.  American Journal of Psychiatry 161:1334–1349, 2004
[CrossRef] | [PubMed]
 
Accreditation Organization Working Group A Proposed Consensus Set of Indicators for Behavioral Health   Albuquerque, NM,  American College of Mental Health Administration, 2001
 
 Core Interventions in the Treatment and Management of Schizophrenia in Primary and Secondary Care.  Clinical Guideline 1   London,  National Institute for Clinical Excellence, 2002
 
 Western Canada Waiting List Project Final Report: From Chaos to Order: Making Sense of Waiting Lists in Canada   Calgary, Alberta,  Western Canada Waiting List Project, 2001
 
McEwan  KL;  Goldner  EM:  Keeping mental health reform on course: selecting indicators of mental health system performance.  Canadian Journal of Community Mental Health 21:5–16, 2002
[PubMed]
 
Lehman  AF;  Kreyenbuhl  J;  Buchanan  RW  et al:  The Schizophrenia Patient Outcomes Research Team (PORT): updated treatment recommendations 2003.  Schizophrenia Bulletin 30:193–217, 2004
[CrossRef] | [PubMed]
 
Royal Australian and New Zealand College of Psychiatrists:  Clinical Practice Guidelines Team for the Treatment of Schizophrenia and Related Disorders.  Australian and New Zealand Journal of Psychiatry 39:1–30, 2005
[PubMed]
 
International Early Psychosis Association Writing Group:  International clinical practice guidelines for early psychosis.  British Journal of Psychiatry Supplement 48:120–124, 2005
 
 Health Care in Canada   Ottawa, Ontario,  Canadian Institute for Health Information, 2001
 
Hospital Report Research Collaborative:  Hospital Report 2001 Preliminary Studies: Volume 1.  Mental Health   Toronto,  University of Toronto, 2001
 
National Committee for Quality Assurance The State of Health Care Quality   Washington DC,  National Committee for Quality Assurance, 2006
 
Druss  B;  Rosenheck  R:  Evaluation of the HEDIS measure of behavioral health care quality: Health Plan Employer Data and Information Set.  Psychiatric Services 48:71–75, 1997
[PubMed]
 
Dickey  B;  Normand  SL;  Eisen  S  et al:  Associations between adherence to guidelines for antipsychotic dose and health status, side effects, and patient care experiences.  Medical Care 44:827–834, 2006
[CrossRef] | [PubMed]
 
Dixon  LB;  Dickerson  F;  Bellack  AS  et al:  The 2009 schizophrenia PORT psychosocial treatment recommendations and summary statements.  Schizophrenia Bulletin 36:48–70, 2010
[CrossRef] | [PubMed]
 
Burns  T:  Hospitalisation as an outcome measure in schizophrenia.  British Journal of Psychiatry Supplement 50:37–41, 2007
[CrossRef]
 
Addington  DE;  Beck  C;  Wang  J  et al:  Predictors of admission in first-episode psychosis: developing a risk adjustment model for service comparisons.  Psychiatric Services 61:483–488, 2010
[CrossRef] | [PubMed]
 
Gleeson  JF;  Alvarez-Jimenez  M;  Cotton  SM  et al:  A systematic review of relapse measurement in randomized controlled trials of relapse prevention in first-episode psychosis.  Schizophrenia Research 119:79–88, 2010
[CrossRef] | [PubMed]
 
References Container
 
Anchor for JumpAnchor for Jump
Table 1

Participation in a three-round Delphi process by 30 members of six stakeholder groups

 
Anchor for JumpAnchor for Jump
Table 2

Performance measures in eight domains rated as essential in the Delphi process and their source, evidence level, and consensus score assessed by semi-interquartile range (SIR)

The degree of consensus achieved in the Delphi was assessed by calculating the semi-interquartile range for each measure after each round. The semi-interquartile range is calculated from the following formula:

Table 1 

Participation in a three-round Delphi process by 30 members of six stakeholder groups

Table 2 

Performance measures in eight domains rated as essential in the Delphi process and their source, evidence level, and consensus score assessed by semi-interquartile range (SIR)

+

References

Hearnshaw  HM;  Harker  RM;  Cheater  FM  et al:  Expert consensus on the desirable characteristics of review criteria for improvement of health care quality.  Quality Health Care 10:173–178, 2001
[CrossRef]
 
McGlynn  EA;  Asch  SM;  Adams  J  et al:  The quality of health care delivered to adults in the United States.  New England Journal of Medicine 348:2635–2645, 2003
[CrossRef] | [PubMed]
 
Druss  BG;  Miller  CL;  Rosenheck  RA  et al:  Mental health care quality under managed care in the United States: a view from the Health Employer Data and Information Set (HEDIS).  American Journal of Psychiatry 159:860–862, 2002
[CrossRef] | [PubMed]
 
Institute of Medicine Committee on Crossing the Quality Chasm:  Adaptation to Mental Health and Addictive Disorders.  Improving the Quality of Health Care for Mental and Substance-Use Conditions   Washington DC,  National Academies Press, 2007
 
Wobrock  T;  Weinmann  S;  Falkai  P  et al:  Quality assurance in psychiatry: quality indicators and guideline implementation.  European Archives of Psychiatry and Clinical Neuroscience 259suppl 2:S219–S226, 2009
[CrossRef] | [PubMed]
 
Andrews  G;  Sanderson  K;  Corry  J  et al:  Cost-effectiveness of current and optimal treatment for schizophrenia.  British Journal of Psychiatry 183:427–435, 2003
[CrossRef] | [PubMed]
 
Buchanan  RW;  Kreyenbuhl  J;  Zito  JM  et al:  The schizophrenia PORT pharmacological treatment recommendations: conformance and implications for symptoms and functional outcome.  Schizophrenia Bulletin 28:63–73, 2002
[CrossRef] | [PubMed]
 
Hermann  RC;  Finnerty  M;  Provost  S  et al:  Process measures for the assessment and improvement of quality of care for schizophrenia.  Schizophrenia Bulletin 28:95–104, 2002
[CrossRef] | [PubMed]
 
 Health Indicators   Ottawa,  Canadian Institute of Health Information, 2011
 
 Roadmap Initiative Launching the Process: The Final Year   Ottawa,  Canadian Institute for Health Information, 2003
 
Measures  TP:  Performance-Based Planning Manual (Preliminary Draft).  Agter NCHRP project 8–32(2)   Washington DC,  Federal Highway Administration Operations Unit, 2002
 
Norquist  G:  Role of outcome measurement in psychiatry.  Outcome Measurement in Psychiatry: A Critical Review IsHak  WW;  Burt  T;  Sederer  LI:  Washington DC,  American Psychiatric Press, 2002
 
Dixon  LB;  Lehman  AF;  Levine  J:  Conventional antipsychotic medications for schizophrenia.  Schizophrenia Bulletin 21:567–577, 1995
[PubMed]
 
Dixon  LB;  Lehman  AF:  Family Interventions for schizophrenia.  Schizophrenia Bulletin 21:631–643, 1995
[PubMed]
 
Tarrier  N;  Yusupoff  L;  Kinney  C  et al:  Randomised controlled trial of intensive cognitive behaviour therapy for patients with chronic schizophrenia.  British Medical Journal 17:303–307, 1998
[CrossRef]
 
 National Service Framework for Mental Health: Modern Standard and Service Models   London, United Kingdom,  Department of Health, 1999
 
 Behavioral Healthcare Performance Measurement System   Alexandria, Va,  National Association of State Mental Health Program Directors Research Institute, 2011
 
Health System Performance Research Network Hospital Report: Mental Health   Toronto, Ontario,  Government of Ontario, Ontario Hospital Association, and the Hospital Report Research Collaborative, 2007
 
McEwan  K;  Goldner  EM:  Accountability and Performance Indicators for Mental Health Services and Supports: A Resource Kit   Ottawa,  Government Services Canada, 2000
 
Canadian Psychiatric Association Working Group:  Clinical practice guidelines: treatment of schizophrenia.  Canadian Journal of Psychiatry 50suppl 1:1S–56S, 2005
 
American Psychiatric Association:  Practice guideline for the treatment of patients with schizophrenia2nd ed.  American Journal of Psychiatry 161suppl2:1–55, 2004
[CrossRef]
 
Canadian Task Force on Preventive Health Care:  New grades for recommendations from the Canadian Task Force on Preventive Health Care.  Canadian Medical Association Journal 169:207–208, 2003
[PubMed]
 
Fitch  K;  Bernstein  SJ;  Aguilar  MS  et al:  The RAND/UCLA Appropriateness Method User's Manual   Santa Monica, Calif,  RAND, 2001
 
Jones  J;  Hunter  D:  Consensus methods for medical and health services research.  British Medical Journal 311:376–380, 1995
[CrossRef] | [PubMed]
 
Black  N;  Murphy  MK;  Lamping  D  et al:  Consensus development methods: a review of best practice in creating clinical guidelines.  Journal of Health Services and Research Policy 4:236–248, 1999
 
Hasson  F;  Keeney  S;  McKenna  H:  Research guidelines for the Delphi survey technique.  Journal of Advanced Nursing 32:1008–1015, 2000
[PubMed]
 
Hermann  RC;  Palmer  H;  Leff  S  et al:  Achieving consensus across diverse stakeholders on quality measures for mental healthcare.  Medical Care 42:1246–1253, 2004
[CrossRef] | [PubMed]
 
Fiander  M;  Burns  T:  Essential components of schizophrenia care: a Delphi approach.  Acta Psychiatrica Scandinavia 98:400–405, 1998
[CrossRef]
 
Shield  T;  Campbell  SM;  Rogers  A  et al:  Quality indicators for primary care mental health services.  Quality and Safety in Health Care 12:100–106, 2003
[CrossRef] | [PubMed]
 
Fiander  M;  Burns  T:  A Delphi approach to describing service models of community mental health practice.  Psychiatric Services 51:656–658, 2000
[CrossRef] | [PubMed]
 
Burns  T;  Fiander  M;  Audini  B:  A Delphi approach to characterising “relapse” as used in UK clinical practice.  International Journal of Social Psychiatry 46:220–230, 2000
[CrossRef] | [PubMed]
 
Addington  D;  Mckenzie  E;  Addington  J  et al:  Performance measures for early psychosis treatment services.  Psychiatric Services 56:1570–1582, 2005
[CrossRef] | [PubMed]
 
Cleary  PD;  Edgman-Levitan  S:  Health care quality: incorporating consumer perspectives.  JAMA 278:1608–1612, 1997
[CrossRef] | [PubMed]
 
Donabedian  A:  The quality of care: how can it be assessed?.  JAMA 260:743–1748, 1988
[CrossRef]
 
Burnam  A:  Selecting performance measures by consensus: an appropriate extension of the Delphi method?.  Psychiatric Services 56:15832005
[CrossRef] | [PubMed]
 
Polit  D;  Hungler  B:  Essentials of Nursing Research   New York,  Lippincott, 1997
 
Murphy  MK;  Black  NA;  Lamping  DL  et al:  Consensus Development Methods and Their Use in Clinical Guideline Development   London,  National Health Services, Research and Development, Health Technology Assessment Programme, 1998
 
Campbell  SM;  Cantrill  JA:  Consensus methods in prescribing research.  Journal of Clinical Pharmacy and Therapeutics 26:5–14, 2001
[CrossRef] | [PubMed]
 
 Statistical Package for Social Sciences, version 11.01   Chicago,  SPSS, 2003
 
Adair  CE;  Simpson  L;  Birdsell  JM  et al:  Performance Measurement Systems in Health and Mental Health Services: Models, Practices and Effectiveness—A State of the Science Review  Alberta, Canada,  Alberta Heritage Foundation for Medical Research, 2003
 
 National Information Priorities and Strategies Under the Second National Mental Health Plan   Canberra, Australia,  Australian Health Ministers' Advisory Council, National Mental Health Working Group, 2003
 
Carlwood  P;  Mason  A;  Goldacre  M:  Health Outcome Indicators: Severe Mental Illness   Oxford, United Kingdom,  National Centre for Outcomes Development, 1999
 
Arns  P;  Rogers  ES;  Cook  J  et al:  The IAPSRS toolkit: development, utility, and relation to other performance measurement systems.  Psychiatric Rehabilitation Journal 25:43–52, 2001
[PubMed]
 
 Report of the American Psychiatric Association Task Force on Quality Indicators   Washington, DC,  American Psychiatric Association, 1999
 
Kessler  RC;  Berglund  PA;  Bruce  ML  et al:  The prevalence and correlates of untreated serious mental illness.  Health Services Research 36:987–1007, 2001
[PubMed]
 
Popkin  MK;  Lurie  N;  Manning  W  et al:  Changes in the process of care for Medicaid patients with schizophrenia in Utah's Prepaid Mental Health Plan.  Psychiatric Services 49:518–523, 1998
[PubMed]
 
Dixon  L:  Dual diagnosis of substance abuse in schizophrenia: prevalence and impact on outcomes.  Schizophrenia Research 35(suppl):S93–100, 1999
[CrossRef] | [PubMed]
 
Young  AS;  Sullivan  G;  Burnam  MA  et al:  Measuring the quality of outpatient treatment for schizophrenia.  Archives of General Psychiatry 55:611–617, 1998
[CrossRef] | [PubMed]
 
Ashton  C;  Del Junco  D;  Soucheck  J  et al:  The association between the quality of inpatient care and early readmission: a meta-analysis of the evidence.  Medical Care 35:1044–1059, 1997
[CrossRef] | [PubMed]
 
Chinman  M;  Young  AS;  Rowe  M  et al:  An instrument to assess competencies of providers treating severe mental illness.  Mental Health Services Research 5:97–108, 2003
[CrossRef] | [PubMed]
 
Marder  SR;  Essock  SM;  Miller  AL  et al:  Physical health monitoring of patients with schizophrenia.  American Journal of Psychiatry 161:1334–1349, 2004
[CrossRef] | [PubMed]
 
Accreditation Organization Working Group A Proposed Consensus Set of Indicators for Behavioral Health   Albuquerque, NM,  American College of Mental Health Administration, 2001
 
 Core Interventions in the Treatment and Management of Schizophrenia in Primary and Secondary Care.  Clinical Guideline 1   London,  National Institute for Clinical Excellence, 2002
 
 Western Canada Waiting List Project Final Report: From Chaos to Order: Making Sense of Waiting Lists in Canada   Calgary, Alberta,  Western Canada Waiting List Project, 2001
 
McEwan  KL;  Goldner  EM:  Keeping mental health reform on course: selecting indicators of mental health system performance.  Canadian Journal of Community Mental Health 21:5–16, 2002
[PubMed]
 
Lehman  AF;  Kreyenbuhl  J;  Buchanan  RW  et al:  The Schizophrenia Patient Outcomes Research Team (PORT): updated treatment recommendations 2003.  Schizophrenia Bulletin 30:193–217, 2004
[CrossRef] | [PubMed]
 
Royal Australian and New Zealand College of Psychiatrists:  Clinical Practice Guidelines Team for the Treatment of Schizophrenia and Related Disorders.  Australian and New Zealand Journal of Psychiatry 39:1–30, 2005
[PubMed]
 
International Early Psychosis Association Writing Group:  International clinical practice guidelines for early psychosis.  British Journal of Psychiatry Supplement 48:120–124, 2005
 
 Health Care in Canada   Ottawa, Ontario,  Canadian Institute for Health Information, 2001
 
Hospital Report Research Collaborative:  Hospital Report 2001 Preliminary Studies: Volume 1.  Mental Health   Toronto,  University of Toronto, 2001
 
National Committee for Quality Assurance The State of Health Care Quality   Washington DC,  National Committee for Quality Assurance, 2006
 
Druss  B;  Rosenheck  R:  Evaluation of the HEDIS measure of behavioral health care quality: Health Plan Employer Data and Information Set.  Psychiatric Services 48:71–75, 1997
[PubMed]
 
Dickey  B;  Normand  SL;  Eisen  S  et al:  Associations between adherence to guidelines for antipsychotic dose and health status, side effects, and patient care experiences.  Medical Care 44:827–834, 2006
[CrossRef] | [PubMed]
 
Dixon  LB;  Dickerson  F;  Bellack  AS  et al:  The 2009 schizophrenia PORT psychosocial treatment recommendations and summary statements.  Schizophrenia Bulletin 36:48–70, 2010
[CrossRef] | [PubMed]
 
Burns  T:  Hospitalisation as an outcome measure in schizophrenia.  British Journal of Psychiatry Supplement 50:37–41, 2007
[CrossRef]
 
Addington  DE;  Beck  C;  Wang  J  et al:  Predictors of admission in first-episode psychosis: developing a risk adjustment model for service comparisons.  Psychiatric Services 61:483–488, 2010
[CrossRef] | [PubMed]
 
Gleeson  JF;  Alvarez-Jimenez  M;  Cotton  SM  et al:  A systematic review of relapse measurement in randomized controlled trials of relapse prevention in first-episode psychosis.  Schizophrenia Research 119:79–88, 2010
[CrossRef] | [PubMed]
 
References Container
+
+

CME Activity

There is currently no quiz available for this resource. Please click here to go to the CME page to find another.
Submit a Comments
Please read the other comments before you post yours. Contributors must reveal any conflict of interest.
Comments are moderated and will appear on the site at the discertion of APA editorial staff.

* = Required Field
(if multiple authors, separate names by comma)
Example: John Doe



Web of Science® Times Cited: 2

Related Content
Articles
Books
Dulcan's Textbook of Child and Adolescent Psychiatry > Chapter 24.  >
Dulcan's Textbook of Child and Adolescent Psychiatry > Chapter 7.  >
The American Psychiatric Publishing Textbook of Geriatric Psychiatry, 4th Edition > Chapter 26.  >
The American Psychiatric Publishing Textbook of Geriatric Psychiatry, 4th Edition > Chapter 26.  >
The American Psychiatric Publishing Textbook of Geriatric Psychiatry, 4th Edition > Chapter 17.  >
Topic Collections
Psychiatric News
APA Guidelines
PubMed Articles