Best Practices: New York's Outpatient Mental Health Clinic Licensing Reform: Using Tracer Methodology to Improve Service Quality

Professional accountability requires a profession to set and maintain credible and useful standards for its members (1) and to implement best practices (2). New York State oversees mental health services provided by state-licensed treatment providers. In response to several incidents of violence that involved people with serious mental illness, New York State and New York City sponsored a Mental Health-Criminal Justice Panel to improve coordination, oversight, and accountability in mental health treatment services. The panel's report was published in June 2008 (3). Although prompted by the incidents of violence, the New York State Office of Mental Health (OMH) took advantage of the call for better assessment of violence to revise the licensing process to underscore recovery-oriented values, such as treatment responsive to individuals' stated needs and family involvement in treatment, in redesigned clinic licensing procedures.

Mental health care has been moving toward a recovery-focused, person-centered, strengths-based system. The President's New Freedom Commission on Mental Health (4) and others (5–7) have called for increasing consumer- and family-driven services, improving quality and accountability in behavioral health services, and incorporating the perspectives of the people who receive services. Licensing processes have not always kept pace with these efforts to improve service delivery.

Outpatient clinics are the “face of the public mental health system” and are the only level of service used by most people. In New York State, for example, more than 500 licensed clinics offer outpatient mental health services at more than 1,200 sites serving more than 400,000 individuals each year. Though the panel's mandate to apply new standards of care was the initial impetus for change, OMH saw this as an opportunity to promote high-quality outpatient services.

The licensing system as it existed at the time of the panel's report in 2008 was a tiered process focused largely on policies and documentation. This licensing process usually included a discussion with the clinic administrator and a review of medical records, policies, meeting minutes, and safety-related procedures. The panel report included guidelines for standards of care at mental health clinics (3). The guidelines focused on new areas of integrated, coordinated care through “clinical homes” (8); risk assessment; information sharing across providers; services based on individual and family needs; and engagement and clinical supervision. The recommendations challenged OMH to identify licensing procedures that would capture these changing responsibilities. This column describes the first steps in redesigning and implementing the clinic licensing system in New York State.

Redesign of the licensing system

OMH proposed a redesign with three aims: to base the licensing process on the guidelines for standards of care, to focus the process on improving care and emphasizing family involvement, and to model strengths-based interactions to reinforce how OMH would like clinics to interact with people with serious mental illness. To accomplish these goals, OMH created a work group that solicited input from central and field office staff; experts on clinical service, psychometrics, engagement, wellness, and co-occurring disorders; people with serious mental illness and family members; family advocacy groups; service providers and provider associations; and special interest groups. In collaboration with these stakeholders, OMH proposed a tracer methodology (9) to create an interactive process to assess current clinic functioning, evaluate systems for providing care, evaluate the experience of services for people with serious mental illness and their families, and provide clinically relevant and actionable feedback to clinics (10).

Tracer methodology, as promoted by the Joint Commission (9), is a tool to assess a facility's compliance with selected standards and to evaluate systems for providing care and services. To use the tracer methodology, individuals with high service needs are selected and, using the medical record as a “road map,” each service contact is traced to evaluate whether program components were compliant with standards of care. Discussion of findings with organizations can also provide relevant feedback to guide mental health service improvements (11).

Throughout the development process, OMH openly shared presentations and data on the OMH Web site, sought feedback from stakeholders, and pilot-tested survey instruments and procedures.

Tools for implementing redesign

OMH developed three tools to support the redesign process.

Site visit protocols

Protocols to select tracer cases directed OMH survey teams to choose individuals with high service needs, such as individuals needing complex care (for example, those with co-occurring medical or substance abuse problems) or with recent use of intensive services, such as hospitalization. To pilot the protocols, OMH survey teams included two central office staff and two or three field office staff from regional offices who conducted site visits and surveys; currently survey teams include two field office staff. OMH developed spreadsheets that listed clients who had used a particular clinic in the past year and who met criteria for having high needs. On the morning of the visit, clinic staff were asked to recommend the most high-need individuals on the appointment list for that day, and the OMH team then compared the clinic's list with names on their spreadsheets and, when possible, selected the matching individuals to trace. Using the medical record as a “road map,” surveyors interviewed people receiving services, their significant others, program clinical and administrative staff, and other service providers to evaluate whether service contacts were compliant with standards of care. After the initial interview, clinic staff and state surveyors reviewed cases together, compared information, compared scoring, and discussed each item to determine areas of consensus, clarity, and ambiguity.

Licensing instrument

The team developed care standards and integrated them into a licensing instrument that would maintain a strengths-based approach, promote person and family centeredness, reflect the current evidence base involving treatment engagement, and include a quality improvement perspective that would continuously push clinics to increase their quality of care. OMH worked with clinicians and administrators to translate standards of care into 31 measurable criteria and examples and to assign them to three levels of competency: exemplary, adequate, and needs improvement. OMH incorporated evidence-based practice in the exemplary and adequate levels. [The criteria are available in an online data supplement to this article.]

Licensing criteria directly reflected standards of care. Criteria were classified into three categories: exemplary, adequate, and needs improvement. The criteria enumerated in the exemplary category enabled OMH to communicate a clear vision to clinics and to provide an aspirational goal. The final licensing instrument included a decision matrix for determining the length of the operating license (which ranges from 0 to 36 months). [A table in the online data supplement presents an example of how a standard of care (violence assessment) was classified into the three categories.]

Implementation toolkit

OMH developed a toolkit to aid providers (12). The toolkit includes documents associated with the licensing process; video presentations on topics related to the standards of care and on implementing best practices in clinics; a questions-and-answers page addressing topics raised by providers and other stakeholders; and a dedicated e-mail address for questions, suggestions, issues, and comments.

Pilot testing and site visits

Central office and field office staff field-tested the licensing instrument and then pilot tested the redesigned process with 11 clinics, soliciting feedback from programs. These activities identified strengths and potential shortcomings of the instrument as a licensing tool, determined the viability of tracer methodology, assessed the validity of the scoring protocol, and provided opportunity for field office and central office staff to work together to ensure consistency in implementation.

Two or three clinics from each region of the state took part in the pilot testing of the new licensing standards, scoring protocol, and tracer methodology. These 11 clinics were promised a minimum of an 18-month operating license to encourage participation. The clinics were a mix of not-for-profit (N=4), county (N=3), hospital-based (N=3), and state-operated (N=1) clinics. The clinics served a variety of populations, including adults (N=4), children and adolescents (N=1), and both adults and children (N=6). Clinics also were a mix of those with satellite offices (N=4) and those without (N=7). Licensing staff reviewed materials and methods with each provider before the site visit and throughout the process.

The OMH survey team conducted pilot visits, spending two days at each of the 11 clinics. At each site, staff examined at least four open records (range 4–8) and the records of two individuals no longer engaged in services (range 2–6) to identify individuals to trace. Interviews were generally conducted by one field office and one central office staff member; some interviews, such as those with administrators, were conducted by the entire OMH survey team. During the visit, staff interviewed two or more individuals receiving services at each site (N=26 interviews). Staff also conducted in-person and telephone interviews with 12 family members or significant others, ten collateral service providers, and 86 clinic staff, including clinicians, administrators, nursing staff, and quality assurance staff. The OMH survey team concluded visits with an exit conference to discuss survey findings and invited clinics to complete an online survey about the visit.

Outcomes of the new process

Under the new process, five of the 11 clinics (45%) received an operating license for fewer months of operation than in the previous licensing process, and six clinics (55%) received a license for more months of operation. The current process was comparable to the previous process, in which most clinics received a license for the same period of operation as previously. A common challenge identified by the tracer methodology, which might not have been identified by the former licensing procedures, was that providers were reluctant to share information about service users with other providers. The new licensing procedures prompted discussion between clinic and OMH staff about confidentiality and privacy laws that allow communication for treatment purposes.

Clinic staff members' responses to the online survey indicated that the new process was more focused and clinically relevant, that the revised standards were reasonable and appropriate, and that the survey team's contact with people receiving services and their family members increased OMH's ability to assess the quality and effectiveness of the treatment offered. (The survey results are available from the authors.) Clinic staff reported that they felt more involved in the process and that the process was more collaborative, affirming, and clinically relevant. They also said that there was more consensus and clarity on clinic functioning than in the previous licensing processes and that the new process was more likely than the previous system to identify clinics' exemplary activities, which was more affirming. One provider wrote, “In my 20+ years of experience, our 2010 recertification process was by far the most patient-centered, clinic-enhancing state process I have overseen.” Implementation of the redesigned licensing procedures began statewide in April 2010.

Conclusions

We are not aware of other state mental health authorities' efforts to radically redesign licensing procedures, aimed at increasing the person-centeredness of licensing processes and evaluating clinically relevant standards of care. The previous process emphasized policies and procedures, minutes of meetings, and adherence to medical record documentation, whereas the tracer methodology allows for more accurate assessment and follow-up of clinical issues. Key collaborations contributed to successful development of the tracer procedures, with all stakeholders in agreement that they would like every clinic to achieve exemplary status. Both the OMH survey team and clinic staff reported a more positive licensing process focused on quality enhancement and clinical issues, rather than the previous focus on paperwork.

There are still challenges in continuously incorporating feedback from clients and providers, ensuring consistency among the OMH survey team, and maintaining consensus. OMH staff anticipate that as research-informed best practices, stakeholder perspectives, OMH priorities, and clinic performance change, the licensing processes will require continued updating. Future efforts will include applying the redesigned procedures to other types of programs and assisting other states that want to adopt similar procedures.