Public-Academic Partnerships: Tools for Mental Health Agencies to Evaluate Research Protocols

Research involving individuals with chronic mental illnesses covers a broad range of areas, from basic biological and genetic studies to practical, community-based therapeutic interventions. Agencies that provide mental health services to this population must decide to what degree and how to participate in these research efforts.

The ultimate goal of most research on chronic mental illnesses is to improve the treatment of and knowledge about these disorders. Many research protocols, however, have potential risks as well as benefits to participants. For providers, these risks can be a significant deterrent to collaborations with external researchers. The potential harm to agency clientele may be viewed as unacceptable. The agency may judge that its own potential liabilities—legal or financial—would be too high if it sanctioned participation of its clients and personnel. These considerations—harm to client and liability to agency—can be particularly prominent with pharmaceutical studies, which may involve placebos and medications not yet approved by the U.S. Food and Drug Administration. Concerns about financial costs to the agency because of uncompensated use of staff time can arise with psychosocial interventions that involve use of agency personnel. Because most mental health agencies do not view research as central to their mission, many avoid involvement with research altogether. Others may limit research to low-risk projects, such as surveys and analyses of deidentified data, and not permit researchers access to clients in their facilities for recruitment into treatment studies.

Although an agency decision to have minimal or no involvement with research is understandable from a risk-avoidance viewpoint, there are potential negative consequences to the agency, to the clients it serves, and to the field. Some research provides substantial direct benefits to participating consumers and agencies in terms of enhanced treatment and resources. Furthermore, if the goal of the research is to develop or test treatments that are intended to help people living with chronic mental illnesses, then conducting the research in usual treatment settings will garner information about feasibility of the interventions, can increase external validity, and will permit evaluation of effectiveness in addition to efficacy. Research conducted solely with consumers who are recruited via advertisements or word of mouth risks enrollment of unrepresentative groups, making extrapolation of results to "typical" agency clients problematic. Finally, clients served by an agency can and do enroll in treatment studies conducted by researchers with no connections to the agency. In these instances, there may be little or no transfer of clinical information between researcher and agency, to the detriment of continuity and quality of care. If medications are changed or clinical status deteriorates, absence of lines of communication between researcher and agency impedes the consumer's transition back to being seen by the agency. A collaborative sharing of clinical information between agency and researcher is in the best interests of the patient who chooses to engage in research conducted outside the agency.

Given these considerations, there are strong reasons for mental health agencies to engage with the research community, but agencies need to accurately assess potential risks and benefits of research protocols to make informed decisions about which protocols are appropriate and what level of involvement to have with them. These are complex judgments that include consideration of questions not assessed by institutional review boards (IRBs). For example, IRBs are not tasked with assessing how research protocols may affect vital functions, such as billing, continuity of care, and integration of care. Yet research protocols may affect these and other functions quite dramatically.

The concept that the IRB system is insufficient when applied to community-based research has been addressed in a survey of 109 community groups and community-institutional partnerships that have set up their own review process, although the focus of the survey was on ethical issues, not community agency issues ( 1 ). The most common reasons given for establishing an additional, separate review process were to ensure that the community directly benefits from the research (85%), to ensure that the community is engaged (75%), to protect the community from possible risks (68%), and to respond to a growing number of researcher requests to support or participate in their research (41%).

The collaboration

As part of a project to improve research collaborations between our academic center and the local community mental health center, we wanted to develop policies and procedures for evaluating research protocols from the agency's perspective. This undertaking was part of a larger initiative funded by the National Institute of Mental Health under the Interventions and Practices—Research Infrastructure Program, which seeks to foster public-academic partnerships to improve quality and relevance of community-based mental health research. The initiative in San Antonio is a collaboration between the community mental health agency for Bexar County, the Center for Health Care Services (CHCS), and the Department of Psychiatry at the University of Texas Health Science Center at San Antonio.

We formed a group of agency administrators and clinicians, clients, family members, and researchers with experience in working with the agency to identify research protocol parameters needing assessment by an entity other than an IRB and to develop criteria for rating these parameters. An important goal was to go beyond an approve-disapprove protocol decision and instead arrive at estimates, along a continuum, of the protocol's value to the agency and its clients. Membership in the group was voluntary. Meetings were held monthly for a year.

To create a set of common understandings, we devoted the first few monthly sessions to reviewing and discussing several kinds of studies (investigational drug, psychosocial intervention, and genetics, for example) to familiarize members with research terminology and processes, as well as to discuss beliefs, concerns, and values associated with the research endeavor. Mastering the vocabulary and structure of research protocols was a daunting task for group members without research backgrounds. Rather than devoting an inordinate amount of time to these technical tasks, the researcher members summarized the studies in lay language. This decision foreshadowed one of the outcomes of the process, in that it became clear that rewording key elements of the protocol into understandable language was not a particularly burdensome task but was essential for group cohesion and participation by nonacademic members.

Stakeholder evaluation of protocols

For purposes of parsing protocols for domains of interest, the clients and family members took the lead in identifying consumer perspectives and concerns; CHCS staff did this task for the agency. The researchers functioned as resource persons, clarifying and explaining the research protocols in response to questions raised by other group members. Separate rating scales were developed for client and agency perspectives. For each domain the group created a set of anchor points, using a 9-point rating scale. The lowest three scores are characterized as negative, the middle three as neutral, and the highest three as positive. Using the anchor points and the descriptions of the domains, group members individually rated summary protocols put together by the researchers. Discussion of these ratings helped to clarify ambiguities and inconsistencies in the domain definitions and anchor points, which were then changed accordingly.

Two main categories emerged from the discussions: payments and resources, and quality of care. These categories were useful in considering agency and client perspectives.

For the agency, hidden costs emerged as an important payment-resource issue. Examples of hidden costs include having administrative staff pull charts solely for research purposes and asking clinical staff to do research-related procedures or interviews. Conversely, a protocol might provide materials or services of value to the agency at no charge. Provision of medications and medical care to study participants, for example, might conserve agency resources. For clients, the principal resource issue was adequacy of compensation for time and effort.

One major quality-of-care consideration for both agency and clients was continuity of care during and after research protocol participation. For clients, concern about the impact of research on quality of care focused mainly on direct services, whereas for the agency a key question was whether agency staff would receive training and support to sustain research-based interventions once the study was over.

A third domain in client and agency perspectives was related to neither payment-resources nor quality of care. Independent of these domains, conducting a research protocol within an agency might affect, for example, administrative processes (such as billing) and community perception of the agency. For clients, the likelihood that research participation might be an enjoyable or uncomfortable experience was an important factor, separate from considerations of financial compensation and quality of care. This domain was termed "nonfinancial."

A domain not encompassed by the client and agency perspectives is the community perspective, where community refers to groups that may be indirectly affected by the research. These include the local community, the mental health community, the scientific community, and society as a whole. In evaluating a protocol, the agency needs to consider its value to these other groups and to its relationships with them. For example, a research protocol might have little immediate value to the agency or clients, yet its results might provide critically important information to guide development of an intervention for a major clinical problem. Such a protocol would be more highly ranked by an agency and clients on the "community" dimension than a protocol of similar value that would provide knowledge of marginal benefit to the field. [Suggested scoring guidelines for all domains are provided in Appendix A, available as an online supplement to this column at ps.psychiatryonline.org .]

Agency personnel and researchers created new forms for investigators to submit in addition to the research protocol, informed consent forms, and evidence of IRB approval. [These forms are provided in Appendix B, available as an online supplement to this column at ps.psychiatryonline.org .] The forms ask the researchers to summarize the significance of the protocol to the mental health field in language understandable to a layperson. Questions about protocol details are organized along the lines of the criteria that are used to judge the protocol. The researchers are asked to list all uses of agency resources, such as the pulling of medical records for research purposes, use of copying machines, use of clinic space, use of agency personnel as sources of research-related information or for discussing the proposed project with clients, and so on. They are asked to specify any services that will be provided by the researchers as well as the plans for sharing research data and for ensuring continuity and integration of care. The forms also request details about the impact on consumers in terms of payments, demands on their time, services received, assessments that may affect their care, and positive aspects of participation. Much of the information about consumer impact is contained in various parts of the informed consent forms, but it is very helpful for agency reviewers to have it summarized in one place. Instructions for filling out the forms are also included in the packet (Appendix B, available at ps.psychiatryonline.org ).

The review guidelines described here were developed by a committee of agency personnel (clinicians and administrators), clients, family members, and researchers, with the researchers serving in a primarily advisory capacity. One option for agencies to operationalize use of these review processes is to form a committee with representation from these same constituencies. There are, however, practical problems with this approach. Protocols arrive episodically, and scheduling meetings to consider them can cause long delays that pose problems for investigators conducting time-sensitive studies. Especially in the case of multicenter studies, such delays put local investigators at a disadvantage relative to other sites.

An alternative adopted by CHCS has been to have the chief operating officer, client rights officer, and chief medical officer review protocols and determine whether input is also needed from others (such as from administrators overseeing medical records). This group then forwards its recommendation to the chief executive officer, who has final approval authority. If any of these four persons thinks that input from clients and family members is needed before acting on the protocol, the executive administrators refer the protocol to a working group of stakeholders whose primary mission is to serve as an external advisory committee for the agency on a wide range of issues. Use of the forms described above, and available online in Appendix B, has made the review process both more thorough and more efficient, because completion of the forms obviates the need to search through lengthy documents for information that is scattered among them. Although there is extra work for investigators to prepare these materials, the result has been more rapid turnaround time from submission to action, which is viewed as a distinct benefit by investigators.

Some of the content areas for evaluating the impact of a research protocol on the client overlap with areas that fall within the domain of an IRB (including compensation and quality of care). Although the goal is not to duplicate the IRB process, the developers thought that these were areas in which more detailed evaluation from the client perspective is useful to form a complete picture of how clients will be affected by participation in the protocol. One task of the IRB, for example, is to determine whether benefits exceed risks. Beyond this, however, the degree to which they do is an important parameter that the agency can evaluate in judging the potential value of a protocol to its clients.

The protocol-scoring metric may have several uses for mental health agencies. First, if protocols compete with one another for enrollment of clients or for agency personnel time, then the score can be used as a prioritization tool. Second, protocols with high scores may warrant explicit agency commitment of resources to them, because of their value to clients and the agency. Third, numerical scoring brings a discipline to the protocol evaluation process that helps clarify and operationalize agency and client values.

Conclusions

There are important reasons for mental health agencies to participate in research. To make informed decisions about individual research protocols, agencies can benefit from tools to realistically assess risks and benefits from their own perspective and that of the clients they serve.

Acknowledgments and disclosures

This work was supported by grant 5R24-MH72830 from the National Institute of Mental Health to Dr. Miller. The members of the Research Review Group were Cheryl Cooper, B.A., Ed Dickey, B.S., Betty Koenig, Mark McGrath, M.D., Cynthia Ortegon, B.A., Martha Ortegon, and Juanita Rodriguez, B.A., M.A., and authors Dr. Miller, Dr. Dassori, Dr. Gonzalez, and Dr. Velligan.

Dr. Miller received research support from Organon, Pfizer, and Roche-Sanofi Aventis. Dr. Velligan received grant support from Ortho-McNeil-Janssen Pharmaceuticals and AstraZeneca. She is also on the advisory board for Abbott Pharmaceuticals and is a consultant for AstraZeneca. The other authors report no competing interests.