Psychiatrists’ Judgments About Antipsychotic Benefit and Risk Outcomes and Formulation in Schizophrenia Treatment
Abstract
Objective
The objectives were to quantify psychiatrists’ judgments of the benefits and risks of antipsychotic treatments of patients with schizophrenia and to evaluate how patient adherence history affects these judgments.
Methods
Weights assigned by respondents to risks, benefits, and alternative drug formulations in the treatment of schizophrenia were assessed via a Web-based survey by using a discrete-choice experiment. Respondents in the United States and the United Kingdom chose among alternative scenarios characterized by various levels of improvement in positive symptoms, negative symptoms, social functioning, weight gain, extrapyramidal symptoms (EPS), hyperprolactinemia, and hyperglycemia and by formulation. The effect of patient adherence history on respondents’ judgments was also assessed. Random-parameters logit and bivariate probit models were estimated.
Results
The sample included 394 psychiatrists. Improvement in positive symptoms from “no improvement” to “very much improved” was the most preferred outcome over the range of improvements included and was assigned a relative importance score of 10. Other outcomes, in decreasing order of importance, were improvement in negative symptoms from “no improvement” to “very much improved” (5.2; 95% confidence interval [CI]=4.2–6.2), social functioning from “severe problems” to “mild problems” (4.6, CI=3.8–5.4), no hyperglycemia (1.9, CI=1.5–2.4), <15% weight gain (1.5, CI=.9–2.0), no hyperprolactinemia (1.3, CI=.8–1.6), and no EPS (1.1, CI=.7–1.5). As adherence decreased, formulation became more important than modest efficacy changes and injections were preferred to daily pills (p<.05).
Conclusions
Psychiatrists favored treatments that primarily improve positive symptoms. Choice of formulation became more important as likely adherence declined.
Treatment of schizophrenia entails careful consideration of the balance between symptom alleviation and avoidance of mild and serious adverse events from second-generation antipsychotics. Clearly, positive and negative symptoms have a huge impact on a patient’s life. What is less clear is the importance to the patient of relieving these symptoms (the benefits) compared with treatment side effects, such as extrapyramidal symptoms (EPS), diabetes, or weight gain (the risks). Physicians carefully consider the importance of benefits and risks when choosing between second-generation antipsychotics or changing a patient’s regimen. Benefits of second-generation antipsychotic treatments include improvements in relapse rates; positive and negative symptoms; and functional measures, such as the Personal and Social Performance Scale. Common risks include weight gain, diabetes or metabolic syndrome, EPS, and prolactin elevation. Two additional key considerations are formulation, typically oral versus long-acting injectables (LAIs), and patient adherence. LAIs have the potential to improve outcomes when adherence has been problematic (1).
Assessing the importance that physicians associate with various benefits and harms is valuable because these preferences influence physicians’ treatment decisions and patients’ comfort with these decisions. The preferences can be quantified with discrete-choice experiments (DCEs), which assess the relative importance of benefit and risk attributes (2–4). DCE is a well-accepted, validated method for assessing how respondents trade off the various properties of alternatives. Studies of schizophrenia treatment with public policy makers, consumers, families, and providers have shown that respondents value reduction of positive symptoms and improved social functioning over reduction of negative symptoms and EPS (5). Other studies have shown that medication side effects are of greater concern to patients and their families than to clinicians (6–10). Therefore, our research question was “Which key benefit and risk attributes of second-generation antipsychotics do physicians trade off and consider when balancing formulation and adherence as adherence changes?”
Methods
DCEs
DCEs ask respondents a series of choice questions, requiring them to indicate which of several hypothetical treatment alternatives they most prefer or judge to be most appropriate. Treatment alternatives are defined by systematically altering the levels of various treatment outcomes or characteristics, such as benefits and risks. This approach is based on the premises that treatments are composed of a set of attributes or outcomes and that the value of a particular treatment is a function of these attributes. This approach has been used in many health care–related applications (11–14), with both face-to-face interviews and, increasingly, online surveys (2,4,15). Analyzing the patterns of responses to the hypothetical choice questions can quantify the relative value that respondents place on each characteristic over the range of outcomes evaluated (2,16).
Study sample
The target sample size was 200 psychiatrists in the United States and 200 psychiatrists in the United Kingdom. Minimum sample sizes in DCEs depend on a number of criteria, including the question format, the complexity of the choice task, the desired precision of the results, and the need to conduct subgroup analyses (17,18). Most DCEs in health care that have included numbers of attributes and levels similar to those in this study used sample sizes between 100 and 300 respondents (16).
Physicians participating in this study were required to be U.S. or U.K. board-eligible or board-certified psychiatrists who treated at least five patients with schizophrenia each month. Psychiatrists were members of Kantar Health’s online physician panel. Kantar Health administered the 20-minute online survey in January 2012. The Office of Research Protection and Ethics at RTI International granted a consent exemption for this study, and the principles outlined in the Declaration of Helsinki were followed.
Survey instrument
To determine attributes to include in the survey, we used an approach based on multicriteria decision analysis and the benefit-risk framework from the Benefit-Risk Action Team (19,20). A list of 35 potential benefits and risks was created and was used in written and telephone interviews with seven key opinion leaders in schizophrenia, which resulted in a ranked list of outcomes that were most critical for treatment evaluations. Working with key opinion leaders, reviews of product inserts, and published literature, seven attributes describing antipsychotic treatments for schizophrenia were identified as the most critical. The seven attributes included improvements in symptomatic response in three domains (positive symptoms, negative symptoms, and social functioning) and incidence of four adverse events (weight gain, EPS, hyperprolactinemia, and hyperglycemia).
The levels describing each attribute were designed to encompass the range of symptoms observed in clinical practice and the range over which respondents are willing to accept trade-offs among attributes (Table 1). To accommodate the different backgrounds of respondents, the levels of positive and negative symptom attributes were defined with three types of language: a level label (for example, “very much improved”), a change in the Positive and Negative Syndrome Scale (PANSS) subscore, and an example. Although the positive and negative subscores of the PANSS provide well-defined degrees of symptoms, respondents without expertise in clinical trials may not know the PANSS scoring system well. The degree of change in PANSS subscore for each level label was based on a linking analysis between PANSS scores and the scores for Clinical Global Impression–Improvements (21).
| Attribute | Level and description |
|---|---|
| Improvement in positive symptoms:a patient initially presents with the positive symptom of severe hallucinations—being almost totally preoccupied with the hallucinations, which virtually dominates patient’s thinking and behavior | Very much improved, 80% reduction in positive symptoms compared with initial condition (example: minimally observable positive symptoms). Much improved, 50% reduction in positive symptoms compared with initial condition (example: hallucinations occur frequently but not continuously and thinking and behavior are affected to only a minor extent). Minimally improved, 25% reduction in positive symptoms compared with initial condition (example: no longer totally preoccupied with hallucinations but continues to have hallucinations frequently or almost continuously; responds to the hallucinations and may treat them as real perceptions). No improvement, <10% reduction in positive symptoms compared with initial condition |
| Improvement in negative symptoms:a patient initially presents with the negative symptom of severe emotional withdrawal—patient is almost totally withdrawn, uncommunicative, and neglecting personal needs as a result of lack of interest and emotional commitment | Very much improved, 80% reduction in negative symptoms compared with initial condition (example: minimally observable negative symptoms). Much improved, 50% reduction in negative symptoms compared with initial condition (example: generally distanced emotionally from the milieu and its challenges but with encouragement can be engaged). Minimally improved, 25% reduction in negative symptoms compared with initial condition (example: almost totally withdrawn, uncommunicative, and neglecting personal needs as a result of lack of interest and emotional commitment; continues to be clearly detached emotionally from persons and events and resists all efforts at engagement). No improvement, <10% reduction in negative symptoms compared with initial condition |
| Social functioning: ability to engage in socially useful activities, including work or study, and in personal and social relationships | Mild problems, impairments are known only to someone very familiar with the patient (for example, spouse or relative). Moderate problems, marked difficulties that are clearly noticeable (still able to perform some functions without professional or social help but does so inadequately; if helped by someone, able to reach a higher level of functioning). Severe problems, severe difficulties in social functioning, and unable to perform socially useful activities, even if helped by someone |
| Weight gain: percentage increase from baseline in the first year of treatment | None, 7%, and 15% |
| Extrapyramidal symptoms: parkinsonism (parkinsonian tremor, rigidity, bradykinesia, or abnormalities of gait or posture), dyskinesia, dystonia, or akathisia in the first year of treatment | No or yes |
| Hyperprolactinemia: ≥2 times the upper limit of normal prolactin levels leading to clinical sequelae in the first year of treatment | No or yes |
| Hyperglycemia: fasting glucose ≥100 mg/dl or 5.5 mmol/l in the first year of treatment for a patient with normal baseline levels (that is, <100 mg/dl) or fasting glucose >125 mg/dl for a patient with baseline fasting glucose 100–125 mg/dl | No or yes |
Before fielding the final survey instrument, a draft instrument was tested in open-ended interviews with 25 U.S. psychiatrists. Interviews were conducted to test the instrument’s clarity, confirm that attributes and levels were appropriate, and assess respondents’ willingness to accept trade-offs among the seven treatment attributes. After these pretests, minor changes were made to the wording.
Each respondent answered eight choice questions on the basis of a statistically efficient experimental design (22–24). The choices consisted of a pair of treatments, each characterized by profiles of various attributes and their levels; physicians were asked to choose which treatment was better for a hypothetical patient with schizophrenia. [An example of a choice question is available in an online data supplement to this article.] In addition, to assess the impact of formulation and adherence, respondents answered a second set of four choice questions. These questions pertained to treatment attributes, such as formulation (daily pill, monthly injection, or injection every three months), percentages of patients with at least 25% improvement in positive symptoms (25% or 50%), and percentages of patients experiencing EPS (5% or 15%). The hypothetical patient in these questions was initially described as fully adherent. After responding to each of these questions, respondents were then informed that the hypothetical patient had missed or skipped either 20% or 50% of his oral antipsychotic medications in the past. Physicians were asked to reevaluate the same treatment, given the new information. [An example of the second set of questions is available in the online supplement.]
Statistical analysis
Responses to the first set of choice questions were analyzed by using a random-parameters logit (RPL) model, where the dependent variable was the medication profile chosen, which was regressed against the attribute levels. The model estimated relative decision weights for the attribute levels shown in Table 1 that best fit the observed pattern of choices. The resulting parameter estimates thus quantified the relative importance of each attribute level (3,25,26). For the second set of choice questions, we used a bivariate-probit model to jointly estimate relative importance weights for the formulation, the chance of improvement in positive symptoms, and the change in EPS, given the patient’s adherence history. All analyses were conducted with NLOGIT 4.0.
In both the RPL and the bivariate-probit models, the model coefficients indicated the change in the relative importance weight that would result from a change in attribute levels. The difference between the importance weights for the best and worst levels for each attribute can be interpreted as the overall mean relative importance between the worst and best levels of that attribute, over the specific ranges included in the hypothetical treatment profiles (11,26,27). Larger mean scores indicate a greater perceived benefit. Therefore, overall mean relative importance is interpreted as the increase in perceived treatment benefit that would result from switching from a medication with the worst level of that attribute to a medication with the best level of that attribute. To facilitate interpretation of the results from both models, we assigned the highest mean relative importance score from each model a value of 10 and scaled the mean relative importance for each of the other attributes relative to the most important attribute. In addition, for each attribute level, we estimated the percentage change in the predicted choice probabilities of physicians selecting a treatment with that attribute level compared with a reference condition (severe positive symptoms, severe negative symptoms, severe social problems, and no adverse events) representing the hypothetical patient not receiving treatment.
Results
Sample characteristics
A total of 1,700 U.S. psychiatrists and e-mail invitations to participate in the survey by Kantar Health. Predefined quotas of 200 psychiatrists per country were collected for the survey, each of whom had to be board eligible or board certified and currently treating at least five schizophrenia patients each month. U.K. respondents received 48€, and U.S. respondents received the equivalent of 43€.
Six respondents, three from each country, always chose outcome A or outcome B for all eight of the first set of choice questions, indicating that they did not pay attention to the choice questions (2). Because of concerns about the validity of these responses, the data were excluded from the final analysis, leaving a final sample of 394 respondents.
Of the 394 respondents, 84% (N=331) had been practicing medicine for ten years or more. Approximately 76% (N=299) were male. Most treated more than 20 patients with schizophrenia each month (68%, N=268) and spent more than 20 hours per week providing direct patient care (94%, N=370). On average, respondents prescribed antipsychotic medications to 90% of their patients with schizophrenia, and 79% of these prescriptions were for oral medications. The respondents believed that, on average, patients treated with oral antipsychotics followed the prescribed dosage 68% of the time.
Mean relative importance weights for outcomes
Statistical analysis of the first set of eight choice questions indicated that respondents considered improvement in positive symptoms to be the most important among the outcomes presented. That is, a treatment that would move a patient from “no improvement in positive symptoms” to “very much improved in positive symptoms” would yield more perceived benefit than any other treatment difference included in the survey. This change was assigned an importance value of 10. The outcome next in importance was improvement in negative symptoms from “no improvement” to “very much improved,” which had a mean overall relative importance of 5.2 (95% confidence interval [CI]=4.2–6.2), indicating that improvement in negative symptoms was approximately half as important as improvement in positive symptoms (p<.05).
The relative importance of the other attributes, in decreasing order, were improvement in social functioning from severe to mild problems (the scale for social functioning extended from mild to severe and did not include a level for no difficulties, because pretesting of the survey indicated that respondents could not accept that a patient with any degree of positive or negative symptoms could have no difficulties in social functioning) (relative importance score=4.6, CI=3.8–5.4), no hyperglycemia (1.9, CI=1.5–2.4), <15% weight gain (1.5, CI=.9–2.0), no hyperprolactinemia (1.3, CI=.8–1.6), and no EPS (1.1, CI=.7–1.5) (Figure 1).
a Extrapyramidal symptoms
The model results yielded additional insights about changes in importance for various levels of improvement. For example, for weight gain, the change from a 7% to a 15% weight gain was about three times more important than the change from 0% to 7%. For positive symptoms, the change from “no improvement” to “minimally improved” was more important than the change from “minimally improved” to “much improved,” which in turn was more important than the change from “much improved” to “very much improved.” [Details are available in the online supplement.]
Initially, results were estimated separately for U.S. and U.K. respondents. However, because no difference was found between the judgments of respondents from the two countries, we present pooled results.
Mean weights for formulation and adherence
In regard to adherence, the percentage of patients with at least a 25% improvement in positive symptoms was the most important attribute, with a mean relative importance score of 10. The attribute next in importance was the percentage of patients with EPS (relative importance score=2.5, CI=1.8–3.2), followed by changing the administration mode from a daily pill to an injection every three months (1.4, CI=.2–2.7). As the hypothetical patient’s adherence decreased, the mode became more important, and injections were increasingly preferred over daily pills (p<.05). For patients who missed or skipped 50% of their doses of oral medication in the past, respondents regarded a 25% improvement in positive symptoms as equal in importance to switching a patient from an oral medication to a monthly injectable. Figure 2 shows the mean relative importance scores and 95% CIs for the three levels of patient adherence.
a Extrapyramidal symptoms
b In a typical week compared with initial presentation
Discussion
This study built on prior work in the area of assessing physicians’ judgments in regard to benefits and harms in three key ways. First, using a structured benefit-risk framework approach with input from key opinion leaders and literature review findings, we identified a set of the critical benefits and risks that physicians consider when making treatment decisions about second-generation antipsychotics. Second, by incorporating formulation into a set of choice questions, we obtained quantitative data about trade-offs (degrees of benefits and degrees of risks) in formulation preference. Third, by providing information on the hypothetical patient’s past adherence to oral antipsychotics, we assessed how formulation preferences depended on physicians’ perception of adherence. These aspects allowed for more refined benefit-risk assessments among second-generation antipsychotics.
A key finding was that physicians regarded minimal improvement in severe positive symptoms as having greater or equal importance compared with any of the four adverse events. Changes in positive symptoms from “severe” to “much improved” were more important than any change in other symptoms or adverse events assessed in the survey. This finding suggested that the main driver in decisions about second-generation antipsychotics was at least some lessening of positive symptoms. Once a patient with severe positive symptoms showed minimal improvement, the difference between gaining further improvement in efficacy and causing adverse events grew larger. A similar argument applied to social functioning, although to a lesser degree. We did not examine the rationale behind the measurements of relative importance. However, a psychiatrist’s first concern might be to stabilize a psychotic patient, considering adverse events as both secondary and much more controllable by fine-tuning the dose and choice of antipsychotic.
A second key finding was the general similarity of the weights given to avoiding adverse events. Although hyperglycemia and weight gain had higher point estimates than the other adverse events, the differences were not statistically significant (Figure 2).
A third key finding was the importance of switching a patient from an oral second-generation antipsychotic to an LAI. For adherent patients, no difference was found between these choices. For a patient with a history of missing 20% of doses, respondents showed a statistically significant preference for both the one- and the three-month LAIs over the oral form. This preference was equal to an approximate 8% change in probability of minimal improvement in positive symptoms. In other words, given the choice between a highly effective oral drug and a somewhat less effective LAI, respondents would choose the LAI. For patients with a history of missing 50% of their doses, the trade-off was even more noteworthy: respondents considered a 25% reduction in a drug’s chance of providing minimal improvement in positive symptoms as less important than switching the patient to an LAI. These results may be of value in both regulatory approval and physician and patient decisions regarding second-generation antipsychotics.
The findings were similar to those of other published studies in which respondents rated reductions in positive symptoms and improvements in social functioning as more important than reductions in negative symptoms and EPS (5). A recent study that compared patient and psychiatrist treatment decisions found that both made treatment decisions primarily on the basis of improvement in positive symptoms but that adverse events were more important to patients than to psychiatrists (28). Other studies have shown that medication side effects are of greater concern to patients and their families than to clinicians (6–10). A particular benefit of DCE studies is that this approach provides numeric data on trade-offs between attributes that are of great value for quantitative benefit-risk analyses.
As with most DCEs, our survey was pretested but not formally validated. We designed the survey so that respondents would interpret the attributes consistently and in the manner intended. However, except for results showing face validity, it is not possible to prove consistent interpretation by respondents. The choice tasks can also be cognitively difficult for untrained respondents, although the training section of the survey included both attribute definitions and practice questions to enable respondents to gain relevant experience before the results were captured. Also common to most DCEs is that all patient scenarios and treatment choices were hypothetical. Unfortunately, DCE tasks limit the number of endpoints that can be considered simultaneously by survey respondents. The structured approach we employed for benefit-risk endpoint selection was a means to mitigate this limitation. The levels for social functioning did not include a “no difficulties” or complete cure level, unlike the levels for positive and negative symptoms. If the full range of social functioning had been included, social functioning might have shown greater importance than negative symptoms. Because of budget and time constraints, this study used a quota sampling approach and surveyed a convenience sample of English-speaking psychiatrists from an online panel, making it difficult to generalize the results of the study to all U.S. and U.K. psychiatrists. Finally, these surveys ideally should be conducted via in-person interviews. However, results from previous online DCEs were, in general, not statistically different from those elicited through face-to-face interviews (29,30), and results from several DCEs that used online physician panels have been published (2,4,15).
Conclusions
Balancing the benefits and risks of treatments is the core of treatment decisions made by health authorities, physicians, and patients. Understanding the importance that physicians and patients place on the various benefits and risks of second-generation antipsychotics and how formulation affects those trade-offs provides insight into past decisions and useful information for future decisions.
1 : New second-generation long-acting injectable antipsychotics for the treatment of schizophrenia. Expert Review of Neurotherapeutics 13:767–783, 2013Crossref, Medline, Google Scholar
2 : How do physicians weigh benefits and risks associated with treatments in patients with osteoarthritis in the United Kingdom? Journal of Rheumatology 39:1056–1063, 2012Crossref, Medline, Google Scholar
3 : A discrete-choice experiment of United Kingdom patients’ willingness to risk adverse events for improved function and pain control in osteoarthritis. Osteoarthritis and Cartilage 21:289–297, 2013Crossref, Medline, Google Scholar
4 : Physicians’ stated trade-off preferences for chronic hepatitis B treatment outcomes in Germany, France, Spain, Turkey, and Italy. European Journal of Gastroenterology and Hepatology 24:419–426, 2012Medline, Google Scholar
5 : Assessing preferences for schizophrenia outcomes: comprehension and decision strategies in three assessment methods. Mental Health Services Research 5:121–135, 2003Crossref, Medline, Google Scholar
6 : Can patients diagnosed with schizophrenia complete choice-based conjoint analysis tasks? Patient 4:267–275, 2011Crossref, Medline, Google Scholar
7 : A test of concordance between patient and psychiatrist valuations of multiple treatment goals for schizophrenia. Health Expectations 16:164–176, 2013Crossref, Medline, Google Scholar
8 : Valuation and attainment of treatment goals in schizophrenia: perspectives of patients, relatives, physicians, and payers. Journal of Psychiatric Practice 18:321–328, 2012Crossref, Medline, Google Scholar
9 : Preference weights for cost-outcome analyses of schizophrenia treatments: comparison of four stakeholder groups. Schizophrenia Bulletin 29:257–266, 2003Crossref, Medline, Google Scholar
10 : Measuring preferences for schizophrenia outcomes with the time tradeoff method. Journal of Behavioral Health Services and Research 32:14–26, 2005Crossref, Medline, Google Scholar
11 : Patients’ preferences for treatment outcomes for advanced non-small cell lung cancer: a conjoint analysis. Lung Cancer 77:224–231, 2012Crossref, Medline, Google Scholar
12 : Magnetic resonance imaging for the investigation of knee injuries: an investigation of preferences. Health Economics 7:595–603, 1998Crossref, Medline, Google Scholar
13 : Factors that affect adherence to bipolar disorder treatments: a stated-preference approach. Medical Care 45:545–552, 2007Crossref, Medline, Google Scholar
14 : Methodological issues in the application of conjoint analysis in health care. Health Economics 7:373–378, 1998Crossref, Medline, Google Scholar
15 : Evaluation of the relative importance of chemotherapeutic and antiemetic efficacy in various oncologic settings. Supportive Care in Cancer 17:405–411, 2009Crossref, Medline, Google Scholar
16 : Conjoint analysis applications in health—how are studies being designed and reported? An update on current practice in the published literature between 2005 and 2008. Patient 3:249–256, 2010Crossref, Medline, Google Scholar
17 Johnson FR, Yang J-C, Mohamed AF: In defense of imperfect experimental designs: statistical efficiency and measurement error in choice-format conjoint analysis. Proceedings of the Sawtooth Software Conference, Orlando, Fla, March 21–23, 2012. Available at www.sawtoothsoftware.com/support/technical-papers/conference-proceedings/proceedings2012Google Scholar
18 : Introduction to stated preference models and methods; in Stated Choice Methods: Analysis and Application. Edited by Louviere JJHensher DASwait JD. Cambridge, United Kingdom, Cambridge University Press, 2000Crossref, Google Scholar
19 : Multi-Criteria Analysis: A Manual. London, Department for Communities and Local Government, 2009Google Scholar
20 : Application of the BRAT framework to case studies: observations and insights. Clinical Pharmacology and Therapeutics 89:217–224, 2011Crossref, Medline, Google Scholar
21 : Linking the PANSS, BPRS, and CGI: clinical implications. Neuropsychopharmacology 31:2318–2325, 2006Crossref, Medline, Google Scholar
22 : Constructing experimental designs for discrete-choice experiments: report of the ISPOR conjoint analysis experimental design good research practices task force. Value in Health 16:3–13, 2013Crossref, Medline, Google Scholar
23 : Marketing Research Methods in SAS: Experimental Design, Choice, Conjoint, and Graphical Techniques. Cary, NC, SAS Institute, 2010Google Scholar
24 : Efficient experimental design with marketing research applications. JMR 31:545–557, 1994Google Scholar
25 : Conjoint analysis applications in health—a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force. Value in Health 14:403–413, 2011bCrossref, Medline, Google Scholar
26 : Patient benefit-risk preferences for targeted agents in the treatment of renal cell carcinoma. PharmacoEconomics 29:977–988, 2011Crossref, Medline, Google Scholar
27 : Treatment preferences and medication adherence of people with Type 2 diabetes using oral glucose-lowering agents. Diabetic Medicine 26:416–424, 2009Crossref, Medline, Google Scholar
28 Markowitz M, Levitan B, Mohamed AF, et al: Psychiatrist and patient preferences for benefit and risk outcomes and formulation in schizophrenia treatments: comparison of two conjoint analyses. Presented at the Annual Meeting of the American Psychiatric Association, San Francisco, May 18–22, 2013Google Scholar
29 : Testing for the survey mode effect on contingent valuation data quality: a case study of web based versus in-person interviews. Ecological Economics 62:388–398, 2007Crossref, Google Scholar
30 : Use of the internet for willingness-to-pay survey: a comparison of face-to-face and web-based interviews. Resource and Energy Economics 33:119–129, 2011Crossref, Google Scholar
