This report is available in PDF (1.4 MB) only. People using assistive technology may not be able to fully access information in this file. For additional assistance, please contact us.

Executive Summary

This summary presents findings from an extensive literature review on the use of public deliberation as a method for increasing public input for health research. We conducted this review to inform a deliberative methods demonstration conducted as part of the Community Forum project, which seeks to expand public input to the Agency for Healthcare Research and Quality’s (AHRQ) Effective Health Care Program and its comparative effectiveness research enterprise, as well as to advance the field of public deliberation. The deliberative methods demonstration, fielded in fall 2012, entailed 72 sessions conducted using four distinct deliberative approaches. Participants were randomly selected members of the public residing in the Sacramento, North Carolina, Chicago, and Washington DC areas; groups were demographically diverse and composed of 12 to 24 members.

In conducting the literature review, we sought to understand the following:

1. What are the fundamental conceptual and philosophical underpinnings of public deliberation?
2. What are the defining characteristics of deliberative methods? What are the distinguishing features among methods?
3. How has public deliberation been used previously, across fields and to address health issues in particular? Has public deliberation been applied to health research?
4. What comparative studies about public deliberation have been conducted, and what has been found?
5. How are outcomes of public deliberation measured?
6. What are the necessary conditions for successful deliberation and what can inhibit it?
7. What are the key implementation considerations that foster successful and high quality deliberation?

Our review was driven by the practical need to design and implement an empirical study of public deliberation. Therefore, the analysis of the available literature focuses on information that inform strategies to obtain public input on questions related to comparative effectiveness research in health.

Methods

To determine the literature review scope and process, we developed a conceptual framework. We also conducted 17 key informant interviews with experts in public deliberation, technology- based deliberation, and CER to obtain information about public deliberation that might not be in the literature and to gain insight into the utility of public input to health care decisionmaking, and more specifically to comparative effectiveness research. We then developed a search strategy and inclusion and exclusion criteria to guide decisionmaking about whether or not to include an article in the final set for abstraction (Exhibit ES.1). We narrowed all searches to English language resources dated 2002 to 2010 to focus on recent examples of public deliberation but included resources dated prior to 2002 if our team members identified them as seminal works.

Based on the final search strategy, we searched literature in PubMed, ERIC, EBSCO, National Bureau of Economic Research, and Cochrane databases. Across all databases, the team reviewed 1,665 abstracts for relevance.

Although we focused primarily on peer-reviewed literature, we expected that many of the most recent applications of public deliberation would be found in grey literature, defined as literature that was not peer-reviewed but met the inclusion criteria. This literature included presentations, white papers, trade publications, issue briefs, and book anthologies. We identified relevant grey literature by seeking publications from organizations that are widely known to be practitioners of public deliberation, such as AmericaSpeaks, Viewpoint Learning, The Jefferson Center, and Public Agenda.

Results

Identified Resources

From the peer-reviewed and grey literature, we identified and reviewed 206 unique resources. The abstraction team determined thirteen of these to be irrelevant, yielding a final set of 193 abstracted peer-reviewed articles and grey literature. We collected and reviewed an additional 106 resources over the course of the project that were primarily conceptual in nature or which elaborated on abstracted articles; we did not abstract these additional resources but cited them when relevant to the discussion below.

Of the 193 abstracted articles, we classified the majority as single empirical studies (n=94). Of these, four included a control group and had a design aimed at a comparison of deliberative outcomes for a deliberating group and its control. A breakdown of the abstracted literature by topic is in Exhibit ES.2 and by strength of evidence in Exhibit ES.3.

Key Points From the Literature

Findings or key points from the literature were compiled by subject area and aggregated to produce a general picture of public deliberation in the following categories:

• Definitions of public deliberation
• Characteristics of the deliberative process
• Goals of public deliberation
• Uses of public deliberation
• Deliberative methods
• Deliberative tasks
• Participation in public deliberation
• Public deliberation in health
• Evaluation of public deliberation, including outcomes
• Critical components to consider in designing and implementing deliberative processes

The following sections present summaries of these findings.

Definitions of Public Deliberation

The literature presents public deliberation as grounded in the philosophy of deliberative democracy, a specific area of political science whose proponents argue that members of the public should be thoughtfully engaged in and informed about the issues that shape their public life. The process of public deliberation involves members of the public in the careful weighing of multiple, often competing arguments on moral or ethical social issues through facilitated discussion. Deliberation is governed by the principle of mutual sharing of perspectives and respect for differing points of view (Jacobs, Cook, and delli Carpini, 2009; Burkhalter, Gastil, and Kelshaw, 2002).

Despite the range of methods of public deliberation described in the literature, the literature is fairly consistent in suggesting that public deliberation is a distinct method of obtaining public input on decisions that are important to society. Public deliberation asks participants to develop solutions about societal problems or issues posed to the group (i.e., the “deliberative topic”) that inherently evoke competing morals, ethics, or values and for which a range of prospective viewpoints and solutions exist.

Characteristics of the Deliberative Process

Deliberative processes share three broad characteristics:

• A sponsor seeks input from members of the public. Sponsors of public deliberation include public agencies and private organizations. Participants are generally lay persons who are engaged in the process with the perspective of members of the public, as opposed to stakeholders who might have a vested interest in the deliberative topic, field, or area of study.
• Participants consider an ethical- or values-based dilemma. Questions appropriate for public deliberation do not have a simple technical answer or solution. Instead, they involve a tension or dilemma, usually characterized by values and ethics inherent in the issue at hand. The dilemma presented tends to be controversial and relevant to current public policy. The process of deliberation is designed to allow the negotiation of competing viewpoints held by the public on societal problems, such as the equitable distribution of scarce resources, which need to be negotiated in order to establish laws and policies that govern people’s public lives (Gutmann and Thompson, 2004; Fishkin, 2009; Daniels and Sabin, 1997; Chambers, 2003; Young, 2000).
• The process includes educational and discussion-based components (i.e., the “information phase” and the “deliberation phase,” Goodin, 2008). In the information phase, participants are given accurate, balanced information about relevant positions and issues through educational materials, experts, or other sources (Fishkin and Farrar, 2005; Abelson, Forest, Eyles et al., 2003; Luskin, 2007). In the deliberation phase, participants engage in dialogue and the active exchange of reasons and justifications for their preferences, opinions, values, and positions about the matter at hand. Participants are asked to conscientiously engage in an intellectual discussion and receive the perspectives of their fellow participants with openness (Fishkin and Farrar, 2005; Burkhalter et al., 2002; DeVries, Stanczyk, Wall et al., 2010; Chambers, 2003; Young, 2000; Luskin, 2007).

Goals of Public Deliberation

Based on the literature, we identified several goals of public deliberation, listed below.

• To obtain informed public opinion. Traditional forms of public consultation elicit “top of mind” responses – reactions that are not necessarily considered or informed. Because public deliberation is grounded in an educational process and the exchange of reasons underlying beliefs and opinions, it generates informed public opinion (Jacobs et al., 2009; Gutmann & Thompson, 2004; Fishkin, 2009; Mansbridge, 2010; Burkhalter et al., 2002).
• To obtain input that includes underrepresented individuals and groups. For a variety of reasons, traditional methods of obtaining public input may not effectively capture the views and values of minority groups. Public deliberation intentionally convenes diverse groups, so that all members have the opportunity to learn from one another in the process of refining their views (Smith & Wales, 2000; Button & Ryfe, 2005). The informed public opinion obtained through deliberation thus includes views informed by a full range of experiences, as well as the views of members of minority groups themselves.
• To garner insights into social values and ethical principles. Public deliberation is designed to allow discussion and negotiation concerning complex social issues. It generates insights into the public’s values and ethical principles on social issues, which can then be used to shape social decisionmaking (Rawlins, 2005).
• To establish the legitimacy and promote the acceptance of public decisions. Because it brings the views of lay people to decision makers, deliberation can promote legitimacy and acceptance of social decisions among members of the public (Gutmann & Thompson, 2004; Arvai, 2003; Button & Ryfe, 2005).

Although the focus of public deliberation is on obtaining input that will affect public policy and programmatic decisions, many authors also highlight the effects of deliberation on participants as important outcomes. These include:

• Improved understanding of the complexity of decisions. Participants in deliberative processes often gain an increased appreciation of the complex tradeoffs inherent in decisions affecting the public, a point underscored in reference to decisions regarding health programs (Abelson, Forest et al., 2003).
• Enhanced civic-mindedness. Deliberation can foster civic-mindedness in individuals involved in the process,—both increasing their concern for issues affecting the community at large and expanding their capacity for altruism and understanding of others’ experiences (Fishkin, 2009; Gutmann & Thompson, 2004; Gastil et al., 2008; Young, 1996).

Uses of Public Deliberation

As described earlier, the issues that are best suited for public deliberation involve fundamental ethical and social dilemmas. Issues appropriate for deliberation remain unresolved or warrant revisiting because of emerging developments or changing perspectives; for example, as medical evidence advances, the various values or ethics involved in decisionmaking may also evolve.

An important requirement for issues addressed through public deliberation is the potential for finding common ground. By asking participants to take a broad perspective, deliberation elicits social values and principles to inform decisionmaking and is therefore appropriate for issues that affect society. However, some issues are so morally fraught that deliberation may not be an option (Mansbridge, 2010; Mendelberg, 2002). For example, issues that are deeply engrained in religious or cultural views, such as physician-assisted suicide, may not be appropriate for deliberation. As described by one set of authors, deliberation cannot be used to make “incompatible values compatible” (Gutmann and Thompson, 2004, p. 11).

The literature includes applications of public deliberation covering a range of social issues in education, environmental policy, civics, biotechnology, bioethics, public health, and health care. Some fields, such as environmental policy and biotechnology, have a much longer history of deliberative methodology than health policy. Topics that have been addressed through public deliberation range from community to global concerns. Topics found in the literature include: whether and how society should address global climate change (The Jefferson Center, 2007), whether and how to obtain surrogate consent for research participation among patients with dementia (Kim, Wall, Stanczyk et al., 2009), and how to prioritize programs and services to address community health concerns (Abelson, Eyles, McLeod et al., 2003).

Deliberative Methods

We identified nine discrete deliberative methods within the literature: citizens’ juries, citizens’ panels, consensus conferences, planning cells, Deliberative Polling®, deliberative focus groups, issues forums, study circles, and 21st Century Town Meetings®. The most important defining features include:

• Number of participant
• Length of individual sessions
• Duration/number of sessions
• Mode of deliberation (face-to-face or online)
• Use of experts and the opportunity to cross-examine experts

Additional variations include the following:

• Recruitment methodologies (e.g., random selection)
• Provision of educational materials
• Use of breakout groups
• Communication between sessions among participants
• Topic or agenda-setting responsibilities afforded to participants
• Consensus as goal
• Use of polling or measurement
• Cost

Deliberative Tasks

Deliberative tasks are the activities that sponsors or conveners of public deliberation ask participants to undertake. Among 105 applications of public deliberation observed in the literature review, we identified five deliberative tasks. Although these tasks are not mutually exclusive, they illustrate the nature of the tasks that have typically been brought to the public as the focus of deliberative activities

• Priority setting and resource allocation. Participants are asked to choose among alternatives (e.g., designing health insurance benefits with limited resources).
• Risk assessment. Participants are asked to consider the risk-benefit tradeoffs of certain actions, interventions, or exposures (e.g., risk to personal privacy).
• Guidance on ethical or values-based dilemmas. Participants are asked to consider the ethics and values involved in an issue (e.g., ethical issues involved in technologies that extend the human lifespan).
• Development of policy guidance, recommendations, or tools. Participants propose or develop guidance that is intended to inform policy (e.g., solutions to community- based issues).
• Determinations of decisionmaking authority. Participants are asked to consider who or which entities have authority (e.g., considering who should determine health priorities).

Fundamentally, all these tasks are undertaken to learn new information from the public as inputs into decisionmaking. The majority of the applications of public deliberation observed in the literature entailed tasks that were classified as the development of policy guidance, recommendations, or tools.

Participation in Public Deliberation

Participants in deliberation can be recruited through a variety of methods, such as random or convenience sampling. As will be discussed later, recruiting a diverse sample is an important component of public deliberation. In some cases, ensuring that the small-scale deliberative process is representative of the broader public may also be a goal (Fishkin, 2009; Parkinson, 2004; Davies, Blackstock, and Rauschmayer, 2005). Recruitment approaches should align with the goals of the particular deliberative process.

Where conveners want a representative sample, random sampling may be used. However, the literature is incomplete with regard to drawing conclusions about how representative deliberative processes have been of the broader population. Few applications that used random sampling compared participants with non-participants. Even when the sample is demographically similar to the population of interest, the opinions and views of participants may not represent those from the population at large because of small sample sizes (Rowe and Frewer, 2000).

Because inclusion of traditionally underrepresented groups is a goal of many deliberative processes, conveners may employ quota or stratified sampling with targets for certain groups to ensure adequate representation (Parkinson, 2004). Many applications of deliberation in the literature relied on volunteer or convenience samples or hybrid approaches to recruitment, often as resources dictated or because inclusion of different populations was easier with certain approaches.

A promising finding in the literature is that, at least in the United States, most individuals say that they are willing to participate in deliberative processes.

Public Deliberation in Health

Deliberation is useful in health policy because dilemmas based in ethical, equity, economic, or other deep-seated concerns are common. The application of deliberative methods to health care began in the 1990s within the United Kingdom’s National Health Service (Abelson, 2010) as an outgrowth of increasing needs on the part of health policymakers and public policy scholars for more informed, effective, and legitimate methods to involve the public and patients in decisions affecting them.

In the area of health and health policy, deliberation has been used to understand public views on how to:

• Allocate scarce resources and assign accountability. (Abelson, 2009; Baum, Jacobson, Goold, 2009; Chafe, Neville, Rathwell et al., 2008b )
• Set priorities for public action. (Rosenbach, Lee, Hwang et al., 1997)
• Assess the risk-benefit tradeoffs and the distribution of risks and benefits across society associated with decisions, policies, or actions. (Willis, MacDonald, Gibson et al., 2010; Nelson, Andow, and Banker, 2009; de Cock Buning, Broerse, and Bunders, 2008; DeVries et al., 2010)
• Increase political leverage. (Citizens’ Health Care Working Group, 2006)
• Resolve public conflict over fundamental ethical and moral dilemmas, such as the conflict between individual liberty and societal well-being and over the definition of what constitutes societal well-being. (Gutmann and Thompson, 2004)
• Consider public ethics and values in the design and implementation of new health care policy or practice. (Abelson, 2010; Murphy, 2005)
• Increase individual and community interest and knowledge. (Fishkin, 2009)
• Determine the appropriate use of information and criteria setting. (e.g., for decision-making) (Lenaghan, 1999)

Among the 105 applications of public deliberation, we found 67 applications relevant to health These studies addressed such topics as the use of telemedicine in the United Kingdom (Mort and Finch, 2005), the use of cost-effectiveness ratios as a criterion in Medicare coverage decisions in the United States (Gold, Franks, Siegelberg et al., 2007), and government provision of mammography for women ages 40–49 in New Zealand (Paul, Nicholls, Priest et al., 2008).

Evaluating Public Deliberation

We identified 28 articles that evaluated public deliberation, including descriptive case studies, evaluative case studies, nonrandomized comparisons of public deliberation, and randomized experiments. The outcome measures used in evaluations of deliberation can be divided into four broad categories, assessing: (1) processes for implementing deliberation, (2) quality of discourse, (3) impact on participants’ knowledge and attitudes, and (4) impact on individual participants’and societal decisions. Most of the outcomes found in the literature focus on the process of deliberation rather than on the ultimate impact of deliberation on decisions.

1. Processes for implementing the deliberation. These measures seek to answer this question: Was the deliberative process implemented in a manner that supported the quality of discourse and the stated goals of deliberation? Evaluations that focus on these measures describe what strategies work best, on the basis of observations about the process, without regard to systematic measurement of the outcome of the process.
2. Procedural elements tend to be assessed qualitatively. They include aspects of deliberation that can be manipulated by the conveners: educational materials, facilitation, representativeness, transparency, early involvement of participants in the process, and the use of the findings from deliberation consistent with the expectations set with participants (Rowe and Frewer, 2000; Rowe and Frewer, 2005; De Stefano, 2010; Carson and Hartz-Karp, 2005; Jacobs et al., 2009, p. 13). Cost-effectiveness has also been measured (Rowe and Frewer, 2005).
3. Findings from the literature review indicate that elements of the deliberative process that promote richer deliberation and increase participants’ satisfaction with the process include clarity of task, equal opportunity to contribute, independence of the process from external influence or bias, and representation of individuals who are potentially affected by the deliberative topic (Rowe and Frewer, 2000, Fishkin, 2009).
4. Quality of discourse. This set of measures focuses on the extent to which deliberation results in high-quality discourse. Measures reflecting the quality of discourse include fairness, competence, equal participation, active participation, civil atmosphere, opportunity for adequate discussion, respect for the opinions of others, and awareness of different perspectives (De Vries et al., 2010; Kim et al., 2009); Rowe and Frewer, 2005; Melville, Willingham, and Dedrick, 2005; Abelson, Forest, Eyles et al., 2007; Timotijevic and Raats, 2007). These constructs are evaluated through participants’ self-reports of their experience, researchers’ observations, and/ or review of session transcripts.
5. Impact on participants’ knowledge and attitudes. These measures assess the effect of the deliberation on the participants’ knowledge, attitudes, perspectives, values, beliefs, opinions, or policy preferences on the deliberative topics. These outcomes are typically measured through pre/post surveys. The knowledge and attitudes assessed are both specific to the topic (i.e., change in knowledge and attitudes about the topic) and generic (i.e., change in attitudinal patterns among the participants).
6. Studies of deliberative processes have consistently reported changes in knowledge as measured by objective pre- and post-knowledge assessments; changes in beliefs, opinions, or values; and change in participant perception of priorities or policy preference (Abelson, Eyles, et al., 2003; Abelson, Forest et al., 2007; Timotijevic and Raats, 2007; Deng and Wu, 2010; Abelson, Forest et al., 2003; De Vries et al., 2010; Fishkin, 2005; Viewpoint Learning, Inc., 2004).
7. Impact on individual participants and societal decisions. Ultimately, deliberation is conducted to obtain information that will influence decisions. As described earlier, a second impact can be the effect on participants’ civic-mindedness. These impacts are the most distal effects of deliberation and the most difficult to measure. Evaluations of these effects are complicated by a broad range of other environmental factors that may be affecting the policies or other decisions that are the subject of deliberative processes. A few evaluations have assessed the impact of deliberation at the individual behavior and societal levels, using measures outlined below:
   • Impact on individual participants. Measurement constructs include civic-mindedness and capacity, engagement in the political process, sense of self-efficacy, sense of empowerment, political efficacy and solidarity, and anticipated post-meeting activity related to deliberation issues (Fishkin, 2005: Fishkin and Farrar, 2005; Deng and Wu, 2010). These constructs are assessed as change between pre- and post-deliberation attitudinal patterns measured with self-reported participant surveys. Further, deliberation has also been shown to encourage people to adopt a societal perspective and to be more civic-minded and engaged (Melville et al., 2005; Jacobs et al., 2009).
   • Impact on society. Measurement constructs include the effect of public input on specific laws, policies, or practices and on decisionmakers’ intentions to act on the results of deliberation. These constructs are usually assessed through case studies or surveys of decisionmakers who may use the findings from the deliberation (Abelson, Forest et al., 2007; Einsiedel, Jelsoe, and Breck, 2001; Ozanne, Corus, and Saatcioglu, 2009).

Few randomized experiments of deliberation or its alternatives and few well designed comparative studies have been conducted. Thus, the evidence base on the effectiveness of one form of deliberation compared to another, on how various aspects of the deliberative process contribute to outcomes, and on the impact of applying different modes of deliberation (i.e., online versus face-to-face) is limited. Discussion of approaches to record and summarize deliberative dialogue is very limited in the literature, as is the discussion of approaches to reporting the results of deliberative activities to the sponsor of the activity or other decisionmakers who would benefit from the information.

Critical Components to Consider in Designing and Implementing Deliberative Processes

Although there are a variety of deliberative methods and approaches, the literature reveals several fundamental conditions needed to foster successful public deliberation:

• Conveners must use balanced, accurate information to educate the participants and inform the deliberation.
• An atmosphere of conscientiousness and reason-giving must be established during the deliberative process, whereby participants engage in the thoughtful exchange of their positions and rely on reason-giving.
• Deliberative groups should have diverse perspectives. Through deliberation with people who have different experiences and worldviews, individuals are exposed to new ideas and become oriented to the collective or to society at large.
• The diversity of perspectives should be given voice and consideration through equal opportunities to participate.

These four conditions underlie decisions made in the design and implementation of deliberative sessions, such as the definition of the study population, recruitment strategies, use of educational materials and experts, and facilitation—the building blocks that together comprise the full design of a deliberation.

Exhibit ES.4 draws from the literature to describe the prerequisites for successful, high quality deliberation.

Implications for the AHRQ Community Forum Deliberative Demonstration

Findings from our literature review, combined with expert input from the Community Forum Technical Expert Panel (see Appendix), contain implications for AHRQ’s Community Forum project—and for others intending to use public deliberation to elicit informed public input to address a policy decision—including the following considerations:

1. Establish clear intent and goals for the use of public input. Extensive literature demonstrates the importance of being clear to participants on how the results of the public deliberative process will be used to inform decisionmaking, public policy, and program directions. Early involvement of key decisionmakers will promote transparency and promote the uptake of the public input.
2. Ensure that the goals for public deliberation inform the deliberative tasks and methods. Consider tradeoffs among deliberative methods and their features, including group size, session duration, and group composition, and assess their implications for quality and feasibility of deliberation.
3. Ensure transparency of the deliberative process to promote legitimacy and public acceptance. An important indicator of transparency is independence of conveners and managers of the deliberation process from the sponsoring organization.
4. Adapt recruitment methods to the goals of deliberation and the intended populations. An approach that appeared repeatedly in the literature was multipronged recruitment, coupling random sampling with targeted recruitment efforts to ensure representation by groups disproportionately affected by the decision under consideration.
5. Design, test, and ensure access to balanced, fair, and factually accurate educational materials. Materials should be highly accessible, meaning that they should be written in plain language and understandable by all participants. Formative research may be useful.
6. Recognize that expert facilitation is central to high quality discourse. The facilitator is charged with establishing an open, safe atmosphere to foster deliberation and to ensure that participants are practicing reason-giving and have an equal opportunity to participate; training is critical.
7. Ensure optimal use of technology within and outside the deliberation, with equal access and ease of use for all parties. Participants must have access to the required resources (e.g. Internet connectivity) or the resulting inequities may detract from the deliberative process.

The literature presents a comprehensive discussion of the principles of public deliberation, the goals and tasks of the deliberative process, and the range of methods that have been developed. The applications identified in the area of health point to interest in the values and ethics underlying public views on the distribution of health resources, priority setting, and other research and policy questions. Public deliberation has a role in providing insights for decisionmakers, in promoting public acceptance of decisions where community members have contributed, and in increasing the civic-mindedness of participants.

This literature review underscores the current lack of information regarding the impact of various design choices on the effectiveness of deliberation. In addition, it will be important to document, going forward, how public input obtained through deliberative process affects policies, programs, and other decisions. This information will both contribute to our ability to assess the value and impact of deliberative input and demonstrate to participants the value of their involvement.

References

1. Abelson J. Opportunities and challenges in the use of public deliberation to inform public health policies. Am J Bioeth. 2009;9;(11):24-5.
2. Abelson J. Using qualitative research methods to inform health policy: The case of public deliberation. In: Bourgeault I, Dingwall R, and De Vries R, eds. The SAGE Handbook of Qualitative Methods in Health Research. London: SAGE Publications; 2010:608-20.
3. Abelson J, Eyles J, McLeod CB, et al. Does deliberation make a difference? Results from a citizens’ panel study of health goals priority setting. Health Policy. 2003;66(1):95-106.
4. Abelson J, Forest PG, Eyles J, et al. Deliberations about deliberative methods: Issues in the design and evaluation of public participation processes. Soc Sci Med. 2003;57(2):239-51.
5. Abelson J, Forest PG, Eyles J, et al. Examining the role of context in the implementation of a deliberative public participation experiment: Results from a Canadian comparative study. Soc Sci Med. 2007;64(10):2115-28.
6. Arvai JL. Using risk communication to disclose the outcome of a participatory decision- making process: Effects on the perceived acceptability of risk-policy decisions. Risk Anal. 2003; 23(2):281-9.
7. Barabas J. How deliberation affects policy opinions. Am Polit Sci Rev. 2004;98(4):687- 701.
8. Baum NM, Jacobson PD, Goold SD. Listen to the people: Public deliberation about social distancing measures in a pandemic. Am J Bioeth. 2009;9(11):4-14.
9. Burkhalter S, Gastil J, Kelshaw T. A conceptual definition and theoretical model of public deliberation in small face-to-face groups. Commun Theory. 2002;12(4):398-422.
10. Button M, Ryfe DM. What can we learn from the practice of deliberative democracy? In: Gastil J and Levine P, eds. The Deliberative Democracy Handbook: Strategies for Effective Civic Engagement in the Twenty-First Century. San Francisco: Jossey-Bass; 2005:20-36.
11. Carson L, Hartz-Karp J. Adapting and combining deliberative designs: Juries, polls, and forums. In: Gastil J and Levine P, eds. The Deliberative Democracy Handbook: Strategies for Effective Civic Engagement in the Twenty-First Century. San Francisco: Jossey-Bass; 2005:120-38.
12. Chafe R, Neville D, Rathwell T, et al. Deciding whether to engage the public on health care issues. Healthc Manage Forum. 2008a;21(3):24-8.
13. Chafe R, Neville D, Rathwell T, et al. A framework for involving the public in health care coverage and resource allocation decisions. Healthc Manage Forum. 2008b;21(4):6- 21.
14. Chambers S. Deliberative democratic theory. Ann Rev Polit Sci. 2003;6(1):307-26.
15. Citizens’ Health Care Working Group. Health Care That Works for All Americans. Bethesda, MD: Citizens’ Health Care Working Group; 2006. http://govinfo.library.unt.edu/chc/recommendations/finalrecommendations_print.pdf. Accessed December 2010.
16. Daniels N, Sabin J. Limits to health care: Fair procedures, democratic deliberation, and the legitimacy problem for insurers. Philos Public Aff. 1997;26(4):303-50.
17. Davies BB, Blackstock K, Rauschmayer F. Recruitment, composition, and mandate issues in deliberative processes: Should we focus on arguments rather than individuals? Environ Plann C Gov Policy. 2005;23(4):599-615.
18. de Cock Buning T, Broerse JE, Bunders JF. Public perception of prenatal genetic testing: Arguments put forward by the public during a participatory policy project in the Netherlands. Community Genet. 2008;11(1):52-62.
19. De Stefano L. Facing the water framework directive challenges: A baseline of stakeholder participation in the European Union. J Environ Manage. 2010;91(6):1332-40.
20. De Vries R, Stanczyk A, Wall IF, et al. Assessing the quality of democratic deliberation: A case study of public deliberation on the ethics of surrogate consent for research. Soc Sci Med. 2010;70(12):1896-1903.
21. Deng CY, Wu CL. An innovative participatory method for newly democratic societies: The civic groups forum on national health insurance reform in Taiwan. Soc Sci Med. 2010;70(6):896-903.
22. Einsiedel EF, Jelsoe E, Breck T. Publics at the technology table: The consensus conference in Denmark, Canada, and Australia. Public Underst Sci. 2001;10(1):83-98.
23. Fishkin J. Overcoming the dilemmas of democratic reform: From thought experiments to real experiments. American Political Science Association Annual Meeting; 2005, September; Washington, DC.
24. Fishkin J, Farrar C. Deliberative polling: From experiment to community resource. In: Gastil J and Levine P, eds. The Deliberative Democracy Handbook: Strategies for Effective Civic Engagement in the Twenty-First Century. San Francisco: Jossey-Bass; 2005:68-79.
25. Fishkin JS. When the People Speak: Deliberative Democracy and Public Consultation. Oxford: Oxford University Press; 2009.
26. Gold M R, Franks P, Siegelberg T, et al. Does providing cost-effectiveness information change coverage priorities for citizens acting as social decision makers? Health Policy. 2007;83(1):65-72.
27. Goodin RE. Innovating democracy: Democratic theory and practice after the deliberative turn. Oxford, UK: Oxford University Press; 2008.
28. Gutmann A, Thompson D. Why deliberative democracy? Princeton, NJ: Princeton University Press; 2004.
29. Jacobs LR, Cook FL, delli Carpini MX. Talking together: Public deliberation and political participation. Chicago: University of Chicago Press; 2009.
30. The Jefferson Center. Citizens jury: Global climate change. Minneapolis, MN:2002. http://jefferson-center.org/wp-content/uploads/2012/10/Global-Climate-Change.pdf Accessed December 2010.
31. Kim SY, Wall IF, Stanczyk A, et al. Assessing the public’s views in research ethics controversies: Deliberative democracy and bioethics as natural allies. J Empir Res Hum Res Ethics. 2009;4(4): 3-16.
32. Kohn M. Language, power, and persuasion: Toward a critique of deliberative democracy. Constellations. 2000;7(3):408-29.
33. Lenaghan J. Involving the public in rationing decisions. The experience of citizens juries. Health Policy. 1999;49(1-2):45-61.
34. Luskin R. Consensus and polarization in small group deliberations. Paper presented at the American Political Science Association annual meeting, Chicago, IL; August-September 2007.
35. Mansbridge J. Deliberative polling as the gold standard. Good Society Journal. 2010;19(1):55-62.
36. Melville K, Willingham TL, Dedrick JR. National issues forums: A network of communities promoting public deliberation. In: Gastil J and Levine P, eds. The Deliberative Democracy Handbook: Strategies for Effective Civic Engagement in the Twenty-First Century. San Francisco: Jossey-Bass; 2005:37-58.
37. Mendelberg T. The deliberative citizen: Theory and evidence. In: Delli Carpini MX, Huddy L, and Shapiro RY, eds. Research in Micropolitics: Political Decision Making, Deliberation and Participation, vol. 6. Amsterdam: JAI Press; 2002:151-93.
38. Mort M, Finch T. Principles for telemedicine and telecare: The perspective of a citizens’ panel. J Telemed Telecare. 2005;11 Suppl 1:66-8.
39. Murphy NJ. Citizen deliberation in setting health-care priorities. Health Expect. 2005;8(2):172-81.
40. Nelson KC, Andow DA, Banker MJ. Problem formulation and option assessment (PFOA) linking governance and environmental risk assessment for technologies: A methodology for problem analysis of nanotechnologies and genetically engineered organisms. J Law Med Ethics. 2009;37(4):732-48.
41. Ozanne JL, Corus C, Saatcioglu B. The philosophy and methods of deliberative democracy: Implications for public policy and marketing. Journal of Public Policy and Marketing. 2009;28(1):29-40.
42. Parkinson J. Hearing voices: Negotiating representation claims in public deliberation. British Journal of Politics and International Relations. 2004;6(3):370-88.
43. Paul C, Nicholls R, Priest P, et al. Making policy decisions about population screening for breast cancer: The role of citizens’ deliberation. Health Policy. 2008;85(3):314-20.
44. Rawlins MD. Pharmacopolitics and deliberative democracy. Clin Med. 2005;5(5):471-5.
45. Rosenbach ML, Lee AJ, Hwang C, et al. The evolution of the Oregon Health Plan: First interim report. Waltham, MA: Health Economics Research, Inc.; 1997
46. Rowe G, Frewer LJ. Public participation methods: A framework for evaluation. Sci Technol Human Values. 2000;25(1):3-29.
47. Rowe G, Frewer LJ. A typology of public engagement mechanisms. Sci Technol Human Values. 2005;30(2):251-90.
48. Smith G, Wales C. Citizens’ juries and deliberative democracy. Political Studies. 2000; 48(1):51-65.
49. Timotijevic L, Raats MM. Evaluation of two methods of deliberative participation of older people in food-policy development. Health Policy. 2007;82(3):302-19.
50. Viewpoint Learning Inc. Citizen dialogues on covering the uninsured: A report to the Robert Wood Johnson Foundation. San Diego:2004. www.viewpointlearning.com/publications/reports/uninsured_0704.pdf. Accessed December 2010.
51. von Lieres B, Kahane D. Inclusion and representation in democratic deliberations: Canada’s Romanow Commission. In: Cornwall A, ed. Spaces for Change? The Politics of Citizen Participation in New Democratic Arenas. London: Zed Books; 2006:131-51.
52. Willis HH, MacDonald Gibson J, Shih RA, et al. Prioritizing environmental health risks in the UAE. Risk Anal. 2010;30(12):1842-56.
53. Young IM. Communication and the other: Beyond deliberative democracy. In: Benhabib S, ed. Democracy and Difference: Contesting the Boundaries of the Political. Princeton, NJ: Princeton University Press; 1996:120-35.
54. Young IM. Inclusion and Democracy. Oxford, UK: Oxford University Press; 2000.

Appendix. Technical Expert Panel

The Community Forum Technical Expert Panel (TEP), comprising the following six experts in public deliberation, CER, or risk communication, provided guidance on implications for the design and implementation of public deliberation on CER topics, at a meeting held on February 15, 2011, and in subsequent communications.

Julia Abelson, Ph.D., Professor, Clinical Epidemiology and Biostatistics; Associate Member, Department of Political Science; Member, Centre for Health Economics and Policy Analysis (CHEPA), McMaster University

Tim Carey, M.D., M.P.H., Director, Sheps Center for Health Service Research; Professor, Social Medicine and Medicine, University of North Carolina-Chapel Hill

Scott Y. Kim, M.D., Ph.D., Associate Professor of Psychiatry; Co-Director of Center for Bioethics and Social Sciences in Medicine, University of Michigan

Peter Muhlberger, Ph.D., Director, Center of Communications Research, Texas Tech University David Ropeik, M.A., Independent Consultant and Instructor, Harvard University

Mark E. Warren, Ph.D., Professor, Harold and Dorrie Merilees Chair in the Study of Democracy, Department of Political Science, University of British Columbia

Journal Publications

Siegel JE, Heeringa JW, Carman KL. Public deliberation in decisions about health research. Virtual Mentor. 2013 Jan 1;15(1):56-64. doi: 10.1001/virtualmentor.2013.15.1.pfor2-1301. PMID: 23356809.

Wang G, Gold MR, Siegel J, et al. Deliberation: obtaining informed input from a diverse public. J Health Care Poor Underserved. 2015 Feb; 26(1):223-42. PMID: 25702739.

Carman K, Mallery C, Maurer M, et al. Effectiveness of public deliberation methods for gathering input on issues in healthcare: results from a randomized trial. Soc Sci Med. 2015 May;133:11-20. PMID: 25828260.