RESEARCH RELEVANT TO SOUND DELIBERATIVE PRACTICE

By Ned Crosby

There are increasing efforts to bring professional responsibility to the field of deliberative democracy. Under the leadership of Carolyn Lukensmeyer, conferences have been held bringing together researchers and practitioners from many disciplines and several nations. The Deliberative Democracy Consortium has evolved from this, as a place for dialogue and the identification of key questions that need to be answered. One element of the DDC is an attempt to set up a Deliberative Advisory Group (DAG) that can summarize to the degree possible best practice for different kinds of deliberative efforts. This paper is written to further that effort.

One line of thinking about deliberative practice is that we should be able to learn quite a bit from social science research. If one reviews some of the standard deliberative methods used in the past couple of decades, it is obvious that they differ in a number of respects, most obviously the number of participants, how these are selected and whether they are paid, the amount of time spent, and how information is presented to them. Wouldn't it be nice if we had some good research to indicate which of these characteristics works best under what circumstances?

The aim of this paper is to reflect on some of the challenges we face in trying to do research on deliberation and what best practice might be. The problem lies not so much in the inappropriateness of research to learn about deliberative methods, as in the expense of doing research of sufficiently high quality to be helpful in analyzing the complex deliberative methods now in use.

What can we learn from the field of psychology?

The field of deliberative democracy is about three decades old. Psychology, as a professional field, is about four times as old. This is a field that has had a strong emphasis on both practice (clinical psychology, industrial psychology, etc.) and research. How have they integrated these two efforts?

I am too far removed from psychology to write anything definitive about this, but did have the good luck to work under one of the most prominent academic psychologists of the 20 th century, Paul Meehl. He was as interested in the relationship between practice and research as anyone in the profession. His reflections on how to make sure that practitioners are being as professional as possible are worth repeating.

He noted that one of the most important rules of thumb for himself and those colleagues he respected was “the general scientific commitment not to be fooled and not to fool anybody else”. He went on to say:

One of the deepest, most pervasive dimensions that separate psychologists in these matters is the famous Russell-Whitehead distinction between the simple-minded and the muddle-headed. This difference has little or nothing to do with being bright or dull, since we find brights and dulls on both sides. In the research context, I sometimes have the impression that simple-minded psychologists have a hard time discovering anything interesting, whereas muddle-headed ones discover all sorts of interesting things that are not so. The simple-minded have a tendency to be hyper-operational, too closely tied to rigid standards of evidence (often based upon misconceptions of both philosophy of science and history of science) and a distaste for explanations that seem to them needlessly complex. The muddle-headed may be on the better grounds ontologically, since the world is complicated and the human brain is at least as complex as the kidney. The problem about the muddle-headed is less in their preference for certain classes of explanatory concepts than it is in their often weak standards of evidence…

I am not suggesting that only scientific data in some quantitative form will warrant an alteration in one's belief system and hence one's clinical practice. One's accumulated clinical experience, including conversations about clinical questions with experienced colleagues, is an admissible source of “soft” evidence, as it was for many years in medicine. But granting this, we should keep in mind how many theories and practices in old-fashioned medicine, before the rise of modern laboratory medicine and controlled experimentation and the application of suitable statistics to clinical trials, turned out to be unwarranted and, in fact, killed a lot of patients. Nobody familiar with the history of medicine can reasonably hold that the mere statement, “Clinical experience shows …” is a fully adequate answer to a skeptic, and it is arrogant to conflate “Clinical experience shows…” with “ My clinical impression is…”, when the very fact that the skeptic is putting the question suffices to prove that different practitioners' clinical impressions have not satisfactorily converged.

The task facing the DAG is to find its way between the dangers of simple-mindedness and muddle-headedness. It is worth noting that some of the recent papers published by researchers taking an interest in deliberative methods indicate a leaning in the simple-minded direction.

For example, Gene Rowe and Lynn J. Frewer wrote a paper entitled, “Evaluating Public-Participation Exercises: A Research Agenda”. In it they do a commendable job of reviewing empirical studies that lay out an explicit definition of effectiveness by which to judge participatory efforts. This means that they are attuned to looking at what those in the field are saying, rather than simply concocting their own categories upon which research should be based. But when they get to their discussion of research techniques, they place a strong emphasis on operational definitions and seem less concerned about validity (that a test or an evaluative technique indeed measures what it is supposed to measure). Indeed, it is very surprising that in their brief discussion of different approaches to validity they note simply that they are not going to discuss in any detail construct validity. Yet construct validity, one of the four main ways of validating a psychological measurement technique, is the approach to validity designed to avoid the problems of simple-mindedness. In a single sentence, without any justification, they leave out the approach to validity most likely to do justice to the complex field of deliberative democracy.

Of greater interest is a review article written by Tali Mendelberg, “The Deliberative Citizen: Theory and Evidence”. In this article she does a careful review of a wide range of literature that examines research relevant to deliberative methods. In the abstract at the beginning of the article, she notes that “a review of several literatures about group discussion yields a mixed prognosis for citizen deliberation”. One of the strong points of the article is her even-handedness: she points out research that highlights the strong points of deliberative methods, but also points out a number of potential weaknesses.

On closer examination, however, I had to wonder how relevant the studies she cites are for current deliberative practice. Take, for example, her section entitled, “Are Several Heads Better Than One”. She notes that many deliberative theorists believe that “two heads are better than one” and cites John Rawls as someone holding that view. She then goes on to say, “It turns out, however, that groups have predictable deficits when it comes to sharing information”.

She then reviews 12 articles that demonstrate some of the ways in which this is true. Group members tend to concentrate on information familiar to most of them, spending much less time on information only one or a few know. Even when investigators warn participants that some of the most important information may not be shared information, they still concentrate on what is known by the majority. Mendelberg ends the section with the comment: “Overall, on the issues that matter in deliberative democracy, two heads are not better than one. Two heads can become better than one, but deliberative success requires a detailed understanding of the many and serious social pitfalls of group's attempts to solve problems.”

Thinking this was a matter to be taken seriously, I decided to look at all 12 articles. To my surprise, the large majority of them relied on studies of introductory psychology or sociology students. Ten of the articles reported on a specific study the authors had performed. Eight of these used students in introductory classes, one used “undergraduates” and the last used medical students and third-year interns who were already MDs. All of the studies were conducted in laboratory settings, rather than real life settings, and none of the experiences lasted for longer than an hour, with the possible exception of one.

It was immediately evident how vastly different these experiences were from the experiences in the deliberative methods with which I am familiar. In all the deliberative methods I know of, people are greeted by name and are given name tags. They are made to feel that they are engaged in important discussions. Taking the Citizens Jury process as an example, an hour or so is taken to helping the 24 people get to know each other and feel comfortable in their surroundings. Care is always taken to select a room which has a gracious feeling to it, something that appeared to be ignored in all ten studies, where no indication was given as to whether the setting made the students feel they were in a significant place, engaged in important work.

Of course, the differences only grow from there. In a Citizens Jury three or four days are devoted to witnesses who present information and can be questioned on their testimony. In the one study of the ten using more mature people, the interns and medical students were shown a videotape for about 20 minutes, told not to take notes and told they could not see it again. This may be functional if one wishes to examine some social science hypothesis, but what person in their right mind, seeking to learn how reasonable people might be, would give them only one piece of evidence, not let them take notes and not let them see it again? Participants in a Citizens Jury not only hear many witnesses, but are allowed to recall a few witnesses, if staff is able to rearrange it. It seems that the Citizens Jury process exists in an entirely different world than that of the experiments that Mendelberg is citing.

The two articles that were review articles seemed to have somewhat broader horizons. In “Pooling of Unshared Information during group discussion”, G. Stasser concludes his review by stating that “group discussion is often an ineffective way of disseminating unshared information”. But he does go on to think of circumstances in which this could be overcome. He notes that in teams specifically assembled for their diverse knowledge and made aware of this might do much better in sharing information. But he says that there is “as yet” little empirical information to demonstrate this.

In the other review article, “Bias in Judgment: Comparing Individuals and Groups”, by N.L. Kerr, there were indications that studies other than those done on undergraduates might be considered. For example, he does cite studies in which jurors were used as the participants. But when I checked out the most recent of his four citations, it turned out to be an experiment in a lab setting where people from a local jury pool were paid $15 to come to the university to participate in the typical less-than-an-hour experiment. Kerr also did note that “many economists have disputed the significance of empirical violations of rational-choice assumptions, offering a number of reasons why laboratory demonstrations might underestimate human rationality in real-life settings”. Interestingly, Kerr did not offer any direct citations for this point of view, in spite of the fact that he took the time to include 158 citations in his list of references.

The suspicion that the psychological lab may not reflect the real world has been around a long time. For example, James P. Kahan and C. Daniel Batson, both wrote articles discussing how real life experiences may not correspond to what is found in lab experiments. These were published in 1975 in the Journal of Personality and Social Psychology. Interestingly, eight of the 12 articles cited by Mendelberg in this section of her article were from that journal. If those authors, all of whom have extensive lists of articles that they cite, choose to ignore literature that discusses whether experiments in laboratories are relevant to real life settings, then one must wonder whether the authors care if they are. I did not subject the rest of Mendelberg's paper to the same close analysis. I looked only at 12 of the over 180 citations in her article, so perhaps she cites literature relevant to this situation that I missed.

Nevertheless, it seems to me that the burden of proof rests with the authors of such articles to show why they think them relevant to a practice such as the Citizens Jury process. Unless someone can show otherwise, I assume that these are the kinds of experiments that Meehl had in mind when he spoke of the dangers of the simple-minded.

Clearly all academic work on deliberation is not based on college students in introductory courses. There is a Deliberative Democracy Consortium task force working in this area that may be aware of research more relevant to sophisticated deliberative methods. But before we take the claims of someone like Mendelberg at face value, thorough and balanced as her review article was, we must look closely at the research upon which the conclusions are based and be sure they really are relevant to our work.

Conclusion

Those of us involved in the search for best deliberative practice need to find some way to talk with each other effectively. We need to identify, as best we can, both simple-mindedness and muddle-headedness. We must find some way of agreeing on what research really is needed and how it can be conducted. In doing this, we must pay attention to finding valid measures that are sophisticated enough to do justice to the complex deliberative efforts that we care about.

It is always difficult to assess someone's contributions to a social science discipline. Meehl served as the President of the American Psychological Association and managed one publication about every 100 days for 58 years. Nevertheless, he claimed there were “two or three dozen psychologists of his age group” that made more contributions to the field than he ( Minnesota Psychologist, May, 2004, p.14). He did admit, however, that his range was unique. It is that range and his sophistication in the philosophy of science that make his reflections on his field worthy of attention. My contact with Meehl arose from what was essentially a masters thesis I wrote for him in 1965, “An Analysis of Construct Validity”. I stayed in occasional contact with him over the years until his death in 2003.

These remarks are from an address to the Minnesota Psychological Association on its 50 th anniversary in 1986 Minnesota Psychologist, May, 2004, pp. 3-12

In making this comment, I must note my leanings toward the muddle-headed direction. It would be foolish for me to write this paper with the pretense that I am above sin, while others are not. I also realize that these terms are likely to be loaded for many people. Meehl could get away with using them since his work clearly fell into both categories. I do not mean to demean the work inclines in the simple-minded direction, but to point out a characteristic we need be aware of, in the same way we need to be on the watch for muddle-headedness. Those interested in my claims that I am not entirely muddle-headed could look at a research proposal I wrote in 1976, two years after setting up the Jefferson Center, showing that I had a strong interest in research at the beginning of my work on the Citizens Jury process. It is posted on www.jefferson-center.org .

Endnotes

It is always difficult to assess someone's contributions to a social science discipline. Meehl served as the President of the American Psychological Association and managed one publication about every 100 days for 58 years. Nevertheless, he claimed there were “two or three dozen psychologists of his age group” that made more contributions to the field than he ( Minnesota Psychologist, May, 2004, p.14). He did admit, however, that his range was unique. It is that range and his sophistication in the philosophy of science that make his reflections on his field worthy of attention. My contact with Meehl arose from what was essentially a masters thesis I wrote for him in 1965, “An Analysis of Construct Validity”. I stayed in occasional contact with him over the years until his death in 2003.

These remarks are from an address to the Minnesota Psychological Association on its 50 th anniversary in 1986 Minnesota Psychologist, May, 2004, pp. 3-12

In making this comment, I must note my leanings toward the muddle-headed direction. It would be foolish for me to write this paper with the pretense that I am above sin, while others are not. I also realize that these terms are likely to be loaded for many people. Meehl could get away with using them since his work clearly fell into both categories. I do not mean to demean the work inclines in the simple-minded direction, but to point out a characteristic we need be aware of, in the same way we need to be on the watch for muddle-headedness. Those interested in my claims that I am not entirely muddle-headed could look at a research proposal I wrote in 1976, two years after setting up the Jefferson Center, showing that I had a strong interest in research at the beginning of my work on the Citizens Jury process. It is posted on www.jefferson-center.org .

Science, Technology and Human Values, Vol. 29, No. 4, Autumn 2004, 512-556

Tali Mendelberg in Research in Mircopolitics , volume 6, by Michael Delli Carpini, Leonie Huddy, and Robert Y. Shapiro (eds.) Elsevier Press. (I was unable to obtain this volume and so have based my comments on a paper dated 7-26-2001 .)

  Printer Friendly View