Should you believe what smart people believe about climate change?
Subscribe to Decision Science News by Email (one email per week, easy unsubscribe)
EVALUATING THE CREDIBILITY OF ENDORSERS AND DOUBTERS OF CLIMATE CHANGE
In science, you are not supposed to believe something simply because other people believe it, even if those other people are really smart. Like the Hollywood narrator, we can think of examples where “one man (1), in a world of doubters, stands up for what he knows to be true”. Galileo was sent before the Roman Inquisition for his views, and mainstream physicists rejected Einstein’s theory of relativity; one Nobel Laureate referred to it as “a Jewish fraud” (2). Thank goodness they didn’t let the prevailing views keep them from publishing what they found.
However, despite what makes a good Hollywood story, the inconvenient truth is that if you think one thing and a lot of smarter and more knowledgeable people think you are wrong, you probably are wrong.
Sure, there’s Galileo, Einstein, the Asch experiments and Tetlock’s book, but where would we be if we didn’t take the word of those with intelligence and experience?
Really stupid, that’s where.
At a certain level of acceptance, a reasonable person will accept something as true enough to believe in and get on with life. We can’t re-run every experiment in the history of science. The good news is that due to homo sapiens’ brilliant capacity to accept some counter-intuitive matters on faith, we gullibly accept fanciful notions like atoms, viruses, and Greenland to make good decisions about chemical engineering, disease prevention, and navigation.
Even rationality, which people in the decision sciences care so deeply about, originated in the Enlightenment as a description of what smart people (les hommes éclairés) (3) believe. Rationality theory at its birth was just a theory of the cognitive psychology of smart people. As the beliefs of smart people changed over time, rationality theory bent in subservience (4).
So, here’s the question of the day. If you are a scientist, what should you believe about your beliefs when they contradict the beliefs of a lot of smart people?
Story time. In graduate school, your Decision Science News editor was chatting with his statistics professor, Steven Stigler (5). The topic was the limited usefulness of p-values. Scientists seem to wish that p-values referred to the probability that a hypothesis is true (and some actually and wrongly believe this, see 6). However, they actually reflect the probability of the data given that the null hypothesis is true. A young Decision Science News remarked that this probability isn’t all that interesting.
“Well”, Stigler said, “When the p-value is very small, it’s either the case that the null hypothesis is false, or that something extraordinary has happened. Both of those seem pretty interesting.”
End of story. Time to link story to the “one man against the world” scenario.
One man believes “not X”, the scientific world believes “X”. We the bystanders want to know the probability that either is right. But we can’t know that. Furthermore, we are not experts in every scientific discipline, and do not have time to become experts.
What we bystanders probably do is run intuitive statistics on the distribution of expert opinions. We guesstimate the probability that we’d observe the data we do (all these smart and knowledgeable standing behind “X”) given that “not X” were true. We estimate this to be a small probability. After all, the smart and knowledgeable people who become scientists are a skeptical bunch. They’re doubters by default and they all want to be Galileos who get immortalized for standing apart from the pack and being proven right. Getting the vast majority of scientists to agree on anything is a feat. We consider this small probability of expert consensus and say “either ‘one man’ is wrong or something extraordinary has happened”. We typically decide that ‘one man’ is wrong, and lo and behold, we’re usually right (7).
Ach, but it gets tricky. Opinions are not i.i.d. Some view overwhelming agreement as less convincing than a bit of disagreement. (Apparently it is written in Maimonides Law of the Sanhedrin (8) “If a Sanhedrin (i.e., a bunch of judges) opens a capital case with a unanimous guilty verdict, he is exempt, until some merit is found to acquit him.” That is, if you’re facing the death penalty and all the judges vote against you, it actually prevents you from being executed. Perhaps the idea is such unanimity is unlikely if the defendant had received a proper defense.)
All of this leads up to this week’s article from Proceedings of the National Academy of Sciences:
Expert credibility in climate change [PDF]
Although preliminary estimates from published literature and expert surveys suggest striking agreement among climate scientists on the tenets of anthropogenic climate change (ACC), the American public expresses substantial doubt about both the anthropogenic cause and the level of scientific agreement underpinning ACC. A broad analysis of the climate scientist community itself, the distribution of credibility of dissenting researchers relative to agreeing researchers, and the level of agreement among top climate experts has not been conducted and would inform future ACC discussions. Here, we use an extensive dataset of 1,372 climate researchers and their publication and citation data to show that (i) 97–98% of the climate researchers most actively publishing in the field support the tenets of ACC outlined by the Intergovernmental Panel on Climate Change, and (ii) the relative climate expertise and scientific prominence of the researchers unconvinced of ACC are substantially below that of the convinced researchers.
The authors claim that not only do most (97-98%) expert climate scientists believe in climate change, but that the small minority who doubt it are of lesser prominence and lower expertise. Publication and citation data are provided to make the argument. The Yahoo Research lunch crowd, all of whom are incredibly smart and all of whom believe in climate change, found the paper to be “awesome” and “hilarious”, but “incredibly fishy”. Sounds like good criteria for inclusion in Decision Science News.
What do you think? [PDF]
NOTES
1) Sorry to the women, but that’s what they say.
2) Einstein: Holton, Gerald (2008). Who was Einstein? Why is he still so alive? In Galison, Peter L., Gerald Holton & Silvan S. Schweber (Eds) “Einstein for the 21st Century: His Legacy in Science, Art, and Modern Culture”. Also, as a Jew I take offense at the Nazi presumption that the Jews couldn’t come up with a better fraud than the theory of relativity.
3) Pardonnez moi, les femmes, main ce qu’on dit.
4) Daston, Lorraine. (1988). Classical Probability in the Enlightenment. Princeton: Princeton University Press.
5) As a graduate student, your Editor become very fond of Statistics and took so many graduate courses, he fulfilled the requirements for a Master’s degree. However, the University of Chicago had a rule that grad student scholarships covered only one Master’s degree and your Editor had already received one in Psychology. Since the costs had already been incurred, your Editor asked if he could give back the Master’s in Psych. The University was not amused.
6) Oakes, M. (1986). Statistical inference: A commentary for the social and behavioral sciences. Chichester, UK: Wiley.
7) Then we die. Sometimes we’re proven wrong after death, but as long as we were correct while alive it’s no grave concern.
8) Chapter 9
There’s a lot of face validity to their argument and my experience would tend to confirm their reasoning, but I know better than to rely on face validity.
My concern is that it seems like a self-fulfilling prophesy. If a scientist tends to buck the trend, of course he or she will be less successful in publishing their research and will receive fewer citations. Thus, the 2-3% may have a history of being out-of-the-mainstream in their field. This could either indicate that they are nuts (and wrong) or simply more creative or countercultural than their peers.
Most of us who are regularly publishing have encountered at least one situation where we’ve tried to publish a paper that doesn’t “fit” the prevalent trends in our field. It can be hard to find a home for these papers and we often have to settle for the less popular journals as an outlet, which means we’ll have fewer citations for that work.
So, while the data is telling, I find the argument to be suggestive and not definitive.
August 21, 2010 @ 5:48 am
I personally trust the smart people in this case because I am by no means an expert, but I still have concerns about this trust. As a lay person, this paper adds to the confidence I place in this trust, however, I doubt it would cause a denier to change their minds. They would easily add it to their evidence for conspiracy folder.
I think Mike brings up a good point about the 2-3% possibly being more counter-cultural than their peers. How pertinent would a “follow the money” analysis be here? For example, the more prominent AGW is as an issue, the more grants become available to climate science research, something the community could possibly be accidentally selecting for out of self interest. What do they do to counteract this?
I am also concerned about the tendency of humans to assume their knowledge is complete and to ignore their ignorance. I know scientists are skeptical by nature, but everyone is susceptible to lapses in judgment. Since I am no expert, I must also trust that climate scientists are actively trying to search for evidence that would disprove their theories.
Also, is it possible to do these types of meta studies on past issues of scientific consensus? For example, for relativity, has anyone looked through historical journal publications to get an idea of how long it took to reach a consensus in the literature? And were there times when 97-98% of physicists agreed, with the 2-3% who disagreed being those with lesser prominence and lower expertise?
August 23, 2010 @ 8:50 pm
The one dissenting opinion I have heard mention of that gave me pause was that of Freeman Dyson, who invented the Dyson Sphere in science fiction (with plenty of science fact). I am still trying to find out more to see why he isn’t convinced that we are seeing climate change caused by human activity. Having experienced things like the massive pollution driving into the San Bernardino valley in CA, it is easy to believe there are consequences of our technology which are bad for the Earth.
November 11, 2010 @ 8:12 pm