On December 7, 2013, the Atlantic Node of Situating Science Strategic Knowledge Cluster hosted a one day public series of discussions exploring six case studies of overselling, misrepresentation or biasing in the presentation of scientific research. Included in these case studies was a discussion of the NASA supported research article published in Science that claimed the discovery of bacteria from arsenic that was subsequently refuted; a discussion of the widely popular and controversial “Liberation Therapy” for MS; and a discussion of what appears to be a contemporary re-inscription of racism in the new science of cognitive genetics. Each of the discussions evoked lively debate and conversation.
Let me start my comments by saying that I am by no means and in no way a scientist. I struggle with basic arithmetic and can never remember what gold is on that chemical thingy (what do you call that?…Oh yeah, the periodic table!! That’s it). The point being that I cannot be trusted to relay any of the theoretical content of the discussions, though I have to admit, the conversations were largely accessible, even to the scientifically challenged. There were, however, several themes that arose throughout the day that concerned the interaction between science the general public, and I am going to focus on one of them now and will return to the others in later posts. Now I am not an expert on the “general public”, whatever that actually is, but as I am not a scientist, I feel that I somehow belong that exo-scientific constellation and I speak from that situated location.
So I hear that the peer-review process is in crises. Now, I am not sure if all would agree with my assessment that the discussions portrayed such a dire situation, but that’s the way it seemed to me. In her presentation “The “Arseniclife” Debacle,” Rosie Redfield (UBC) discussed an article in Science that reported the existence of arsenic-based bacteria. After this research was published it was subject to intense critique and was subsequently refuted based on a number of outstanding problems (don’t ask me what they were, I didn’t get it and I don’t know). Dr. Redfield led the audience in questioning how such major scientific problems in this, a NASA-supported research project, could have made its way through the series of checks and balances that have been traditionally trusted to safeguard against this very thing. While certainly the peer-review was not the only culprit (and indeed Dr. Redfield suggested other factors that were likely of greater consequence), the starkness of the error is outstanding.
So what’s the problem with peer-review? Well, nothing new here. Writers want readers, sellers want buyers, and everyone wants something new and exciting. Sometimes reviewers are selected based on their known sympathies for the subject of the pending paper, some reviewers are swayed by academic charisma of the research being presented, and sometimes even the pool of potential reviewers is assembled based on political prospects rather than on academic credentials and expertise. I’m sure there are many more reasons why this process is problematic, and I hope readers will bring me up to speed on all the other pitfalls, but the point is that the word on the street is this: Houston, we do indeed have a problem!
What struck me as particularly interesting is the way in which the conversation oscillated between how essential the peer-review process is and how problematic it is. If the issues are as severe as I heard, why not just abandon the whole thing? This reminds of the problem with marriage—it fails somewhere between 40-60 percent of the time (depending who you ask), and these stats do not even include those who may remain married even though they are miserable or those who remain married even though the partners are practically living separately. Yet everyone just keeps doing it, over and over. Abandon the whole idea? What? Society would crumble! Abandon peer review? The science sky would surely fall.
In my own research, I’ve recently been considering a legal case in the US in which the court was trying to sort out the scientific status (and legal status though it seemed that the legal status was secondary) of intelligent design. In this case, one scientist made the argument that science is as close as it comes to a universal language, and that the peer-review process is the essence of this language. ID is not considered science by most scientists because, among other things, its practitioners do not participate in traditional peer-review research forums (though ID advocates have built up a network of their own journals, conferences, etc). But if the peer-review process is what keeps ID ‘science’ at bay, and peer-review is not really working well, then I, the general publican in the room, am nervous.
I wonder if it is not really the peer-review process itself that enraptures scientists but the ideal of peer-review. Scientists, to me, often seem to deal in prestige and credibility, and so it’s perhaps not the impact of peer-review on knowledge claims that is so important but rather the impact on the credibility of knowledge claimers. Perhaps the greatest value of peer-review is that it marks validation and acceptance into a selective social sphere entailing rights and privileges that diminish as the boundaries become more porous. If this is indeed the case, then it’s the presence not the practice of peer-review that really matters, and little wonder scientists are not really interested in seeing it abolished (I have a fun parallel argument for [or against, actually] marriage, but alas I must save it for another time and place).
I am actually not being cynical here at all. As a wary watcher of alternative religious ‘science,’ particularly those alternatives that are widely endorsed by a large proportion of the ‘general public,’ I am nervous about the democratization of science for lots of reasons. But on the other hand, this democratization has to a large extent already taken place. This was spectacularly evident in the presentation by Dr. Murray on “Liberation Therapy” for MS. Dr. Murray discussed how this therapy was widely endorsed and taken up by members of the general public and even the mainstream medical community despite a significant lack of evidence for its effectiveness and a good deal of evidence for its potential harmfulness. People want to and do judge for themselves, and the governing bodies can easily find themselves at the mercy of great public pressure.
The main point that I want to make here is this: If scientists want to revamp the peer-review process, it might be more effective to be clear about what it is they are dealing with. Is it a process of knowledge adjudication, a process of knower adjudication, or both? That much of the conversation at this event focused on protecting/improving knowledge and knowledge claims seemed to me to be a shill for protecting the power and prestige of the scientific community (and the trustworthiness of science by extension), and there’s nothing wrong with that (at least not for me so long as it works to keep a rein on the political powers of pseudo-scientific initiatives, but this is a huge topic for another time). The task is made cumbersome by the tacit cloak of epistemic value. I have no idea whether or not good science can be done without peer-review, but I think that a lot of good science gets done in spite of it.
Limitless 