When Michelle Tseng (founding editor of Evolutionary Applications) asked me many years back how I felt about double blind peer review, I was fairly agnostic. Wouldn’t most reviewers be able to guess anyway? Surely the system isn’t biased enough to warrant such an obstacle? How will reviewers know what sort of overlap the study has with other work the authors have published? And so forth. I am now changing my stance. And yes, this change is based on only an N of 1, so I would love to hear the thoughts and experiences of others who’ve gone the process lately (translation: comments welcome!! And no need to agree, of course).
A year and a half or so ago, I was contacted by Derek Lin, a previous undergraduate in John Thompson’s lab at UC Santa Cruz with whom I had collaborated on an experiment examining bacterial resistance against multiple phages. He was contemplating his next career move, and was considering both graduate school and a medical degree, but was currently enjoying his job as a teacher in the Bay Area. Derek also said he was interested in collaborating again and working on another paper, so we set up a time to Skype and compared ideas. After some brainstorming and looking around, Derek came up with the idea for a review article focusing on the human-associated pathogen, Helicobacter pylori, as he had been intrigued by a recent paper suggesting that the range of beneficial to pathogenic symptoms correlating with H. Pylori infection might be due to mismatched strains and hosts (see Kodamam et Al. 2014). I agreed that this would be an interesting topic to explore, and thus began the collaboration.
After over a year of research and back and forth of who knows how many drafts, Derek and I were ready to submit (but were both a bit nervous, as neither of us had ever worked on this particular pathogen before. Would the reviewers wonder why we thought we were in a position to write such a piece?). At the same time, I had an email from Craig Primmer, the Evolutionary Applications Reviews Editor, reminding me that the journal now had a special reviews and synthesis section. I thought to myself: everything’s coming up Milhouse! It was the perfect fit, as we had worked to take an evolutionary angle in reviewing the literature and to put forward some ways in which evolutionary theory could be applied to this topic. We submitted, and a little over a month later had our reviews back.
Okay… Here begins my conversion. Like many of you, I imagine, I am used to fairly patronising reviews that seem to always use the working assumption that I have not thought about alternative interpretations and hypotheses, do not have the expertise needed to write a paper, or am generally a numpty. These are always hard to read because, of course, I do have imposters syndrome and find putting my ideas and research out there into the public domain to be judged hard enough already. It takes me quite a while and a lot of work before I feel confident enough to submit a paper… So having negative reviews, especially when I find them unconstructive and occasionally just plain wrong, but worded strongly, is hard to swallow. I thought this was just how the review process worked, and that I needed tougher skin to stay in this field. This may still be the hard and fast truth, but I have just had the first seed of doubt planted.
When I got the recent reviews back from Evol Apps, I read through them and smiled. Not because they were overwhelmingly positive; they had some really useful criticisms and pointed out key gaps we needed to fill. Rather, my smile was due to the fact that they seemed to have been written with the underlying assumption that we knew what we were talking about (even referred to the piece as an “extensive and up-to-date review” – Rev 1, which “offers a welcome, balanced perspective” – Rev 2). How refreshing! I don’t know who the referees were, but I do know that I suggested 5 names in the field whom I’ve never met but seemed to be leading the way in H. Pylori research. After all, the point of this peer review process was to ensure we had not misrepresented or misunderstood the current state of the field. Later that day I was sharing this story at lunch where it was pointed out to me that the reviewers may have thought the authors were big shots in the field, or at least that they couldn’t rule this possibility out. Indeed, the real benefit of double blind peer review, to my mind, became obvious: the referees had to review the work on the science behind it, not the authors. I won’t speculate here as to whether the difference in the tone of these reviews from so many of my others comes down to a gender issue, or is simply due to my early(ish) career status, or is just a general phenomenon (although there are some empirical reasons to believe the first option may be true in some cases, e.g. here and here). All I can say is that this experience has made me much more likely to consider a journal with double blind peer review in the future. In part because I am a scientist, and don’t believe any study (or blog post) that is not based on properly replicated data.
More resources of interest:
Kodaman, Nuri, et al. “Human and Helicobacter pylori coevolution shapes the risk of gastric disease.” Proceedings of the National Academy of Sciences 111.4 (2014): 1455-1460.
http://blog.sciencewomen.com/2008/01/peer-review-and-gender-bias.html
http://bioscience.oxfordjournals.org/content/56/9/712.short
http://www.nature.com/news/journals-weigh-up-double-blind-peer-review-1.15564
Koskella Lab at UC Berkeley 
Great post, Britt! So, I am a big fan of the idea. I think it would actually help to break up some of the nepotistic trenches dug by some editorial teams who are otherwise so resistant to changes in dogma from “outsiders”.
To not expose any names, myself and colleagues had a pretty important paper on “X in Y” rejected, with two insightful and positive reviews, simply because the third reviewer – whose major comment showed clear lack of methodological understanding of X, but whose more pervasive undertone was “leave the Y research to the Y researchers” – was a well respected Y-ologist. So, despite two great reviews, the editor informed us that he sided with the third reviewer, were not given the opportunity to respond, and upon appeal, were informed of the high status of the Y researcher rather than pointed to which remaining flaw we had failed to convincingly defend. Sending it, along with the reviews and our responses, to another journal, it was accepted in a little over 24 hours.
Thanks for the comment. I agree that seniority in a field is often used by editors to help with decisions, and don’t think this is a bad thing per se… As long as the prominent reviewer is judging based on the science and not the author list! Glad to hear your paper was accepted so rapidly elsewhere!!
Completely agree – my recent experience of double blind peer review at Evol Apps was (sloooow but) good: very constructive reviews with helpful criticisms. It was a paper that combined a couple of fields we hadn’t worked on before, which had clearly led to reviewer concerns when they knew our identities, but mysteriously posed no trouble at all when we were anonymous… as with you n=1 but I’m certainly attracted to DBPR especially for work on new systems/research areas
Thanks for the note and additional datapoint! And nice to hear youve had a good (albeit slow) experience at Evol Apps too.