Double-Blind Review (Nov 2011)

DOUBLE-BLIND REVIEW IN THE AGE OF GOOGLE AND POWERPOINT

By Lorraine Eden, Texas A&M University, leden@tamu.edu

Blog No. 2011-03 (November 1, 2011)

NOTE: This posting first appeared on THE ETHICIST, the Academy of Management's ethics blog, http://connect.aomonline.org/resources/614c81fe4f/summary, in July 2011. The AOM's THE ETHICIST has three contributors, each one writing a monthly blog in sequence, on (1) research (Lorraine Eden), (2) teaching (Kathy Lund Dean) and (3) professional life (Paul Vaaler). Additional copies of my blog posts, in PDF format, are downloadable from my LinkedIn page at http://www.linkedin.com/in/transferpricingaggies.

Key Insight:  Double-blind peer review is one of the academy’s most cherished principles.   Its purpose is to ensure that our scholarly journals make decisions to accept or reject manuscripts based solely on the quality, fit and contribution of the paper. Double-blind review, however, has costs as well as benefits, and may be more fiction than fact in today’s world of Google and PowerPoint. 

1.       Case Examples

Example 1: I was sitting in a job talk where the candidate was reviewing his research activities. The activities were nicely organized into “papers published”, “papers under review” and “in progress”. What caught my attention was the list of journals attached to the “under review” category. The job candidate had listed author names, the paper title, the journal where the paper was under review, and the stage in the reviewing process (first round, R&R, etc.). He then proceeded to briefly discuss each of the papers – even though there was a fairly high probability that one or more of his reviewers was in the room.  

Example 2: I was on a search committee, reading job candidate applications and CVs. I was also writing a couple of P&T letters for candidates up for tenure and promotion at their universities.  In both situations, almost all the CVs listed “work in progress” and provided full details of authors, paper titles, names of journals, and stage in the reviewing process.

Example 3: I’ve also had to recently update and post my own CV on the Texas A&M internal “Howdy” web portal.  State law in Texas now requires that all faculty teaching undergraduate courses must post their CV inside the University web portal so that it is accessible to anyone inside the portal.  I just checked the system, and yes, I am following the herd here. My CV includes full details on papers under review also. My CV is behind a closed electronic door, but still accessible to anyone inside the TAMU electronic walls.

Is there an ethical issue here? What about double-blind peer review?

2.       The benefits of double-blind review

Double-blind review (DBR) requires that authors and reviewers be anonymous to each other throughout the reviewing process.  DBR means that, as an author, I do not know who is reviewing my paper, and, as a reviewer, I do not know who authored the paper.  Quoting Kathryn McKinley,  “The purpose of double-blind reviewing is to focus the evaluation process on the quality of the submission by reducing human biases with respect to the authors’ reputation, gender, and institution, by not revealing those details”.  McKinley reviews several studies that find DBR does reduce biases and improve the quality editorial process outcomes. 

A Nature editorial in 2008, on the other hand, reported much more mixed results, arguing there was only one clear benefit: reducing the bias against female authors.  In a subsequent editorial, Nature even retracted that statement, noting the DBR studies on gender bias also had mixed results.  (Whether the absence of DBR induces biases based on gender and/or race still appears to be an ongoing debate, however.)

Still, a 2008 survey by the Publishing Research Consortium of 3,000 academics in the sciences and humanities found that 71% of the respondents had “confidence in the double-blind review process” and 56% preferred it to other forms of review (e.g., single-blind review where the author is known but the reviewers are not).  An updated survey in 2009 by Sense About Science  of over 4,000 academics found that over three-quarters of respondents favored double-blind review on the grounds that it was “the most effective form of peer review because it eliminates bias, encourages forthright opinion and allows the reviewer to focus on the quality of the manuscript”.

3.       Problems with double-blind review

Double-blind review is not without its problems, however.  I provide some examples.  First, the Nature editorial noted that reviewers, on average, can identify at least one of the authors on about 40% of journal submissions.  Moreover, reviewers can google the title of a manuscript and often discover the full paper or a conference abstract paper on the internet. This suggests the reviewing process may not be as “double-blind” as we think. 

Second, an article by Xiao-Ping Chen, “Author ethical dilemmas in the research publication process”,  in a forthcoming special issue on journal ethics in Management and Organization Review, provides another criticism. Chen argues that authors can use various unethical strategies to “game” the reviewing system. For example, by sending papers out for review before journal submission, authors may be able to assess which individuals are likely to provide negative reviews. Listing the names of negative reviewers in the paper’s Acknowledgements might persuade the journal editor to not select them as reviewers on the grounds that this would violate double-blind review. Authors might even acknowledge names of individuals known to be hard reviewers, without sending them the paper or receiving comments, deliberately and unethically hoping to influence reviewer selection by a journal editor.

Third, there are some advantages that come with single-blind review. If reviewers know the identity of authors, the questions asked by reviewers can be more pointed. For example, if a team of authors already has two papers published on the same topic with the same database, the reviewers would be more likely to know this and therefore better able to judge the novelty of the current journal submission.  At present, if authors do not fully report their prior research (whether to preserve double-blind review or for the gaming reasons cited by Chen), reviewers may be more likely to over-estimate novelty and more inclined towards a positive review.  My JIBS editorial in issue 41.4, “Scientists behaving Badly”, call this ethical dilemma the “failure to cross-reference”.  

Lastly, email discussions with the advisory panel for THE ETHICIST raised additional concerns. Suppose we wanted to become very serious as authors about maintaining the sanctity of double-blind review. Suppose authors did NOT include information on where their papers were in the reviewing process, either on their CVs or in conference presentations or job talks.  The forced lack of disclosure might be particularly harsh treatment for fresh PhD graduates and junior faculty who need to show their work-in-progress to recruiters.  Moreover, discussing our work-in-progress with colleagues is what we do as scholars; it’s a normal part of the creativity process. Also, given the long lag time between journal submission and publication (often years), it is not surprising that authors present their work at conferences during the reviewing process.  Even at AOM conferences, the rule is that: “Submitted papers must NOT have been previously presented, scheduled for presentation, published, or accepted for publication. If a paper is under review, it must NOT appear in print before the Academy meeting.” These practical concerns suggest that asking authors to “not disclose” information about their papers under review would not only inconvenience the authors, but also damage our professional and educational activities.

4.       Editorial policy options

So, the jury is out: Double-blind review has both benefits and costs. It is not surprising therefore to find that journals vary in terms of their policy choices. The range of policy options can range from no peer review (the editor decides) to the full double-blind review process.  Let me comment on three of the possible policy options.  

First, let’s look at two examples of prestigious organizations that use single-blind review in their journals. 

Nature practices single-blind peer review in all of its journals, where the author is known to the reviewers but not vice versa.  The reason given by Nature for using single-blind review is that: “Nature's policies over the years have generally moved towards greater transparency. Coupling that with the lack of evidence that double-anonymity is beneficial makes this journal resistant to adopting it as the default refereeing policy any time soon.”

In July 2011, the American Economic Association replaced double-blind review with single-blind review (SBR) for all of its journals including the prestigious American Economic Review. The reason given by the AEA for shifting to single-blind review, was: “Easy access to search engines increasingly limits the effectiveness of the double-blind process in maintaining author anonymity. Double-blind refereeing also increases administrative costs of the journals and makes it harder for referees to identify an author’s potential conflicts of interest arising, for example, from consulting”. Commenting in the Chronicle of Higher Education on the AEA policy shift, Jonathan Katz, co-editor of the journal Political Analysis, made the memorable quip that “in the age of Google, double-blind has become a fiction”. 

Second, moving to the opposite polar case, the Code of Ethics of the Journal of International Business Studies (JIBS), which I developed as editor-in-chief of the journal, is strongly in favor of double-blind review. The code even has a separate section on double-blind review, and mentions DBR several times.  JIBS does occasionally publish single-blind review articles, and these are identified in the journal as single-blind reviewed. Authors, editors and reviewers are told them must “ensure the confidentiality of the double-blind review process”. Authors are told they must explicitly cite their own earlier work and ideas, but “avoid self-citation that might violate the double-blind review process”. In addition, during the manuscript submission process, authors are requested to “check the box” that their papers will not be posted on the internet while under journal review so as to lessen the likelihood of Katz’s quip about fact becoming fiction. Having said this, I see nothing in the JIBS Code of Ethics that restricts authors from “spilling the beans” to other individuals at conferences, job talks or in their CVs about the status of their papers under review at JIBS, so even JIBS is not as harsh as it could be about DBR.

The third policy option – the “in the middle” one – may be the Academy of Management.  Interestingly, I could not find anything in the AOM Code of Ethics specifically on double-blind review.  AOM journal editors must “ensure the confidentiality of the review process” (4.2.4.2), “ensure the anonymity of reviewers” (4.2.4.4) and “ensure the anonymity of authors” (4.2.4.5). But nowhere in the AOM Code of Ethics are authors restricted from “tooting their own horn” in the ways I have described above. AOM journals, however, do follow the DBR process, according to the journal websites and occasional references to DBR in editorial letters.

My belief (based on asking/emailing a small sample of individuals what they think, discussions at AOM ethics workshops, and so on) is that – if asked – most AOM members would strongly uphold the importance of double-blind review. I suspect they would vote overwhelmingly against AOM journals following Nature and the American Economic Association and shifting from double to single-blind review.   However, this is just a guesstimate on my part.

Still, nowhere do I see admonitions to authors to NOT include the particulars of their work-in-progress and under review articles in their CVs, PowerPoint presentations, and so on.  And regularly I do see individuals as authors violating the DBR process  - and now including myself, a former journal editor who wrote DBR into a code of ethics!

Based on my three mini-cases above, let me rephrase Katz’s remark: In the age of Google and PowerPoint, has double-blind review become a fiction?

5.       Questions for discussion

I hope that my blog posting on double-blind review provides us with some food for thought. Some possible questions for discussion:

1. As an author, if you have a paper under review at a journal that enforces double-blind review, is it OK for you to provide the information about the status of your paper on your CV or in PowerPoint presentations at conferences, job talks, and the like?

2. As a reviewer, how often has an author “spilled the beans” to you about the status of his/her paper in the reviewing process at a journal, and you realized that you were one of the reviewers? If that happened to you, what did you do afterwards? Did you tell the journal editor that you now knew the author? What did the editor decide?

3. As a journal editor, what is your position on double-blind review? If your journal enforces a double-blind reviewing process, what advice as the journal editor do you give authors and reviewers in terms of self-monitoring so that they maintain the double-blind review process? What is your view on authors including full information on the status of their papers in CVs, job talks, and the like? Should authors take “under review at X” and “second R&R at Y” off their CVs, conference presentations and job talks?

4. As a teacher, what do you tell your PhD students? Should or should they not include full information on papers under review, on their websites, CVs, job applications, and in job talks and conference presentations?

5. As an AOM member, what is your position on double-blind review? Should AOM follow the path of Nature and the American Economic Association and shift from double to single-blind review? Should we move to an approach where authors self-monitor and do not share information about their papers under review? Or, like Goldilocks and the Three Bears, is the status quo “middle-sized bowl” just right?

The Ethicist Terms of Use: AOM, contributors to THE ETHICIST, and AOM officers, staff and volunteers accept no responsibility for the content of all postings on THE ETHICIST, including the opinions and information posted or circulated by users on THE ETHICIST.  The content of all postings is solely the responsibility of the users. AOM cannot warrant the accuracy of any information posted on THE ETHICIST and disclaims all warranties with regard to information circulated on THE ETHICIST.  This disclaimer includes all warranties of merchantability and fitness.