In the last two weeks I’ve noticed a lot of discussion on twitter about the failings of the private and anonymous peer review system as administered by traditional scientific journals. I am a fan of open peer review because I think the transparency has some major advantages over anonymous and private peer review (which I like to call the black hole of science): 1. readers can read the reviews and understand what aspects of the paper were reviewed and where weaknesses might be, 2. having the reviewer’s name attached creates an incentive for careful, honest work and opens the possibility that good review work can be given due credit, and 3. it makes explicit the social network of review so that it can be compared to, e.g. the social network of collaboration.

Among practicing bioinformaticians, a recent focal point for discussing open peer review was the rejection by Bioinformatics of Heng Li’s paper on bwa mem, as described on the opinionomics blog. In that post, Mick Watson suggested that a few people could create open peer reviews and that some action might be taken based on them.

At the same time, I have been working to launch a Bioinformatics discussion group at the University of Technology Sydney where I am just getting started. It occurred to me that a natural extension of a journal club would be the production of a short written review of the paper being discussed. And if that paper happened to be a preprint manuscript from a place like arxiv.org, the feedback could even reach the author early enough to help them improve the manuscript before “official” publication. These thoughts were churning in my head at the time the bwa mem paper rejection was publicized, and since the manuscript is available on arxiv it seemed like a natural choice for discussion and review at our first journal club meeting yesterday.

So I spent several hours dissecting bwa mem, writing about it, and conducting my own evaluation of its accuracy. I’ve posted the review to figshare, where it has been assigned a citeable DOI. As part of this, I decided to compare bwa mem’s accuracy to one of my favorite pairwise sequence aligners, LAST.

At this stage I have no idea whether Heng Li will read what I’ve written, much less like what I’ve written. If I were the journal editor I would definitely ask for revisions to the manuscript. But there were no unfounded claims and the program appears to work as advertised, so a rejection seems unjustifiable.

In the long run I would like to see a slick web system for linking together reviews on preprints, the preprints, and the final published manuscript. For now though, I will suggest to Heng that if he finds any of my feedback useful for improving the manuscript that he should include a citation to the review DOI from within the Acknowledgements section of the final paper. I think this would be a nice way to encourage people to start contributing open review feedback on preprints until a smarter system emerges.

Citation:
Review of bwa mem. Aaron Darling. figshare.
http://dx.doi.org/10.6084/m9.figshare.714096