A new take on Peer Review
Today I was reading freethinking blog samizdata, and got to thinking. A lot of people have trouble accepting the work of peer review, especially when the results of peer review don’t ‘jive’ with their political stripe, or perhaps when they suspect that there is a significant bias involved against their theory of choice.
My idea to try to bring some clarity into the real bias would be for journals with peer review in place should post not only the articles that made it through, but each article/edit of an article which did not. Not in the same place, of course–but there should be a tool which generates a random paper, and you, the reader are supposed to act as a judge, as to whether or not you would accept the paper in the journal if it were up to you. Not whether you accept the article as right, but whether you accept it as reasonable enough to make its way into the journal.
The next step would be, to allow you, as the general public to try to rate a good deal many of these papers, and then compare your results with the results of the actual body of ‘peers’ which does the ‘official’ reviewing. Comment threads could be allowed to provide places for differences / errors to be discussed. This would allow a Bayesian individual to gauge to some extent how rational the peers of that journal are, and in turn, how much respect to give that journal.
Of course, one could still be wrong in assuming what is and is not rational, however being given the access to the inner workings of a peer review journal would give you more reason to trust/not trust a journal than exists today. Climate change skeptics, for example, might find that the science involved really is well thought through, or at the very least beyond their ability to understand and that they should perhaps either trust the IPCC more or trust their perhaps uninformed views less. Non-climate change skeptics could also use the service to ’sanity check’ their beliefs in the IPCC’s results.
The best part about such a system is that the hard part–gathering the data–is already done. Archiving the data might be a little bit more costly than the current approach(toss the crap, only publish the good stuff), but even a 1% acceptance rate by a journal would leave only two orders of magnitude; ie, hard drives are advancing in capacity such that this would only be a big deal for a few years before the drive capacity made the archival a non-issue again. The engine for rating / creating Bayesian results wouldn’t be too difficult to write, and would look a lot like existing tools such as the Open Source engine running Reddit.
Open Access journals and journal aggregators such as arxiv are the first step of a reformation of how science is done, all levels including social and technical ones, that computer technology has started.
Thoughts?
Tags: open access, peer review, science, scientific journals, scigen
December 19th, 2008 at 3:30 pm
I came over here following your post on Samizdata.
The idea you suggest is a very interesting one but I am not sure how well it would work in practice. You might find that most people might not be able to judge the paper properly because, as one of the links I posted on that Samizdata thread says, scientific papers are so turgid that readers who are not specialists in the relevant field may “die of boredom.”
The only way to find out would be for someone to give it a try.
December 19th, 2008 at 11:56 pm
I like the idea. Old journals which played the role of taste-makers based their value on exclusivity – the fact that you couldn’t get their content anywhere else.
But today, and in the future, the problem when anyone is trivially enabled to publish their own work, is that there’s too much content. The problem isn’t that you can’t find work elsewhere because it isn’t published, but because you can’t find work because it’s //buried in a sea of other work//.
The role, then, as a trusted taste-maker, as a source of information that retains certain qualities, remains relevant, even though the conditions of scarcity and abundance have changed.