The Fugue State

Today I was reading freethinking blog samizdata, and got to thinking. A lot of people have trouble accepting the work of peer review, especially when the results of peer review don’t ‘jive’ with their political stripe, or perhaps when they suspect that there is a significant bias involved against their theory of choice.

My idea to try to bring some clarity into the real bias would be for journals with peer review in place should post not only the articles that made it through, but each article/edit of an article which did not.  Not in the same place, of course–but there should be a tool which generates a random paper, and you, the reader are supposed to act as a judge, as to whether or not you would accept the paper in the journal if it were up to you.  Not whether you accept the article as right, but whether you accept it as reasonable enough to make its way into the journal.

The next step would be, to allow you, as the general public to try to rate a good deal many of these papers, and then compare your results with the results of the actual body of ‘peers’ which does the ‘official’ reviewing.  Comment threads could be allowed to provide places for differences / errors to be discussed.  This would allow a Bayesian individual to gauge to some extent how rational the peers of that journal are, and in turn, how much respect to give that journal.

Of course, one could still be wrong in assuming what is and is not rational, however being given the access to the inner workings of a peer review journal would give you more reason to trust/not trust a journal than exists today.  Climate change skeptics, for example, might find that the science involved really is well thought through, or at the very least beyond their ability to understand and that they should perhaps either trust the IPCC more or trust their perhaps uninformed views less.  Non-climate change skeptics could also use the service to ’sanity check’ their beliefs in the IPCC’s results.

The best part about such a system is that the hard part–gathering the data–is already done.  Archiving the data might be a little bit more costly than the current approach(toss the crap, only publish the good stuff), but even a 1% acceptance rate by a journal would leave only two orders of magnitude; ie, hard drives are advancing in capacity such that this would only be a big deal for a few years before the drive capacity made the archival a non-issue again.  The engine for rating / creating Bayesian results wouldn’t be too difficult to write, and would look a lot like existing tools such as the Open Source engine running Reddit.

Open Access journals and journal aggregators such as arxiv are the first step of a reformation of how science is done, all levels including social and technical ones, that computer technology has started.

Thoughts?