Why do we still have the old system of peer review?

The scientific publishing industry is dominated by some big profit-making organisations who are apparently not well liked but are still embraced by the scientific community. Every now and then one of the publishing houses tries to raise its subscription prices by a zillion percent, there is a big outcry from the university libraries and finally they settle for a mere yillion percent and everyone carries on. Why is this?

The answer seems to be that the publishing businesses are quite clever. They bundle journals and sell them to libraries rather than individuals. These days the subscription includes electronic access. Few scientists now pop over to the library to get down a volume of a periodical, they look it up online instead. If the subscription is stopped the research centre loses electronic access to all the past issues as well as current ones. It is just not possible to give that up, so they pay up instead.

Scientists have tried to combat this by setting up their own open access journals and some of these are working quite well. But now there is a concept of impact factor that measures how good a journal is according to how well cited its articles are. The most prestigious journals can afford to filter our submissions that are not likely to get many quick citations so they keep their impact factors high. The impact factors are used to measure how good people’s publication histories are when they look for a new position. This creates a feedback loop that makes the big publishers very powerful.

So is peer review needed? If so, should it, and can it be wrestled from the grasp of big business? could it be done differently? Over at Quantum Diaries Survivor, Tommaso Dorigo informs us that he is due to give a presentation on such questions and he wants to know what we think. The comment section has some interesting discussions.

These days everyone with access to the internet can publish online without peer-review. if they are excluded by arXiv.org they can use viXra.org and of course there are many other archives that scientists use, or they can just publish on their own blog. But these are not regarded as real publications by the scientific community until they have been peer-reviewed. Despite the internet, the system is still based on the principle that you can “publish” when and if you pass the test of peer review. Peer-review is still important primarily because careful verification is essential (especially in mathematics, experimental physics, medicine etc) but also because of the role peer-review plays in assessing the worthiness of scientists when it comes to job promotion. The ability to make your work available as a pre-print before peer-review exists only as a compromise because the publication process is otherwise too slow for many fast-moving areas of research. Of course, not everyone sees it that way. That’s just one traditional view.

The existing peer-review process is imperfect in many ways aside from its cost. Good papers are rejected by peer-review and this has a real effect on the pace of acceptance. A good case study would be the science of climate research where some people argue that peer review has become corrupted and is biased towards one side of an important scientific debate. (see e.g. what Lubos writes at Reference Frame)

In a perfect world things would be done very differently. Repositories like arXiv and viXra would become the publishing medium and peer-review would become an open and public process of critical review by relevant experts. One simple approach would be to have ratings for articles using a system like Digg. This works nicely for news articles and is a good way to filter out stuff of little interest, but it is far short of peer-review. By the way, there is a site called scirate.com which allows you to rate articles on arXiv in this way but it demonstrates one of the most basic problems with these approaches: It is very hard to get anyone interested. Another site that suffers the same fate is arXiv1.org where you can freely make comments on arXiv papers, but very few people do. Another system that almost works is the trackback system where blog comments are recorded on arXiv itself, but the comments are moderated in a way that some think is biased, so it does not qualify as any kind or review. Citation counts form another indicator that is used, but they usually trace back to positive responses. Peer-review also needs to be negative when appropriate

A proper system of open peer review would have to go beyond basic rating and commenting. The process needs to come to some kind of consensus about the validity and general worthiness of a paper. The people who do the reviewing need to be experts on the subject. This means you need a system of identifying experts. This can be done by looking at their qualifications and position to classify their areas and levels of expertise, but that would be open to bias and the corrupt rule of authority, precisely what we want to avoid. A more open system might allow anyone to review and rate a paper, but the ratings would be weighted according to the reviewer’s reputation which is earned according to the ratings of their own papers in the same subject area. Could such a system work or is it just a Utopian dream? This is the question we discussed over at Tomasso’s blog with some interesting comments but no real conclusion.

There is no doubt that such a system would be hard to get working. You would have to overcome the reviewers reluctance to criticise in public. The existing peer-review system is mostly anonymous and for a good reason. Scientists are human and don’t want to be attacked for their negative reviews. If the process is not anonymous they may not be willing to air their criticism. This is a real issue but anonymity and privacy in the peer-review process also make it hard to challenge a review. The system takes on various forms of corruption, for example, journal editors have a lot of power to influence the peer-review process either by directly affecting the result or by selecting reviewers who they know will be for or against the article. This works in both ways, either creating journals where a group of people can publish low quality research, or excluding a valid opposing view to an area of research. More openness where the reviewers judgement can be further criticised in public should be an important goal. This does not necessarily mean that anonymity must be given up. That could remain as an option. It is really the privacy that creates the problems.

It is understandable that there is skepticism about the possibility of establishing a working system of open review. It is hard to get people interested, However, there are some websites that indicate that this problem is not insurmountable. Despite the initial odds, Wikipedia has established a huge system for building works of reference that attracts considerable expertise in many areas. Wikipedia specifically excludes original research and is not a suitable system for peer-review, but it shows that the right people will get involved if a system has the right features.

Another system that is closer in some ways to what we seek is stackoverflow.com and derivative sites such as mathoverflow.net. These are sites for submitting questions in a particular subject area. That is very different from the requirements of peer-review, but the rating system used is getting close to something that might work. On these sites questions are rated and so are answers. People can also comment on answers so anything can be challenged and this feeds back into the rating. People build up reputations according to how well their answers are rated and the reputations increase their rights to rate questions. There are also moderators who are elected democratically with reputation being an influence. These moderators can shut down off-topic discussions. Not only does this system generate a high level of discussion, it also attracts some well-known experts in the field to make contributions. A peer-review system would be something on a grander scale but the principle might be similar and the evidence is that it could work if the details are right.

The technology on which to base such a system exists. Many archives adhere to the Open Archives Initiatives which means they have an API so that you can query them and integrate them with other repositories into conglomerate systems. A peer review website could work that way.  There would be no need to start a new repository for the purpose . Despite Lubos’s flattering suggestion over at AQDS, it is not likely to be me who makes the first attempt at building such a system. It requires more than the technical expertise of one person I think. A small group with the backing of some big organisations would be more suitable.

Meanwhile I am looking forward to hearing what comes out of Tomasso’s presentation. He is a smart and reasonable person so perhaps he can kick-start something that will grow to be the future of peer-review. I just hope it will be a system that is open and not doomed to a fate of elitism and corruption. I hope it promotes honest scientific progress without stifling a valid new approach just because it does not fit the prevailing dogma. We can only hope.


Comments are closed.

%d bloggers like this: