Peer review – are they really even trying anymore?

by Carl V Phillips

As part of our research on journal peer review in public health — the practice and interpretation of which is a dire threat to THR and other policies based on good science — my colleagues and I found ourselves contemplating this report at Retraction Watch from March, about BioMed Central (BMC) retracting 43 published articles for improprieties in the peer review process. We were bitterly reminded of BMC’s lack of retraction for the travesty of an article by Lucy Popova and Pamela M Ling in BMC Public Health. Those authors claimed that they had demonstrated that harsher warning labels about smokeless tobacco and e-cigarettes were warranted, when actually their results — such as they were — supported the opposite conclusion. You may recall from my previous posts that the Popova-Ling paper was excoriated as unethical, utterly useless, and misleading by me, Brian Carter, Clive Bates, and others. The journal considered retracting it after our complaints, but ultimately decided that was all fine by them.

Popova and Ling did not merely lie in the everyday ANTZ manner. Instead, their article was characterized by:

• Unethical treatment of human subjects (misleading them without any apparent corrective briefing, making and breaking promises to them even though this did nothing to further the research, generally wasting their time given the uselessness of the research),
• A research plan that was incapable of generating any result other than the completely uninteresting, “if you give people a stronger warning about something, they worry about it more”,
• Other serious methodological flaws,
• A rambling political diatribe that reached conclusions that included declaring that stronger warnings were warranted, which were in no way supported by the results,
• Failure to declare their conflicts of interest or make clear the political premises which their conclusions were based on,
• Suppression of the only interesting research finding.

• Moreover, if that suppressed finding, which appeared in a previous version of the manuscript, is considered, it makes quite clear that stronger warnings are not warranted because people already overestimate the risks (though this can also be surmised from what appeared in the article alone).

This is the article that BMC decided was just fine. If mistreating your research subjects and making up claims that are actually contradicted by the only useful finding from your otherwise completely uninformative research is ok, what do you have to do to get retracted? Did the authors of the 43 retracted articles strangle kittens and just make up their data?

It appears that what they did was hire a service that helps English-challenged international authors edit their research papers and submit them. [Update: I should disclose that my wife is an editor for a legitimate one of those services, and thus would be a competitor of those who engage in the nefarious activities described in what follows.] But it seems that this service was also recommending peer-reviewers to the journal (as journals typically ask authors to do) who were not real people or giving fake contact information. The contact information led back to the service, which then wrote the reviews which led to publication of the papers. The report does not make clear whether the authors were fully aware of what they were buying, completely unaware of this scam, or somewhere in the “nudge nudge, wink wink” range in between. So they, unlike Popova and Ling, theoretically could have been innocent victims who were not attempting to mislead anyone. (Given that there has not been an outcry by the authors involved in this scandal, which went beyond BMC journals, this seems unlikely in retrospect, but it was conceivable.)

Here’s the thing: I found a few of these “fake” reviews that still appear online, and they were pretty much typical reviews, arguably above average for what we are seeing in our (Igor Burstyn, Carter, me) analysis of peer-review reports at BMC Public Health. (BMC, to their great credit, publishes the reviews behind the published articles, unlike most health science journals where the entire peer-review process is conducted in secrecy, leaving the reader no idea who reviewed the paper (if anyone) and what they said about it. Not to their credit, they seem to have removed that record for most of the retracted articles.) The reviews recommended some minor improvements though, of course, they did not offer any fundamental criticisms or instruct the authors to make any major changes.

I have seen no claims that any of the retracted articles actively harmed the scientific record or that the authors misreported the research results or declared conclusions that were not supported by the research. So whatever was wrong with the papers, it was not at the level of the active lies and damage to legitimate science caused by Popova-Ling, or the BMC Public Health article by Hughes on e-cigarette use that I wrote about here, of others that we have reviewed in our research. It is possible there were serious flaws in the retracted papers, which the reviewers would have “overlooked”, of course. But it is not as if the “real” reviews of Popova-Ling, Hughes, and others were any different.

The reviews of the retracted papers were written by insiders whose real goal was to make sure the paper got published. But how is that any different from the Popova-Ling or Hughes cases? The Popova-Ling reviewers — presumably also chosen based on the recommendation of the authors, though the published record does not tell us for sure — were also committed to making sure the paper got published: They (Israel Agaku and Saida Sharapova) were both employed by an organization (CDC) that is politically committed to the policies Popova and Ling recommended in their non sequitur conclusions, a clear conflict of interest that both reviewers denied existed. The Hughes reviewers were also apparently unqualified and hugely biased (which they also lied about). The “fake” reviews of the retracted paper I was able to view were better than any of the “real” reviews of Popova-Ling and Hughes, which identified none of the glaring major flaws and overall made the papers worse rather than better.

The analogy I came up with (sportier suggestions welcome) is someone playing a competitive online action video game, in which spotting subtle information on the screen is part of the challenge. If someone hacks the code so that those details are highlighted during play, it violates the rules and he will be kicked out of the competition if caught. But if someone merely has a friend sitting beside him watching for those details and calling them out, it has the exact same effect but no rule-breaking has occurred. Oh, and my likening of the process of getting papers published in health science journals (not health science research itself, but the publication process) to a constructed artificial competition whose only benefits are self-indulgence was no accident.

While there is no excuse for journals not retracting actively harmful articles also, it is easy to understand why they reacted those they did. No one likes to be duped by a clever hacker exploiting the design vulnerabilities in their system. Except, regarding that clever bit…

The above link is to the reviews of a retracted paper about neurology (the reviews had already been removed for the only public health paper in the retracted collection). Apparently three copies of each review were submitted — not a smoking gun that something was amiss, but perhaps enough to trigger a second look by an editor. An editor might have also noticed that one of the reviews was supposedly from Vida Demarin. She is a researcher in the field, but it appears that everything she has ever written is in Croatian; the only English language writing by her that turns up in a web search is in substantially flawed English. (This is no criticism of her, of course — we all know what languages we know.) Yet “her” review was written in nearly flawless English and even recommended that the authors have their paper edited by a native English speaker to get rid of the many language errors in it.

The other review might be considered an even better clue that something was amiss. It was written by “sd T d”. Yes, that really is what was entered as the name of the reviewer (the hackers’ bot seems to have had a few problems). He(?) also wrote a few seemingly useful minor suggestions and recommended a native English review. I am not sure if he actually speaks English because a web search for that name only turned up a bunch of algebra. Ironically, the two reviews agreed (correctly) that the paper was in need of serious language editing, but the version that BMC published was still in need of serious language editing.

So basically the journal and its editors got duped into accepting reviews whose status as fakes was rather apparent upon viewing them, and this could have been confirmed beyond a doubt with a few minutes’ work (by emailing Demarin at her real address, which shows up in any web search, to ask if she had really written the review). But, hey, who can be bothered to actually read the reviews or write emails? Worse, the journal then just let the authors ignore the useful instructions that appeared in those reviews. It is almost as if they have given up even trying to pretend the journal peer-review process means anything.