When people describe legitimate research, they tend to preface it with the term “peer review.” Because peer review is a critical part of scholarly publishing, it’s worth taking a few hundred words and diving into its meaning.

Peer review is exactly what it sounds like: academic peers review an individual’s work in order to determine if the research is strong enough to publish. All articles published in legitimate research journals are peer-reviewed, which is why scholarly journals are deemed a reliable source of information. This is also why predatory journals are a problem. They don’t follow the peer review protocol, which means there aren’t any gatekeepers to stop unethical or fraudulent research from getting out into the public.

Peer review follows a standard process:

If you find research in a reputable journal, the article has been peer reviewed. However, sometimes researchers bypass the peer review process and instead submit research directly to their university or for use at an industry conference.

Having watched Justin go through multiple rounds of article submission, I feel the need to highlight the difficulty and glacial pace of publication. This shit is hard and slow. Justin has work he finished years ago that has only recently been accepted. It’s not that it takes all that long to read a paper, but because reviewers aren’t paid and they have other things to do, sometimes the work gets lost in the slush.

One survey suggested that 50% percent of articles are ultimately published, but only 9% are accepted without a revise and resubmit. While 50/50 odds aren’t the worst, the competition for publication in top journals is vicious. The journal Science only accepts 8% of submissions, while the New England Journal of Medicine publishes just 6%.

In short, no. Critics of the peer review system say it’s slow and expensive, inconsistent and subjective, and often filled with bias and abuse. However, with no viable alternative, both researchers and the general public must continue to believe in the system. The irony of course, as summed up by peer review critic Richard Smith: “How odd that science should be rooted in belief.”

Click here for Part One of Where to Find Scientific Research Papers (and How to Know if They’re Legit).

---

Yesterday, I wrote about predatory journals. I suppose I shouldn’t have been surprised to learn that there are shitty people in the research world who get off on exploiting academics and undermining science, but I was. Blame my mother. She raised me in a world where all people, on some level, are good. I never quite bought it, but I also didn’t learn to look at everything and everyone with skepticism. I tended to assume that people were just doing the best they can. They may be severely annoying in the process, but ultimately it was all with good intention.

The internet has shattered that illusion. People are fucked.

And so the burden falls on to the individual to see through the bullshit. Historically, we’re not great at that, but when it comes to sussing out whether or not a research paper is legitimate, there are a few quick and easy ways to verify your science.

Google Scholar is one of my favorite ways to source research, but because Google Scholar is a search engine and not a curated database, articles published in known predatory journals may pop up in your search results.

The quickest way to determine if the article is legit is to check the “Cited by” number at the bottom of the search. If an article has multiple citations, it means other researchers are referring to the research in their own articles, which indicates legitimacy. It’s rare that articles are cited hundreds of thousands of times like Eugene Paykel’s excellent study in the photo above. (Paykel’s study is the research equivalent of a New York Times bestselling book.) According to my smarty-pants academic boyfriend Justin, even mid-single digits is enough to assume the research isn’t bunk.

While citations are a great place to start, they benefit from time in the system. Paykel’s article has been around since 1976, which means it has nearly half a century of research built upon it. New research won’t come with shiny citations, so you need to look at the journal it’s published in to see if it’s legitimate.

Academic journals are ranked for impact and quality. Think of it like the college system. Harvard isn’t the same as Iowa State, but that doesn’t mean that Iowa State isn’t capable of producing damn good citizens (and we all know question marks who graduated from top tier universities.) The top journals produce great work, but there is still plenty of meaningful work to be found in smaller journals.

Find journal rankings by googling the name of the journal and the word “ranking.” The Scimago Journal & Country Rank (SJR) should be the first result, and that will take you to a list with the journal in question buried somewhere in there. The rank is determined by the H-Index, the details of which I don’t entirely understand. The H-Index is determined by the number of publications and citations, and higher H-Index indicated a higher ranking. However, the H-Index is not standardized across subject areas, so you can’t cross-compare.

For our purposes, the H-Index doesn’ matter too much. In Justin’s words, “A low ranking isn’t necessarily a problem. No ranking is a problem.”

If the journal article doesn’t appear on the SJR, your predatory journal spidey sense should go off. Cross-reference the journal against Beall’s List, an archive of predatory journals created by librarian Jeffrey Beall. The sheer number of journals listed on Beall’s List is astounding, and it’s easy to see how naive readers could be duped.

I’ve been feeling dejected since yesterday’s post. In fact, the whole research system has got me down. Let me be clear: I am not against research or science concept. What I am frustrated with, like everyone else, is the fucking system.

This all began when I came across something called predatory journals during my research for yesterday’s post. Predatory journals are junk journals at best and scams at worst. They exist because academics are under so much pressure to not only produce research but to publish research. You don’t get tenure without publishing. You don’t get a shot at tenure without publishing either. And given that the odds of landing a tenure-track job at a decent university are on par with making it to the NBA, young academics are under immense get work into the world.

To get a research article published in a competitive journal is an exercise in patience and will. I haven’t done it myself, but as the other half of a research academic trying to land a tenure track job, the process strikes me as the brainy equivalent of peeling your own skin off, letting it scab over, and peeling it off again. Some clever yet undeniably shitty people saw an opportunity to take advantage of struggling academics, so predatory journals were born. Think of them like the academic equivalent of late-night infomercials. For just $1700, you too can have your important research published in our official journal!

Yes, you read that right. Predatory journals often charge academics a fee for their work to appear in a bogus journal. That’s the hallmark of any good scam, right? They make you think you need them and then they take your money. It’s not always so obvious, though. Some journals are more sophisticated than others, leading well-intentioned researchers to unintentionally publish in hack journals. One study found that 5% of Italian researchers were duped into publishing in predatory journals.

In addition to exploiting academics, these journals don’t perform any sort of quality check on the work that gets submitted, leaving ample room for plagiarism, fraud, ethics, conflicts of interest, and general shitty science. But naive readers don’t know the difference, leading to a cancer of misinformation.

If this isn’t bad enough, predatory journals also create a sinkhole for funding and resources. One analysis found that 17% of articles sampled from predatory journals reported that their funding for the study was from the US Nationals Institute of Health (NIH). This means that one of the world’s foremost medical research center is funneling resources towards studies that end up in scientifically questionable journals.

“Little of this work will advance science,” the authors of the analysis say. “It is too dodgily reported (and possibly badly conducted) and too hard to find.”

When I asked my partner, Justin, about predatory journals, he laughed and brushed it off with an, “I get spam emails from them all the time!” Then he giggled and pulled up a journal article entitled, “What’s the Deal with Birds?” published by Daniel T. Baldassarre in the Scientific Journal of Research and Reviews. The author, fed up with predatory journals, submitted a fancy-looking yet totally bogus “study” to a known predatory journal just to prove a point. And they actually published it, proving that just because something looks like research, doesn’t mean it’s legit.