How to identify reliable scientific sources: The top six questions to ask

As a clinician, you know how important it is to keep up to date on medical literature. Only by reading scientific literature will you be able to practice evidence-based medicine. But we also know that there is a lot of literature out there, and as a busy professional there’s only so much time you can spend sifting through it. That said, what good is your ability to stay current on the latest research findings and literature if you pick the wrong articles?

This Mini Guide summarizes the top six questions to ask yourself when evaluating a scientific article. These six questions will help you determine if a study is credible and beneficial to your patients. By identifying reliable scientific articles, you’ll be equipped to reject faulty articles and provide the latest evidence-based medicine to your patients.

These are the six questions you should ask when reviewing an article:

1. Is the article published in a credible journal?
2. Are the authors of the study qualified and free of conflicts of interest?
3. Are the study participants a good representation of the target population?
4. Are the study designs used appropriately for the article’s research question?
5. Are the results interpreted with the correct statistical methods?
6. Are the right conclusions drawn based on the results presented?

Let’s take a look at each of these questions in a little more detail.

The majority of scientific articles are published in academic journals. The highest quality journals are those that provide peer-reviews before publishing any articles. This gives you, the reader, the assurance that a group of your scientific peers has approved the paper’s content and methodology before publication.

But, there are journal publications that prey on unsuspecting researchers with the promise of publishing the researchers work—regardless of the quality. These are known as predatory journals that may have broad names and often cover multiple disciplines. You can use impact factors to help identify these less-than-stellar journals.

In fact, you can use a journal’s impact factor to get an idea of the quality of articles that a journal prints. An impact factor is the average number of citations that an article in the journal receives in a year. A journal full of highly cited articles will have a high impact factor, and a journal full of low-cited articles will have a low impact factor. Think of it like this: a great product will generate positive reviews, and people will recommend it to others. By choosing to reference an article in their own work, a researcher is giving credibility to an article and the article’s publication.

Since a journal’s impact factor is based on an average, it’s also a good idea to check the article’s citation frequency. You can use a tool like Google Scholar to get an idea about the credibility and quality of a specific article by looking at how many times the article has been cited by other authors.

Here’s the take-home message: if you find an article that interests you, take a minute to look into where the paper is published. Journal publications with high impact factors and a peer review process are less likely to publish shoddy scientific articles.

Are you still unsure if you would be able to tell a good article from a bad one? Check out this article to delve deeper into how to use peer reviews and impact factors to assess an article's credibility with peer reviews and impact factors.

As a clinician, you’ve spent years learning and honing your skills to reach this point in your career. Shouldn’t you expect the authors of the studies you’re relying on to also spend a significant amount of time researching their subject matters?

An expert author is more likely to produce reliable scientific articles. A look into the background of a study’s author can give you an idea of the study’s quality. For example, if you’re interested in a new drug for HIV mentioned in a promising article, it would help to know that the study’s author has spent decades researching infectious diseases.

What if you discovered that the same author holds stocks in the pharmaceutical company that manufactures this new drug? Can you be sure that the author is unbiased in their interpretation for this new treatment, and not simply looking to line their own pockets? This is why it’s important to watch out for any potential conflicts that an author may have.

Conflicts of interest are supposed to be listed at the end of a study article (after the conclusions section). However, a look at the study’s funding is also a good idea. This information is typically found in the acknowledgments section of a paper.

Want to dig deeper into what makes an author credible? We discuss analyzing an author’s background, potential conflicts of interest, and funding in our article on how to assess an author’s credibility.

You’ve probably heard of the common phrase, you can’t compare apples to oranges. Let’s say that you find a case study of a 4-year-old patient exhibiting the same symptoms as a 61-year-old patient. Would you be comfortable relying on the case study findings to determine a treatment protocol for your own patient?

The answer is ultimately up to you. But, taking the time to examine the study’s sample population will help you determine the article’s usefulness in your clinical practice. An article with a study population that closely matches your patient would, of course, be better.

You should also consider whether the article's study population is optimal for the research question being studied. For example, we know that the prevalence of type 2 diabetes is almost two times higher in African Americans than in Caucasian Americans (Marshall Jr 2005). So, a study testing a new diabetes drug that only included Caucasian participants would not be a representative study since a significant portion of the population is missing.

Most of the information on the study’s population demographics (such as age, race, and socioeconomic status) can usually be found in Table 1 of an article.

Do you want to get better at evaluating sample populations? Check out our article on sample populations where we expand on how participant selection, exclusion criteria, inclusion criteria, and how these factors affect an article’s credibility.

Let’s say that you wanted to bake a cake. Would you look up a soup recipe? Probably not! You know that a soup recipe would not give you the information you need. What if the recipe promised a cake but instead gave you the ingredients for soup? The end product would not be the promised (and desired) result. You’d want a cake recipe that included the right ingredients and methods to bake a cake.

Well, in research it’s the same! The correct study design must be used to answer a study question. Otherwise, an article may not properly answer the research question.

For example, if you want to know whether a specific drug works better than a placebo at decreasing blood sugar levels in diabetic patients, a study asking people with diabetes how they feel about taking medicine wouldn’t be helpful. Likewise, if you want to evaluate how people with diabetes feel about increasing the number of medications they take, a study showing blood insulin levels wouldn’t give you an answer.

A good research article will clearly state the research question in the introduction. The methods section should also include steps that detail which research design was chosen, why it was chosen, and the steps that are taken to carry out the design. Reading over these sections will give you an idea of the factors that the authors considered when conducting their study. You can then decide whether they chose the most suitable methods to address the research question. It’s all about making sure that the study can answer the question that you need answered.

So, how do you know if the correct study design was used for a particular study? It helps to know the most common study designs and what types of questions they answer. If you’re feeling a little rusty when it comes to study designs—we’ve got you covered. In our article on study designs, we delve into different research questions and highlight the most commonly used study designs.

When you’re searching through the latest research findings to help your patients, you’re probably looking for an answer to a question. When you find a study with results answering your question, you want to know how confident you can be with those results so you can apply them in the clinic.

Researchers rely on statistical measures like the p value to determine whether the study results are statistically significant. For you to evaluate if the results are clinically significant, you’ll need to wade through the results, interpretations, and basic statistical methods to understand what the findings mean in a clinical setting.

Let’s say that you want to know whether prescribing a novel drug to your patients with the flu would help them recover. You find a study claiming that the novel drug helps patients experiencing flu symptoms. The results show a p value of less than 0.05, which is considered statistically significant when the alpha level is set to 0.05. Sounds great, right?

Well, what if a further look into the results showed that the novel drug only shortened the flu symptom duration by a few hours? How helpful would these results be in the clinical setting?

As clinicians, you’ll need toassess the results and determine whether they are statistically significant and clinically relevant to your patients. In our article on statistical methods, we discuss what a p value really means, and other factors that should be considered when evaluating an article’s results.

Imagine that you come across a scientific study claiming that eating pizza could help you lose weight. The study results showed that participants who ate one slice of pizza lost weight compared to their controls who didn’t eat pizza. Sounds promising, right?

But maybe something isn’t right and there is something else going on in the study! Just because this study found a relationship between eating a slice of pizza and losing weight doesn’t mean that eating the pizza caused the participants to lose weight. Researchers explain this concept as correlation versus causation. Put simply, two variables being correlated does not guarantee that one variable causes the other.

What if there was another reason for the weight loss that had nothing to do with the pizza? Could there be another variable that is affecting the results? Did the authors not account for some of the data, or incorrectly interpret the statistical analyses?

It’s possible for researchers to share their study design, methods, and results in a paper and still draw the wrong conclusions. In fact, it’s also possible that the authors did everything by the book and still drew incorrect conclusions. This is why it’s important to read over the results and determine if the right conclusions are reached.

Want to be more efficient at picking out possible misinterpretations in the data? Our article on data interpretation examines incorrect conclusions that researchers may draw about correlations and causations, missing data, and possible confounders.

Well, there you have it: the top six questions you should ask yourself when reading medical literature.

Carefully assessing the quality of the article and the publication, as well as checking out the author's credibility, will help you select reliable scientific articles. As well, a quick look at the sample populations, study designs, analyses, and conclusions can save you from relying on inaccurate findings for your patients.

With so many articles being published today, it’s become even more important to be critical of the papers you come across. Remember that these questions will help you save time as you wade through the mass of literature. Be sure to check out the other articles in this series for more information on each of these topics.

Are you interested in learning more about epidemiological study designs and statistics, but reading isn’t really your thing? Good news! We’ve got a short video series that answers all the questions and more. Sign up for a free trial and check out our Epidemiology Essentials Course today!