Fifteen Eighty Four

In June on this blog, I wrapped up my post by asking which of the five CERIC elements we use to analyze scientific articles feels the toughest. The questions we use focus on the Claim (as in, the new knowledge being presented), the Evidence that supports it, the Reasoning that ties them together, the Implications for future research, and the Context that started it all. MY working guess, after a decade of teaching, was Reasoning, and the checking has gone on since—around the world.

Since the book came out in July, we have run workshops on the CERIC method in fifteen countries with undergrads, grads, postdocs, faculty, and librarians. We taught the method in seminar rooms, big lecture halls, and even once on a cruise ship lounge. The answer stayed pretty consistent with what I expected, but with a twist.

Here’s the reveal: the hardest CERIC element to find is Reasoning, by a wide margin. Reasoning keeps coming up as, by far, the hardest of an argument. The other four elements tend show up somewhere in the text. You can spot the Claim in the title or abstract, Evidence in the figures and results, Implications near the end of the discussion, and Context in the first lines of the introduction. Once students know where to look for the CERIC elements, they mow through a paper’s argument in a fraction of the time.

Part of the difficulty with Reasoning is that it can be everywhere and nowhere. It is the logic that says why this Evidence actually backs this specific Claim instead of something else. It shows up scattered through the methods, in the figures, in the discussion, or sometimes just hinted at with a statistic or a single sentence. Readers have to reconstruct it themselves, and if nobody ever points that out, the job feels finished once the easier, more visible pieces of Claim and Evidence are found.

A similar moment happened in almost every workshop when we asked people to explain the logic out loud. The room goes quiet. It was not that so many bright people couldn’t think, it was more like they suddenly noticed a step they had been skipping for years. Even a basic question about Reasoning was a stumper—for instance, What is the main kind of Reasoning used in this paper, deductive, inductive, or adductive?

My informal poll showed nearly everyone had picked up paper reading on their own through trial and error. I did the same, and so did my co-author. This skill of critical reading, which matters most dearly for a career in research, is the one least likely to get taught directly. When trial and error leaves Reasoning overlooked, the mental muscle never develops because it’s much easier to grab the Claim and the Evidence table and stop there.

The twist comes in more complex articles with multiple lines of sophisticated Evidence and Reasoning; then, students really struggle to tell them apart. My favorite teaching example (even though I am not an astronomer) is the seminal gravitational wave detection paper by Abbott et al. (2016), the LIGO and Virgo collaborations’ report in Physical Review Letters of the first observed merger of two black holes. Its figures are beautiful precisely because they interleave Evidence and Reasoning in the same frame. No, it’s very a difficult paper. But for a novice critical reader in astronomy, these elements are often extremely difficult to detangle. 

This gap between Evidence and Reasoning stands out to me more now than ever with AI summaries around. They handle the obvious parts that jump on the page well enough, giving a clean Context and some tidy Implications. They generally do not rebuild the Reasoning well because that part is often never written out in full by the article’s authors. An AI summary can sound confident and fluent, while the main argument is missing, distorted, or just hallucinated. Nothing in the output flags those issues. In fact, the book tour workshops ended up spending most time just on noticing when the Reasoning was missing or weak.

The exercise that helped most on tour came from my days as a high school chemistry teacher: predict-observe-explain or POE from Richard White and Richard Gunstone of Monash University, whose 1992 book Probing Understanding, guided a generation of science teachers, me included. Before looking at the methods, tables, or figures in an article, I ask students to predict what the authors would need to show for the Claim to be true and why. Then, they hunt for the elements and notice what is there and what is not (observe). Finally, they compare what they predicted with what the authors actually wrote. The difference between what they expected and what actually happened makes the Reasoning—or lack thereof—visible in a way that feels concrete and validating. Students who used to struggle saying why an article worked or didn’t could suddenly describe to the logic.

The difficulty with Reasoning and separating from Evidence persisted across languages and fields, which surprised me a little. I had wondered whether it was something about American schooling, and I expected it to shift by discipline; neither turned out to be so, and the silence sounded the same in Greek or Vietnamese. But I have yet to put the question to anyone with a proper survey instrument. 

So, if you teach students, you might try a two-minute POE exercise before diving into the next scientific article and see if the same challenge with Reasoning (or Reasoning and Evidence) shows up.

Title: How to Critically Read the Scientific Research Literature

Author: Genevive Bjorn and Adam J. Burgasser

ISBN: 9781009631273