Making sense of science in the news

By David Steele, PhD. (Originally published in the January/February 2008 issue of Canada EarthSaver)

The pros and cons of what we eat have been in the news a lot in the past few months. But, when we put them all together, they don’t make sense. One study says that overweight people suffer from more disease. Another tells us that, at any given time, overweight people are less likely to die. Several tell us that vegan diets are best for the planet, that they can feed many more people than meat-based diets can. Another reportedly tells us that including a little meat in our diets will allow more to be fed. Contradictory information is everywhere. Just what are we to believe?

Well, my plan is to give you a few tools you can use to make your own judgments. As a scientist myself, I know how hard it can be to sort out claims made in the media. So, I’ll give a few simple rules of thumb you can use to get a pretty good idea of what to trust and what to doubt. You won’t necessarily get it right, but you’ll do pretty well. Just remember that your judgments are provisional. That’s not so bad. Science is always provisional.

The biggest problem is that the mass media often get it wrong. They get basic news wrong often enough. Just think how likely they are to get complicated science reporting wrong. Sometimes they’ll even intentionally mislead you. Some commentators are particularly adept at it. They’ll push a story line that favors their own ideological or financial bent, regardless of what the data actually say.

Take the study out of Cornell that was all over the news this fall. It compared the number of people who could be fed locally from New York State lands if they ate diets ranging from low-fat vegetarian fare to very high fat, largely meat and dairy diets. The news would have us believe that diets containing a little meat can actually feed more people than vegetarian diets can. Problem is, it’s not true. The media latched on to a special case: one that, one might cynically point out, supports the predilections of many an advertiser.

Tip number one. If you can, get your hands on the original study. You can sometimes get it just by Googling. Enter the name of one of the people reported to have done the work alongside a key word or two about the results. While the original papers are sometimes written in scientific gobbledygook, other times they’re pretty easy to read. The Cornell report, for example, is particularly easy to figure out. The whole thing’s summarized in a one simple graph.

If you had that graph in front of you, you’d see that to a certain extent, the news reports are right. New York State land could indeed feed slightly more people eating low fat diets and a tiny bit of meat (2 ounces per day) than it could feed people eating very high fat vegetarian diets. Yes, eating a tiny bit of local beef alongside local fruits and vegetables may be better, food security-wise, than eating a diet of omelets and pizzas. But that’s really just an aside. It ignores the most important information in the article.

Up at the very top of the graph is the low fat vegetarian diet. Ignored by all the news reports, this data at the top of the graph shows unequivocally that the low fat vegetarian diet is the best by far. It can feed hundreds of thousands more New Yorkers than any of the other diets the researchers investigated. If look at the original paper, it jumps right out at you.

Tip number two. Sometimes scientific papers are hard to get. Journals want a lot of money for access. So, if you can’t get a hold of it, or if it’s just plain too hard to read, consider instead, the quality of the source. Who wrote the paper? Where was it published? And, most importantly, who paid for the work? You can make some pretty good judgments even if armed with only this information.

Google the journal’s name alongside “impact factor” and find out if the work was published in a good journal. An impact factor of 10 or more means that you’re looking at a journal that maintains very high standards. Work published there is carefully reviewed by independent scientists. It has been found to be thorough, carefully done and scientifically interesting. Work published in a journal with an impact factor higher than 4 or 5 likely isn’t bad either. It too has been carefully reviewed. The work is probably just a little less ‘exciting.’ An impact factor of much less than 4, though, means you may be looking at less than stellar work. Very low impact journals often have low standards of review as well.

Go on to find the summary of the paper at www.pubmed.org. Find out where the authors work and who paid for their research. Are they in academia or do they work for a pharmaceutical company or an agricultural giant? Industry scientists are generally constrained in what they can publish. You won’t see much negative data from them. Ditto for studies carried out by some industry-funded academic scientists. A landmark report in the New England Journal of Medicine a few years back showed that studies paid for by pharmaceutical money were much more likely to find a drug to be effective than studies carried out by academic scientists independently funded by government agencies or charitable foundations.

Unlike their industry-funded cohorts, scientists funded by charitable foundations, by the Canadian Institutes of Health Research or, in the United States, by the National Institutes of Health, have remarkable freedom in carrying out their research. While overall spending levels are determined by the agencies dispensing the money, the actual funding decisions are made, through a rigorous panel review process, by other scientists.

Once funded, the scientist is free to follow the data wherever it takes him or her. Often, she’ll find herself on a tack she could never have predicted when she began her work. No one is looking over her shoulder telling her what she can and cannot do. If her work is good, it’ll appear in good journals. No one is standing in her way.

Tip number three. Finally, consider the kind of study that’s being reported. Was it a small controlled study or a large statistical study? In controlled studies of diet, a relatively small number of people are divided into two or more groups. One group is fed, say, Diet A and another is fed Diet B, which preferably differs in only one component from Diet A. Controlled studies are scientifically most rigorous and they tend to give fairly clearcut results but frequently it is not practical to study human diet this way.

Most research into human diets are statistical in nature. In the best studies of this type, large numbers of people are recruited to report back over a number of years what they’ve eaten and what health problems they encounter. Vegetarians are compared to omnivores; the effects of high fat diets are compared to low fat ones, etc. Look for big studies here. The bigger it is, the more sensitive and accurate it is likely to be. Make sure the results are ‘statistically significant.’ If they’re not, there’s little reason to believe them. There’s always the chance that an unseen factor is behind even highly significant results but, if the study is large and covers a broad swath of people, that chance is greatly diminished.

Look for large ‘meta studies.’ These are analyses of large numbers of previously published studies. In the best of these works, the data from high quality previous studies are compiled and sophisticated statistical tests are applied to sort out what holds true in all of them. This greatly reduces the chance of an unconsidered factor confounding the results and increases the chance of finding important effects that were missed in the uncompiled, original reports.

The two big studies out last fall – the one that found that overweight people have more health problems and the one showing that moderately overweight people have lower death rates – were both huge meta studies. Since the data in both seems very good indeed, it seems likely that overweight people get sick more often but they can weather illness better. Conflict resolved. The first study, by the way, also found that that routinely eating even a little bit of processed meat –bacon, ham, sausage – substantially increases the risk of cancer, especially bowel cancer. The data on that looks very good, too. Don’t trust me, though. Check it out for yourself. That study is available free online. It’s at www.dietandcancerreport.org