Recognizing Rubbish

(Photo from PngImg)

Just in case this didn’t come through strongly enough in my previous post, I get loads of new information by reading studies, but I don’t believe everything I read. One of the most important aspects of learning to read scientific “papers” is figuring out how to distinguish high quality from rubbish.

Rubbish as food is not good for your body. Rubbish as information is not good for your mind. It belongs in the trash can.

Methodology

I’ve mentioned that I set aside papers when their methodology doesn’t look sound. Deciding whether methodology is sound mostly requires some thought and common sense.

A little over 20 years ago, I accompanied a friend to medical appointments at a premier diagnostic facility in the Texas Medical Center in Houston. She had a diagnosis of myalgic encephalomyelitis (M.E.) in the UK, but her symptoms didn’t quite fit with the rest of the patients in that section of the Chronic Illness forum. This was a quest to find out what was really going on. In her age bracket, there was one treatment that worked for some M.E. patients, but only if it began within four years of onset. She was near the four year mark. Should she get that treatment or not?

In the previous couple of years a badly done study had been published on M.E. It promoted a non-pharmaceutical treatment that was actually harmful. Insurers and national health care systems loved it because it gave them a quick way to shove out the door patients who would otherwise need repeated appointments. The methodology was terrible. The forum had been in an uproar about it, so I read it.

By the time my friend and I got to the diagnostic center, it had been adopted into a medical textbook. The doctor leading her investigations tried a few standard tests and then seemed to get frustrated by not having a tidy diagnosis. We went in for an appointment and he used that study, the one that was so badly flawed, to tell her that he thought he should stop running tests. She should just accept that her problem was psychological and she just needed to do graded exercise therapy (GET) and cognitive behavioral therapy (CBT).

GET and CBT can turn M.E. patients who are somewhat limited into people who are housebound or bedbound for the rest of their lives.

My friend was so shocked, she couldn’t think of anything to say. I was furious, but simply started asking questions as dryly as I could. I had read the study recently enough to remember it. After so many years I don’t remember everything I asked any more. I do remember asking how the treatment could be deemed successful when two thirds of the patients in the study had to drop out.

The doctor didn’t have any good answers for my questions. He had accepted the study’s summary without looking at its methodology. It became embarrassing. He suddenly announced that he would run more tests and make more referrals after all.

All it took to see the flaws in the study was a little thought. Seriously, if you offer a cure to someone who has been hopeless, how likely are they to say “No thanks, I’d rather stay miserably sick for the rest of my life” and drop out of your study? Would two thirds of the study participants need to drop out if the treatment worked? The criteria for declaring success obviously hadn’t been determined appropriately.

A paper can flag itself as unreliable in other ways, too.

Are the Researchers Vulnerable to Pressure?

If research relies heavily on funding from a source that has a strong interest in getting a particular outcome, I’m skeptical. It takes nerve and power to publish results the funder doesn’t like, and the funding source undercuts the researcher’s power. Should I trust research funded by the tobacco industry about the health impact of smoking or the oil industry about climate change? I think not.

At a much more mundane level, twice in my career whoever was paying for my work wanted me to falsify technical test results when they didn’t like the reality my tests revealed. The first time it happened, I avoided letting them get in a position to say it to my face. The supervisor they went to first warned me. I was able to protect the data before the head honcho reached me. But I never got to set foot in that lab again.

The second time was a huge row. I told my client to publish the test report as theirs, with their team lead as the lead author and me as a secondary author even though I did the work… and take control of the report away from me. The data was protected. But when that contract term expired, I had to find another client.

Not everyone is in a financial position to take a stand against whoever controls the money. Think about whether the researchers are vulnerable to pressure.

Human Foibles

Researchers are human. Occasionally bias, a personal agenda or raw ego distorts their work. That can make deciding whether to believe them tricky.

Robert Gallo’s page on Wikipedia says, in part, that he is “best known for his role in establishing the human immunodeficiency virus as the infectious agent responsible for acquired immune deficiency syndrome.” Wikipedia doesn’t fully tell what his role was.

In science, to be fully relied upon, a study must be replicated by a different scientist and support the same conclusion. Gallo personally delayed research into HIV by a year with his insistence that he must be the one to replicate the very first isolation of the virus by Luc Montagnier’s team at the Pasteur Institute. When Gallo at last revealed his results, the virus he claimed to have isolated from a different patient was almost identical to the reference sample he had gotten from the Institute for comparison. HIV mutates so rapidly, over the course of that year it wouldn’t have been the same even in the original patient who provided the reference material. Reputedly the room fell silent when he showed that result. It couldn’t be correct. (An excellent article about this is behind a paywall at the journal Nature, but it is also well documented in the book And The Band Played On: Politics, People and the AIDS Epidemic by Randy Shilts. Disclosure: This is an affiliate link.)

But Gallo got the reputation and clout to demand to be the one to replicate the Institute’s work by making a truly astounding advance in medical science. There is a virus called human T-lymphotrophic virus (HTLV). It spreads similarly to the way HIV spreads, so it isn’t easy to catch. It isn’t common. About thirty years after a person is infected with it, HTLV causes a fatal blood cancer. It isn’t just correlated with the cancer. It causes the cancer. Until Gallo proved that, we didn’t know a virus can cause cancer.

As I said, researchers are people. Even the most skilled of scientists can make a mistake and be wrong. Getting back on track can take time, but eventually science makes a course correction.

Research into HIV ramped up again as quickly as possible after Gallo’s error and here we are, knowing much more about that virus, with treatments for the infected.

As for Gallo’s earlier work on HTLV, how much longer would it have taken to realize the link between HPV and cervical cancer without his HTLV discovery? How much longer would it have taken to develop a vaccine against HPV?

So… I believe Gallo’s work about HTLV. I am more skeptical about his work on HIV. In the late 1980s, he and his closest cronies made noise about what was beginning to be called Chronic Fatigue Syndrome (CFS) in the USA and M.E. in the UK. They were saying almost exactly the same things they had said about AIDS, trying to position themselves for any funding that went toward research into the new topic. I thought about what happened with HIV and how they were behaving the same way again. I chose to ignore anything they published about their new area of interest.

Wrap-Up

That gives you a clearer picture of what I mean when I say I don’t believe everything I read. It tells you more about how I decide what to take in and what to set aside. You can make your decisions about this in your own way, but I hope getting more of a view into my approach can give you some ideas.