07 January 2011

Fraud and science

Everyone else is writing about it, so I may as well too. Everyone is reporting that not only does the MMR vaccine not cause autism, but the data purporting to show that it did was fraudulent.

While everyone seems terribly outraged by this, they also seem to be treating the whole thing as some sort of crazy anomaly -- the scientific equivalent of that high school reunion your wife couldn't go to and where you had a bit too much to drink and the (still) hot girl you pined for long ago is now impressed at what you've become and somehow you end up with her panties in your luggage. I mean, why tell your wife? It'll turn both your lives upside down, hurt everyone involved, and it was just a once in a lifetime confluence of alcohol, chance, and reminiscence...

Except that isn't the truth and everyone knows it. While it's not tenable to deploy the mutaween to the thousands of high school reunions in this country, it does seem feasible that we come up with a better accounting system in the world of science (and retractions in particular). At present the editors of surgical journals seem to have no problem telling Retraction Watch that the reason a paper was retracted is, "None of your damn business."

This is why some context is important. The media treating this Wakefield debacle as though it's of singular import casts a shadow that obfuscates the huge number of articles that ought to be retracted. For those too lazy to click that link the authors speculated that between 10,000 and 100,000 unretracted ought to be. (Ah, but maybe their own is among them? Shut up.)

Oh, but this Wakefield thing is just a fluke. Most of those crappy journal articles don't actually hurt people. Right? Right???

Wrong. Let me tell you the story of Werner Bezwoda (as if to emphasize my point, he doesn't even have a Wikipedia page). Back in the 1980s doctors had come up with a new theory for cancer treatment. Well, it wasn't really a new theory, it was just the extension of an old one to an unfathomable degree. The limiting factor on chemotherapy effectiveness at the time was the patient's bone marrow. If you destroyed all of the bone marrow cells (along with the cancer cells) your patient invariably died (although not from cancer). Medicine being what it is, doctors decided, well we need to destroy the bone marrow to kill the cancer, so what we'll do is extract bone marrow from these folks, give them the super megadose (it was actually called that) chemo, and then implant the bone marrow back.

The guys at the NIH who first came up with this idea tried to set up some clinical trials to rigorously test the idea before putting it into practice. Unfortunately they were undermined by a media blitzkrieg that cast them as villains for keeping a potentially lifesaving therapy from dying cancer patients. The public outcry was sufficient that the FDA gave the whole procedure a compassionate use exemption. Public demand for the procedure was fed by our man Bezwoda, a physician claiming extraordinary results with the autologous transplantation after the super megadose chemo protocol he had developed. Clients from around the world were regularly flying down to his clinic at Witwatersrand in Johannesburg, South Africa.

As a metric for how quickly this protocol swept the scientific community, in 1993 alone there were 1,177 papers published on it. In any case, those NIH guys were stubbornly continuing with their trials (try recruiting for a randomized trial when your subjects can jump ship if they don't like their assigned lot).

Then in 1999 Bezwoda opened the annual cancer meeting in Atlanta with a presentation on his results: He found that 8.5 years after megadose chemo and transplantation 60% of his patients were alive, whereas only 20% survived in the control arm. During the afternoon session those guys from the NIH presented their results. Except that their results were, let's say, not good. In one of their studies the researchers found "not even a modest improvement," and complication rates considerably higher than the control arm.

Later in the year a team of researchers pulled together by the president of the American Society of Clinical Oncology flew off to South Africa to take a look at Bezwoda's data. Upon arrival they requested the log books for the 158 patients Bezwoda reported treating. He gave them log books for 58, and said the rest had been lost (oh to live in a world without paperwork retainment requirements). The data he did give them was horrible. One of Bezwoda's purported breast cancer patients was actually a man. The entire thing had been a sham. In essence, Bezwoda's protocol was completely fabricated and his fraud was the sole thing holding up a $4 billion industry that performed the protocol on approximately 40,000 women.

To recap: There was a new experimental procedure for treating breast cancer. The media created such a furor at its restricted use that the government was browbeaten into allowing anyone who could pay (or litigate their insurers into paying, a whole different tangent) have a completely unproven procedure. The procedure turned out to be an epic failure and the exposition thereof went almost completely unnoticed by the mainstream media. (Incidentally, in this case scientists were the ones who exposed the fraud, in the Wakefield case it took a journalist to do so).

I do not have a solution to this madness, but until we, collectively, realize that the status quo is madness we're never going to get any closer to coming up with one.

As a parting note I'm going to quote an email that was sent to Jonah Lehrer by a former academic scientist who now works for a large biotech company:
When I worked in a university lab, we’d find all sorts of ways to get a significant result. We’d adjust the sample size after the fact, perhaps because some of the mice were outliers or maybe they were handled incorrectly, etc. This wasn’t considered misconduct. It was just the way things were done. Of course, once these animals were thrown out [of the data] the effect of the intervention was publishable. 
Here we have to be explicit, in advance, of how many mice we are going to use, and what effect we expect to find. We can’t fudge the numbers after the experiment has been done… That’s because companies don’t want to begin an expensive clinical trial based on basic research that is fundamentally flawed or just a product of randomness.
Reagan famously said, "Trust but verify." That's not sufficient any more. The new maxim is closer to, "Doubt until you've figured out their incentives to lie."

No comments:

Post a Comment