“There are three kinds of lies: lies, damned lies and statistics.”
— Attributed to Benjamin Disraeli
President Donald Trump held a press conference Aug. 23 to announce “a very historic breakthrough” that would “save countless lives” in patients with severe COVID-19 infections. The breakthrough was Food and Drug Administration approval of an emergency-use authorization for convalescent plasma, the antibody-rich plasma harvested from patients infected with COVID-19.
Patients would now have easier access to this therapy. Present were Alex Azar, secretary of Health and Human Services, and Dr. Stephen Hahn, commissioner of the FDA. This announcement of a historic breakthrough turned into a historic breakdown.
The president’s statement that convalescent plasma had been “proven to reduce mortality by 35%” was explained by Dr. Hahn.
“What that means is 100 people who are sick with COVID-19, 35 would have been saved because of the administration of plasma,” Mr. Azar chimed in. “This is a major advance in the treatment of patients. I don’t want you to gloss over this number.”
No one is glossing over this number, but not for the reasons Mr. Azar intended.
There are two problems with the assertions made in the president’s news conference. First, the study was unpublished and was uncontrolled. The results could not be compared to patient outcomes in a group not treated with plasma. Second, Dr. Hahn’s attempt to clarify the significance of the study significantly misses the mark. He confused the difference between relative and absolute risk. These two terms can lead to widely divergent conclusions.
The figure illustrates the difference between relative and absolute risk. For example, if a drug is effective in preventing heart attacks, there are two ways to convey this message — the drug reduces heart attacks by 50% (relative risk reduction), or it reduces heart attacks from 2 per 100 people to 1 per 100 people (absolute risk reduction).
Which sounds more newsworthy?
The 35% reported reduction in mortality is the relative risk reduction, and was a subgroup analysis from 35,000 patients. When the data were analyzed (some say “tortured”) for a small cohort of 1,018 selected patients — those under 80 years of age, not on ventilators and treated within 72 hours — who received low-dose plasma had a 13.7% seven-day mortality vs. an 8.9% mortality in the high-dose group. The absolute risk reduction is 4.8%, so perhaps five persons out of 100 would be helped, but not 35. And this is only true if the study itself was valid. These are not breakthrough conclusions.
Criticism of Dr. Hahn’s comments and the press conference was immediate.
“I can’t remember a mistake by FDA or the commissioner as serious as this one,” said a prominent critic, Dr. Eric Topol, founder and director of the Scripps Research Translational Institute, professor of molecular medicine and editor in chief of Medscape.
Dr. Hahn issued an immediate retraction.
“The criticism is entirely justified,” he said. “What I should have said better is that the data show a relative risk reduction, not an absolute risk reduction.”
On Aug. 31, Dr. Topol wrote an open letter, “Dear Commissioner Hahn: Tell the Truth or Resign,” that outlined concerns of many.
“Not only has your credibility been diminished but so has that of the FDA, its 15,000-plus staff members, and, most importantly, your ability to oversee the health interests of the American people,” the letter read.
On Sept. 1 the COVID-19 Treatment Guidelines Panel of the NIH issued its statement on convalescent plasma. Because of a lack of reliable clinical data they concluded, “Convalescent plasma should not be considered standard of care for the treatment of patients with COVID-19.”
Convalescent plasma has gone from a “historic breakthrough” to “should not be considered” in one week, and dramatically reflects the contamination of good science by bad politics.