Much has been written over the last twenty years about quantitative (e.g. turnkey) assessment of risk and benefit, but it is very difficult to do so. In clinical trials, the benefit is usually a fixed statistical endpoint - survival was increased by 4 months plus or minus one month, or strokes per year were reduced by 20% plus or minus 3 percent. On the other hand, "risks" or adverse events are ungainly, random, unpredictable events of all kinds differing widely in impact and frequency. It would be easy if we could say, the benefit is "ten plus minus two" and the risk is "negative five, plus minus one" so that the net benefit is favorable, in this case, about plus five with a modest standard deviation and certainly greater than zero. Comparing risks and benefits is never easy largely because "benefits" are well trained animals in a tight box, and "risks" are an ungainly wild menagerie.
Although the terminology may vary, the same issues face payers and regulators, with the payers typically adding more concerns about comparative effectiveness, overutilization, and external validity (will it work outside the trial). In a word, payers see or infer much bigger "error bars" that the original trialist and statistician.
Given the difficulty in adding and subtracting risk and benefits against each other, a core problem is uncertainty. This year, the FDA and Institute of Medicine held two full day public workshops specifically tied to "uncertainty" in risk-benefit decisions. (See my April 2014 blog, here.) IOM has now released a 123 page ebook for free download, here.
The IOM also has dedicated webpages for the workshop on February 12, 2014, here, and the workshop on May 12, 2014, here. These February and May webpages provide access to speakers' presentations and biographies.
Still available from IOM is also a 2007 workshop report, here.
In the new book, a flavor of the type of discussion comes from an extended case study of problems occuring after the release of Tysabri for treating multiple sclerosis. It was approved with accelerated approval, but a very few patients developed later cases of progressive multifocal leukoencephalopathy. Was this a freak occurrence? Was it a harbinger of a frightening impending volume of such patients any month? Should the drug be pulled off? What should be done next? These were the uncertainties.
Steven Woloshin of Dartmouth suggested four types of uncertainty:
- Standard uncertainty. For example, all new drugs have a limited track record. Nobody knows the effect of ten years' use of the drug, for example.
- Extra uncertainty due to accelerated approval. Early release of a drug that seems beneficial for patients with limited treatment options, and pending further FDA review on later data.
- Extra uncertainty based on surrogate primary outcomes. This affects only the uncertainty of the "benefit" scale.
- Extra uncertainty based on harm signals. These might be dealt with by ensuring further information is collected rigorously, such as in REMS studies.
PS
The contrast between benefits and risks was noted by evidence and policy expert Harold Sox in a September 24, 2014, article about CT lung cancer screening policy in JAMA (here). He writes, " [J]udgments about the balance of harms and benefits of screening [are] necessarily subjective because harms and benefits are typically measured in different units. If the study protocol included an assessment of the patients’ utilities, the benefits and harms can be expressed in the same units (quality-adjusted life-years) using modeling." As Sox well knows, it's somewhere between the pot at the end of the rainbow, the Holy Grail, and a mirage to do so - usually very difficult in the real world to achieve the seemingly simple conversion to same-units, although progress is being made.
At the April 30, 2014 MEDCAC on CT lung cancer screening, Dr Mark Grant registered the same ideal near the end of the day. "I really think a gap is our metric in which we discuss net benefits and harms, and it would be very helpful if something were adopted and used that could be communicated in a transparent way that placed them all in a similar scale, albeit with all the limitations thereof, but I think it would make the conversations a little bit easier. I think it would allow quantifying uncertainty and what the value of future research might be in a particular areas to reduce that uncertainty, yet throwing the balls around, it's always challenging without at least some common scale, at least for me." Again, there is a lot of literature in the field now, and if it were easy, we'd be doing it, but the ongoing dialog is good.
