Theranos: A Library of Articles and Links 2006-2016

This blog began in December 2014, when a no-longer-existing blog ran a critical review of the favorable New Yorker article on Elizabeth Holmes and Theranos.

From December 2014-December 2016, this blog entry records an informal log of online news articles about Theranos.   I do not review it for the dead links that likely accumulate over time.

The original lead paragraphs of this blog, written back in December 2014, were archived here.
For an update on the March 2018 SEC action against Theranos, here.

I'll add one note from June 2018 - an article that, "Pathologists predicted the Theranos debacle, but their voices were missing from most news coverage"!!  Here.  (Archived here).

FDA to hold workshop on NGS regulation: February 20, 2015; Publishes 9-p white paper

The FDA has announced a workshop on regulation of genetic tests and next-generation sequencing, to be held in Bethesda on February 20, 2015.  The FDA announcement is here.  The FDA conference page is here.

The agenda for the day has not yet been announced.  However, FDA has published a nine page white paper outlining the agency's view of key topics and issues in NGS regulation, here.

• Update 2/23/2015: The meeting was held on February 20; for the Genomeweb meeting review by Turna Ray, see here (subscription).

Key points are:

SUMMARY: 
The Food and Drug Administration (FDA) is announcing the following public workshop entitled “Optimizing FDA’s Regulatory Oversight of Next Generation Sequencing Diagnostic Tests.” (here) The purpose of this workshop is to discuss and receive feedback from the community on the questions in the discussion paper on diagnostic tests for human genetics or genomics using next generation sequencing (NGS) technology.

WHEN:  February 20, 2015, from 8:30 a.m. to 5 p.m.
ADDRESS:  Natcher Center at the National Institutes of Health Campus, 9000 Rockville Pike, Bldg. 45 Auditorium, Bethesda, MD 20814.

WHITE PAPER:  Section 1 is a brief introduction, Section 2 discusses the FDA's regulation of laboratory tests.  Section 3 is a two-page overview of "challenge and opportunities" raised by NGS.  It also notes FDA's several workshops and prior efforts in this area, and the clearance of Illumnia cystic fibrosis NGS tests.  The longest section, Section IV, discusses NGS analytical validity topics (IV-A) and clinical validity topics (IV-B).  There is a discussion of CLINVAR and other public data repositories, and a discussion of how the FDA, clinicians, and public should handle variants of uncertain significance.  There is a short closing section V.

I've also provided online the full-text of the FDA discussion paper on NGS, after the break.

For an NIH workshop on Precision Medicine and whole genome sequencing, Feb 11-12, 2015, see here.

On February 18, 2015, Eric Lander of the President's Council of Advisors on Science and Technology (PCAST) published an Op Ed in the NEJM on the need for special FDA treatment of NGS (here.)  Trade press at Genomeweb (here, subscription).

Can “Clinical Utility" Vary For Two Tests, if “Clinical Validity" is the Same?

Last summer, I was discussing molecular tests with a commercial plan medical director, and he referred to two gene panel breast cancer tests. In his opinion, he felt that the two tests had equal clinical validity, or if anything, the second test had higher clinical validity. However, he felt the first test had “greater clinical utility."

This puzzled me, because I had just published a paper with coauthor Felix Frueh on bringing some structure and order to communications about clinical utility. It had never occurred to me that a test could have greater clinical validity than another, similar test, yet lower clinical utility. In order to draw this conclusion, I believe there is too much “slippage" in the way clinical validity and clinical utility are defined and used. It is more useful to define them in a way such that clinical utility depends on clinical validity plus a use context. If the clinical validity for one of the tests is higher, and the use context is the same, the clinical utility for the other test cannot be better, although it might be the same.

Returning to the Definitions

In a 2014 comprehensive review, Parkinson and colleagues defined clinical validity as “the association between the biomarker and the pathophysiological state or clinical presentation of illness."1 This is essentially the same as that used by Hayes and colleagues in 2013: clinical validity is “how well the test relates to the clinical outcome of interest, such as survival or response to therapy."2

Following these definitions, the test relates what happens in a test tube (a chemical or molecular analysis) to a clinically relevant phenomenon. To be comprehensive, sometimes the test report is analytical (glucose = 125), sometimes it is genomic (we find mutation BRAF V600E is present), sometimes it is an abstract score (“recurrence score = 35"), and sometimes it is binary (a strep lateral flow immunotest is “positive").

Purely analytical tests are related to clinical states by common knowledge or definitions. For combination diagnostics, a clinical correlation is statistically (BRAF V600E is strongly associated with a high chance of response to sorafenib.) For multiple analyte tests with algorithms (MAAA tests), there is usually a double report, one being algorithmic and one being a clinical variable (recurrence score of 35 correlates with a 23% chance of 10 year recurrence).

Test	Analytical Report	Clinical Validity
Glucose	Glucose = 125	Common knowledge or definitions or protocols (not on report)
BRAF	BRAF mutation = V600E	High correlation with sorafenib response in malignant melanoma (usually on report, “interpretation")
MAAA	Recurrence score = 35	Correlated with 23% risk of 10 year recurrence in ER+, Node- breast cancer if tamoxifen treated (on report)
Strep Test	Binary (immunoreaction positive)	Strep present (on report)

We do something with tests, which brings about their clinical utility. For Parkinson et al. (2014), “the results of the assay lead to a clinical decision that has been shown with a high level of evidence to improve outcomes."3 For Hayes et al. (2013), “whether the results of the test provide information that contributes to and improves current optimal management of the patient’s disease."4

The commercial payer’s question, with which I opened this essay, led me to think that these definitions of clinical validity and clinical utility were not designed to help people agree where the two concepts start and stop, or how they meet at a border zone. However, it is possible to think about these concepts in such a way that there is a bright-line border between them.

The Three Buckets for “AV," “CV," and “CU"

A metaphor that makes the difference clear is this:

• Analytical validity lives in a test tube.
• Clinical validity lives on a report and in a data file.
• Clinical utility lives in a patient.

The simplest concept is probably analytical validity: it relates to expressions in test measurements, repeatability, reproducibility, interfering substances, or analyte sensitivity such as ng/ml. Thus, “analytical validity lives in a test tube," or at least inside the equipment that is doing the measuring.

Clinical validity lives in a report, in a data file, on a chalk board. We know that BRAF V600E is associated with sorafenib response in malignant melanoma because clinical trials showed us this was the case. We know that a range of breast cancer genomic tests – including Mammaprint, Oncotype DX, the BCI, and Prosigna – correlate with breast cancer recurrence rates because well-designed large databases tell us so.

It’s best to view clinical validity and clinical utility as separate categories even when the contents, for a test in question and its use case, are similar. For example, we might say offhand that the clinical validity and the clinical utility of a BRAF test are the same – a V600E mutation is associated with clinical response, and the lack of this mutation is associated with non response. In health technology assessments, tests like BRAF get a fast pass, because the clinical validity and clinical utility are so similar.5 However, we avoid a host of later problems if we take the position that even for these tests, the clinical validity and clinical utility are not “the same."

Test	Analytical Report	Clinical Validity	Clinical Utility
Combination Diagnostics
BRAF	BRAF mutation = V600E	V600E is correlated with increased survival when treated with sorafenib	When V600E patient is treated with sorafenib, he/she lives longer.
Her2Neu	Her2neu Positive	Positive Her2Neu is correlatedwith increased survival if treated with Herceptin	When Her2neu positive patient is treated with Herceptin, she lives longer.

Here, while “clinical validity" and “clinical utility" sound the same, they are not the same thing. Clinical validity is a correlation with an analytical test report and the outside clinical world. Clinical utility is something that really happens as a result of what we “do" for a patient. This carries out the idea that clinical validity “lives" on a test report, on a chalk board, or in a paper, whereas clinical utility “lives" in the patient herself or himself. Another way of using our metaphor is to say that analytical validity happens in the real world – although in a very tiny real world inside a test tube – and clinical validity “lives" on a report that we are confident is true because of various prior data. Clinical utility again lives in the real three dimensional living world of drugs, therapies, and patients. Yet another way of saying this – if only one factory could make Herceptin, and it blew up, and there were no more Herceptin drug supplies, the clinical utility of a Her2neu report for a doctor and his patient would be gone (at least for now, and in regards to prescribing Herceptin). However, the clinical validity would be just as true: her test report is in our hand, and cohorts of test-positive patients were reported to live “N" months longer, and it had varied greatly based on Her2neu status.

A Graphic Model of Clinical Utility

In 2014, Frueh and Quinn published a six-question approach to communications about clinical utility, including a figure that shows a one-way relationship between clinical validity and clinical utility. The model focuses on decision making for new tests, where the clinical utility is comparative to the status quo:

However, there is only an increase in clinical utility for that patient because we did something different than the status quo, a “change in management:"

And there could only be a change in management because there was something different about the information we had with the new test, relative to the status quo with the old test (or no test):

This last graphic, shown above, only makes sense if two tests that had “the same clinical validity" would have the same impact on clinical utility in the same use case. This is easy to see if we have two thermometers, or two glucose meters, that have exactly the same reports. It’s also easy to see if we have two different BRAF tests that have the same (or very nearly the same) V600E mutation or wild type reports. The model shown above is generalized for diagnostic tests. For example, if an old MRI scanner has 0.5 cm resolution, and a new MRI scanner has 0.2 cm resolution, the new scanner may have more accurate radiology reports (clinical validity) which lead to better outcomes (correct surgical decisions and other clinical decisions not misled by false positives and false negatives). On the other hand, if two MRI scanners have exact the same image field size, slice thickness, contrast ratio, and resolution, it would be difficult to imagine that two indistinguishable imaging results from, say, a Siemens and a Philips MRI scanner, would have different diagnoses in the reading room or different clinical actions.

-------------------------

1 Parkinson DR et al. (2014) Evidence of clinical utility: an unmet need in molecular diagnostics for patients with cancer. Clin Cancer Res 20:1428-44.2 Hayes DF et al. (2013) Tumor biomarker diagnostics: Breaking a vicious cycle. Science Translat Med 5:196cm63 Parkinson et al. (ref. 1); citing Olson S, Berger AC (2012) Genome based diagnostics: clarifying pathways to clinical use [Workshop]. Institute of Medicine.4 Hayes et al. (ref. 2).5 The Palmetto MolDX evaluation process provides a “fast track" for combination diagnostics approved with drugs in FDA pivotal drug trials. The BCBS Tech Evaluation Center provided a rapid satisfactory report on BRAF kinase testing in 2011.http://www.bcbs.com/blueresources/tec/vols/26/26_07.pdf

House Energy & Commerce: Committee Wants Feedback on Regulation of Lab Tests

On December 9, 2014, the "21st Century Cures" subcommittee of the House Energy & Commerce Committee released ten questions on which it seeks public feedback by Monday, January 5, 2015.   The 21st Century Cures committee website is here, the feedback request is here.  The request for comments is online as a PDF, here.

The request for comments notes that the committee held a roundtable on personalized medicine on July 23, 2014, and a meeting specific to the FDA's proposal to regulate LDTs on September 9, 2014.  (For my coverage of the latter, see here, including a very detailed discussion report, here.)

I've also cut-pasted the eleven questions in full, below the break.

My talk at California Society of Pathologists

On December 5, I had the chance to give a short talk as part of a panel on changes in payer policy and how they are impacting practicing pathologists.  The location was the nationally popular winter California Society of Pathologists meeting, which gives both Californians and pathologists from cold wintry states a December trip to San Francisco for CME credits.   The final program is available here.

I was pinch hitting for a senior Medicare speaker, but did my best both to talk about what is going on at Medicare vis-a-vis pathology and to convey how the field, profession, or industry (take your pick) looks from the viewpoint of a Medicare policymaker.  My deck for the presentation is available in the cloud, here.

My key points were...

•  Federal policy changes for pathology are legion

•  Local changes to pathology policy

•  MolDX Program

–MolDX LCDs

•  How pathology looks to policymakers

•  Can changes be more selective?

FDA announces two-day public LDT Regulation meeting, January 8-9, 2015.

On November 24, 2014, the Federal Register published the FDA's announcement of a two-day public workshop on January 8-9, 2015, on its proposal to regulate laboratory developed tests.   The meeting announcement is available here.  The workshop will be held at the Natcher Center at the NIH.

UPDATE:

The FDA posted transcripts webcasts  after the meeting, here.

The rest of the original blog continues...

Registration is first come first serve until December 12, 2014, at 4 pm, but may sell out earlier.  The FDA may "limit the participants from each organization."   A streaming webcast is expected to be available.

Oral presentations must be requested by December 12, and winners will be selected by December 17.  Presentation materials would be due by January 6, 2015.

Comments may be submitted before or after the meeting, up until February 2, 2015.

The meeting announcement is four pages long and contains additional commentary on the FDA's plans and goals.

Medicare Finalizes an Updated "Coverage with Evidence Development" Guidance

In 2006, Medicare released its first guidance document for Coverage with Evidence Development.  In 2012, CMS released a draft update of its guidance (for a discussion from 2013, see here).

On November 20, 2014, CMS finalized its updated guidance document.   The full CMS text is online, here.  CMS also published a public "summary of comments received" document, available here.

Analysis to follow.

For those with subscription access, the Gray Sheet trade press coverage is here.

Medicare MolDX Program Issues "Molecular Test Panel Alert"

On Friday, November 14, Medicare's program for molecular test coding, pricing, and coverage - MolDX - issued an "Alert" for molecular test coding when a laboratory performs a "set of medical tests composed of individual laboratory tests" that is completed "on a single sample."  Labs performing such tests should start to register such tests as panels beginning on Monday, November 17, 2014, to obtain a "unique MolDX identifier" for each panel.

Effective in six weeks, January 1, 2015, MolDX will "reject tests" submitted with a single biomarker CPT code for each biomarker.

Full text after the break.

FDA Law Blog: Summary of November 6 CDC-FDA meeting on LDT regulation

At the FDA Law Blog, Jamie K. Wolszon & Jeffrey N. Gibbs of Hyman Phelps provided an extended summary of the November 6 FDA-CDC public workshop on the FDA's plan to regulate laboratory developed tests.  Their report is here.  "OIR Head Alberto Gutierrez Discusses Draft LDT Framework at Federal Laboratory Advisory Committee Meeting; Provides Additional Insights on Agency Plans to Regulate LDTs."

For those with access, Genomeweb provided a detailed report previewing the Association for Molecular Pathology "Visit the Hill" day this week, focusing on the AMP's position on LDT regulation....here.  Turna Ray writes:  "Following Republican Election Gain, Detractors of FDA Lab Test Regulation Seize Chance for Support."

CMS Releases Proposed NCD for Lung Cancer Screening

On November 10, 2014, CMS released a proposed NCD that covers annual low-dose CT lung cancer screening for beneficiaries with a long smoking history.

For detailed coverage of the public advisory panel held at CMS last spring, see here.

The topic was controversial because the screening test received a "B" rating earlier from the USPSTF, and the Affordable Care Act requires coverage for individuals with commercial health insurance.   Medicare law allows coverage under Medicare, but requires a separate review and National Coverage Determination, the process that is underway.

In my reading, the topic was controversial because while the pivotal trial was very, very large (about 50,000 patients), the absolute survival benefit was very small (about 0.4%), suggesting that if implementation was a little bit less effective than the trial, there wouldn' t be a benefit.  This is almost always the case with very large trials; if you study an anti-platelet drug in 30,000 patients, it's because the benefit of the drug is likely in the 1% range.

CMS proposes to require that all beneficiaries covered for low dose CT screening have a standardized, multi-faceted training and awareness counseling session, and that each eligible center must participate in a nationwide Medicare registry.  

The national coverage decision is online here.   Coverage at the radiology/imaging industry website AuntMinnie is here (free registration required).  New England Journal ran an article on cost-effectiveness a few weeks ago, here.  JAMA Internal Medicine ran recent articles on the topic, here ("Overdiagnosis in CT screening") and here ("CT Screening, How strong is the evidence?")  JAMA Internal Medicine is edited by Dr. Rita Redberg (here) who also is chairperson of the federal advisory committee MedCAC (here).

Coverage at NY Times here; at Medscape here.  At "Radiology Business," here.

A blog at Health Affairs by Dora Hughes MD, here.

Some interesting parallels between this NCD and some recent LCDs on molecular diagnostics will be discussed in a future post.

CMS Rulemaking for CY2015 Released: Outpatient and Physician Settings

On Friday evening, October 31, CMS released its final policies and spreadsheets for CY2015 for both the physician fee schedule (here) and the hospital outpatient service (here).  The full set of PFS files is here and the full set of outpatient data files is here.

The final typeset versions of the rules will appear in the Federal Register in about two weeks.

My Talk at Harvard

On Friday, October 17, I had the opportunity to give a talk on genomic medicine policy in the U.S. at Harvard (Brigham & Women's Genomic Medicine seminar series.)   The deck I presented is here.(Click on the small black down arrow to download).

While the slides are meant to be spoken to and not just read, my "Prolog" is that in a traditional business, you design a product and use sales and marketing to transfer it to customers, and if they like it and want it, you get return on investment.   In healthcare, and especially in molecular diagnostics, there are many diverse policy headwinds that make this process more complex and circuitous.

The rest of the deck talks about the transition from molecular stack codes to today's CPT coding and pricing and future PAMA market-based pricing.

I give some examples of potentially shifting business models - while also noting that one man's barriers to innovation are another's protections against excess cost and overutilization.   As an example, should out-of-network payment policies for startup labs become onerous enough, there will be more incentive to create on site hospital-based delivery systems, because the hospital is (almost) always in network for its own patients.  My point from an MBA perspective is that this "policy based" market force (starve out-of-network labs) may intersect and synergize with new technologic capabilities (e.g., FDA's approval of Prosigna.)

Along the way, I cite the excellent Harvard Business School case study from 2013 by Robert Higgins on Claritas Genomics (here).  The last few slides discuss approaches to clinical utility that my coauthor Felix Frueh and I have worked on (here).

CMS Releases Prices for New CPT CLFS Codes - Gene Panels Get Gapfill

On October 9, 2014, CMS released its pricing decisions for all new CPT laboratory codes that will be active in 2015.

It finalized its proposed in on November 25, 2014.

The CY2015 Excel spreadsheet fee scheduled, released about January 1, 2015, is at the CMS website, here.  (See: 15CLAB.zip).

* Drug testing codes - use by Medicare is deferred, as is pricing by Medicare.
* Genomic tests - gapfilled.
* Exact Sciences ColoGuard - $502.

UPDATE:
Codes were finalized by CMS as proposed.  Final CMS positions are posted here.  

Note that for specific drug tests, CMS produced some two dozen new G-codes to use, replicating 2014 CPT codes, instead of using new 2015 CPT codes. 

Details after the break.

FDA and LDT’s: Should it be "Guidance" or "Regulation"?

Since a public meeting in 2010, the FDA has been discussing its plans to bring laboratory tests at local hospitals and laboratories under its scope of regulation.   This activity escalated quickly with the release of a planned regulatory framework to Congress on July 31, 2014, and the official release of the same documents on October 3, 2014, kicking off a 120 day public comment period (here).

On September 9 (here), the House Ways & Means committee held a hearing on the proposed new scope of FDA activities.  In that hearing, some lawmakers and public stakeholders repeatedly suggested that the FDA should undertake these activities only after formal federal notice & comment rulemaking.   But Dr. Shuren of the FDA repeatedly emphasized that the FDA had solid legal grounds allowing it to regulate laboratory developed tests by issuing and finalizing a “guidance document.”  

What would really be gained by the rulemaking process?   

1. For one thing, for regulations proposed that would have a greater than $100M economic impact, a formal budgetary analysis must be made by the agency and reviewed by the Office of Management and Budget.   
2. For another, new regulations (law made by an agency and codified in the Code of Federal Regulations) would be proposed, open to comment, and then finalized verbatim and rarely changed.  
3. Finally, policies that are related to, but not literally laid down in the C.F.R. regulation would also be presented  and generally followed by the agency until changed through another round of agency proposal and public comment.  

We discuss the FDA LDT proposals from this perspective after the break, borrowing some analogies from CMS regulations and policy and from the FDA formal rulemaking for Analyte Specific Reagents (ASRs) in 1996.

A Critical Look at "Choosing Wisely" - But it's not Today's NEJM.

Update 10/2017:
  Health Affairs "festival" on Choosing Wisely held in DC, here.
  Example of later quantitative health services article by Colla (2017), here.
  3 October 2017 Articles in Health Affairs here, here, here.


On October 2, 2014, the New England Journal published an article on reducing low value healthcare, one of the approaches being the Choosing Wisely campaign.

Unfortunately, the author did not have the time and space to cite another New England Journal article on Choosing Wisely - one that I read with enthusiastic agreement, because that earlier article encapsulated what I'd been telling colleagues and friends for a couple years as I've read press about Choosing Wisely.  The earlier article is open access and worth looking up.

FDA Releases Official Draft Guidance for LDTs; 120 Day Comment Period

On September 30, 2014, the FDA officially released its two major guidance documents on LDT regulation: The first guidance covers a framework for FDA review, for risk classification and for proposed LDT safe harbors such as unmet needs.   The second guidance covers "reporting" for LDTs as medical devices, i.e. adverse event reporting.

The guidances were released to Congress on July 30, 2014, under a legislative requirement that Congress be given 60 days notice before the "official" release and comment period.  Which starts now.

The documents are dated October 3, 2014, so the 120 day comment period will extend until about January 1, 2015.

FDA Draft Guidance for the LDT Regulatory Framework, here.
FDA Draft Guidacne for LDT Adverse Event Reporting, here.

Federal Register notice of the release and comment period - for Framework, here; for Reporting, here.

The FDA commented it made no changes to July documents, except in the definition of combination diagnostic and in the exact grammar for "traditional LDTs" composed of components legally marketed for clinical LDT use (e.g. ASRs).  They clarify that they are proposing special allowances when LDTs are made from components marketed legally ^for clinical test use.  RUOs are marketed legally, but not marketed legally ^for clinical test use.

IOM Publishes Ebook: Characterizing the Uncertainty of Risks and Benefits.

A fundamental problem shared by both regulatory agencies like the FDA and payers like CMS is assessing the level of net benefit for a new product or service.  

Much has been written over the last twenty years about quantitative (e.g. turnkey) assessment of risk and benefit, but it is very difficult to do so.  In clinical trials, the benefit is usually a fixed statistical endpoint - survival was increased by 4 months plus or minus one month, or strokes per year were reduced by 20% plus or minus 3 percent.  On the other hand, "risks" or adverse events are ungainly, random, unpredictable events of all kinds differing widely in impact and frequency.   It would be easy if we could say, the benefit is "ten plus minus two" and the risk is "negative five, plus minus one" so that the net benefit is favorable, in this case, about plus five with a modest standard deviation and certainly greater than zero.  Comparing risks and benefits is never easy largely because "benefits" are well trained animals in a tight box, and "risks" are an ungainly wild menagerie.

Although the terminology may vary, the same issues face payers and regulators, with the payers typically adding more concerns about comparative effectiveness, overutilization, and external validity (will it work outside the trial).  In a word, payers see or infer much bigger "error bars" that the original trialist and statistician.

Given the difficulty in adding and subtracting risk and benefits against each other, a core problem is uncertainty.  This year, the FDA and Institute of Medicine held two full day public workshops specifically tied to "uncertainty" in risk-benefit decisions.  (See my April 2014 blog, here.)   IOM has now released a 123 page ebook for free download, here.