Is the FDA FMI CDx Approval "Really" A Very Elaborate Analytical Validity Assessment?

In mid-December, the FDA released its main safety & effectiveness review document on the Foundation Medicine CDX 384-gene test (here).   As commentators have noted, CMS gives very high priority to FMI's PMA approval - "This tells us that it really has clinical utility."   Far lower status is given to tests with published data, NYSDOH approval, or the new 510(k) somatic gene panel approval.   As proposed, those tests get coverage only if $10,000 or more, such as RECIST studies, are deployed for each test paid by CMS. 

The 58-page FDA review is worth reading in detail, which probably few people have invested several hours in doing.   The review is mostly an exhaustive analytical validity assessment.   FMI CDx is verified to have 700-800X coverage for targeted regions; which was information already used in publications, its website, I assume its CAP validations, NYS approval, and MolDX approval.   FMI CDx has a net sequencing error rate of .003 percent (below the FDA's stated threshold of .01 percent or 1 base in 10,000).   But again, this is sequencing accuracy level that is known to be met for clinical genetic tests.    And it's analytical data.

The heart of the review assesses the performance of six very specific outputs of the FMI CDx test against six prior art PMA comparators, as diverse as the Ventana ERBB2/Her2 FISH test and the Roche EGFR T790M point mutation test. 

Two of the comparisons involved clinical blocks, but these were much noisier (with missing blocks, differing results, etc) than the four comparisons that involved running the predicate test on merely available tumor blocks. 

The two comparisons that are cross-modality (between ALK FISH and ERBB2 FISH and sequencing)  are much noisier than accuracy between an FMI point mutation and a PMA kit point mutation.  I summarize the results in the first figure below by gene reviewed, and give representative examples of "very tight" and "noisier" concordance in the second figure below.

My Brief Informal Summary of Results from FDA S&E Document

Examples of Tight (Left) and Noisier (Right) Concordance

I provide a more detailed, but still informal, summary of the S&E key findings in a sidebar blog, here.  In general, whether clinical trial or "available" blocks were used, about 100-150 positive blocks and a similar number of negative blocks were required.    However, even when clinical trial archives were used, the assessing for a passing grade is based on the analytical validity concordance of FMI and prior art results.

The FDA approval is, no question, an exhaustive view that gives a great deal of confidence in the test.  But few commenters actually read it.   The key pivotal tables are mostly based on "available" blocks.  All the data on concordance to prior art is "analytical" and "statistical."   The bar is within 10% noninferior to a comparator. 

It's a big, big achievement.   But I'm unconvinced it's light-years and night and day different than an academic NGS test which has, say, 99%  concordance to Sanger in hundreds of specimens and 95-99% concordance to an IVD in dozens of specimens.   Or for that matter, the 99.9% accurate MSK IMPACT test with its new 510(k) approval.

As I noted earlier (here), the FDA review defines away any issues about accuracy outside the claims.   The FDA tested accuracy in some cases to match a single point mutation; in writing up its pivotal risk-benefit decision, FDA states that: the test matches those several cases of prior PMA art and therefore the whole test (which also reports sequencing on some 377 other genes and things like Total Mutational Burden) introduces no new medical issues or risks beyond the prior art such as a point mutation T790M test or a point mutation L858R test.