The point of this blog is to give an example where we feed a molecular medicine paper to AI (here, ChatGPT), and compare the results to what we'd get from the abstract alone.
##
The paper is Petak et al 2021, which I ran across last week. The title is: A computational method for prioritizing targeted therapies in precision oncology: performance analysis in the SHIVA01 trial. It appeared in Nature Precision Oncology 5:59, here. Digitally driven drug assignment could double PFS, but on a small scale (4 vs 2 months). I would classify it as a proof of concept paper.
Here's the article's own abstract:
Precision oncology is currently based on pairing molecularly targeted agents (MTA) to predefined single driver genes or biomarkers. Each tumor harbors a combination of a large number of potential genetic alterations of multiple driver genes in a complex system that limits the potential of this approach. We have developed an artificial intelligence (AI)-assisted computational method, the digital drug-assignment (DDA) system, to prioritize potential MTAs for each cancer patient based on the complex individual molecular profile of their tumor.
We analyzed the clinical benefit of the DDA system on the molecular and clinical outcome data of patients treated in the SHIVA01 precision oncology clinical trial with MTAs matched to individual genetic alterations or biomarkers of their tumor. We found that the DDA score assigned to MTAs was significantly higher in patients experiencing disease control than in patients with progressive disease (1523 versus 580, P = 0.037). The median PFS was also significantly longer in patients receiving MTAs with high (1000+ <) than with low (<0) DDA scores (3.95 versus 1.95 months, P = 0.044). Our results indicate that AI-based systems, like DDA, are promising new tools for oncologists to improve the clinical benefit of precision oncology.
###
OK, now let's feed the article to ChatGPT (model 3.5). We use a text splitter to split the text into chunks of about 2000 words (3 chunks). From here on, my words are in red, and the unaltered machine genereated ChatGPT responses are in black.
I am Dr Bruce Quinn, a molecular pathologist and policy expert for precision medicine. I need to summarize this article in detail in a lecture next week. Please provide me a detailed summary of this article. I am an MD PhD.
ChatGPT
Certainly, Dr. Quinn. I can provide you with a detailed summary of the article you shared. Here's a summary of the key points:
The article presents a computational method called Digital Drug Assignment (DDA) that aims to prioritize molecularly targeted agents (MTAs) for precision oncology. DDA is an explainable AI system designed to assist in personalized treatment strategies for cancer patients.
The DDA system works by analyzing the molecular profiles of patients and identifying potential driver genes, associated targets, and MTAs that have positive or negative associations with the molecular alterations present in the patient's tumor.
The system calculates an Aggregated Evidence Level (AEL) score for each MTA based on the strength of associations with driver genes and targets. The higher the AEL score, the more likely the MTA is to be effective in the given patient's tumor.
The study evaluated the clinical performance of the DDA system using data from the SHIVA01 clinical trial, where patients were treated with MTAs based on predefined molecular alterations. The analysis showed that MTAs with higher AEL scores had a significantly higher disease control rate and longer progression-free survival compared to those with lower AEL scores.
The DDA system demonstrated the ability to identify effective MTAs even when the molecular alterations in the patient's tumor did not match the specific targets of the MTAs. This suggests that the network-based approach of DDA, considering multiple associations and functional relevance, can predict treatment outcomes more accurately.
The article emphasizes that DDA should be used as a supportive tool for treatment decision-making, especially when multiple MTAs are available at the same evidence level. It can help prioritize MTAs based on their AEL scores, but further clinical trials are needed to validate the clinical benefits of MTAs selected by DDA.
The DDA system is designed to be transparent, open to human supervision, and continuously updated with new evidence. It can be used by healthcare professionals trained in its utility and limitations to aid molecular tumor boards in planning personalized treatment strategies.
The article highlights the potential of AI-based systems like DDA in precision oncology, but it also emphasizes the need for rigorous evaluation, randomized trials, and regulatory frameworks to ensure their reliable and effective use in clinical practice.
In conclusion, the DDA system presented in the article offers a promising approach to prioritize MTAs for precision oncology based on the molecular profiles of patients. It demonstrates the potential to improve treatment outcomes by considering multiple associations and functional relevance. However, further validation and integration into clinical practice are necessary before widespread adoption.
What are the greatest strengths of this approach?
ChatGPT
The DDA approach outlined in the article has several notable strengths:
- Network-based Analysis: The DDA system utilizes a network-based approach that considers multiple associations between driver genes, targets, and MTAs. This comprehensive analysis allows for a more holistic understanding of the molecular landscape of a patient's tumor and the potential interactions between different molecular alterations and therapeutic targets.
- Aggregated Evidence Level (AEL) Scoring: The system calculates an AEL score for each MTA, which takes into account the strength and relevance of the associations between the MTA, driver genes, and targets. This scoring system helps prioritize MTAs based on their potential efficacy, allowing for more informed treatment decisions.
- Transparent and Explainable AI: DDA is designed to be an "open box" AI system, meaning its rationale and mechanism of action are transparent and explainable. The system generates lists of published evidence used for scoring and provides text descriptions of molecular alterations and associated MTAs. This transparency allows healthcare professionals to review and verify the system's decisions, enhancing trust and facilitating human supervision.
- Integration of Functional Evidence: DDA incorporates functional evidence, including clinical, preclinical, and in silico data, to assess the functional relevance of molecular alterations and their associations with MTAs. By considering the functional significance of specific mutations and their targets, DDA can predict treatment outcomes even when there is no direct match between the molecular alterations and the targeted therapies.
- Clinical Performance Validation: The DDA system was evaluated using data from the SHIVA01 clinical trial, where the clinical benefits of MTAs selected by DDA were assessed. The analysis demonstrated that MTAs with higher AEL scores had better disease control rates and longer progression-free survival. This validation provides evidence for the clinical utility and effectiveness of the DDA approach.
- Potential for Personalized Treatment Strategies: DDA has the potential to assist molecular tumor boards in developing personalized treatment strategies for cancer patients. By considering the unique molecular profiles of individual patients, DDA can help identify the most appropriate MTAs based on the available evidence, optimizing treatment outcomes.
Overall, the greatest strengths of the DDA approach lie in its comprehensive analysis, transparent methodology, incorporation of functional evidence, demonstrated clinical performance, and potential to guide personalized treatment decisions in precision oncology.
What are potential weaknesses? Might the work be hard to replicate? Is there any "hype" in the discussion? How might an expert oncologist criticize the approach?
ChatGPT
Potential weaknesses of the DDA approach and areas that could be subject to criticism include:
- Limited Data Validation: The validation of the DDA system's performance was primarily based on data from the SHIVA01 clinical trial. While the results were promising, the evaluation was retrospective and conducted on a specific patient population. Further prospective clinical trials and external validation using diverse patient cohorts are necessary to establish the generalizability and reproducibility of the approach.
- Complexity and Replication Challenges: The DDA system relies on a vast network of associations and algorithms, making it a complex computational tool. Replicating such a system with similar performance and accuracy might pose challenges for other research groups or institutions. Detailed information about the system's architecture, algorithms, and databases would be crucial to enable replication, but these specifics may not be readily available.
- Interpretation of AEL Scores: The AEL scoring system used in DDA may require careful interpretation and validation. Assigning weights and determining the thresholds for positive and negative associations can introduce subjectivity and potential biases. The optimal thresholds and cutoffs for AEL scores may vary depending on the specific patient population, tumor types, and treatment contexts.
- Limited Clinical Evidence: The article acknowledges that the current version of DDA should not replace on-label therapies with MTAs that have higher clinical evidence levels. The system's effectiveness relies heavily on the available evidence-based associations, which might be limited, especially for emerging or investigational therapies. As new evidence emerges and more MTAs become available, the system would require continuous updates to maintain relevance.
- Generalizability and Bias: The article focuses on the SHIVA01 trial and its specific patient population. The performance and applicability of DDA to other cancer types, treatment settings, and populations with different genetic backgrounds and healthcare systems need further exploration. The system's ability to address potential biases, such as overfitting, selection bias, or data bias, should also be assessed to ensure unbiased and equitable treatment recommendations.
- "Hype" in Discussion: The discussion section of the article may contain elements of enthusiasm and anticipation for the future of AI-based precision oncology. While the results from SHIVA01 are encouraging, it is important to remain cautious and recognize that further clinical trials, real-world evidence, and regulatory evaluations are necessary before widespread adoption of DDA or similar AI-driven approaches.
An expert oncologist might criticize the approach by highlighting:
- The need for larger, randomized clinical trials comparing DDA-guided treatment decisions against standard practices to assess its true clinical benefit and superiority.
- Concerns about potential overreliance on computational algorithms without sufficient emphasis on individual patient factors, clinical judgment, and shared decision-making.
- The challenge of integrating DDA into existing clinical workflows and the potential for increased complexity and decision-making burden on healthcare providers.
- The limitations of evidence-based associations, as the quality and reliability of the underlying evidence can vary, and evidence gaps may exist for certain rare mutations or specific patient subgroups.
- The importance of considering other factors beyond molecular profiles, such as patient preferences, comorbidities, treatment toxicity, and cost-effectiveness, when making treatment decisions.
These criticisms aim to highlight the need for further research, clinical validation, and a careful balance between AI-driven recommendations and the expertise and individualized care provided by oncologists.
I read the following sentence in the abstract. The PFS is around 4 vs 2 months. It seems the effect size is pretty small. ### The median PFS was also significantly longer in patients receiving MTAs with high (1000+ <) than with low (<0) DDA scores (3.95 versus 1.95 months, P = 0.044)
ChatGPT
You are correct that the difference in median progression-free survival (PFS) between patients receiving MTAs with high DDA scores (1000+ <) and those with low DDA scores (<0) is relatively small. The reported median PFS for patients with high DDA scores was 3.95 months, compared to 1.95 months for patients with low DDA scores. The p-value of 0.044 suggests a statistically significant difference between the two groups.
It's important to note that while the difference in median PFS may appear small, statistical significance indicates that this difference is unlikely to have occurred by chance. However, the clinical significance of this small improvement in PFS needs to be carefully considered, as it may have limited impact on overall patient outcomes. The study would benefit from further exploration of overall survival (OS) data and additional clinical trials to provide a more comprehensive assessment of the clinical benefit associated with DDA-guided treatment decisions.
## END
See newly created AMA CPT code 0794T, pharmaco-oncologic treatment ranking.
