Answering real-world clinical questions using large language model based systems
Authors:
Yen Sia Low,
Michael L. Jackson,
Rebecca J. Hyde,
Robert E. Brown,
Neil M. Sanghavi,
Julian D. Baldwin,
C. William Pike,
Jananee Muralidharan,
Gavin Hui,
Natasha Alexander,
Hadeel Hassan,
Rahul V. Nene,
Morgan Pike,
Courtney J. Pokrzywa,
Shivam Vedak,
Adam Paul Yan,
Dong-han Yao,
Amy R. Zipursky,
Christina Dinh,
Philip Ballentine,
Dan C. Derieg,
Vladimir Polony,
Rehan N. Chawdry,
Jordan Davies,
Brigham B. Hyde
, et al. (2 additional authors not shown)
Abstract:
Evidence to guide healthcare decisions is often limited by a lack of relevant and trustworthy literature as well as difficulty in contextualizing existing research for a specific patient. Large language models (LLMs) could potentially address both challenges by either summarizing published literature or generating new studies based on real-world data (RWD). We evaluated the ability of five LLM-bas…
▽ More
Evidence to guide healthcare decisions is often limited by a lack of relevant and trustworthy literature as well as difficulty in contextualizing existing research for a specific patient. Large language models (LLMs) could potentially address both challenges by either summarizing published literature or generating new studies based on real-world data (RWD). We evaluated the ability of five LLM-based systems in answering 50 clinical questions and had nine independent physicians review the responses for relevance, reliability, and actionability. As it stands, general-purpose LLMs (ChatGPT-4, Claude 3 Opus, Gemini Pro 1.5) rarely produced answers that were deemed relevant and evidence-based (2% - 10%). In contrast, retrieval augmented generation (RAG)-based and agentic LLM systems produced relevant and evidence-based answers for 24% (OpenEvidence) to 58% (ChatRWD) of questions. Only the agentic ChatRWD was able to answer novel questions compared to other LLMs (65% vs. 0-9%). These results suggest that while general-purpose LLMs should not be used as-is, a purpose-built system for evidence summarization based on RAG and one for generating novel evidence working synergistically would improve availability of pertinent evidence for patient care.
△ Less
Submitted 29 June, 2024;
originally announced July 2024.
Risk Assessment, Threat Modeling and Security Testing in SDLC
Authors:
Alya Hannah Ahmad Kamal,
Caryn Chuah Yi Yen,
Gan Jia Hui,
Pang Sze Ling,
Fatima-tuz-Zahra
Abstract:
The software development process is considered as one of the key guidelines in the creation of said software and this approach is necessary for providing a more efficient yet satisfactory output. Without separation of work into distinct stages, it may lead to many delays and inefficiency of the project process where this disorganization can directly affect the product quality and reliability. More…
▽ More
The software development process is considered as one of the key guidelines in the creation of said software and this approach is necessary for providing a more efficient yet satisfactory output. Without separation of work into distinct stages, it may lead to many delays and inefficiency of the project process where this disorganization can directly affect the product quality and reliability. Moreover, with this methodology established as the standard for any project, there are bound to be missteps specifically in regard to the involvement of security due to the lack of awareness. Therefore, the aim of this research is to identify and elaborate the findings and understanding of the security integrated into the process of software development as well as the related individual roles in ensuring that this security is maintained. Through thorough analysis and review of literature, an effort has been made through this paper to showcase the correct processes and ways for securing the software development process. At the same time, certain issues that pertain to this subject have been discussed together with proposing appropriate solutions. Furthermore, in depth discussion is carried out regarding methods such as security testing, risk assessment, threat modeling and other techniques that are able to create a more secure environment and systematic approach in a software development process.
△ Less
Submitted 13 December, 2020;
originally announced December 2020.
