Merlin: A Vision Language Foundation Model for 3D Computed Tomography
Authors:
Louis Blankemeier,
Joseph Paul Cohen,
Ashwin Kumar,
Dave Van Veen,
Syed Jamal Safdar Gardezi,
Magdalini Paschali,
Zhihong Chen,
Jean-Benoit Delbrouck,
Eduardo Reis,
Cesar Truyts,
Christian Bluethgen,
Malte Engmann Kjeldskov Jensen,
Sophie Ostmeier,
Maya Varma,
Jeya Maria Jose Valanarasu,
Zhongnan Fang,
Zepeng Huo,
Zaid Nabulsi,
Diego Ardila,
Wei-Hung Weng,
Edson Amaro Junior,
Neera Ahuja,
Jason Fries,
Nigam H. Shah,
Andrew Johnston
, et al. (6 additional authors not shown)
Abstract:
Over 85 million computed tomography (CT) scans are performed annually in the US, of which approximately one quarter focus on the abdomen. Given the current radiologist shortage, there is a large impetus to use artificial intelligence to alleviate the burden of interpreting these complex imaging studies. Prior state-of-the-art approaches for automated medical image interpretation leverage vision la…
▽ More
Over 85 million computed tomography (CT) scans are performed annually in the US, of which approximately one quarter focus on the abdomen. Given the current radiologist shortage, there is a large impetus to use artificial intelligence to alleviate the burden of interpreting these complex imaging studies. Prior state-of-the-art approaches for automated medical image interpretation leverage vision language models (VLMs). However, current medical VLMs are generally limited to 2D images and short reports, and do not leverage electronic health record (EHR) data for supervision. We introduce Merlin - a 3D VLM that we train using paired CT scans (6+ million images from 15,331 CTs), EHR diagnosis codes (1.8+ million codes), and radiology reports (6+ million tokens). We evaluate Merlin on 6 task types and 752 individual tasks. The non-adapted (off-the-shelf) tasks include zero-shot findings classification (31 findings), phenotype classification (692 phenotypes), and zero-shot cross-modal retrieval (image to findings and image to impressions), while model adapted tasks include 5-year disease prediction (6 diseases), radiology report generation, and 3D semantic segmentation (20 organs). We perform internal validation on a test set of 5,137 CTs, and external validation on 7,000 clinical CTs and on two public CT datasets (VerSe, TotalSegmentator). Beyond these clinically-relevant evaluations, we assess the efficacy of various network architectures and training strategies to depict that Merlin has favorable performance to existing task-specific baselines. We derive data scaling laws to empirically assess training data needs for requisite downstream task performance. Furthermore, unlike conventional VLMs that require hundreds of GPUs for training, we perform all training on a single GPU.
△ Less
Submitted 10 June, 2024;
originally announced June 2024.
GeoSES -- um Índice Socioeconômico para Estudos de Saúde no Brasil
Authors:
Ligia Vizeu Barrozo,
Michel Fornaciali,
Carmen Diva Saldiva de André,
Guilherme Augusto Zimeo Morais,
Giselle Mansur,
William Cabral-Miranda,
João Ricardo Sato,
Edson Amaro Júnior
Abstract:
Objective: to define an index that summarizes the main dimensions of the socioeconomic context for research purposes, evaluation and monitoring health inequalities. Methods: the index was created from the 2010 Brazilian Demographic Census, whose variables selection was guided by theoretical references for health studies, including seven socioeconomic dimensions: education, mobility, poverty, wealt…
▽ More
Objective: to define an index that summarizes the main dimensions of the socioeconomic context for research purposes, evaluation and monitoring health inequalities. Methods: the index was created from the 2010 Brazilian Demographic Census, whose variables selection was guided by theoretical references for health studies, including seven socioeconomic dimensions: education, mobility, poverty, wealth, income, segregation and deprivation of resources and services. The index was developed using principal component analysis, and was evaluated for its construct, content and applicability components. Results: GeoSES-BR dimensions showed good association with HDI-M (above 0.85). The model with the poverty dimension best explained the relative risk of avoidable cause mortality in Brazil. In the intraurban scale, the model with GeoSES-IM was the one that best explained the relative risk of mortality from circulatory system diseases. Conclusion: GeoSES showed significant explanatory potential in the studied scales.
△ Less
Submitted 9 October, 2019;
originally announced October 2019.
