-
Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biology
Authors:
Oren Kraus,
Kian Kenyon-Dean,
Saber Saberian,
Maryam Fallah,
Peter McLean,
Jess Leung,
Vasudev Sharma,
Ayla Khan,
Jia Balakrishnan,
Safiye Celik,
Dominique Beaini,
Maciej Sypetkowski,
Chi Vicky Cheng,
Kristen Morse,
Maureen Makes,
Ben Mabey,
Berton Earnshaw
Abstract:
Featurizing microscopy images for use in biological research remains a significant challenge, especially for large-scale experiments spanning millions of images. This work explores the scaling properties of weakly supervised classifiers and self-supervised masked autoencoders (MAEs) when training with increasingly larger model backbones and microscopy datasets. Our results show that ViT-based MAEs…
▽ More
Featurizing microscopy images for use in biological research remains a significant challenge, especially for large-scale experiments spanning millions of images. This work explores the scaling properties of weakly supervised classifiers and self-supervised masked autoencoders (MAEs) when training with increasingly larger model backbones and microscopy datasets. Our results show that ViT-based MAEs outperform weakly supervised classifiers on a variety of tasks, achieving as much as a 11.5% relative improvement when recalling known biological relationships curated from public databases. Additionally, we develop a new channel-agnostic MAE architecture (CA-MAE) that allows for inputting images of different numbers and orders of channels at inference time. We demonstrate that CA-MAEs effectively generalize by inferring and evaluating on a microscopy image dataset (JUMP-CP) generated under different experimental conditions with a different channel structure than our pretraining data (RPI-93M). Our findings motivate continued research into scaling self-supervised learning on microscopy data in order to create powerful foundation models of cellular biology that have the potential to catalyze advancements in drug discovery and beyond.
△ Less
Submitted 15 April, 2024;
originally announced April 2024.
-
Masked Autoencoders are Scalable Learners of Cellular Morphology
Authors:
Oren Kraus,
Kian Kenyon-Dean,
Saber Saberian,
Maryam Fallah,
Peter McLean,
Jess Leung,
Vasudev Sharma,
Ayla Khan,
Jia Balakrishnan,
Safiye Celik,
Maciej Sypetkowski,
Chi Vicky Cheng,
Kristen Morse,
Maureen Makes,
Ben Mabey,
Berton Earnshaw
Abstract:
Inferring biological relationships from cellular phenotypes in high-content microscopy screens provides significant opportunity and challenge in biological research. Prior results have shown that deep vision models can capture biological signal better than hand-crafted features. This work explores how self-supervised deep learning approaches scale when training larger models on larger microscopy d…
▽ More
Inferring biological relationships from cellular phenotypes in high-content microscopy screens provides significant opportunity and challenge in biological research. Prior results have shown that deep vision models can capture biological signal better than hand-crafted features. This work explores how self-supervised deep learning approaches scale when training larger models on larger microscopy datasets. Our results show that both CNN- and ViT-based masked autoencoders significantly outperform weakly supervised baselines. At the high-end of our scale, a ViT-L/8 trained on over 3.5-billion unique crops sampled from 93-million microscopy images achieves relative improvements as high as 28% over our best weakly supervised baseline at inferring known biological relationships curated from public databases. Relevant code and select models released with this work can be found at: https://github.com/recursionpharma/maes_microscopy.
△ Less
Submitted 27 November, 2023; v1 submitted 27 September, 2023;
originally announced September 2023.
-
RxRx1: A Dataset for Evaluating Experimental Batch Correction Methods
Authors:
Maciej Sypetkowski,
Morteza Rezanejad,
Saber Saberian,
Oren Kraus,
John Urbanik,
James Taylor,
Ben Mabey,
Mason Victors,
Jason Yosinski,
Alborz Rezazadeh Sereshkeh,
Imran Haque,
Berton Earnshaw
Abstract:
High-throughput screening techniques are commonly used to obtain large quantities of data in many fields of biology. It is well known that artifacts arising from variability in the technical execution of different experimental batches within such screens confound these observations and can lead to invalid biological conclusions. It is therefore necessary to account for these batch effects when ana…
▽ More
High-throughput screening techniques are commonly used to obtain large quantities of data in many fields of biology. It is well known that artifacts arising from variability in the technical execution of different experimental batches within such screens confound these observations and can lead to invalid biological conclusions. It is therefore necessary to account for these batch effects when analyzing outcomes. In this paper we describe RxRx1, a biological dataset designed specifically for the systematic study of batch effect correction methods. The dataset consists of 125,510 high-resolution fluorescence microscopy images of human cells under 1,138 genetic perturbations in 51 experimental batches across 4 cell types. Visual inspection of the images alone clearly demonstrates significant batch effects. We propose a classification task designed to evaluate the effectiveness of experimental batch correction methods on these images and examine the performance of a number of correction methods on this task. Our goal in releasing RxRx1 is to encourage the development of effective experimental batch correction methods that generalize well to unseen experimental batches. The dataset can be downloaded at https://rxrx.ai.
△ Less
Submitted 13 January, 2023;
originally announced January 2023.
-
MolE: a molecular foundation model for drug discovery
Authors:
Oscar Méndez-Lucio,
Christos Nicolaou,
Berton Earnshaw
Abstract:
Models that accurately predict properties based on chemical structure are valuable tools in drug discovery. However, for many properties, public and private training sets are typically small, and it is difficult for the models to generalize well outside of the training data. Recently, large language models have addressed this problem by using self-supervised pretraining on large unlabeled datasets…
▽ More
Models that accurately predict properties based on chemical structure are valuable tools in drug discovery. However, for many properties, public and private training sets are typically small, and it is difficult for the models to generalize well outside of the training data. Recently, large language models have addressed this problem by using self-supervised pretraining on large unlabeled datasets, followed by fine-tuning on smaller, labeled datasets. In this paper, we report MolE, a molecular foundation model that adapts the DeBERTa architecture to be used on molecular graphs together with a two-step pretraining strategy. The first step of pretraining is a self-supervised approach focused on learning chemical structures, and the second step is a massive multi-task approach to learn biological information. We show that fine-tuning pretrained MolE achieves state-of-the-art results on 9 of the 22 ADMET tasks included in the Therapeutic Data Commons.
△ Less
Submitted 3 November, 2022;
originally announced November 2022.
-
WILDS: A Benchmark of in-the-Wild Distribution Shifts
Authors:
Pang Wei Koh,
Shiori Sagawa,
Henrik Marklund,
Sang Michael Xie,
Marvin Zhang,
Akshay Balsubramani,
Weihua Hu,
Michihiro Yasunaga,
Richard Lanas Phillips,
Irena Gao,
Tony Lee,
Etienne David,
Ian Stavness,
Wei Guo,
Berton A. Earnshaw,
Imran S. Haque,
Sara Beery,
Jure Leskovec,
Anshul Kundaje,
Emma Pierson,
Sergey Levine,
Chelsea Finn,
Percy Liang
Abstract:
Distribution shifts -- where the training distribution differs from the test distribution -- can substantially degrade the accuracy of machine learning (ML) systems deployed in the wild. Despite their ubiquity in the real-world deployments, these distribution shifts are under-represented in the datasets widely used in the ML community today. To address this gap, we present WILDS, a curated benchma…
▽ More
Distribution shifts -- where the training distribution differs from the test distribution -- can substantially degrade the accuracy of machine learning (ML) systems deployed in the wild. Despite their ubiquity in the real-world deployments, these distribution shifts are under-represented in the datasets widely used in the ML community today. To address this gap, we present WILDS, a curated benchmark of 10 datasets reflecting a diverse range of distribution shifts that naturally arise in real-world applications, such as shifts across hospitals for tumor identification; across camera traps for wildlife monitoring; and across time and location in satellite imaging and poverty mapping. On each dataset, we show that standard training yields substantially lower out-of-distribution than in-distribution performance. This gap remains even with models trained by existing methods for tackling distribution shifts, underscoring the need for new methods for training models that are more robust to the types of distribution shifts that arise in practice. To facilitate method development, we provide an open-source package that automates dataset loading, contains default model architectures and hyperparameters, and standardizes evaluations. Code and leaderboards are available at https://wilds.stanford.edu.
△ Less
Submitted 16 July, 2021; v1 submitted 14 December, 2020;
originally announced December 2020.
