-
Adapting Conformal Prediction to Distribution Shifts Without Labels
Authors:
Kevin Kasa,
Zhiyu Zhang,
Heng Yang,
Graham W. Taylor
Abstract:
Conformal prediction (CP) enables machine learning models to output prediction sets with guaranteed coverage rate, assuming exchangeable data. Unfortunately, the exchangeability assumption is frequently violated due to distribution shifts in practice, and the challenge is often compounded by the lack of ground truth labels at test time. Focusing on classification in this paper, our goal is to impr…
▽ More
Conformal prediction (CP) enables machine learning models to output prediction sets with guaranteed coverage rate, assuming exchangeable data. Unfortunately, the exchangeability assumption is frequently violated due to distribution shifts in practice, and the challenge is often compounded by the lack of ground truth labels at test time. Focusing on classification in this paper, our goal is to improve the quality of CP-generated prediction sets using only unlabeled data from the test domain. This is achieved by two new methods called ECP and EACP, that adjust the score function in CP according to the base model's uncertainty on the unlabeled test data. Through extensive experiments on a number of large-scale datasets and neural network architectures, we show that our methods provide consistent improvement over existing baselines and nearly match the performance of supervised algorithms.
△ Less
Submitted 3 June, 2024;
originally announced June 2024.
-
Mixture-Models: a one-stop Python Library for Model-based Clustering using various Mixture Models
Authors:
Siva Rajesh Kasa,
Hu Yijie,
Santhosh Kumar Kasa,
Vaibhav Rajan
Abstract:
\texttt{Mixture-Models} is an open-source Python library for fitting Gaussian Mixture Models (GMM) and their variants, such as Parsimonious GMMs, Mixture of Factor Analyzers, MClust models, Mixture of Student's t distributions, etc. It streamlines the implementation and analysis of these models using various first/second order optimization routines such as Gradient Descent and Newton-CG through au…
▽ More
\texttt{Mixture-Models} is an open-source Python library for fitting Gaussian Mixture Models (GMM) and their variants, such as Parsimonious GMMs, Mixture of Factor Analyzers, MClust models, Mixture of Student's t distributions, etc. It streamlines the implementation and analysis of these models using various first/second order optimization routines such as Gradient Descent and Newton-CG through automatic differentiation (AD) tools. This helps in extending these models to high-dimensional data, which is first of its kind among Python libraries. The library provides user-friendly model evaluation tools, such as BIC, AIC, and log-likelihood estimation. The source-code is licensed under MIT license and can be accessed at \url{https://github.com/kasakh/Mixture-Models}. The package is highly extensible, allowing users to incorporate new distributions and optimization techniques with ease. We conduct a large scale simulation to compare the performance of various gradient based approaches against Expectation Maximization on a wide range of settings and identify the corresponding best suited approach.
△ Less
Submitted 8 February, 2024;
originally announced February 2024.
-
How Robust are LLMs to In-Context Majority Label Bias?
Authors:
Karan Gupta,
Sumegh Roychowdhury,
Siva Rajesh Kasa,
Santhosh Kumar Kasa,
Anish Bhanushali,
Nikhil Pattisapu,
Prasanna Srinivasa Murthy
Abstract:
In the In-Context Learning (ICL) setup, various forms of label biases can manifest. One such manifestation is majority label bias, which arises when the distribution of labeled examples in the in-context samples is skewed towards one or more specific classes making Large Language Models (LLMs) more prone to predict those labels. Such discrepancies can arise from various factors, including logistic…
▽ More
In the In-Context Learning (ICL) setup, various forms of label biases can manifest. One such manifestation is majority label bias, which arises when the distribution of labeled examples in the in-context samples is skewed towards one or more specific classes making Large Language Models (LLMs) more prone to predict those labels. Such discrepancies can arise from various factors, including logistical constraints, inherent biases in data collection methods, limited access to diverse data sources, etc. which are unavoidable in a real-world industry setup. In this work, we study the robustness of in-context learning in LLMs to shifts that occur due to majority label bias within the purview of text classification tasks. Prior works have shown that in-context learning with LLMs is susceptible to such biases. In our study, we go one level deeper and show that the robustness boundary varies widely for different models and tasks, with certain LLMs being highly robust (~90%) to majority label bias. Additionally, our findings also highlight the impact of model size and the richness of instructional prompts contributing towards model robustness. We restrict our study to only publicly available open-source models to ensure transparency and reproducibility.
△ Less
Submitted 27 December, 2023;
originally announced December 2023.
-
Empirically Validating Conformal Prediction on Modern Vision Architectures Under Distribution Shift and Long-tailed Data
Authors:
Kevin Kasa,
Graham W. Taylor
Abstract:
Conformal prediction has emerged as a rigorous means of providing deep learning models with reliable uncertainty estimates and safety guarantees. Yet, its performance is known to degrade under distribution shift and long-tailed class distributions, which are often present in real world applications. Here, we characterize the performance of several post-hoc and training-based conformal prediction m…
▽ More
Conformal prediction has emerged as a rigorous means of providing deep learning models with reliable uncertainty estimates and safety guarantees. Yet, its performance is known to degrade under distribution shift and long-tailed class distributions, which are often present in real world applications. Here, we characterize the performance of several post-hoc and training-based conformal prediction methods under these settings, providing the first empirical evaluation on large-scale datasets and models. We show that across numerous conformal methods and neural network families, performance greatly degrades under distribution shifts violating safety guarantees. Similarly, we show that in long-tailed settings the guarantees are frequently violated on many classes. Understanding the limitations of these methods is necessary for deployment in real world and safety-critical applications.
△ Less
Submitted 3 July, 2023;
originally announced July 2023.
-
Two-color photoassociation spectroscopy of ytterbium atoms and the precise determinations of s-wave scattering lengths
Authors:
Masaaki Kitagawa,
Katsunari Enomoto,
Kentaro Kasa,
Yoshiro Takahashi,
Roman Ciurylo,
Pascal Naidon,
Paul S. Julienne
Abstract:
By performing high-resolution two-color photoassociation spectroscopy, we have successfully determined the binding energies of several of the last bound states of the homonuclear dimers of six different isotopes of ytterbium. These spectroscopic data are in excellent agreement with theoretical calculations based on a simple model potential, which very precisely predicts the s-wave scattering len…
▽ More
By performing high-resolution two-color photoassociation spectroscopy, we have successfully determined the binding energies of several of the last bound states of the homonuclear dimers of six different isotopes of ytterbium. These spectroscopic data are in excellent agreement with theoretical calculations based on a simple model potential, which very precisely predicts the s-wave scattering lengths of all 28 pairs of the seven stable isotopes. The s-wave scattering lengths for collision of two atoms of the same isotopic species are 13.33(18) nm for ^{168}Yb, 3.38(11) nm for ^{170}Yb, -0.15(19) nm for ^{171}Yb, -31.7(3.4) nm for ^{172}Yb, 10.55(11) nm for ^{173}Yb, 5.55(8) nm for ^{174}Yb, and -1.28(23) nm for ^{176}Yb. The coefficient of the lead term of the long-range van der Waals potential of the Yb_2 molecule is C_6=1932(30) atomic units $(E_h a_0^6 \approx 9.573\times 10^{-26}$ J nm^6).
△ Less
Submitted 6 August, 2007;
originally announced August 2007.
