-
PAC-Bayes Analysis for Recalibration in Classification
Authors:
Masahiro Fujisawa,
Futoshi Futami
Abstract:
Nonparametric estimation with binning is widely employed in the calibration error evaluation and the recalibration of machine learning models. Recently, theoretical analyses of the bias induced by this estimation approach have been actively pursued; however, the understanding of the generalization of the calibration error to unknown data remains limited. In addition, although many recalibration al…
▽ More
Nonparametric estimation with binning is widely employed in the calibration error evaluation and the recalibration of machine learning models. Recently, theoretical analyses of the bias induced by this estimation approach have been actively pursued; however, the understanding of the generalization of the calibration error to unknown data remains limited. In addition, although many recalibration algorithms have been proposed, their generalization performance lacks theoretical guarantees. To address this problem, we conduct a generalization analysis of the calibration error under the probably approximately correct (PAC) Bayes framework. This approach enables us to derive a first optimizable upper bound for the generalization error in the calibration context. We then propose a generalization-aware recalibration algorithm based on our generalization theory. Numerical experiments show that our algorithm improves the Gaussian-process-based recalibration performance on various benchmark datasets and models.
△ Less
Submitted 10 June, 2024;
originally announced June 2024.
-
Information-theoretic Generalization Analysis for Expected Calibration Error
Authors:
Futoshi Futami,
Masahiro Fujisawa
Abstract:
While the expected calibration error (ECE), which employs binning, is widely adopted to evaluate the calibration performance of machine learning models, theoretical understanding of its estimation bias is limited. In this paper, we present the first comprehensive analysis of the estimation bias in the two common binning strategies, uniform mass and uniform width binning. Our analysis establishes u…
▽ More
While the expected calibration error (ECE), which employs binning, is widely adopted to evaluate the calibration performance of machine learning models, theoretical understanding of its estimation bias is limited. In this paper, we present the first comprehensive analysis of the estimation bias in the two common binning strategies, uniform mass and uniform width binning. Our analysis establishes upper bounds on the bias, achieving an improved convergence rate. Moreover, our bounds reveal, for the first time, the optimal number of bins to minimize the estimation bias. We further extend our bias analysis to generalization error analysis based on the information-theoretic approach, deriving upper bounds that enable the numerical evaluation of how small the ECE is for unknown data. Experiments using deep learning models show that our bounds are nonvacuous thanks to this information-theoretic generalization analysis approach.
△ Less
Submitted 24 May, 2024;
originally announced May 2024.
-
ProgrammableGrass: A Shape-Changing Artificial Grass Display Adapted for Dynamic and Interactive Display Features
Authors:
Kojiro Tanaka,
Akito Mizuno,
Toranosuke Kato,
Masahiko Mikawa,
Makoto Fujisawa
Abstract:
There are various proposals for employing grass materials as a green landscape-friendly display. However, it is difficult for current techniques to display smooth animations using 8-bit images and to adjust display resolution, similar to conventional displays. We present ProgrammableGrass, an artificial grass display with scalable resolution, capable of swiftly controlling grass color at 8-bit lev…
▽ More
There are various proposals for employing grass materials as a green landscape-friendly display. However, it is difficult for current techniques to display smooth animations using 8-bit images and to adjust display resolution, similar to conventional displays. We present ProgrammableGrass, an artificial grass display with scalable resolution, capable of swiftly controlling grass color at 8-bit levels. This grass display can control grass colors linearly at the 8-bit level, similar to an LCD display, and can also display not only 8-bit-based images but also videos. This display enables pixel-by-pixel color transitions from yellow to green using fixed-length yellow and adjustable-length green grass. We designed a grass module that can be connected to other modules. Utilizing a proportional derivative control, the grass colors are manipulated to display animations at approximately 10 [fps]. Since the relationship between grass lengths and colors is nonlinear, we developed a calibration system for ProgrammableGrass. We revealed that this calibration system allows ProgrammableGrass to linearly control grass colors at 8-bit levels through experiments under multiple conditions. Lastly, we demonstrate ProgrammableGrass to show smooth animations with 8-bit grayscale images. Moreover, we show several application examples to illustrate the potential of ProgrammableGrass. With the advancement of this technology, users will be able to treat grass as a green-based interactive display device.
△ Less
Submitted 18 March, 2024;
originally announced March 2024.
-
Time-Independent Information-Theoretic Generalization Bounds for SGLD
Authors:
Futoshi Futami,
Masahiro Fujisawa
Abstract:
We provide novel information-theoretic generalization bounds for stochastic gradient Langevin dynamics (SGLD) under the assumptions of smoothness and dissipativity, which are widely used in sampling and non-convex optimization studies. Our bounds are time-independent and decay to zero as the sample size increases, regardless of the number of iterations and whether the step size is fixed. Unlike pr…
▽ More
We provide novel information-theoretic generalization bounds for stochastic gradient Langevin dynamics (SGLD) under the assumptions of smoothness and dissipativity, which are widely used in sampling and non-convex optimization studies. Our bounds are time-independent and decay to zero as the sample size increases, regardless of the number of iterations and whether the step size is fixed. Unlike previous studies, we derive the generalization error bounds by focusing on the time evolution of the Kullback--Leibler divergence, which is related to the stability of datasets and is the upper bound of the mutual information between output parameters and an input dataset. Additionally, we establish the first information-theoretic generalization bound when the training and test loss are the same by showing that a loss function of SGLD is sub-exponential. This bound is also time-independent and removes the problematic step size dependence in existing work, leading to an improved excess risk bound by combining our analysis with the existing non-convex optimization error bounds.
△ Less
Submitted 2 November, 2023;
originally announced November 2023.
-
Initialization Bias of Fourier Neural Operator: Revisiting the Edge of Chaos
Authors:
Takeshi Koshizuka,
Masahiro Fujisawa,
Yusuke Tanaka,
Issei Sato
Abstract:
This paper investigates the initialization bias of the Fourier neural operator (FNO). A mean-field theory for FNO is established, analyzing the behavior of the random FNO from an \emph{edge of chaos} perspective. We uncover that the forward and backward propagation behaviors exhibit characteristics unique to FNO, induced by mode truncation, while also showcasing similarities to those of densely co…
▽ More
This paper investigates the initialization bias of the Fourier neural operator (FNO). A mean-field theory for FNO is established, analyzing the behavior of the random FNO from an \emph{edge of chaos} perspective. We uncover that the forward and backward propagation behaviors exhibit characteristics unique to FNO, induced by mode truncation, while also showcasing similarities to those of densely connected networks. Building upon this observation, we also propose an edge of chaos initialization scheme for FNO to mitigate the negative initialization bias leading to training instability. Experimental results show the effectiveness of our initialization scheme, enabling stable training of deep FNO without skip-connection.
△ Less
Submitted 15 February, 2024; v1 submitted 10 October, 2023;
originally announced October 2023.
-
A Lightweight Transmission Parameter Selection Scheme Using Reinforcement Learning for LoRaWAN
Authors:
Aohan Li,
Ikumi Urabe,
Minoru Fujisawa,
So Hasegawa,
Hiroyuki Yasuda,
Song-Ju Kim,
Mikio Hasegawa
Abstract:
The number of IoT devices is predicted to reach 125 billion by 2023. The growth of IoT devices will intensify the collisions between devices, degrading communication performance. Selecting appropriate transmission parameters, such as channel and spreading factor (SF), can effectively reduce the collisions between long-range (LoRa) devices. However, most of the schemes proposed in the current liter…
▽ More
The number of IoT devices is predicted to reach 125 billion by 2023. The growth of IoT devices will intensify the collisions between devices, degrading communication performance. Selecting appropriate transmission parameters, such as channel and spreading factor (SF), can effectively reduce the collisions between long-range (LoRa) devices. However, most of the schemes proposed in the current literature are not easy to implement on an IoT device with limited computational complexity and memory. To solve this issue, we propose a lightweight transmission-parameter selection scheme, i.e., a joint channel and SF selection scheme using reinforcement learning for low-power wide area networking (LoRaWAN). In the proposed scheme, appropriate transmission parameters can be selected by simple four arithmetic operations using only Acknowledge (ACK) information. Additionally, we theoretically analyze the computational complexity and memory requirement of our proposed scheme, which verified that our proposed scheme could select transmission parameters with extremely low computational complexity and memory requirement. Moreover, a large number of experiments were implemented on the LoRa devices in the real world to evaluate the effectiveness of our proposed scheme. The experimental results demonstrate the following main phenomena. (1) Compared to other lightweight transmission-parameter selection schemes, collisions between LoRa devices can be efficiently avoided by our proposed scheme in LoRaWAN irrespective of changes in the available channels. (2) The frame success rate (FSR) can be improved by selecting access channels and using SFs as opposed to only selecting access channels. (3) Since interference exists between adjacent channels, FSR and fairness can be improved by increasing the interval of adjacent available channels.
△ Less
Submitted 2 August, 2022;
originally announced August 2022.
-
Dynamic Grass Color Scale Display Technique Based on Grass Length for Green Landscape-Friendly Animation Display
Authors:
Kojiro Tanaka,
Yuichi Kato,
Akito Mizuno,
Masahiko Mikawa,
Makoto Fujisawa
Abstract:
Recently, public displays such as liquid crystal displays (LCDs) are often used in urban green spaces, however, the display devices can spoil green landscape of urban green spaces because they look like artificial materials. We previously proposed a green landscape-friendly grass animation display method by controlling a pixel-by-pixel grass color dynamically. The grass color can be changed by mov…
▽ More
Recently, public displays such as liquid crystal displays (LCDs) are often used in urban green spaces, however, the display devices can spoil green landscape of urban green spaces because they look like artificial materials. We previously proposed a green landscape-friendly grass animation display method by controlling a pixel-by-pixel grass color dynamically. The grass color can be changed by moving a green grass length in yellow grass, and the grass animation display can play simple animations using grayscale images. In the previous research, the color scale was mapped to the green grass length subjectively, however, this method has not achieved displaying the grass colors corresponding to the color scale based on objective evaluations. Here, we introduce a dynamic grass color scale display technique based on a grass length. In this paper, we developed a grass color scale setting procedure to map the grass length to the color scale with five levels through image processing. Through the outdoor experiment of the grass color scale setting procedure, the color scale can correspond to the green grass length based on a viewpoint. After the experiments, we demonstrated a grass animation display to show the animations with the color scale using the experiment results.
△ Less
Submitted 18 December, 2022; v1 submitted 16 March, 2022;
originally announced March 2022.
-
$γ$-ABC: Outlier-Robust Approximate Bayesian Computation Based on a Robust Divergence Estimator
Authors:
Masahiro Fujisawa,
Takeshi Teshima,
Issei Sato,
Masashi Sugiyama
Abstract:
Approximate Bayesian computation (ABC) is a likelihood-free inference method that has been employed in various applications. However, ABC can be sensitive to outliers if a data discrepancy measure is chosen inappropriately. In this paper, we propose to use a nearest-neighbor-based $γ$-divergence estimator as a data discrepancy measure. We show that our estimator possesses a suitable theoretical ro…
▽ More
Approximate Bayesian computation (ABC) is a likelihood-free inference method that has been employed in various applications. However, ABC can be sensitive to outliers if a data discrepancy measure is chosen inappropriately. In this paper, we propose to use a nearest-neighbor-based $γ$-divergence estimator as a data discrepancy measure. We show that our estimator possesses a suitable theoretical robustness property called the redescending property. In addition, our estimator enjoys various desirable properties such as high flexibility, asymptotic unbiasedness, almost sure convergence, and linear-time computational complexity. Through experiments, we demonstrate that our method achieves significantly higher robustness than existing discrepancy measures.
△ Less
Submitted 5 March, 2021; v1 submitted 13 June, 2020;
originally announced June 2020.
-
Multilevel Monte Carlo Variational Inference
Authors:
Masahiro Fujisawa,
Issei Sato
Abstract:
We propose a variance reduction framework for variational inference using the Multilevel Monte Carlo (MLMC) method. Our framework is built on reparameterized gradient estimators and "recycles" parameters obtained from past update history in optimization. In addition, our framework provides a new optimization algorithm based on stochastic gradient descent (SGD) that adaptively estimates the sample…
▽ More
We propose a variance reduction framework for variational inference using the Multilevel Monte Carlo (MLMC) method. Our framework is built on reparameterized gradient estimators and "recycles" parameters obtained from past update history in optimization. In addition, our framework provides a new optimization algorithm based on stochastic gradient descent (SGD) that adaptively estimates the sample size used for gradient estimation according to the ratio of the gradient variance. We theoretically show that, with our method, the variance of the gradient estimator decreases as optimization proceeds and that a learning rate scheduler function helps improve the convergence. We also show that, in terms of the \textit{signal-to-noise} ratio, our method can improve the quality of gradient estimation by the learning rate scheduler function without increasing the initial sample size. Finally, we confirm that our method achieves faster convergence and reduces the variance of the gradient estimator compared with other methods through experimental comparisons with baseline methods using several benchmark datasets.
△ Less
Submitted 2 December, 2021; v1 submitted 1 February, 2019;
originally announced February 2019.
