-
Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models
Authors:
Xiaofang Wang,
Dan Kondratyuk,
Eric Christiansen,
Kris M. Kitani,
Yair Alon,
Elad Eban
Abstract:
Committee-based models (ensembles or cascades) construct models by combining existing pre-trained ones. While ensembles and cascades are well-known techniques that were proposed before deep learning, they are not considered a core building block of deep model architectures and are rarely compared to in recent literature on developing efficient models. In this work, we go back to basics and conduct…
▽ More
Committee-based models (ensembles or cascades) construct models by combining existing pre-trained ones. While ensembles and cascades are well-known techniques that were proposed before deep learning, they are not considered a core building block of deep model architectures and are rarely compared to in recent literature on developing efficient models. In this work, we go back to basics and conduct a comprehensive analysis of the efficiency of committee-based models. We find that even the most simplistic method for building committees from existing, independently pre-trained models can match or exceed the accuracy of state-of-the-art models while being drastically more efficient. These simple committee-based models also outperform sophisticated neural architecture search methods (e.g., BigNAS). These findings hold true for several tasks, including image classification, video classification, and semantic segmentation, and various architecture families, such as ViT, EfficientNet, ResNet, MobileNetV2, and X3D. Our results show that an EfficientNet cascade can achieve a 5.4x speedup over B7 and a ViT cascade can achieve a 2.3x speedup over ViT-L-384 while being equally accurate.
△ Less
Submitted 17 February, 2022; v1 submitted 3 December, 2020;
originally announced December 2020.
-
Compact 200 line MATLAB code for inverse design in photonics by topology optimization: tutorial
Authors:
Rasmus E. Christiansen,
Ole Sigmund
Abstract:
We provide a compact 200 line MATLAB code demonstrating how topology optimization (TopOpt) as an inverse design tool may be used in photonics, targeting the design of two-dimensional dielectric metalenses and a metallic reflector as examples. The physics model is solved using the finite element method, and the code utilizes MATLAB's fmincon algorithm to solve the optimization problem. In addition…
▽ More
We provide a compact 200 line MATLAB code demonstrating how topology optimization (TopOpt) as an inverse design tool may be used in photonics, targeting the design of two-dimensional dielectric metalenses and a metallic reflector as examples. The physics model is solved using the finite element method, and the code utilizes MATLAB's fmincon algorithm to solve the optimization problem. In addition to presenting the code itself, we briefly discuss a number of extensions and provide the code required to implement some of these. Finally, we demonstrate the superiority of using a gradient-based method compared to a genetic-algorithm-based method (using MATLAB's ga algorithm) for solving inverse design problems in photonics. The MATLAB software is freely available in the paper and may be downloaded from https://www.topopt.mek.dtu.dk.
△ Less
Submitted 23 February, 2021; v1 submitted 23 August, 2020;
originally announced September 2020.
-
NAS-Bench-101: Towards Reproducible Neural Architecture Search
Authors:
Chris Ying,
Aaron Klein,
Esteban Real,
Eric Christiansen,
Kevin Murphy,
Frank Hutter
Abstract:
Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to researchers without access to large-scale computation. We aim to ameliorate these problems by introducing NAS-Bench-101, the first public architecture dataset for NAS research. To build NAS-Bench-101, we carefully constru…
▽ More
Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to researchers without access to large-scale computation. We aim to ameliorate these problems by introducing NAS-Bench-101, the first public architecture dataset for NAS research. To build NAS-Bench-101, we carefully constructed a compact, yet expressive, search space, exploiting graph isomorphisms to identify 423k unique convolutional architectures. We trained and evaluated all of these architectures multiple times on CIFAR-10 and compiled the results into a large dataset of over 5 million trained models. This allows researchers to evaluate the quality of a diverse range of models in milliseconds by querying the pre-computed dataset. We demonstrate its utility by analyzing the dataset as a whole and by benchmarking a range of architecture optimization algorithms.
△ Less
Submitted 14 May, 2019; v1 submitted 25 February, 2019;
originally announced February 2019.
-
An upper bound on prototype set size for condensed nearest neighbor
Authors:
Eric Christiansen
Abstract:
The condensed nearest neighbor (CNN) algorithm is a heuristic for reducing the number of prototypical points stored by a nearest neighbor classifier, while keeping the classification rule given by the reduced prototypical set consistent with the full set. I present an upper bound on the number of prototypical points accumulated by CNN. The bound originates in a bound on the number of times the dec…
▽ More
The condensed nearest neighbor (CNN) algorithm is a heuristic for reducing the number of prototypical points stored by a nearest neighbor classifier, while keeping the classification rule given by the reduced prototypical set consistent with the full set. I present an upper bound on the number of prototypical points accumulated by CNN. The bound originates in a bound on the number of times the decision rule is updated during training in the multiclass perceptron algorithm, and thus is independent of training set size.
△ Less
Submitted 29 September, 2013;
originally announced September 2013.
