-
SOccDPT: Semi-Supervised 3D Semantic Occupancy from Dense Prediction Transformers trained under memory constraints
Authors:
Aditya Nalgunda Ganesh
Abstract:
We present SOccDPT, a memory-efficient approach for 3D semantic occupancy prediction from monocular image input using dense prediction transformers. To address the limitations of existing methods trained on structured traffic datasets, we train our model on unstructured datasets including the Indian Driving Dataset and Bengaluru Driving Dataset. Our semi-supervised training pipeline allows SOccDPT…
▽ More
We present SOccDPT, a memory-efficient approach for 3D semantic occupancy prediction from monocular image input using dense prediction transformers. To address the limitations of existing methods trained on structured traffic datasets, we train our model on unstructured datasets including the Indian Driving Dataset and Bengaluru Driving Dataset. Our semi-supervised training pipeline allows SOccDPT to learn from datasets with limited labels by reducing the requirement for manual labelling by substituting it with pseudo-ground truth labels to produce our Bengaluru Semantic Occupancy Dataset. This broader training enhances our model's ability to handle unstructured traffic scenarios effectively. To overcome memory limitations during training, we introduce patch-wise training where we select a subset of parameters to train each epoch, reducing memory usage during auto-grad graph construction. In the context of unstructured traffic and memory-constrained training and inference, SOccDPT outperforms existing disparity estimation approaches as shown by the RMSE score of 9.1473, achieves a semantic segmentation IoU score of 46.02% and operates at a competitive frequency of 69.47 Hz. We make our code and semantic occupancy dataset public.
△ Less
Submitted 19 November, 2023;
originally announced November 2023.
-
OCTraN: 3D Occupancy Convolutional Transformer Network in Unstructured Traffic Scenarios
Authors:
Aditya Nalgunda Ganesh,
Dhruval Pobbathi Badrinath,
Harshith Mohan Kumar,
Priya SS,
Surabhi Narayan
Abstract:
Modern approaches for vision-centric environment perception for autonomous navigation make extensive use of self-supervised monocular depth estimation algorithms that output disparity maps. However, when this disparity map is projected onto 3D space, the errors in disparity are magnified, resulting in a depth estimation error that increases quadratically as the distance from the camera increases.…
▽ More
Modern approaches for vision-centric environment perception for autonomous navigation make extensive use of self-supervised monocular depth estimation algorithms that output disparity maps. However, when this disparity map is projected onto 3D space, the errors in disparity are magnified, resulting in a depth estimation error that increases quadratically as the distance from the camera increases. Though Light Detection and Ranging (LiDAR) can solve this issue, it is expensive and not feasible for many applications. To address the challenge of accurate ranging with low-cost sensors, we propose, OCTraN, a transformer architecture that uses iterative-attention to convert 2D image features into 3D occupancy features and makes use of convolution and transpose convolution to efficiently operate on spatial information. We also develop a self-supervised training pipeline to generalize the model to any scene by eliminating the need for LiDAR ground truth by substituting it with pseudo-ground truth labels obtained from boosted monocular depth estimation.
△ Less
Submitted 20 July, 2023;
originally announced July 2023.
-
Multiplexed gradient descent: Fast online training of modern datasets on hardware neural networks without backpropagation
Authors:
Adam N. McCaughan,
Bakhrom G. Oripov,
Natesh Ganesh,
Sae Woo Nam,
Andrew Dienstfrey,
Sonia M. Buckley
Abstract:
We present multiplexed gradient descent (MGD), a gradient descent framework designed to easily train analog or digital neural networks in hardware. MGD utilizes zero-order optimization techniques for online training of hardware neural networks. We demonstrate its ability to train neural networks on modern machine learning datasets, including CIFAR-10 and Fashion-MNIST, and compare its performance…
▽ More
We present multiplexed gradient descent (MGD), a gradient descent framework designed to easily train analog or digital neural networks in hardware. MGD utilizes zero-order optimization techniques for online training of hardware neural networks. We demonstrate its ability to train neural networks on modern machine learning datasets, including CIFAR-10 and Fashion-MNIST, and compare its performance to backpropagation. Assuming realistic timescales and hardware parameters, our results indicate that these optimization techniques can train a network on emerging hardware platforms orders of magnitude faster than the wall-clock time of training via backpropagation on a standard GPU, even in the presence of imperfect weight updates or device-to-device variations in the hardware. We additionally describe how it can be applied to existing hardware as part of chip-in-the-loop training, or integrated directly at the hardware level. Crucially, the MGD framework is highly flexible, and its gradient descent process can be optimized to compensate for specific hardware limitations such as slow parameter-update speeds or limited input bandwidth.
△ Less
Submitted 5 March, 2023;
originally announced March 2023.
-
From Quantifying Vagueness To Pan-niftyism
Authors:
Natesh Ganesh
Abstract:
In this short paper, we will introduce a simple model for quantifying philosophical vagueness. There is growing interest in this endeavor to quantify vague concepts of consciousness, agency, etc. We will then discuss some of the implications of this model including the conditions under which the quantification of `nifty' leads to pan-nifty-ism. Understanding this leads to an interesting insight -…
▽ More
In this short paper, we will introduce a simple model for quantifying philosophical vagueness. There is growing interest in this endeavor to quantify vague concepts of consciousness, agency, etc. We will then discuss some of the implications of this model including the conditions under which the quantification of `nifty' leads to pan-nifty-ism. Understanding this leads to an interesting insight - the reason a framework to quantify consciousness like Integrated Information Theory implies (forms of) panpsychism is because there is favorable structure already implicitly encoded in the construction of the quantification metric.
△ Less
Submitted 1 March, 2021;
originally announced March 2021.
-
No Substitute for Functionalism -- A Reply to 'Falsification & Consciousness'
Authors:
Natesh Ganesh
Abstract:
In their paper 'Falsification and Consciousness' [1], Kleiner and Hoel introduced a formal mathematical model of the process of generating observable data from experiments and using that data to generate inferences and predictions onto an experience space. The resulting substitution argument built on this framework was used to show that any theory of consciousness with independent inference and pr…
▽ More
In their paper 'Falsification and Consciousness' [1], Kleiner and Hoel introduced a formal mathematical model of the process of generating observable data from experiments and using that data to generate inferences and predictions onto an experience space. The resulting substitution argument built on this framework was used to show that any theory of consciousness with independent inference and prediction data are pre-falsified, if the inference reports are considered valid. If this argument does indeed pre-falsify many of the leading theories of consciousness, it would indicate a fundamental problem affecting the field of consciousness as a whole that would require radical changes to how consciousness science is performed. In this reply, the author will identify avenues of expansion for the model proposed in [1], allowing us to distinguish between different types of variation. Motivated by examples from neural networks, state machines and Turing machines, we will prove that substitutions do not exist for a very broad class of Level-1 functionalist theories, rendering them immune to the aforementioned substitution argument.
△ Less
Submitted 30 April, 2021; v1 submitted 28 May, 2020;
originally announced June 2020.
-
A Non-equilibrium Thermodynamic Framework of Consciousness
Authors:
Natesh Ganesh
Abstract:
In this paper, we take a brief look at the advantages and disadvantages of dominant frameworks in consciousness studies -- functionalist and causal structure theories, and use it to motivate a new non-equilibrium thermodynamic framework of consciousness. The main hypothesis in this paper will be two thermodynamic conditions obtained from the non-equilibrium fluctuation theorems -- TCC 1 and 2, tha…
▽ More
In this paper, we take a brief look at the advantages and disadvantages of dominant frameworks in consciousness studies -- functionalist and causal structure theories, and use it to motivate a new non-equilibrium thermodynamic framework of consciousness. The main hypothesis in this paper will be two thermodynamic conditions obtained from the non-equilibrium fluctuation theorems -- TCC 1 and 2, that the author proposes as necessary conditions that a system will have to satisfy in order to be 'conscious'. These descriptions will look to specify the functions achieved by a conscious system and restrict the physical structures that achieve them without presupposing either of the two. These represent an attempt to integrate consciousness into established physical law (without invoking untested novel frameworks in quantum mechanics and/or general relativity). We will also discuss it's implications on a wide range of existing questions, including a stance on the hard problem. The paper will also explore why this framework might offer a serious path forward to understanding consciousness (and perhaps even realizing it in artificial systems) as well as laying out some problems and challenges that lie ahead.
△ Less
Submitted 4 May, 2020;
originally announced May 2020.
-
Rebooting Neuromorphic Hardware Design -- A Complexity Engineering Approach
Authors:
Natesh Ganesh
Abstract:
As the compute demands for machine learning and artificial intelligence applications continue to grow, neuromorphic hardware has been touted as a potential solution. New emerging devices like memristors, atomic switches, etc have shown tremendous potential to replace CMOS-based circuits but have been hindered by multiple challenges with respect to device variability, stochastic behavior and scalab…
▽ More
As the compute demands for machine learning and artificial intelligence applications continue to grow, neuromorphic hardware has been touted as a potential solution. New emerging devices like memristors, atomic switches, etc have shown tremendous potential to replace CMOS-based circuits but have been hindered by multiple challenges with respect to device variability, stochastic behavior and scalability. In this paper we will introduce a Description<->Design framework to analyze past successes in computing, understand current problems and identify solutions moving forward. Engineering systems with these emerging devices might require the modification of both the type of descriptions of learning that we will design for, and the design methodologies we employ in order to realize these new descriptions. We will explore ideas from complexity engineering and analyze the advantages and challenges they offer over traditional approaches to neuromorphic design with novel computing fabrics. A reservoir computing example is used to understand the specific changes that would accompany in moving towards a complexity engineering approach. The time is ideal for a significant reboot of our design methodologies and success will represent a radical shift in how neuromorphic hardware is designed and pave the way for a new paradigm.
△ Less
Submitted 22 September, 2020; v1 submitted 22 April, 2020;
originally announced May 2020.
-
Thermodynamic Computing
Authors:
Tom Conte,
Erik DeBenedictis,
Natesh Ganesh,
Todd Hylton,
John Paul Strachan,
R. Stanley Williams,
Alexander Alemi,
Lee Altenberg,
Gavin Crooks,
James Crutchfield,
Lidia del Rio,
Josh Deutsch,
Michael DeWeese,
Khari Douglas,
Massimiliano Esposito,
Michael Frank,
Robert Fry,
Peter Harsha,
Mark Hill,
Christopher Kello,
Jeff Krichmar,
Suhas Kumar,
Shih-Chii Liu,
Seth Lloyd,
Matteo Marsili
, et al. (14 additional authors not shown)
Abstract:
The hardware and software foundations laid in the first half of the 20th Century enabled the computing technologies that have transformed the world, but these foundations are now under siege. The current computing paradigm, which is the foundation of much of the current standards of living that we now enjoy, faces fundamental limitations that are evident from several perspectives. In terms of hard…
▽ More
The hardware and software foundations laid in the first half of the 20th Century enabled the computing technologies that have transformed the world, but these foundations are now under siege. The current computing paradigm, which is the foundation of much of the current standards of living that we now enjoy, faces fundamental limitations that are evident from several perspectives. In terms of hardware, devices have become so small that we are struggling to eliminate the effects of thermodynamic fluctuations, which are unavoidable at the nanometer scale. In terms of software, our ability to imagine and program effective computational abstractions and implementations are clearly challenged in complex domains. In terms of systems, currently five percent of the power generated in the US is used to run computing systems - this astonishing figure is neither ecologically sustainable nor economically scalable. Economically, the cost of building next-generation semiconductor fabrication plants has soared past $10 billion. All of these difficulties - device scaling, software complexity, adaptability, energy consumption, and fabrication economics - indicate that the current computing paradigm has matured and that continued improvements along this path will be limited. If technological progress is to continue and corresponding social and economic benefits are to continue to accrue, computing must become much more capable, energy efficient, and affordable. We propose that progress in computing can continue under a united, physically grounded, computational paradigm centered on thermodynamics. Herein we propose a research agenda to extend these thermodynamic foundations into complex, non-equilibrium, self-organizing systems and apply them holistically to future computing systems that will harness nature's innate computational capacity. We call this type of computing "Thermodynamic Computing" or TC.
△ Less
Submitted 14 November, 2019; v1 submitted 5 November, 2019;
originally announced November 2019.
