-
Unmemorization in Large Language Models via Self-Distillation and Deliberate Imagination
Authors:
Yijiang River Dong,
Hongzhou Lin,
Mikhail Belkin,
Ramon Huerta,
Ivan Vulić
Abstract:
While displaying impressive generation capabilities across many tasks, Large Language Models (LLMs) still struggle with crucial issues of privacy violation and unwanted exposure of sensitive data. This raises an essential question: how should we prevent such undesired behavior of LLMs while maintaining their strong generation and natural language understanding (NLU) capabilities? In this work, we…
▽ More
While displaying impressive generation capabilities across many tasks, Large Language Models (LLMs) still struggle with crucial issues of privacy violation and unwanted exposure of sensitive data. This raises an essential question: how should we prevent such undesired behavior of LLMs while maintaining their strong generation and natural language understanding (NLU) capabilities? In this work, we introduce a novel approach termed deliberate imagination in the context of LLM unlearning. Instead of trying to forget memorized data, we employ a self-distillation framework, guiding LLMs to deliberately imagine alternative scenarios. As demonstrated in a wide range of experiments, the proposed method not only effectively unlearns targeted text but also preserves the LLMs' capabilities in open-ended generation tasks as well as in NLU tasks. Our results demonstrate the usefulness of this approach across different models and sizes, and also with parameter-efficient fine-tuning, offering a novel pathway to addressing the challenges with private and sensitive data in LLM applications.
△ Less
Submitted 15 February, 2024;
originally announced February 2024.
-
Analyzing and Improving Hardware Modeling of Accel-Sim
Authors:
Rodrigo Huerta,
Mojtaba Abaie Shoushtary,
Antonio González
Abstract:
GPU architectures have become popular for executing general-purpose programs. Their many-core architecture supports a large number of threads that run concurrently to hide the latency among dependent instructions. In modern GPU architectures, each SM/core is typically composed of several sub-cores, where each sub-core has its own independent pipeline.
Simulators are a key tool for investigating…
▽ More
GPU architectures have become popular for executing general-purpose programs. Their many-core architecture supports a large number of threads that run concurrently to hide the latency among dependent instructions. In modern GPU architectures, each SM/core is typically composed of several sub-cores, where each sub-core has its own independent pipeline.
Simulators are a key tool for investigating novel concepts in computer architecture. They must be performance-accurate and have a proper model related to the target hardware to explore the different bottlenecks properly.
This paper presents a wide analysis of different parts of Accel-sim, a popular GPGPU simulator, and some improvements of its model. First, we focus on the front-end and developed a more realistic model. Then, we analyze the way the result bus works and develop a more realistic one. Next, we describe the current memory pipeline model and propose a model for a more cost-effective design. Finally, we discuss other areas of improvement of the simulator.
△ Less
Submitted 18 January, 2024;
originally announced January 2024.
-
Hedonic Prices and Quality Adjusted Price Indices Powered by AI
Authors:
Patrick Bajari,
Zhihao Cen,
Victor Chernozhukov,
Manoj Manukonda,
Suhas Vijaykumar,
Jin Wang,
Ramon Huerta,
Junbo Li,
Ling Leng,
George Monokroussos,
Shan Wan
Abstract:
Accurate, real-time measurements of price index changes using electronic records are essential for tracking inflation and productivity in today's economic environment. We develop empirical hedonic models that can process large amounts of unstructured product data (text, images, prices, quantities) and output accurate hedonic price estimates and derived indices. To accomplish this, we generate abst…
▽ More
Accurate, real-time measurements of price index changes using electronic records are essential for tracking inflation and productivity in today's economic environment. We develop empirical hedonic models that can process large amounts of unstructured product data (text, images, prices, quantities) and output accurate hedonic price estimates and derived indices. To accomplish this, we generate abstract product attributes, or ``features,'' from text descriptions and images using deep neural networks, and then use these attributes to estimate the hedonic price function. Specifically, we convert textual information about the product to numeric features using large language models based on transformers, trained or fine-tuned using product descriptions, and convert the product image to numeric features using a residual network model. To produce the estimated hedonic price function, we again use a multi-task neural network trained to predict a product's price in all time periods simultaneously. To demonstrate the performance of this approach, we apply the models to Amazon's data for first-party apparel sales and estimate hedonic prices. The resulting models have high predictive accuracy, with $R^2$ ranging from $80\%$ to $90\%$. Finally, we construct the AI-based hedonic Fisher price index, chained at the year-over-year frequency. We contrast the index with the CPI and other electronic indices.
△ Less
Submitted 28 April, 2023;
originally announced May 2023.
-
Online Decorrelation of Humidity and Temperature in Chemical Sensors for Continuous Monitoring
Authors:
Ramon Huerta,
Thiago S. Mosqueiro,
Jordi Fonollosa,
Nikolai F Rulkov,
Irene Rodriguez-Lujan
Abstract:
A method for online decorrelation of chemical sensor signals from the effects of environmental humidity and temperature variations is proposed. The goal is to improve the accuracy of electronic nose measurements for continuous monitoring by processing data from simultaneous readings of environmental humidity and temperature. The electronic nose setup built for this study included eight metal-oxide…
▽ More
A method for online decorrelation of chemical sensor signals from the effects of environmental humidity and temperature variations is proposed. The goal is to improve the accuracy of electronic nose measurements for continuous monitoring by processing data from simultaneous readings of environmental humidity and temperature. The electronic nose setup built for this study included eight metal-oxide sensors, temperature and humidity sensors with a wireless communication link to external computer. This wireless electronic nose was used to monitor air for two years in the residence of one of the authors and it collected data continuously during 537 days with a sampling rate of 1 samples per second. To estimate the effects of variations in air humidity and temperature on the chemical sensors signals, we used a standard energy band model for an n-type metal-oxide (MOX) gas sensor. The main assumption of the model is that variations in sensor conductivity can be expressed as a nonlinear function of changes in the semiconductor energy bands in the presence of external humidity and temperature variations. Fitting this model to the collected data, we confirmed that the most statistically significant factors are humidity changes and correlated changes of temperature and humidity. This simple model achieves excellent accuracy with a coefficient of determination $R^2$ close to 1. To show how the humidity-temperature correction model works for gas discrimination, we constructed a model for online discrimination among banana, wine and baseline response. This shows that pattern recognition algorithms improve performance and reliability by including the filtered signal of the chemical sensors.
△ Less
Submitted 7 August, 2016; v1 submitted 4 August, 2016;
originally announced August 2016.
