Showing 1–50 of 77 results for author: Iyer, S

QMViT: A Mushroom is worth 16x16 Words

Authors: Siddhant Dutta, Hemant Singh, Kalpita Shankhdhar, Sridhar Iyer

Abstract: Consuming poisonous mushrooms can have severe health consequences, even resulting in fatality and accurately distinguishing edible from toxic mushroom varieties remains a significant challenge in ensuring food safety. So, it's crucial to distinguish between edible and poisonous mushrooms within the existing species. This is essential due to the significant demand for mushrooms in people's daily me… ▽ More Consuming poisonous mushrooms can have severe health consequences, even resulting in fatality and accurately distinguishing edible from toxic mushroom varieties remains a significant challenge in ensuring food safety. So, it's crucial to distinguish between edible and poisonous mushrooms within the existing species. This is essential due to the significant demand for mushrooms in people's daily meals and their potential contributions to medical science. This work presents a novel Quantum Vision Transformer architecture that leverages quantum computing to enhance mushroom classification performance. By implementing specialized quantum self-attention mechanisms using Variational Quantum Circuits, the proposed architecture achieved 92.33% and 99.24% accuracy based on their category and their edibility respectively. This demonstrates the success of the proposed architecture in reducing false negatives for toxic mushrooms, thus ensuring food safety. Our research highlights the potential of QMViT for improving mushroom classification as a whole. △ Less

Submitted 10 May, 2024; originally announced July 2024.

An XOR Lemma for Deterministic Communication Complexity

Authors: Siddharth Iyer, Anup Rao

Abstract: We prove a lower bound on the communication complexity of computing the $n$-fold xor of an arbitrary function $f$, in terms of the communication complexity and rank of $f$. We prove that $D(f^{\oplus n}) \geq n \cdot \Big(\frac{Ω(D(f))}{\log \mathsf{rk}(f)} -\log \mathsf{rk}(f)\Big )$, where here $D(f), D(f^{\oplus n})$ represent the deterministic communication complexity, and $\mathsf{rk}(f)$ is… ▽ More We prove a lower bound on the communication complexity of computing the $n$-fold xor of an arbitrary function $f$, in terms of the communication complexity and rank of $f$. We prove that $D(f^{\oplus n}) \geq n \cdot \Big(\frac{Ω(D(f))}{\log \mathsf{rk}(f)} -\log \mathsf{rk}(f)\Big )$, where here $D(f), D(f^{\oplus n})$ represent the deterministic communication complexity, and $\mathsf{rk}(f)$ is the rank of $f$. Our methods involve a new way to use information theory to reason about deterministic communication complexity. △ Less

Submitted 1 July, 2024; originally announced July 2024.

Comprehensive Lipidomic Automation Workflow using Large Language Models

Authors: Connor Beveridge, Sanjay Iyer, Caitlin E. Randolph, Matthew Muhoberac, Palak Manchanda, Amy C. Clingenpeel, Shane Tichy, Gaurav Chopra

Abstract: Lipidomics generates large data that makes manual annotation and interpretation challenging. Lipid chemical and structural diversity with structural isomers further complicates annotation. Although, several commercial and open-source software for targeted lipid identification exists, it lacks automated method generation workflows and integration with statistical and bioinformatics tools. We have d… ▽ More Lipidomics generates large data that makes manual annotation and interpretation challenging. Lipid chemical and structural diversity with structural isomers further complicates annotation. Although, several commercial and open-source software for targeted lipid identification exists, it lacks automated method generation workflows and integration with statistical and bioinformatics tools. We have developed the Comprehensive Lipidomic Automated Workflow (CLAW) platform with integrated workflow for parsing, detailed statistical analysis and lipid annotations based on custom multiple reaction monitoring (MRM) precursor and product ion pair transitions. CLAW contains several modules including identification of carbon-carbon double bond position(s) in unsaturated lipids when combined with ozone electrospray ionization (OzESI)-MRM methodology. To demonstrate the utility of the automated workflow in CLAW, large-scale lipidomics data was collected with traditional and OzESI-MRM profiling on biological and non-biological samples. Specifically, a total of 1497 transitions organized into 10 MRM-based mass spectrometry methods were used to profile lipid droplets isolated from different brain regions of 18-24 month-old Alzheimer's disease mice and age-matched wild-type controls. Additionally, triacyclglycerols (TGs) profiles with carbon-carbon double bond specificity were generated from canola oil samples using OzESI-MRM profiling. We also developed an integrated language user interface with large language models using artificially intelligent (AI) agents that permits users to interact with the CLAW platform using a chatbot terminal to perform statistical and bioinformatic analyses. We envision CLAW pipeline to be used in high-throughput lipid structural identification tasks aiding users to generate automated lipidomics workflows ranging from data acquisition to AI agent-based bioinformatic analysis. △ Less

Submitted 22 March, 2024; originally announced March 2024.

Comments: 53 pages, 4 main figures, 23 Supporting figures, 10 Supporting Tables

Real-Time Multimodal Cognitive Assistant for Emergency Medical Services

Authors: Keshara Weerasinghe, Saahith Janapati, Xueren Ge, Sion Kim, Sneha Iyer, John A. Stankovic, Homa Alemzadeh

Abstract: Emergency Medical Services (EMS) responders often operate under time-sensitive conditions, facing cognitive overload and inherent risks, requiring essential skills in critical thinking and rapid decision-making. This paper presents CognitiveEMS, an end-to-end wearable cognitive assistant system that can act as a collaborative virtual partner engaging in the real-time acquisition and analysis of mu… ▽ More Emergency Medical Services (EMS) responders often operate under time-sensitive conditions, facing cognitive overload and inherent risks, requiring essential skills in critical thinking and rapid decision-making. This paper presents CognitiveEMS, an end-to-end wearable cognitive assistant system that can act as a collaborative virtual partner engaging in the real-time acquisition and analysis of multimodal data from an emergency scene and interacting with EMS responders through Augmented Reality (AR) smart glasses. CognitiveEMS processes the continuous streams of data in real-time and leverages edge computing to provide assistance in EMS protocol selection and intervention recognition. We address key technical challenges in real-time cognitive assistance by introducing three novel components: (i) a Speech Recognition model that is fine-tuned for real-world medical emergency conversations using simulated EMS audio recordings, augmented with synthetic data generated by large language models (LLMs); (ii) an EMS Protocol Prediction model that combines state-of-the-art (SOTA) tiny language models with EMS domain knowledge using graph-based attention mechanisms; (iii) an EMS Action Recognition module which leverages multimodal audio and video data and protocol predictions to infer the intervention/treatment actions taken by the responders at the incident scene. Our results show that for speech recognition we achieve superior performance compared to SOTA (WER of 0.290 vs. 0.618) on conversational data. Our protocol prediction component also significantly outperforms SOTA (top-3 accuracy of 0.800 vs. 0.200) and the action recognition achieves an accuracy of 0.727, while maintaining an end-to-end latency of 3.78s for protocol prediction on the edge and 0.31s on the server. △ Less

Submitted 11 March, 2024; originally announced March 2024.

Comments: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible

Instruction-tuned Language Models are Better Knowledge Learners

Authors: Zhengbao Jiang, Zhiqing Sun, Weijia Shi, Pedro Rodriguez, Chunting Zhou, Graham Neubig, Xi Victoria Lin, Wen-tau Yih, Srinivasan Iyer

Abstract: In order for large language model (LLM)-based assistants to effectively adapt to evolving information needs, it must be possible to update their factual knowledge through continued training on new data. The standard recipe for doing so involves continued pre-training on new documents followed by instruction-tuning on question-answer (QA) pairs. However, we find that LLMs trained with this recipe s… ▽ More In order for large language model (LLM)-based assistants to effectively adapt to evolving information needs, it must be possible to update their factual knowledge through continued training on new data. The standard recipe for doing so involves continued pre-training on new documents followed by instruction-tuning on question-answer (QA) pairs. However, we find that LLMs trained with this recipe struggle to answer questions, even though the perplexity of documents is minimized. We found that QA pairs are generally straightforward, while documents are more complex, weaving many factual statements together in an intricate manner. Therefore, we hypothesize that it is beneficial to expose LLMs to QA pairs before continued pre-training on documents so that the process of encoding knowledge from complex documents takes into account how this knowledge is accessed through questions. Based on this, we propose pre-instruction-tuning (PIT), a method that instruction-tunes on questions prior to training on documents. This contrasts with standard instruction-tuning, which learns how to extract knowledge after training on documents. Extensive experiments and ablation studies demonstrate that pre-instruction-tuning significantly enhances the ability of LLMs to absorb knowledge from new documents, outperforming standard instruction-tuning by 17.8%. △ Less

Submitted 25 May, 2024; v1 submitted 20 February, 2024; originally announced February 2024.

Comments: ACL 2024. The reproduced data for this paper is available at https://github.com/Edward-Sun/PIT

Knowledge Graph Enhanced Aspect-Level Sentiment Analysis

Authors: Kavita Sharma, Ritu Patel, Sunita Iyer

Abstract: In this paper, we propose a novel method to enhance sentiment analysis by addressing the challenge of context-specific word meanings. It combines the advantages of a BERT model with a knowledge graph based synonym data. This synergy leverages a dynamic attention mechanism to develop a knowledge-driven state vector. For classifying sentiments linked to specific aspects, the approach constructs a me… ▽ More In this paper, we propose a novel method to enhance sentiment analysis by addressing the challenge of context-specific word meanings. It combines the advantages of a BERT model with a knowledge graph based synonym data. This synergy leverages a dynamic attention mechanism to develop a knowledge-driven state vector. For classifying sentiments linked to specific aspects, the approach constructs a memory bank integrating positional data. The data are then analyzed using a DCGRU to pinpoint sentiment characteristics related to specific aspect terms. Experiments on three widely used datasets demonstrate the superior performance of our method in sentiment classification. △ Less

Submitted 26 January, 2024; v1 submitted 1 December, 2023; originally announced December 2023.

Household navigation and manipulation for everyday object rearrangement tasks

Authors: Shrutheesh R. Iyer, Anwesan Pal, Jiaming Hu, Akanimoh Adeleye, Aditya Aggarwal, Henrik I. Christensen

Abstract: We consider the problem of building an assistive robotic system that can help humans in daily household cleanup tasks. Creating such an autonomous system in real-world environments is inherently quite challenging, as a general solution may not suit the preferences of a particular customer. Moreover, such a system consists of multi-objective tasks comprising -- (i) Detection of misplaced objects an… ▽ More We consider the problem of building an assistive robotic system that can help humans in daily household cleanup tasks. Creating such an autonomous system in real-world environments is inherently quite challenging, as a general solution may not suit the preferences of a particular customer. Moreover, such a system consists of multi-objective tasks comprising -- (i) Detection of misplaced objects and prediction of their potentially correct placements, (ii) Fine-grained manipulation for stable object grasping, and (iii) Room-to-room navigation for transferring objects in unseen environments. This work systematically tackles each component and integrates them into a complete object rearrangement pipeline. To validate our proposed system, we conduct multiple experiments on a real robotic platform involving multi-room object transfer, user preference-based placement, and complex pick-and-place tasks. Project page: https://sites.google.com/eng.ucsd.edu/home-robot △ Less

Submitted 11 December, 2023; originally announced December 2023.

Comments: Paper accepted at IEEE IRC-2023

XOR Lemmas for Communication via Marginal Information

Authors: Siddharth Iyer, Anup Rao

Abstract: We define the $\textit{marginal information}$ of a communication protocol, and use it to prove XOR lemmas for communication complexity. We show that if every $C$-bit protocol has bounded advantage for computing a Boolean function $f$, then every $\tilde Ω(C \sqrt{n})$-bit protocol has advantage $\exp(-Ω(n))$ for computing the $n$-fold xor $f^{\oplus n}$. We prove exponentially small bounds in the… ▽ More We define the $\textit{marginal information}$ of a communication protocol, and use it to prove XOR lemmas for communication complexity. We show that if every $C$-bit protocol has bounded advantage for computing a Boolean function $f$, then every $\tilde Ω(C \sqrt{n})$-bit protocol has advantage $\exp(-Ω(n))$ for computing the $n$-fold xor $f^{\oplus n}$. We prove exponentially small bounds in the average case setting, and near optimal bounds for product distributions and for bounded-round protocols. △ Less

Submitted 2 July, 2024; v1 submitted 5 December, 2023; originally announced December 2023.

Comments: Fixed typos

SplatArmor: Articulated Gaussian splatting for animatable humans from monocular RGB videos

Authors: Rohit Jena, Ganesh Subramanian Iyer, Siddharth Choudhary, Brandon Smith, Pratik Chaudhari, James Gee

Abstract: We propose SplatArmor, a novel approach for recovering detailed and animatable human models by `armoring' a parameterized body model with 3D Gaussians. Our approach represents the human as a set of 3D Gaussians within a canonical space, whose articulation is defined by extending the skinning of the underlying SMPL geometry to arbitrary locations in the canonical space. To account for pose-dependen… ▽ More We propose SplatArmor, a novel approach for recovering detailed and animatable human models by `armoring' a parameterized body model with 3D Gaussians. Our approach represents the human as a set of 3D Gaussians within a canonical space, whose articulation is defined by extending the skinning of the underlying SMPL geometry to arbitrary locations in the canonical space. To account for pose-dependent effects, we introduce a SE(3) field, which allows us to capture both the location and anisotropy of the Gaussians. Furthermore, we propose the use of a neural color field to provide color regularization and 3D supervision for the precise positioning of these Gaussians. We show that Gaussian splatting provides an interesting alternative to neural rendering based methods by leverging a rasterization primitive without facing any of the non-differentiability and optimization challenges typically faced in such approaches. The rasterization paradigms allows us to leverage forward skinning, and does not suffer from the ambiguities associated with inverse skinning and warping. We show compelling results on the ZJU MoCap and People Snapshot datasets, which underscore the effectiveness of our method for controllable human synthesis. △ Less

Submitted 17 November, 2023; originally announced November 2023.

An Experience-based TAMP Framework for Foliated Manifolds

Authors: Jiaming Hu, Shrutheesh R. Iyer, Henrik I. Christensen

Abstract: Due to their complexity, foliated structure problems often pose intricate challenges to task and motion planning in robotics manipulation. To counter this, our study presents the ``Foliated Repetition Roadmap.'' This roadmap assists task and motion planners by transforming the complex foliated structure problem into a more accessible graph format. By leveraging query experiences from different fol… ▽ More Due to their complexity, foliated structure problems often pose intricate challenges to task and motion planning in robotics manipulation. To counter this, our study presents the ``Foliated Repetition Roadmap.'' This roadmap assists task and motion planners by transforming the complex foliated structure problem into a more accessible graph format. By leveraging query experiences from different foliated manifolds, our framework can dynamically and efficiently update this graph. The refined graph can generate distribution sets, optimizing motion planning performance in foliated structure problems. In our paper, we lay down the theoretical groundwork and illustrate its practical applications through real-world examples. △ Less

Submitted 12 October, 2023; originally announced October 2023.

Attention and Pooling based Sigmoid Colon Segmentation in 3D CT images

Authors: Md Akizur Rahman, Sonit Singh, Kuruparan Shanmugalingam, Sankaran Iyer, Alan Blair, Praveen Ravindran, Arcot Sowmya

Abstract: Segmentation of the sigmoid colon is a crucial aspect of treating diverticulitis. It enables accurate identification and localisation of inflammation, which in turn helps healthcare professionals make informed decisions about the most appropriate treatment options. This research presents a novel deep learning architecture for segmenting the sigmoid colon from Computed Tomography (CT) images using… ▽ More Segmentation of the sigmoid colon is a crucial aspect of treating diverticulitis. It enables accurate identification and localisation of inflammation, which in turn helps healthcare professionals make informed decisions about the most appropriate treatment options. This research presents a novel deep learning architecture for segmenting the sigmoid colon from Computed Tomography (CT) images using a modified 3D U-Net architecture. Several variations of the 3D U-Net model with modified hyper-parameters were examined in this study. Pyramid pooling (PyP) and channel-spatial Squeeze and Excitation (csSE) were also used to improve the model performance. The networks were trained using manually annotated sigmoid colon. A five-fold cross-validation procedure was used on a test dataset to evaluate the network's performance. As indicated by the maximum Dice similarity coefficient (DSC) of 56.92+/-1.42%, the application of PyP and csSE techniques improves segmentation precision. We explored ensemble methods including averaging, weighted averaging, majority voting, and max ensemble. The results show that average and majority voting approaches with a threshold value of 0.5 and consistent weight distribution among the top three models produced comparable and optimal results with DSC of 88.11+/-3.52%. The results indicate that the application of a modified 3D U-Net architecture is effective for segmenting the sigmoid colon in Computed Tomography (CT) images. In addition, the study highlights the potential benefits of integrating ensemble methods to improve segmentation precision. △ Less

Submitted 25 September, 2023; originally announced September 2023.

Comments: 8 Pages, 6 figures, Accepted at IEEE DICTA 2023

Energy-Sustainable IoT Connectivity: Vision, Technological Enablers, Challenges, and Future Directions

Authors: Onel A. López, Osmel M. Rosabal, David Ruiz-Guirola, Prasoon Raghuwanshi, Konstantin Mikhaylov, Lauri Lovén, Sridhar Iyer

Abstract: Technology solutions must effectively balance economic growth, social equity, and environmental integrity to achieve a sustainable society. Notably, although the Internet of Things (IoT) paradigm constitutes a key sustainability enabler, critical issues such as the increasing maintenance operations, energy consumption, and manufacturing/disposal of IoT devices have long-term negative economic, soc… ▽ More Technology solutions must effectively balance economic growth, social equity, and environmental integrity to achieve a sustainable society. Notably, although the Internet of Things (IoT) paradigm constitutes a key sustainability enabler, critical issues such as the increasing maintenance operations, energy consumption, and manufacturing/disposal of IoT devices have long-term negative economic, societal, and environmental impacts and must be efficiently addressed. This calls for self-sustainable IoT ecosystems requiring minimal external resources and intervention, effectively utilizing renewable energy sources, and recycling materials whenever possible, thus encompassing energy sustainability. In this work, we focus on energy-sustainable IoT during the operation phase, although our discussions sometimes extend to other sustainability aspects and IoT lifecycle phases. Specifically, we provide a fresh look at energy-sustainable IoT and identify energy provision, transfer, and energy efficiency as the three main energy-related processes whose harmonious coexistence pushes toward realizing self-sustainable IoT systems. Their main related technologies, recent advances, challenges, and research directions are also discussed. Moreover, we overview relevant performance metrics to assess the energy-sustainability potential of a certain technique, technology, device, or network and list some target values for the next generation of wireless systems. Overall, this paper offers insights that are valuable for advancing sustainability goals for present and future generations. △ Less

Submitted 27 October, 2023; v1 submitted 4 June, 2023; originally announced June 2023.

Comments: 25 figures, 12 tables, submitted to IEEE Open Journal of the Communications Society

MSC Class: 94-02; 68-02

GAT-GAN : A Graph-Attention-based Time-Series Generative Adversarial Network

Authors: Srikrishna Iyer, Teng Teck Hou

Abstract: Generative Adversarial Networks (GANs) have proven to be a powerful tool for generating realistic synthetic data. However, traditional GANs often struggle to capture complex relationships between features which results in generation of unrealistic multivariate time-series data. In this paper, we propose a Graph-Attention-based Generative Adversarial Network (GAT-GAN) that explicitly includes two g… ▽ More Generative Adversarial Networks (GANs) have proven to be a powerful tool for generating realistic synthetic data. However, traditional GANs often struggle to capture complex relationships between features which results in generation of unrealistic multivariate time-series data. In this paper, we propose a Graph-Attention-based Generative Adversarial Network (GAT-GAN) that explicitly includes two graph-attention layers, one that learns temporal dependencies while the other captures spatial relationships. Unlike RNN-based GANs that struggle with modeling long sequences of data points, GAT-GAN generates long time-series data of high fidelity using an adversarially trained autoencoder architecture. Our empirical evaluations, using a variety of real-time-series datasets, show that our framework consistently outperforms state-of-the-art benchmarks based on \emph{Frechet Transformer distance} and \emph{Predictive score}, that characterizes (\emph{Fidelity, Diversity}) and \emph{predictive performance} respectively. Moreover, we introduce a Frechet Inception distance-like (FID) metric for time-series data called Frechet Transformer distance (FTD) score (lower is better), to evaluate the quality and variety of generated data. We also found that low FTD scores correspond to the best-performing downstream predictive experiments. Hence, FTD scores can be used as a standardized metric to evaluate synthetic time-series data. △ Less

Submitted 3 June, 2023; originally announced June 2023.

Comments: 9 pages, 1 figure, 3 tables, preprint under review

LIMA: Less Is More for Alignment

Authors: Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, Omer Levy

Abstract: Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences. We measure the relative importance of these two stages by training LIMA, a 65B parameter LLaMa language model fine-tuned with the standard supervis… ▽ More Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences. We measure the relative importance of these two stages by training LIMA, a 65B parameter LLaMa language model fine-tuned with the standard supervised loss on only 1,000 carefully curated prompts and responses, without any reinforcement learning or human preference modeling. LIMA demonstrates remarkably strong performance, learning to follow specific response formats from only a handful of examples in the training data, including complex queries that range from planning trip itineraries to speculating about alternate history. Moreover, the model tends to generalize well to unseen tasks that did not appear in the training data. In a controlled human study, responses from LIMA are either equivalent or strictly preferred to GPT-4 in 43% of cases; this statistic is as high as 58% when compared to Bard and 65% versus DaVinci003, which was trained with human feedback. Taken together, these results strongly suggest that almost all knowledge in large language models is learned during pretraining, and only limited instruction tuning data is necessary to teach models to produce high quality output. △ Less

Submitted 18 May, 2023; originally announced May 2023.

TinyML: Tools, Applications, Challenges, and Future Research Directions

Authors: Rakhee Kallimani, Krishna Pai, Prasoon Raghuwanshi, Sridhar Iyer, Onel L. A. López

Abstract: In recent years, Artificial Intelligence (AI) and Machine learning (ML) have gained significant interest from both, industry and academia. Notably, conventional ML techniques require enormous amounts of power to meet the desired accuracy, which has limited their use mainly to high-capability devices such as network nodes. However, with many advancements in technologies such as the Internet of Thin… ▽ More In recent years, Artificial Intelligence (AI) and Machine learning (ML) have gained significant interest from both, industry and academia. Notably, conventional ML techniques require enormous amounts of power to meet the desired accuracy, which has limited their use mainly to high-capability devices such as network nodes. However, with many advancements in technologies such as the Internet of Things (IoT) and edge computing, it is desirable to incorporate ML techniques into resource-constrained embedded devices for distributed and ubiquitous intelligence. This has motivated the emergence of the TinyML paradigm which is an embedded ML technique that enables ML applications on multiple cheap, resource- and power-constrained devices. However, during this transition towards appropriate implementation of the TinyML technology, multiple challenges such as processing capacity optimization, improved reliability, and maintenance of learning models' accuracy require timely solutions. In this article, various avenues available for TinyML implementation are reviewed. Firstly, a background of TinyML is provided, followed by detailed discussions on various tools supporting TinyML. Then, state-of-art applications of TinyML using advanced technologies are detailed. Lastly, various research challenges and future directions are identified. △ Less

Submitted 23 March, 2023; originally announced March 2023.

Comments: 12 pags, 3 tables, 4 figures

Journal ref: Multimedia Tools and Applications, 2023

LEVER: Learning to Verify Language-to-Code Generation with Execution

Authors: Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen-tau Yih, Sida I. Wang, Xi Victoria Lin

Abstract: The advent of large language models trained on code (code LLMs) has led to significant progress in language-to-code generation. State-of-the-art approaches in this area combine LLM decoding with sample pruning and reranking using test cases or heuristics based on the execution results. However, it is challenging to obtain test cases for many real-world language-to-code applications, and heuristics… ▽ More The advent of large language models trained on code (code LLMs) has led to significant progress in language-to-code generation. State-of-the-art approaches in this area combine LLM decoding with sample pruning and reranking using test cases or heuristics based on the execution results. However, it is challenging to obtain test cases for many real-world language-to-code applications, and heuristics cannot well capture the semantic features of the execution results, such as data type and value range, which often indicates the correctness of the program. In this work, we propose LEVER, a simple approach to improve language-to-code generation by learning to verify the generated programs with their execution results. Specifically, we train verifiers to determine whether a program sampled from the LLMs is correct or not based on the natural language input, the program itself and its execution results. The sampled programs are reranked by combining the verification score with the LLM generation probability, and marginalizing over programs with the same execution results. On four datasets across the domains of table QA, math QA and basic Python programming, LEVER consistently improves over the base code LLMs(4.6% to 10.9% with code-davinci-002) and achieves new state-of-the-art results on all of them. △ Less

Submitted 1 September, 2023; v1 submitted 16 February, 2023; originally announced February 2023.

Comments: ICML'23; code available at https://github.com/niansong1996/lever

OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization

Authors: Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O'Horo, Gabriel Pereyra, Jeff Wang, Christopher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, Ves Stoyanov

Abstract: Recent work has shown that fine-tuning large pre-trained language models on a collection of tasks described via instructions, a.k.a. instruction-tuning, improves their zero and few-shot generalization to unseen tasks. However, there is a limited understanding of the performance trade-offs of different decisions made during the instruction-tuning process. These decisions include the scale and diver… ▽ More Recent work has shown that fine-tuning large pre-trained language models on a collection of tasks described via instructions, a.k.a. instruction-tuning, improves their zero and few-shot generalization to unseen tasks. However, there is a limited understanding of the performance trade-offs of different decisions made during the instruction-tuning process. These decisions include the scale and diversity of the instruction-tuning benchmark, different task sampling strategies, fine-tuning with and without demonstrations, training using specialized datasets for reasoning and dialogue, and finally, the fine-tuning objectives themselves. In this paper, we characterize the effect of instruction-tuning decisions on downstream task performance when scaling both model and benchmark sizes. To this end, we create OPT-IML Bench: a large benchmark for Instruction Meta-Learning (IML) of 2000 NLP tasks consolidated into task categories from 8 existing benchmarks, and prepare an evaluation framework to measure three types of model generalizations: to tasks from fully held-out categories, to held-out tasks from seen categories, and to held-out instances from seen tasks. Through the lens of this framework, we first present insights about instruction-tuning decisions as applied to OPT-30B and further exploit these insights to train OPT-IML 30B and 175B, which are instruction-tuned versions of OPT. OPT-IML demonstrates all three generalization abilities at both scales on four different evaluation benchmarks with diverse tasks and input formats -- PromptSource, FLAN, Super-NaturalInstructions, and UnifiedSKG. Not only does it significantly outperform OPT on all benchmarks but is also highly competitive with existing models fine-tuned on each specific benchmark. We release OPT-IML at both scales, together with the OPT-IML Bench evaluation framework. △ Less

Submitted 30 January, 2023; v1 submitted 22 December, 2022; originally announced December 2022.

Comments: 56 pages. v2->v3: fix OPT-30B evaluation results across benchmarks (previously we reported lower performance of this model due to an evaluation pipeline bug)

Demystifying Prompts in Language Models via Perplexity Estimation

Authors: Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith, Luke Zettlemoyer

Abstract: Language models can be prompted to perform a wide variety of zero- and few-shot learning problems. However, performance varies significantly with the choice of prompt, and we do not yet understand why this happens or how to pick the best prompts. In this work, we analyze the factors that contribute to this variance and establish a new empirical hypothesis: the performance of a prompt is coupled wi… ▽ More Language models can be prompted to perform a wide variety of zero- and few-shot learning problems. However, performance varies significantly with the choice of prompt, and we do not yet understand why this happens or how to pick the best prompts. In this work, we analyze the factors that contribute to this variance and establish a new empirical hypothesis: the performance of a prompt is coupled with the extent to which the model is familiar with the language it contains. Over a wide range of tasks, we show that the lower the perplexity of the prompt is, the better the prompt is able to perform the task. As a result, we devise a method for creating prompts: (1) automatically extend a small seed set of manually written prompts by paraphrasing using GPT3 and backtranslation and (2) choose the lowest perplexity prompts to get significant gains in performance. △ Less

Submitted 7 December, 2022; originally announced December 2022.

Complementary Explanations for Effective In-Context Learning

Authors: Xi Ye, Srinivasan Iyer, Asli Celikyilmaz, Ves Stoyanov, Greg Durrett, Ramakanth Pasunuru

Abstract: Large language models (LLMs) have exhibited remarkable capabilities in learning from explanations in prompts, but there has been limited understanding of exactly how these explanations function or why they are effective. This work aims to better understand the mechanisms by which explanations are used for in-context learning. We first study the impact of two different factors on the performance of… ▽ More Large language models (LLMs) have exhibited remarkable capabilities in learning from explanations in prompts, but there has been limited understanding of exactly how these explanations function or why they are effective. This work aims to better understand the mechanisms by which explanations are used for in-context learning. We first study the impact of two different factors on the performance of prompts with explanations: the computation trace (the way the solution is decomposed) and the natural language used to express the prompt. By perturbing explanations on three controlled tasks, we show that both factors contribute to the effectiveness of explanations. We further study how to form maximally effective sets of explanations for solving a given test query. We find that LLMs can benefit from the complementarity of the explanation set: diverse reasoning skills shown by different exemplars can lead to better performance. Therefore, we propose a maximal marginal relevance-based exemplar selection approach for constructing exemplar sets that are both relevant as well as complementary, which successfully improves the in-context learning performance across three real-world tasks on multiple LLMs. △ Less

Submitted 12 June, 2023; v1 submitted 24 November, 2022; originally announced November 2022.

Comments: ACL Findings 2023 Camera-Ready

A Comprehensive Survey on Spectrum Sharing Techniques for 5G/B5G Intelligent Wireless Networks: Opportunities, Challenges and Future Research Directions

Authors: Anita Patil, Sridhar Iyer, Onel L. A. Lopez, Rahul J Pandya, Krishna Pai, Anshuman Kalla, Rakhee Kallimani

Abstract: The increasing popularity of Internet of Everything and small-cell devices has enormously accelerated traffic loads. Consequently, increased bandwidth and high data rate requirements stimulate the operation at the millimeter wave and the Tera-Hertz spectrum bands in the fifth generation (5G) and beyond 5G (B5G) wireless networks. Furthermore, efficient spectrum allocation, maximizing the spectrum… ▽ More The increasing popularity of Internet of Everything and small-cell devices has enormously accelerated traffic loads. Consequently, increased bandwidth and high data rate requirements stimulate the operation at the millimeter wave and the Tera-Hertz spectrum bands in the fifth generation (5G) and beyond 5G (B5G) wireless networks. Furthermore, efficient spectrum allocation, maximizing the spectrum utilization, achieving efficient spectrum sharing (SS), and managing the spectrum to enhance the system performance remain challenging. To this end, recent studies have implemented artificial intelligence and machine learning techniques, enabling intelligent and efficient spectrum leveraging. However, despite many recent research advances focused on maximizing utilization of the spectrum bands, achieving efficient sharing, allocation, and management of the enormous available spectrum remains challenging. Therefore, the current article acquaints a comprehensive survey on intelligent SS methodologies for 5G and B5G wireless networks, considering the applications of artificial intelligence for efficient SS. Specifically, a thorough overview of SS methodologies is conferred, following which the various spectrum utilization opportunities arising from the existing SS methodologies in intelligent wireless networks are discussed. Subsequently, to highlight critical limitations of the existing methodologies, recent literature on existing SS methodologies is reviewed in detail, classifying them based on the implemented technology, i.e., cognitive radio, machine learning, blockchain, and multiple other techniques. Moreover, the related SS techniques are reviewed to highlight significant challenges in the B5G intelligent wireless network. Finally, to provide an insight into the prospective research avenues, the article is concluded by presenting several potential research directions and proposed solutions. △ Less

Submitted 17 November, 2022; v1 submitted 16 November, 2022; originally announced November 2022.

The Friendship Paradox and Social Network Participation

Authors: Ahmed Medhat, Shankar Iyer

Abstract: The friendship paradox implies that a person will, on average, have fewer friends than their friends do. Prior work has shown how the friendship paradox can lead to perception biases regarding behaviors that correlate with the number of friends: for example, people tend to perceive their friends as being more socially engaged than they are. Here, we investigate the consequences of this type of soc… ▽ More The friendship paradox implies that a person will, on average, have fewer friends than their friends do. Prior work has shown how the friendship paradox can lead to perception biases regarding behaviors that correlate with the number of friends: for example, people tend to perceive their friends as being more socially engaged than they are. Here, we investigate the consequences of this type of social comparison in the conceptual setting of content creation ("sharing") in an online social network. Suppose people compare the amount of feedback that their content receives to the amount of feedback that their friends' content receives, and suppose they modify their sharing behavior as a result of that comparison. How does that impact overall sharing on the social network over time? We run simulations over model-generated synthetic networks, assuming initially uniform sharing and feedback rates. Thus, people's initial modifications of their sharing behavior in response to social comparisons are entirely driven by the friendship paradox. These modifications induce inhomogeneities in sharing rates that can further alter perception biases. If people's responses to social comparisons are monotonic (i.e., the larger the disparity, the larger the modification in sharing behavior), our simulations suggest that overall sharing in the network gradually declines. Meanwhile, convex responses can sustain or grow overall sharing in the network. We focus entirely on synthetic graphs in the present work and have not yet extended our simulations to real-world network topologies. Nevertheless, we do discuss practical implications, such as how interventions can be tailored to sustain long-term sharing, even in the presence of adverse social-comparison effects. △ Less

Submitted 9 November, 2022; originally announced November 2022.

Comments: 10 pages, 15 figures

ZEUS: An Experimental Toolkit for Evaluating Congestion Control Algorithms in 5G Environments

Authors: Rohail Asim, Muhammad Khan, Luis Diez, Shiva Iyer, Ramon Aguero, Lakshmi Subramanian, Yasir Zaki

Abstract: As global cellular networks converge to 5G, one question lingers: Are we ready for the 5G challenge? A growing concern surrounds how well do existing congestion control algorithms perform in diverse 5G networks. Given that 5G networks are not yet widely deployed, assessing the performance of existing congestion control algorithms in realistic 5G settings presents several challenges. Moreover, exis… ▽ More As global cellular networks converge to 5G, one question lingers: Are we ready for the 5G challenge? A growing concern surrounds how well do existing congestion control algorithms perform in diverse 5G networks. Given that 5G networks are not yet widely deployed, assessing the performance of existing congestion control algorithms in realistic 5G settings presents several challenges. Moreover, existing network simulation and emulation environments are also not ideally suited to address the unique challenges of 5G network environments. Therefore, building a simple and easily accessible platform becomes crucial to allow testing and comparison of congestion control algorithms under different testing conditions. This paper makes two main contributions. First, we present Zeus, an open-source testbed that emulates 5G channels to evaluate congestion control algorithms in a repeatable and reproducible manner. Second, we assess and characterize ten of today's state-of-the-art congestion control algorithms under different 5G environments and show the difficulties of these solutions to achieve high performance under highly varying 5G channel conditions. In addition, we also utilize the recently proposed harm metric to characterize the detriment done by such algorithms to TCP Cubic cross traffic in 5G environments. △ Less

Submitted 30 August, 2022; originally announced August 2022.

Comments: 21 pages, 18 figures

Searching for Regularity in Bounded Functions

Authors: Siddharth Iyer, Michael Whitmeyer

Abstract: Given a function $f$ on $\mathbb{F}_2^n$, we study the following problem. What is the largest affine subspace $\mathcal{U}$ such that when restricted to $\mathcal{U}$, all the non-trivial Fourier coefficients of $f$ are very small? For the natural class of bounded Fourier degree $d$ functions $f:\mathbb{F}_2^n \to [-1,1]$, we show that there exists an affine subspace of dimension at least… ▽ More Given a function $f$ on $\mathbb{F}_2^n$, we study the following problem. What is the largest affine subspace $\mathcal{U}$ such that when restricted to $\mathcal{U}$, all the non-trivial Fourier coefficients of $f$ are very small? For the natural class of bounded Fourier degree $d$ functions $f:\mathbb{F}_2^n \to [-1,1]$, we show that there exists an affine subspace of dimension at least $ \tildeΩ(n^{1/d!}k^{-2})$, wherein all of $f$'s nontrivial Fourier coefficients become smaller than $ 2^{-k}$. To complement this result, we show the existence of degree $d$ functions with coefficients larger than $2^{-d\log n}$ when restricted to any affine subspace of dimension larger than $Ω(dn^{1/(d-1)})$. In addition, we give explicit examples of functions with analogous but weaker properties. Along the way, we provide multiple characterizations of the Fourier coefficients of functions restricted to subspaces of $\mathbb{F}_2^n$ that may be useful in other contexts. Finally, we highlight applications and connections of our results to parity kill number and affine dispersers. △ Less

Submitted 3 May, 2023; v1 submitted 27 July, 2022; originally announced July 2022.

Comments: 27 pages

Survey on Wireless Information Energy Transfer (WIET) and Related Applications in 6G Internet of NanoThings (IoNT)

Authors: Pragati Sharma, Rahul Jashvantbhai Pandya, Sridhar Iyer, Anubhav Sharma

Abstract: This article contains an overview of WIET and the related applications in 6G IoNT. Specifically, to explore the following, we: (i) introduce the 6G network along with the implementation challenges, possible techniques, THz communication and related research challenges, (ii) focus on the WIET architecture, and different energy carrying code words for efficient charging through WIET, (iii) discuss I… ▽ More This article contains an overview of WIET and the related applications in 6G IoNT. Specifically, to explore the following, we: (i) introduce the 6G network along with the implementation challenges, possible techniques, THz communication and related research challenges, (ii) focus on the WIET architecture, and different energy carrying code words for efficient charging through WIET, (iii) discuss IoNT with techniques proposed for communication of nano-devices, and (iv) conduct a detailed literature review to explore the implicational aspects of the WIET in the 6G nano-network. In addition, we also investigate the expected applications of WIET in the 6G IoNT based devices and discuss the WIET implementation challenges in 6G IoNT for the optimal use of the technology. Lastly, we overview the expected design challenges which may occur during the implementation process, and identify the key research challenges which require timely solutions and which are significant to spur further research in this challenging area. Overall, through this survey, we discuss the possibility to maximize the applications of WIET in 6G IoNT. △ Less

Submitted 1 July, 2022; originally announced July 2022.

Journal ref: Proceedings of the Indian National Science Academy 2023

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

Authors: Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza , et al. (426 additional authors not shown)

Abstract: Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-futur… ▽ More Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting. △ Less

Submitted 12 June, 2023; v1 submitted 9 June, 2022; originally announced June 2022.

Comments: 27 pages, 17 figures + references and appendices, repo: https://github.com/google/BIG-bench

Journal ref: Transactions on Machine Learning Research, May/2022, https://openreview.net/forum?id=uyTL5Bvosj

Applied Federated Learning: Architectural Design for Robust and Efficient Learning in Privacy Aware Settings

Authors: Branislav Stojkovic, Jonathan Woodbridge, Zhihan Fang, Jerry Cai, Andrey Petrov, Sathya Iyer, Daoyu Huang, Patrick Yau, Arvind Sastha Kumar, Hitesh Jawa, Anamita Guha

Abstract: The classical machine learning paradigm requires the aggregation of user data in a central location where machine learning practitioners can preprocess data, calculate features, tune models and evaluate performance. The advantage of this approach includes leveraging high performance hardware (such as GPUs) and the ability of machine learning practitioners to do in depth data analysis to improve mo… ▽ More The classical machine learning paradigm requires the aggregation of user data in a central location where machine learning practitioners can preprocess data, calculate features, tune models and evaluate performance. The advantage of this approach includes leveraging high performance hardware (such as GPUs) and the ability of machine learning practitioners to do in depth data analysis to improve model performance. However, these advantages may come at a cost to data privacy. User data is collected, aggregated, and stored on centralized servers for model development. Centralization of data poses risks, including a heightened risk of internal and external security incidents as well as accidental data misuse. Federated learning with differential privacy is designed to avoid the server-side centralization pitfall by bringing the ML learning step to users' devices. Learning is done in a federated manner where each mobile device runs a training loop on a local copy of a model. Updates from on-device models are sent to the server via encrypted communication and through differential privacy to improve the global model. In this paradigm, users' personal data remains on their devices. Surprisingly, model training in this manner comes at a fairly minimal degradation in model performance. However, federated learning comes with many other challenges due to its distributed nature, heterogeneous compute environments and lack of data visibility. This paper explores those challenges and outlines an architectural design solution we are exploring and testing to productionize federated learning at Meta scale. △ Less

Submitted 7 June, 2022; v1 submitted 1 June, 2022; originally announced June 2022.

ToKen: Task Decomposition and Knowledge Infusion for Few-Shot Hate Speech Detection

Authors: Badr AlKhamissi, Faisal Ladhak, Srini Iyer, Ves Stoyanov, Zornitsa Kozareva, Xian Li, Pascale Fung, Lambert Mathias, Asli Celikyilmaz, Mona Diab

Abstract: Hate speech detection is complex; it relies on commonsense reasoning, knowledge of stereotypes, and an understanding of social nuance that differs from one culture to the next. It is also difficult to collect a large-scale hate speech annotated dataset. In this work, we frame this problem as a few-shot learning task, and show significant gains with decomposing the task into its "constituent" parts… ▽ More Hate speech detection is complex; it relies on commonsense reasoning, knowledge of stereotypes, and an understanding of social nuance that differs from one culture to the next. It is also difficult to collect a large-scale hate speech annotated dataset. In this work, we frame this problem as a few-shot learning task, and show significant gains with decomposing the task into its "constituent" parts. In addition, we see that infusing knowledge from reasoning datasets (e.g. Atomic2020) improves the performance even further. Moreover, we observe that the trained models generalize to out-of-distribution datasets, showing the superiority of task decomposition and knowledge infusion compared to previously used methods. Concretely, our method outperforms the baseline by 17.83% absolute gain in the 16-shot case. △ Less

Submitted 20 May, 2023; v1 submitted 25 May, 2022; originally announced May 2022.

Comments: Accepted at EMNLP 2022

Journal ref: In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2109-2120, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics

Improving In-Context Few-Shot Learning via Self-Supervised Training

Authors: Mingda Chen, Jingfei Du, Ramakanth Pasunuru, Todor Mihaylov, Srini Iyer, Veselin Stoyanov, Zornitsa Kozareva

Abstract: Self-supervised pretraining has made few-shot learning possible for many NLP tasks. But the pretraining objectives are not typically adapted specifically for in-context few-shot learning. In this paper, we propose to use self-supervision in an intermediate training stage between pretraining and downstream few-shot usage with the goal to teach the model to perform in-context few shot learning. We p… ▽ More Self-supervised pretraining has made few-shot learning possible for many NLP tasks. But the pretraining objectives are not typically adapted specifically for in-context few-shot learning. In this paper, we propose to use self-supervision in an intermediate training stage between pretraining and downstream few-shot usage with the goal to teach the model to perform in-context few shot learning. We propose and evaluate four self-supervised objectives on two benchmarks. We find that the intermediate self-supervision stage produces models that outperform strong baselines. Ablation study shows that several factors affect the downstream performance, such as the amount of training data and the diversity of the self-supervised objectives. Human-annotated cross-task supervision and self-supervision are complementary. Qualitative analysis suggests that the self-supervised-trained models are better at following task requirements. △ Less

Submitted 6 June, 2022; v1 submitted 3 May, 2022; originally announced May 2022.

Comments: NAACL 2022

A Survey on Brain-Computer Interface and Related Applications

Authors: Krishna Pai, Rakhee Kallimani, Sridhar Iyer, B. Uma Maheswari, Rajashri Khanai, Dattaprasad Torse

Abstract: BCI systems are able to communicate directly between the brain and computer using neural activity measurements without the involvement of muscle movements. For BCI systems to be widely used by people with severe disabilities, long-term studies of their real-world use are needed, along with effective and feasible dissemination models. In addition, the robustness of the BCI systems' performance shou… ▽ More BCI systems are able to communicate directly between the brain and computer using neural activity measurements without the involvement of muscle movements. For BCI systems to be widely used by people with severe disabilities, long-term studies of their real-world use are needed, along with effective and feasible dissemination models. In addition, the robustness of the BCI systems' performance should be improved so they reach the same level of robustness as natural muscle-based health monitoring. In this chapter, we review the recent BCI related studies, followed by the most relevant applications of BCI systems. We also present the key issues and challenges which exist in regard to the BCI systems and also provide future directions. △ Less

Submitted 17 March, 2022; originally announced March 2022.

Journal ref: Machine Intelligence for Internet of Medical Things: Applications and Future Trends, Computational Intelligence for Data Analysis (2023) 2:210-228 (19)

A Survey of Machine Learning Algorithms for 6G Wireless Networks

Authors: Anita Patil, Sridhar Iyer, Rahul Jashvantbhai Pandya

Abstract: The primary focus of Artificial Intelligence/Machine Learning (AI/ML) integration within the wireless technology is to reduce capital expenditures, optimize network performance, and build new revenue streams. Replacing traditional algorithms with deep learning AI techniques have dramatically reduced the power consumption and improved the system performance. Further, implementation of ML algorithms… ▽ More The primary focus of Artificial Intelligence/Machine Learning (AI/ML) integration within the wireless technology is to reduce capital expenditures, optimize network performance, and build new revenue streams. Replacing traditional algorithms with deep learning AI techniques have dramatically reduced the power consumption and improved the system performance. Further, implementation of ML algorithms also enables the wireless network service providers to (i) offer high automation levels from distributed AI/ML architectures applicable at the network edge, (ii) implement application-based traffic steering across the access networks, (iii) enable dynamic network slicing for addressing different scenarios with varying quality of service requirements, and (iv) enable ubiquitous connectivity across the various 6G communication platforms. In this chapter, we review/survey the ML techniques which are applicable to the 6G wireless networks. and also list the open problems of research which require timely solutions. △ Less

Submitted 16 March, 2022; originally announced March 2022.

Survey on Internet of Things enabled by 6G Wireless Networks

Authors: Sridhar Iyer, Rahul Jashvantbhai Pandya, Rakhee Kallimani, Krishna Pai, Rajashri Khanai, Dattaprasad Torse, Swati Mavinkattimath

Abstract: The 6G wireless technology is visualized to revolutionize multiple customer services with the Internet of Things (IoT), thereby contributing to a ubiquitous intelligent society comprising autonomous systems. In this chapter, we conduct a detailed survey on the IoT networks with 6G wireless networks and investigate the trending possibilities provided by the 6G technology within the IoT networks and… ▽ More The 6G wireless technology is visualized to revolutionize multiple customer services with the Internet of Things (IoT), thereby contributing to a ubiquitous intelligent society comprising autonomous systems. In this chapter, we conduct a detailed survey on the IoT networks with 6G wireless networks and investigate the trending possibilities provided by the 6G technology within the IoT networks and the related utilization; Firstly, we detail the breakthrough IoT technologies and the technological drivers which are anticipated to strengthen IoT networks in future. Next, we present the relevant use cases detailing the discussion on the role of the 6G technology within a broad spectrum of IoT potential applications. Lastly, we highlight the several research scope and challenges and list the potential research needs and encourage further research within the thrust area of IoT enabled by 6G networks. △ Less

Submitted 16 March, 2022; originally announced March 2022.

A Survey on Technological Trends to Enhance Spectrum Efficiency in 6G Communications

Authors: Sridhar Iyer, Anita Patil, Shilpa Bhairanatti, Soumya Halagatti, Rahul Jashvantbhai Pandya

Abstract: The research community has already identified that, by 2030, 5G networks will reach the capacity limits, and hence, will be inadequate to support next generation bandwidth-hungry, ubiquitous, intelligent services, and applications. Therefore, in view of sustaining the competitive edge of wireless technology and stratifying the next decade's communication requirements both, industry and research co… ▽ More The research community has already identified that, by 2030, 5G networks will reach the capacity limits, and hence, will be inadequate to support next generation bandwidth-hungry, ubiquitous, intelligent services, and applications. Therefore, in view of sustaining the competitive edge of wireless technology and stratifying the next decade's communication requirements both, industry and research community have already begun conceptualizing the 6G technology. This article presents a detailed survey on the recent technological trends which address the capacity issues and enhance the spectrum-efficiency in 6G Communications. We present these trends in detail and then identify the challenges that need solutions before the practical deployment to realize 6G communications. Our survey article attempts to significantly contribute to initiating future research directions in the area of spectrum-efficiency in 6G communications. △ Less

Submitted 23 February, 2022; originally announced February 2022.

Journal ref: 2022

A Survey on Semantic Communications for Intelligent Wireless Networks

Authors: Sridhar Iyer, Rajashri Khanai, Dattaprasad Torse, Rahul Jashvantbhai Pandya, Khaled Rabie, Krishna Pai, Wali Ullah Khan, Zubair Fadlullah

Abstract: With deployment of 6G technology, it is envisioned that competitive edge of wireless networks will be sustained and next decade's communication requirements will be stratified. Also 6G will aim to aid development of a human society which is ubiquitous and mobile, simultaneously providing solutions to key challenges such as, coverage, capacity, etc. In addition, 6G will focus on providing intellige… ▽ More With deployment of 6G technology, it is envisioned that competitive edge of wireless networks will be sustained and next decade's communication requirements will be stratified. Also 6G will aim to aid development of a human society which is ubiquitous and mobile, simultaneously providing solutions to key challenges such as, coverage, capacity, etc. In addition, 6G will focus on providing intelligent use-cases and applications using higher data-rates over mill-meter waves and Tera-Hertz frequency. However, at higher frequencies multiple non-desired phenomena such as atmospheric absorption, blocking, etc., occur which create a bottleneck owing to resource (spectrum and energy) scarcity. Hence, following same trend of making efforts towards reproducing at receiver, exact information which was sent by transmitter, will result in a never ending need for higher bandwidth. A possible solution to such a challenge lies in semantic communications which focuses on meaning (context) of received data as opposed to only reproducing correct transmitted data. This in turn will require less bandwidth, and will reduce bottleneck due to various undesired phenomenon. In this respect, current article presents a detailed survey on recent technological trends in regard to semantic communications for intelligent wireless networks. We focus on semantic communications architecture including model, and source and channel coding. Next, we detail cross-layer interaction, and various goal-oriented communication applications. We also present overall semantic communications trends in detail, and identify challenges which need timely solutions before practical implementation of semantic communications within 6G wireless technology. Our survey article is an attempt to significantly contribute towards initiating future research directions in area of semantic communications for intelligent 6G wireless networks. △ Less

Submitted 10 August, 2022; v1 submitted 8 February, 2022; originally announced February 2022.

Journal ref: Wireless Personal Communications 129, 569-611 (2023)

Efficient Large Scale Language Modeling with Mixtures of Experts

Authors: Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giri Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, Ves Stoyanov

Abstract: Mixture of Experts layers (MoEs) enable efficient scaling of language models through conditional computation. This paper presents a detailed empirical study of how autoregressive MoE language models scale in comparison with dense models in a wide range of settings: in- and out-of-domain language modeling, zero- and few-shot priming, and full-shot fine-tuning. With the exception of fine-tuning, we… ▽ More Mixture of Experts layers (MoEs) enable efficient scaling of language models through conditional computation. This paper presents a detailed empirical study of how autoregressive MoE language models scale in comparison with dense models in a wide range of settings: in- and out-of-domain language modeling, zero- and few-shot priming, and full-shot fine-tuning. With the exception of fine-tuning, we find MoEs to be substantially more compute efficient. At more modest training budgets, MoEs can match the performance of dense models using $\sim$4 times less compute. This gap narrows at scale, but our largest MoE model (1.1T parameters) consistently outperforms a compute-equivalent dense model (6.7B parameters). Overall, this performance gap varies greatly across tasks and domains, suggesting that MoE and dense models generalize differently in ways that are worthy of future study. We make our code and models publicly available for research use. △ Less

Submitted 26 October, 2022; v1 submitted 20 December, 2021; originally announced December 2021.

Comments: EMNLP 2022

PATO: Producibility-Aware Topology Optimization using Deep Learning for Metal Additive Manufacturing

Authors: Naresh S. Iyer, Amir M. Mirzendehdel, Sathyanarayanan Raghavan, Yang Jiao, Erva Ulu, Morad Behandish, Saigopal Nelaturi, Dean M. Robinson

Abstract: In this paper, we propose PATO-a producibility-aware topology optimization (TO) framework to help efficiently explore the design space of components fabricated using metal additive manufacturing (AM), while ensuring manufacturability with respect to cracking. Specifically, parts fabricated through Laser Powder Bed Fusion are prone to defects such as warpage or cracking due to high residual stress… ▽ More In this paper, we propose PATO-a producibility-aware topology optimization (TO) framework to help efficiently explore the design space of components fabricated using metal additive manufacturing (AM), while ensuring manufacturability with respect to cracking. Specifically, parts fabricated through Laser Powder Bed Fusion are prone to defects such as warpage or cracking due to high residual stress values generated from the steep thermal gradients produced during the build process. Maturing the design for such parts and planning their fabrication can span months to years, often involving multiple handoffs between design and manufacturing engineers. PATO is based on the a priori discovery of crack-free designs, so that the optimized part can be built defect-free at the outset. To ensure that the design is crack free during optimization, producibility is explicitly encoded within the standard formulation of TO, using a crack index. Multiple crack indices are explored and using experimental validation, maximum shear strain index (MSSI) is shown to be an accurate crack index. Simulating the build process is a coupled, multi-physics computation and incorporating it in the TO loop can be computationally prohibitive. We leverage the current advances in deep convolutional neural networks and present a high-fidelity surrogate model based on an Attention-based U-Net architecture to predict the MSSI values as a spatially varying field over the part's domain. Further, we employ automatic differentiation to directly compute the gradient of maximum MSSI with respect to the input design variables and augment it with the performance-based sensitivity field to optimize the design while considering the trade-off between weight, manufacturability, and functionality. We demonstrate the effectiveness of the proposed method through benchmark studies in 3D as well as experimental validation. △ Less

Submitted 8 December, 2021; originally announced December 2021.

Organ localisation using supervised and semi supervised approaches combining reinforcement learning with imitation learning

Authors: Sankaran Iyer, Alan Blair, Laughlin Dawes, Daniel Moses, Christopher White, Arcot Sowmya

Abstract: Computer aided diagnostics often requires analysis of a region of interest (ROI) within a radiology scan, and the ROI may be an organ or a suborgan. Although deep learning algorithms have the ability to outperform other methods, they rely on the availability of a large amount of annotated data. Motivated by the need to address this limitation, an approach to localisation and detection of multiple… ▽ More Computer aided diagnostics often requires analysis of a region of interest (ROI) within a radiology scan, and the ROI may be an organ or a suborgan. Although deep learning algorithms have the ability to outperform other methods, they rely on the availability of a large amount of annotated data. Motivated by the need to address this limitation, an approach to localisation and detection of multiple organs based on supervised and semi-supervised learning is presented here. It draws upon previous work by the authors on localising the thoracic and lumbar spine region in CT images. The method generates six bounding boxes of organs of interest, which are then fused to a single bounding box. The results of experiments on localisation of the Spleen, Left and Right Kidneys in CT Images using supervised and semi supervised learning (SSL) demonstrate the ability to address data limitations with a much smaller data set and fewer annotations, compared to other state-of-the-art methods. The SSL performance was evaluated using three different mixes of labelled and unlabelled data (i.e.30:70,35:65,40:60) for each of lumbar spine, spleen left and right kidneys respectively. The results indicate that SSL provides a workable alternative especially in medical imaging where it is difficult to obtain annotated data. △ Less

Submitted 6 December, 2021; originally announced December 2021.

Comments: 16 pages, 12 figures

Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs

Authors: Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, Srinivasan Iyer

Abstract: Do language models have beliefs about the world? Dennett (1995) famously argues that even thermostats have beliefs, on the view that a belief is simply an informational state decoupled from any motivational state. In this paper, we discuss approaches to detecting when models have beliefs about the world, and we improve on methods for updating model beliefs to be more truthful, with a focus on meth… ▽ More Do language models have beliefs about the world? Dennett (1995) famously argues that even thermostats have beliefs, on the view that a belief is simply an informational state decoupled from any motivational state. In this paper, we discuss approaches to detecting when models have beliefs about the world, and we improve on methods for updating model beliefs to be more truthful, with a focus on methods based on learned optimizers or hypernetworks. Our main contributions include: (1) new metrics for evaluating belief-updating methods that focus on the logical consistency of beliefs, (2) a training objective for Sequential, Local, and Generalizing model updates (SLAG) that improves the performance of learned optimizers, and (3) the introduction of the belief graph, which is a new form of interface with language models that shows the interdependencies between model beliefs. Our experiments suggest that models possess belief-like qualities to only a limited extent, but update methods can both fix incorrect model beliefs and greatly improve their consistency. Although off-the-shelf optimizers are surprisingly strong belief-updating baselines, our learned optimizers can outperform them in more difficult settings than have been considered in past work. Code is available at https://github.com/peterbhase/SLAG-Belief-Updating △ Less

Submitted 26 November, 2021; originally announced November 2021.

Comments: 19 pages

AnswerSumm: A Manually-Curated Dataset and Pipeline for Answer Summarization

Authors: Alexander R. Fabbri, Xiaojian Wu, Srini Iyer, Haoran Li, Mona Diab

Abstract: Community Question Answering (CQA) fora such as Stack Overflow and Yahoo! Answers contain a rich resource of answers to a wide range of community-based questions. Each question thread can receive a large number of answers with different perspectives. One goal of answer summarization is to produce a summary that reflects the range of answer perspectives. A major obstacle for this task is the absenc… ▽ More Community Question Answering (CQA) fora such as Stack Overflow and Yahoo! Answers contain a rich resource of answers to a wide range of community-based questions. Each question thread can receive a large number of answers with different perspectives. One goal of answer summarization is to produce a summary that reflects the range of answer perspectives. A major obstacle for this task is the absence of a dataset to provide supervision for producing such summaries. Recent works propose heuristics to create such data, but these are often noisy and do not cover all answer perspectives present. This work introduces a novel dataset of 4,631 CQA threads for answer summarization curated by professional linguists. Our pipeline gathers annotations for all subtasks of answer summarization, including relevant answer sentence selection, grouping these sentences based on perspectives, summarizing each perspective, and producing an overall summary. We analyze and benchmark state-of-the-art models on these subtasks and introduce a novel unsupervised approach for multi-perspective data augmentation that boosts summarization performance according to automatic evaluation. Finally, we propose reinforcement learning rewards to improve factual consistency and answer coverage and analyze areas for improvement. △ Less

Submitted 29 April, 2022; v1 submitted 11 November, 2021; originally announced November 2021.

Comments: NAACL 2022; arXiv admin note: substantial text overlap with arXiv:2104.08536

Proceedings of KDD 2020 Workshop on Data-driven Humanitarian Mapping: Harnessing Human-Machine Intelligence for High-Stake Public Policy and Resilience Planning

Authors: Snehalkumar, S. Gaikwad, Shankar Iyer, Dalton Lunga, Yu-Ru Lin

Abstract: Humanitarian challenges, including natural disasters, food insecurity, climate change, racial and gender violence, environmental crises, the COVID-19 coronavirus pandemic, human rights violations, and forced displacements, disproportionately impact vulnerable communities worldwide. According to UN OCHA, 235 million people will require humanitarian assistance in 2021 . Despite these growing perils,… ▽ More Humanitarian challenges, including natural disasters, food insecurity, climate change, racial and gender violence, environmental crises, the COVID-19 coronavirus pandemic, human rights violations, and forced displacements, disproportionately impact vulnerable communities worldwide. According to UN OCHA, 235 million people will require humanitarian assistance in 2021 . Despite these growing perils, there remains a notable paucity of data science research to scientifically inform equitable public policy decisions for improving the livelihood of at-risk populations. Scattered data science efforts exist to address these challenges, but they remain isolated from practice and prone to algorithmic harms concerning lack of privacy, fairness, interpretability, accountability, transparency, and ethics. Biases in data-driven methods carry the risk of amplifying inequalities in high-stakes policy decisions that impact the livelihood of millions of people. Consequently, proclaimed benefits of data-driven innovations remain inaccessible to policymakers, practitioners, and marginalized communities at the core of humanitarian actions and global development. To help fill this gap, we propose the Data-driven Humanitarian Mapping Research Program, which focuses on developing novel data science methodologies that harness human-machine intelligence for high-stakes public policy and resilience planning. The proceedings of the 1st Data-driven Humanitarian Mapping workshop at the 26th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, August 24th, 2020. △ Less

Submitted 7 September, 2021; v1 submitted 1 September, 2021; originally announced September 2021.

Comments: The proceedings of the 1st Data-driven Humanitarian Mapping workshop at the 26th ACM SIGKDD Conference on Knowledge Discovery & Data Mining

Proceedings of KDD 2021 Workshop on Data-driven Humanitarian Mapping: Harnessing Human-Machine Intelligence for High-Stake Public Policy and Resilience Planning

Authors: Snehalkumar, S. Gaikwad, Shankar Iyer, Dalton Lunga, Elizabeth Bondi

Abstract: Humanitarian challenges, including natural disasters, food insecurity, climate change, racial and gender violence, environmental crises, the COVID-19 coronavirus pandemic, human rights violations, and forced displacements, disproportionately impact vulnerable communities worldwide. According to UN OCHA, 235 million people will require humanitarian assistance in 2021. Despite these growing perils,… ▽ More Humanitarian challenges, including natural disasters, food insecurity, climate change, racial and gender violence, environmental crises, the COVID-19 coronavirus pandemic, human rights violations, and forced displacements, disproportionately impact vulnerable communities worldwide. According to UN OCHA, 235 million people will require humanitarian assistance in 2021. Despite these growing perils, there remains a notable paucity of data science research to scientifically inform equitable public policy decisions for improving the livelihood of at-risk populations. Scattered data science efforts exist to address these challenges, but they remain isolated from practice and prone to algorithmic harms concerning lack of privacy, fairness, interpretability, accountability, transparency, and ethics. Biases in data-driven methods carry the risk of amplifying inequalities in high-stakes policy decisions that impact the livelihood of millions of people. Consequently, proclaimed benefits of data-driven innovations remain inaccessible to policymakers, practitioners, and marginalized communities at the core of humanitarian actions and global development. To help fill this gap, we propose the Data-driven Humanitarian Mapping Research Program, which focuses on developing novel data science methodologies that harness human-machine intelligence for high-stakes public policy and resilience planning. The proceedings of the 2nd Data-driven Humanitarian Mapping workshop at the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. August 15th, 2021 △ Less

Submitted 7 September, 2021; v1 submitted 31 August, 2021; originally announced September 2021.

Comments: The proceedings of the 2nd Data-driven Humanitarian Mapping workshop at the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. August 15th, 2021

Tight bounds on the Fourier growth of bounded functions on the hypercube

Authors: Siddharth Iyer, Anup Rao, Victor Reis, Thomas Rothvoss, Amir Yehudayoff

Abstract: We give tight bounds on the degree $\ell$ homogenous parts $f_\ell$ of a bounded function $f$ on the cube. We show that if $f: \{\pm 1\}^n \rightarrow [-1,1]$ has degree $d$, then $\| f_\ell \|_\infty$ is bounded by $d^\ell/\ell!$, and $\| \hat{f}_\ell \|_1$ is bounded by $d^\ell e^{\binom{\ell+1}{2}} n^{\frac{\ell-1}{2}}$. We describe applications to pseudorandomness and learning theory. We use s… ▽ More We give tight bounds on the degree $\ell$ homogenous parts $f_\ell$ of a bounded function $f$ on the cube. We show that if $f: \{\pm 1\}^n \rightarrow [-1,1]$ has degree $d$, then $\| f_\ell \|_\infty$ is bounded by $d^\ell/\ell!$, and $\| \hat{f}_\ell \|_1$ is bounded by $d^\ell e^{\binom{\ell+1}{2}} n^{\frac{\ell-1}{2}}$. We describe applications to pseudorandomness and learning theory. We use similar methods to generalize the classical Pisier's inequality from convex analysis. Our analysis involves properties of real-rooted polynomials that may be useful elsewhere. △ Less

Submitted 19 July, 2021; v1 submitted 13 July, 2021; originally announced July 2021.

EASE: Extractive-Abstractive Summarization with Explanations

Authors: Haoran Li, Arash Einolghozati, Srinivasan Iyer, Bhargavi Paranjape, Yashar Mehdad, Sonal Gupta, Marjan Ghazvininejad

Abstract: Current abstractive summarization systems outperform their extractive counterparts, but their widespread adoption is inhibited by the inherent lack of interpretability. To achieve the best of both worlds, we propose EASE, an extractive-abstractive framework for evidence-based text generation and apply it to document summarization. We present an explainable summarization system based on the Informa… ▽ More Current abstractive summarization systems outperform their extractive counterparts, but their widespread adoption is inhibited by the inherent lack of interpretability. To achieve the best of both worlds, we propose EASE, an extractive-abstractive framework for evidence-based text generation and apply it to document summarization. We present an explainable summarization system based on the Information Bottleneck principle that is jointly trained for extraction and abstraction in an end-to-end fashion. Inspired by previous research that humans use a two-stage framework to summarize long documents (Jing and McKeown, 2000), our framework first extracts a pre-defined amount of evidence spans as explanations and then generates a summary using only the evidence. Using automatic and human evaluations, we show that explanations from our framework are more relevant than simple baselines, without substantially sacrificing the quality of the generated summary. △ Less

Submitted 14 May, 2021; originally announced May 2021.

Multi-Perspective Abstractive Answer Summarization

Authors: Alexander R. Fabbri, Xiaojian Wu, Srini Iyer, Mona Diab

Abstract: Community Question Answering (CQA) forums such as Stack Overflow and Yahoo! Answers contain a rich resource of answers to a wide range of questions. Each question thread can receive a large number of answers with different perspectives. The goal of multi-perspective answer summarization is to produce a summary that includes all perspectives of the answer. A major obstacle for multi-perspective, ab… ▽ More Community Question Answering (CQA) forums such as Stack Overflow and Yahoo! Answers contain a rich resource of answers to a wide range of questions. Each question thread can receive a large number of answers with different perspectives. The goal of multi-perspective answer summarization is to produce a summary that includes all perspectives of the answer. A major obstacle for multi-perspective, abstractive answer summarization is the absence of a dataset to provide supervision for producing such summaries. This work introduces a novel dataset creation method to automatically create multi-perspective, bullet-point abstractive summaries from an existing CQA forum. Supervision provided by this dataset trains models to inherently produce multi-perspective summaries. Additionally, to train models to output more diverse, faithful answer summaries while retaining multiple perspectives, we propose a multi-reward optimization technique coupled with a sentence-relevance prediction multi-task loss. Our methods demonstrate improved coverage of perspectives and faithfulness as measured by automatic and human evaluations compared to a strong baseline. △ Less

Submitted 17 April, 2021; originally announced April 2021.

The case for model-driven interpretability of delay-based congestion control protocols

Authors: Muhammad Khan, Yasir Zaki, Shiva Iyer, Talal Ahamd, Thomas Pötsch, Jay Chen, Anirudh Sivaraman, Lakshmi Subramanian

Abstract: Analyzing and interpreting the exact behavior of new delay-based congestion control protocols with complex non-linear control loops is exceptionally difficult in highly variable networks such as cellular networks. This paper proposes a Model-Driven Interpretability (MDI) congestion control framework, which derives a model version of a delay-based protocol by simplifying a congestion control protoc… ▽ More Analyzing and interpreting the exact behavior of new delay-based congestion control protocols with complex non-linear control loops is exceptionally difficult in highly variable networks such as cellular networks. This paper proposes a Model-Driven Interpretability (MDI) congestion control framework, which derives a model version of a delay-based protocol by simplifying a congestion control protocol's response into a guided random walk over a two-dimensional Markov model. We demonstrate the case for the MDI framework by using MDI to analyze and interpret the behavior of two delay-based protocols over cellular channels: Verus and Copa. Our results show a successful approximation of throughput and delay characteristics of the protocols' model versions across variable network conditions. The learned model of a protocol provides key insights into an algorithm's convergence properties. △ Less

Submitted 9 February, 2021; originally announced February 2021.

FiD-Ex: Improving Sequence-to-Sequence Models for Extractive Rationale Generation

Authors: Kushal Lakhotia, Bhargavi Paranjape, Asish Ghoshal, Wen-tau Yih, Yashar Mehdad, Srinivasan Iyer

Abstract: Natural language (NL) explanations of model predictions are gaining popularity as a means to understand and verify decisions made by large black-box pre-trained models, for NLP tasks such as Question Answering (QA) and Fact Verification. Recently, pre-trained sequence to sequence (seq2seq) models have proven to be very effective in jointly making predictions, as well as generating NL explanations.… ▽ More Natural language (NL) explanations of model predictions are gaining popularity as a means to understand and verify decisions made by large black-box pre-trained models, for NLP tasks such as Question Answering (QA) and Fact Verification. Recently, pre-trained sequence to sequence (seq2seq) models have proven to be very effective in jointly making predictions, as well as generating NL explanations. However, these models have many shortcomings; they can fabricate explanations even for incorrect predictions, they are difficult to adapt to long input documents, and their training requires a large amount of labeled data. In this paper, we develop FiD-Ex, which addresses these shortcomings for seq2seq models by: 1) introducing sentence markers to eliminate explanation fabrication by encouraging extractive generation, 2) using the fusion-in-decoder architecture to handle long input contexts, and 3) intermediate fine-tuning on re-structured open domain QA datasets to improve few-shot performance. FiD-Ex significantly improves over prior work in terms of explanation metrics and task accuracy, on multiple tasks from the ERASER explainability benchmark, both in the fully supervised and in the few-shot settings. △ Less

Submitted 31 December, 2020; originally announced December 2020.

Human Evaluation of Spoken vs. Visual Explanations for Open-Domain QA

Authors: Ana Valeria Gonzalez, Gagan Bansal, Angela Fan, Robin Jia, Yashar Mehdad, Srinivasan Iyer

Abstract: While research on explaining predictions of open-domain QA systems (ODQA) to users is gaining momentum, most works have failed to evaluate the extent to which explanations improve user trust. While few works evaluate explanations using user studies, they employ settings that may deviate from the end-user's usage in-the-wild: ODQA is most ubiquitous in voice-assistants, yet current research only ev… ▽ More While research on explaining predictions of open-domain QA systems (ODQA) to users is gaining momentum, most works have failed to evaluate the extent to which explanations improve user trust. While few works evaluate explanations using user studies, they employ settings that may deviate from the end-user's usage in-the-wild: ODQA is most ubiquitous in voice-assistants, yet current research only evaluates explanations using a visual display, and may erroneously extrapolate conclusions about the most performant explanations to other modalities. To alleviate these issues, we conduct user studies that measure whether explanations help users correctly decide when to accept or reject an ODQA system's answer. Unlike prior work, we control for explanation modality, e.g., whether they are communicated to users through a spoken or visual interface, and contrast effectiveness across modalities. Our results show that explanations derived from retrieved evidence passages can outperform strong baselines (calibrated confidence) across modalities but the best explanation strategy in fact changes with the modality. We show common failure cases of current explanations, emphasize end-to-end evaluation of explanations, and caution against evaluating them in proxy modalities that are different from deployment. △ Less

Submitted 30 December, 2020; originally announced December 2020.

Comments: pre-print

RECONSIDER: Re-Ranking using Span-Focused Cross-Attention for Open Domain Question Answering

Authors: Srinivasan Iyer, Sewon Min, Yashar Mehdad, Wen-tau Yih

Abstract: State-of-the-art Machine Reading Comprehension (MRC) models for Open-domain Question Answering (QA) are typically trained for span selection using distantly supervised positive examples and heuristically retrieved negative examples. This training scheme possibly explains empirical observations that these models achieve a high recall amongst their top few predictions, but a low overall accuracy, mo… ▽ More State-of-the-art Machine Reading Comprehension (MRC) models for Open-domain Question Answering (QA) are typically trained for span selection using distantly supervised positive examples and heuristically retrieved negative examples. This training scheme possibly explains empirical observations that these models achieve a high recall amongst their top few predictions, but a low overall accuracy, motivating the need for answer re-ranking. We develop a simple and effective re-ranking approach (RECONSIDER) for span-extraction tasks, that improves upon the performance of large pre-trained MRC models. RECONSIDER is trained on positive and negative examples extracted from high confidence predictions of MRC models, and uses in-passage span annotations to perform span-focused re-ranking over a smaller candidate set. As a result, RECONSIDER learns to eliminate close false positive passages, and achieves a new state of the art on four QA tasks, including 45.5% Exact Match accuracy on Natural Questions with real user questions, and 61.7% on TriviaQA. △ Less

Submitted 21 October, 2020; originally announced October 2020.

Agent-based Simulation Model and Deep Learning Techniques to Evaluate and Predict Transportation Trends around COVID-19

Authors: Ding Wang, Fan Zuo, Jingqin Gao, Yueshuai He, Zilin Bian, Suzana Duran Bernardes, Chaekuk Na, Jingxing Wang, John Petinos, Kaan Ozbay, Joseph Y. J. Chow, Shri Iyer, Hani Nassif, Xuegang Jeff Ban

Abstract: The COVID-19 pandemic has affected travel behaviors and transportation system operations, and cities are grappling with what policies can be effective for a phased reopening shaped by social distancing. This edition of the white paper updates travel trends and highlights an agent-based simulation model's results to predict the impact of proposed phased reopening strategies. It also introduces a re… ▽ More The COVID-19 pandemic has affected travel behaviors and transportation system operations, and cities are grappling with what policies can be effective for a phased reopening shaped by social distancing. This edition of the white paper updates travel trends and highlights an agent-based simulation model's results to predict the impact of proposed phased reopening strategies. It also introduces a real-time video processing method to measure social distancing through cameras on city streets. △ Less

Submitted 23 September, 2020; originally announced October 2020.

Efficient One-Pass End-to-End Entity Linking for Questions

Authors: Belinda Z. Li, Sewon Min, Srinivasan Iyer, Yashar Mehdad, Wen-tau Yih

Abstract: We present ELQ, a fast end-to-end entity linking model for questions, which uses a biencoder to jointly perform mention detection and linking in one pass. Evaluated on WebQSP and GraphQuestions with extended annotations that cover multiple entities per question, ELQ outperforms the previous state of the art by a large margin of +12.7% and +19.6% F1, respectively. With a very fast inference time (1… ▽ More We present ELQ, a fast end-to-end entity linking model for questions, which uses a biencoder to jointly perform mention detection and linking in one pass. Evaluated on WebQSP and GraphQuestions with extended annotations that cover multiple entities per question, ELQ outperforms the previous state of the art by a large margin of +12.7% and +19.6% F1, respectively. With a very fast inference time (1.57 examples/s on a single CPU), ELQ can be useful for downstream question answering systems. In a proof-of-concept experiment, we demonstrate that using ELQ significantly improves the downstream QA performance of GraphRetriever (arXiv:1911.03868). Code and data available at https://github.com/facebookresearch/BLINK/tree/master/elq △ Less

Submitted 5 October, 2020; originally announced October 2020.

Comments: 9 pages, EMNLP 2020

Toward the "New Normal": A Surge in Speeding, New Volume Patterns, and Recent Trends in Taxis/For-Hire Vehicles

Authors: Jingqin Gao, Abhinav Bhattacharyya, Ding Wang, Nick Hudanich, Siva Sooryaa, Muruga Thambiran, Suzana Duran Bernardes, Chaekuk Na, Fan Zuo, Zilin Bian, Kaan Ozbay, Shri Iyer, Hani Nassif, Joseph Y. J. Chow

Abstract: Six months into the pandemic and one month after the phase four reopening in New York City (NYC), restrictions are lifting, businesses and schools are reopening, but global infections are still rising. This white paper updates travel trends observed in the aftermath of the COVID-19 outbreak in NYC and highlight some findings toward the "new normal." Six months into the pandemic and one month after the phase four reopening in New York City (NYC), restrictions are lifting, businesses and schools are reopening, but global infections are still rising. This white paper updates travel trends observed in the aftermath of the COVID-19 outbreak in NYC and highlight some findings toward the "new normal." △ Less

Submitted 23 September, 2020; originally announced September 2020.