-
Improving Opinion-based Question Answering Systems Through Label Error Detection and Overwrite
Authors:
Xiao Yang,
Ahmed K. Mohamed,
Shashank Jain,
Stanislav Peshterliev,
Debojeet Chatterjee,
Hanwen Zha,
Nikita Bhalla,
Gagan Aneja,
Pranab Mohanty
Abstract:
Label error is a ubiquitous problem in annotated data. Large amounts of label error substantially degrades the quality of deep learning models. Existing methods to tackle the label error problem largely focus on the classification task, and either rely on task specific architecture or require non-trivial additional computations, which is undesirable or even unattainable for industry usage. In this…
▽ More
Label error is a ubiquitous problem in annotated data. Large amounts of label error substantially degrades the quality of deep learning models. Existing methods to tackle the label error problem largely focus on the classification task, and either rely on task specific architecture or require non-trivial additional computations, which is undesirable or even unattainable for industry usage. In this paper, we propose LEDO: a model-agnostic and computationally efficient framework for Label Error Detection and Overwrite. LEDO is based on Monte Carlo Dropout combined with uncertainty metrics, and can be easily generalized to multiple tasks and data sets. Applying LEDO to an industry opinion-based question answering system demonstrates it is effective at improving accuracy in all the core models. Specifically, LEDO brings 1.1% MRR gain for the retrieval model, 1.5% PR AUC improvement for the machine reading comprehension model, and 0.9% rise in the Average Precision for the ranker, on top of the strong baselines with a large-scale social media dataset. Importantly, LEDO is computationally efficient compared to methods that require loss function change, and cost-effective as the resulting data can be used in the same continuous training pipeline for production. Further analysis shows that these gains come from an improved decision boundary after cleaning the label errors existed in the training data.
△ Less
Submitted 12 June, 2023;
originally announced June 2023.
-
Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?
Authors:
Xilun Chen,
Kushal Lakhotia,
Barlas Oğuz,
Anchit Gupta,
Patrick Lewis,
Stan Peshterliev,
Yashar Mehdad,
Sonal Gupta,
Wen-tau Yih
Abstract:
Despite their recent popularity and well-known advantages, dense retrievers still lag behind sparse methods such as BM25 in their ability to reliably match salient phrases and rare entities in the query and to generalize to out-of-domain data. It has been argued that this is an inherent limitation of dense models. We rebut this claim by introducing the Salient Phrase Aware Retriever (SPAR), a dens…
▽ More
Despite their recent popularity and well-known advantages, dense retrievers still lag behind sparse methods such as BM25 in their ability to reliably match salient phrases and rare entities in the query and to generalize to out-of-domain data. It has been argued that this is an inherent limitation of dense models. We rebut this claim by introducing the Salient Phrase Aware Retriever (SPAR), a dense retriever with the lexical matching capacity of a sparse model. We show that a dense Lexical Model Λ can be trained to imitate a sparse one, and SPAR is built by augmenting a standard dense retriever with Λ. Empirically, SPAR shows superior performance on a range of tasks including five question answering datasets, MS MARCO passage retrieval, as well as the EntityQuestions and BEIR benchmarks for out-of-domain evaluation, exceeding the performance of state-of-the-art dense and sparse retrievers. The code and models of SPAR are available at: https://github.com/facebookresearch/dpr-scale/tree/main/spar
△ Less
Submitted 11 November, 2022; v1 submitted 13 October, 2021;
originally announced October 2021.
-
Decoupled Transformer for Scalable Inference in Open-domain Question Answering
Authors:
Haytham ElFadeel,
Stan Peshterliev
Abstract:
Large transformer models, such as BERT, achieve state-of-the-art results in machine reading comprehension (MRC) for open-domain question answering (QA). However, transformers have a high computational cost for inference which makes them hard to apply to online QA systems for applications like voice assistants. To reduce computational cost and latency, we propose decoupling the transformer MRC mode…
▽ More
Large transformer models, such as BERT, achieve state-of-the-art results in machine reading comprehension (MRC) for open-domain question answering (QA). However, transformers have a high computational cost for inference which makes them hard to apply to online QA systems for applications like voice assistants. To reduce computational cost and latency, we propose decoupling the transformer MRC model into input-component and cross-component. The decoupling allows for part of the representation computation to be performed offline and cached for online use. To retain the decoupled transformer accuracy, we devised a knowledge distillation objective from a standard transformer model. Moreover, we introduce learned representation compression layers which help reduce by four times the storage requirement for the cache. In experiments on the SQUAD 2.0 dataset, a decoupled transformer reduces the computational cost and latency of open-domain MRC by 30-40% with only 1.2 points worse F1-score compared to a standard transformer.
△ Less
Submitted 5 August, 2021;
originally announced August 2021.
-
Robustly Optimized and Distilled Training for Natural Language Understanding
Authors:
Haytham ElFadeel,
Stan Peshterliev
Abstract:
In this paper, we explore multi-task learning (MTL) as a second pretraining step to learn enhanced universal language representation for transformer language models. We use the MTL enhanced representation across several natural language understanding tasks to improve performance and generalization. Moreover, we incorporate knowledge distillation (KD) in MTL to further boost performance and devise…
▽ More
In this paper, we explore multi-task learning (MTL) as a second pretraining step to learn enhanced universal language representation for transformer language models. We use the MTL enhanced representation across several natural language understanding tasks to improve performance and generalization. Moreover, we incorporate knowledge distillation (KD) in MTL to further boost performance and devise a KD variant that learns effectively from multiple teachers. By combining MTL and KD, we propose Robustly Optimized and Distilled (ROaD) modeling framework. We use ROaD together with the ELECTRA model to obtain state-of-the-art results for machine reading comprehension and natural language inference.
△ Less
Submitted 15 March, 2021;
originally announced March 2021.
-
Conversational Answer Generation and Factuality for Reading Comprehension Question-Answering
Authors:
Stan Peshterliev,
Barlas Oguz,
Debojeet Chatterjee,
Hakan Inan,
Vikas Bhardwaj
Abstract:
Question answering (QA) is an important use case on voice assistants. A popular approach to QA is extractive reading comprehension (RC) which finds an answer span in a text passage. However, extractive answers are often unnatural in a conversational context which results in suboptimal user experience. In this work, we investigate conversational answer generation for QA. We propose AnswerBART, an e…
▽ More
Question answering (QA) is an important use case on voice assistants. A popular approach to QA is extractive reading comprehension (RC) which finds an answer span in a text passage. However, extractive answers are often unnatural in a conversational context which results in suboptimal user experience. In this work, we investigate conversational answer generation for QA. We propose AnswerBART, an end-to-end generative RC model which combines answer generation from multiple passages with passage ranking and answerability. Moreover, a hurdle in applying generative RC are hallucinations where the answer is factually inconsistent with the passage text. We leverage recent work from summarization to evaluate factuality. Experiments show that AnswerBART significantly improves over previous best published results on MS MARCO 2.1 NLGEN by 2.5 ROUGE-L and NarrativeQA by 9.4 ROUGE-L.
△ Less
Submitted 11 March, 2021;
originally announced March 2021.
-
NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
Authors:
Sewon Min,
Jordan Boyd-Graber,
Chris Alberti,
Danqi Chen,
Eunsol Choi,
Michael Collins,
Kelvin Guu,
Hannaneh Hajishirzi,
Kenton Lee,
Jennimaria Palomaki,
Colin Raffel,
Adam Roberts,
Tom Kwiatkowski,
Patrick Lewis,
Yuxiang Wu,
Heinrich Küttler,
Linqing Liu,
Pasquale Minervini,
Pontus Stenetorp,
Sebastian Riedel,
Sohee Yang,
Minjoon Seo,
Gautier Izacard,
Fabio Petroni,
Lucas Hosseini
, et al. (28 additional authors not shown)
Abstract:
We review the EfficientQA competition from NeurIPS 2020. The competition focused on open-domain question answering (QA), where systems take natural language questions as input and return natural language answers. The aim of the competition was to build systems that can predict correct answers while also satisfying strict on-disk memory budgets. These memory budgets were designed to encourage conte…
▽ More
We review the EfficientQA competition from NeurIPS 2020. The competition focused on open-domain question answering (QA), where systems take natural language questions as input and return natural language answers. The aim of the competition was to build systems that can predict correct answers while also satisfying strict on-disk memory budgets. These memory budgets were designed to encourage contestants to explore the trade-off between storing retrieval corpora or the parameters of learned models. In this report, we describe the motivation and organization of the competition, review the best submissions, and analyze system predictions to inform a discussion of evaluation for open-domain QA.
△ Less
Submitted 19 September, 2021; v1 submitted 31 December, 2020;
originally announced January 2021.
-
UniK-QA: Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answering
Authors:
Barlas Oguz,
Xilun Chen,
Vladimir Karpukhin,
Stan Peshterliev,
Dmytro Okhonko,
Michael Schlichtkrull,
Sonal Gupta,
Yashar Mehdad,
Scott Yih
Abstract:
We study open-domain question answering with structured, unstructured and semi-structured knowledge sources, including text, tables, lists and knowledge bases. Departing from prior work, we propose a unifying approach that homogenizes all sources by reducing them to text and applies the retriever-reader model which has so far been limited to text sources only. Our approach greatly improves the res…
▽ More
We study open-domain question answering with structured, unstructured and semi-structured knowledge sources, including text, tables, lists and knowledge bases. Departing from prior work, we propose a unifying approach that homogenizes all sources by reducing them to text and applies the retriever-reader model which has so far been limited to text sources only. Our approach greatly improves the results on knowledge-base QA tasks by 11 points, compared to latest graph-based methods. More importantly, we demonstrate that our unified knowledge (UniK-QA) model is a simple and yet effective way to combine heterogeneous sources of knowledge, advancing the state-of-the-art results on two popular question answering benchmarks, NaturalQuestions and WebQuestions, by 3.5 and 2.6 points, respectively.
The code of UniK-QA is available at: https://github.com/facebookresearch/UniK-QA.
△ Less
Submitted 3 May, 2022; v1 submitted 29 December, 2020;
originally announced December 2020.
-
Self-Attention Gazetteer Embeddings for Named-Entity Recognition
Authors:
Stanislav Peshterliev,
Christophe Dupuy,
Imre Kiss
Abstract:
Recent attempts to ingest external knowledge into neural models for named-entity recognition (NER) have exhibited mixed results. In this work, we present GazSelfAttn, a novel gazetteer embedding approach that uses self-attention and match span encoding to build enhanced gazetteer embeddings. In addition, we demonstrate how to build gazetteer resources from the open source Wikidata knowledge base.…
▽ More
Recent attempts to ingest external knowledge into neural models for named-entity recognition (NER) have exhibited mixed results. In this work, we present GazSelfAttn, a novel gazetteer embedding approach that uses self-attention and match span encoding to build enhanced gazetteer embeddings. In addition, we demonstrate how to build gazetteer resources from the open source Wikidata knowledge base. Evaluations on CoNLL-03 and Ontonotes 5 datasets, show F1 improvements over baseline model from 92.34 to 92.86 and 89.11 to 89.32 respectively, achieving performance comparable to large state-of-the-art models.
△ Less
Submitted 18 April, 2020; v1 submitted 8 April, 2020;
originally announced April 2020.
-
F10-SGD: Fast Training of Elastic-net Linear Models for Text Classification and Named-entity Recognition
Authors:
Stanislav Peshterliev,
Alexander Hsieh,
Imre Kiss
Abstract:
Voice-assistants text classification and named-entity recognition (NER) models are trained on millions of example utterances. Because of the large datasets, long training time is one of the bottlenecks for releasing improved models. In this work, we develop F10-SGD, a fast optimizer for text classification and NER elastic-net linear models. On internal datasets, F10-SGD provides 4x reduction in tr…
▽ More
Voice-assistants text classification and named-entity recognition (NER) models are trained on millions of example utterances. Because of the large datasets, long training time is one of the bottlenecks for releasing improved models. In this work, we develop F10-SGD, a fast optimizer for text classification and NER elastic-net linear models. On internal datasets, F10-SGD provides 4x reduction in training time compared to the OWL-QN optimizer without loss of accuracy or increase in model size. Furthermore, we incorporate biased sampling that prioritizes harder examples towards the end of the training. As a result, in addition to faster training, we were able to obtain statistically significant accuracy improvements for NER.
On public datasets, F10-SGD obtains 22% faster training time compared to FastText for text classification. And, 4x reduction in training time compared to CRFSuite OWL-QN for NER.
△ Less
Submitted 27 February, 2019;
originally announced February 2019.
-
Active Learning for New Domains in Natural Language Understanding
Authors:
Stanislav Peshterliev,
John Kearney,
Abhyuday Jagannatha,
Imre Kiss,
Spyros Matsoukas
Abstract:
We explore active learning (AL) for improving the accuracy of new domains in a natural language understanding (NLU) system. We propose an algorithm called Majority-CRF that uses an ensemble of classification models to guide the selection of relevant utterances, as well as a sequence labeling model to help prioritize informative examples. Experiments with three domains show that Majority-CRF achiev…
▽ More
We explore active learning (AL) for improving the accuracy of new domains in a natural language understanding (NLU) system. We propose an algorithm called Majority-CRF that uses an ensemble of classification models to guide the selection of relevant utterances, as well as a sequence labeling model to help prioritize informative examples. Experiments with three domains show that Majority-CRF achieves 6.6%-9% relative error rate reduction compared to random sampling with the same annotation budget, and statistically significant improvements compared to other AL approaches. Additionally, case studies with human-in-the-loop AL on six new domains show 4.6%-9% improvement on an existing NLU system.
△ Less
Submitted 30 March, 2019; v1 submitted 3 October, 2018;
originally announced October 2018.
-
Statistical Model Compression for Small-Footprint Natural Language Understanding
Authors:
Grant P. Strimel,
Kanthashree Mysore Sathyendra,
Stanislav Peshterliev
Abstract:
In this paper we investigate statistical model compression applied to natural language understanding (NLU) models. Small-footprint NLU models are important for enabling offline systems on hardware restricted devices, and for decreasing on-demand model loading latency in cloud-based systems. To compress NLU models, we present two main techniques, parameter quantization and perfect feature hashing.…
▽ More
In this paper we investigate statistical model compression applied to natural language understanding (NLU) models. Small-footprint NLU models are important for enabling offline systems on hardware restricted devices, and for decreasing on-demand model loading latency in cloud-based systems. To compress NLU models, we present two main techniques, parameter quantization and perfect feature hashing. These techniques are complementary to existing model pruning strategies such as L1 regularization. We performed experiments on a large scale NLU system. The results show that our approach achieves 14-fold reduction in memory usage compared to the original models with minimal predictive performance impact.
△ Less
Submitted 19 July, 2018;
originally announced July 2018.
-
Just ASK: Building an Architecture for Extensible Self-Service Spoken Language Understanding
Authors:
Anjishnu Kumar,
Arpit Gupta,
Julian Chan,
Sam Tucker,
Bjorn Hoffmeister,
Markus Dreyer,
Stanislav Peshterliev,
Ankur Gandhe,
Denis Filiminov,
Ariya Rastrow,
Christian Monson,
Agnika Kumar
Abstract:
This paper presents the design of the machine learning architecture that underlies the Alexa Skills Kit (ASK) a large scale Spoken Language Understanding (SLU) Software Development Kit (SDK) that enables developers to extend the capabilities of Amazon's virtual assistant, Alexa. At Amazon, the infrastructure powers over 25,000 skills deployed through the ASK, as well as AWS's Amazon Lex SLU Servic…
▽ More
This paper presents the design of the machine learning architecture that underlies the Alexa Skills Kit (ASK) a large scale Spoken Language Understanding (SLU) Software Development Kit (SDK) that enables developers to extend the capabilities of Amazon's virtual assistant, Alexa. At Amazon, the infrastructure powers over 25,000 skills deployed through the ASK, as well as AWS's Amazon Lex SLU Service. The ASK emphasizes flexibility, predictability and a rapid iteration cycle for third party developers. It imposes inductive biases that allow it to learn robust SLU models from extremely small and sparse datasets and, in doing so, removes significant barriers to entry for software developers and dialogue systems researchers.
△ Less
Submitted 2 March, 2018; v1 submitted 1 November, 2017;
originally announced November 2017.