-
Understanding Subjectivity through the Lens of Motivational Context in Model-Generated Image Satisfaction
Authors:
Senjuti Dutta,
Sherol Chen,
Sunny Mak,
Amnah Ahmad,
Katherine Collins,
Alena Butryna,
Deepak Ramachandran,
Krishnamurthy Dvijotham,
Ellie Pavlick,
Ravi Rajakumar
Abstract:
Image generation models are poised to become ubiquitous in a range of applications. These models are often fine-tuned and evaluated using human quality judgments that assume a universal standard, failing to consider the subjectivity of such tasks. To investigate how to quantify subjectivity, and the scale of its impact, we measure how assessments differ among human annotators across different use…
▽ More
Image generation models are poised to become ubiquitous in a range of applications. These models are often fine-tuned and evaluated using human quality judgments that assume a universal standard, failing to consider the subjectivity of such tasks. To investigate how to quantify subjectivity, and the scale of its impact, we measure how assessments differ among human annotators across different use cases. Simulating the effects of ordinarily latent elements of annotators subjectivity, we contrive a set of motivations (t-shirt graphics, presentation visuals, and phone background images) to contextualize a set of crowdsourcing tasks. Our results show that human evaluations of images vary within individual contexts and across combinations of contexts. Three key factors affecting this subjectivity are image appearance, image alignment with text, and representation of objects mentioned in the text. Our study highlights the importance of taking individual users and contexts into account, both when building and evaluating generative models
△ Less
Submitted 26 February, 2024;
originally announced March 2024.
-
Modeling subjectivity (by Mimicking Annotator Annotation) in toxic comment identification across diverse communities
Authors:
Senjuti Dutta,
Sid Mittal,
Sherol Chen,
Deepak Ramachandran,
Ravi Rajakumar,
Ian Kivlichan,
Sunny Mak,
Alena Butryna,
Praveen Paritosh
Abstract:
The prevalence and impact of toxic discussions online have made content moderation crucial.Automated systems can play a vital role in identifying toxicity, and reducing the reliance on human moderation.Nevertheless, identifying toxic comments for diverse communities continues to present challenges that are addressed in this paper.The two-part goal of this study is to(1)identify intuitive variances…
▽ More
The prevalence and impact of toxic discussions online have made content moderation crucial.Automated systems can play a vital role in identifying toxicity, and reducing the reliance on human moderation.Nevertheless, identifying toxic comments for diverse communities continues to present challenges that are addressed in this paper.The two-part goal of this study is to(1)identify intuitive variances from annotator disagreement using quantitative analysis and (2)model the subjectivity of these viewpoints.To achieve our goal, we published a new dataset\footnote{\url{https://github.com/XXX}} with expert annotators' annotations and used two other public datasets to identify the subjectivity of toxicity.Then leveraging the Large Language Model(LLM),we evaluate the model's ability to mimic diverse viewpoints on toxicity by varying size of the training data and utilizing same set of annotators as the test set used during model training and a separate set of annotators as the test set.We conclude that subjectivity is evident across all annotator groups, demonstrating the shortcomings of majority-rule voting. Moving forward, subjective annotations should serve as ground truth labels for training models for domains like toxicity in diverse communities.
△ Less
Submitted 31 October, 2023;
originally announced November 2023.
-
LaMDA: Language Models for Dialog Applications
Authors:
Romal Thoppilan,
Daniel De Freitas,
Jamie Hall,
Noam Shazeer,
Apoorv Kulshreshtha,
Heng-Tze Cheng,
Alicia Jin,
Taylor Bos,
Leslie Baker,
Yu Du,
YaGuang Li,
Hongrae Lee,
Huaixiu Steven Zheng,
Amin Ghafouri,
Marcelo Menegali,
Yanping Huang,
Maxim Krikun,
Dmitry Lepikhin,
James Qin,
Dehao Chen,
Yuanzhong Xu,
Zhifeng Chen,
Adam Roberts,
Maarten Bosma,
Vincent Zhao
, et al. (35 additional authors not shown)
Abstract:
We present LaMDA: Language Models for Dialog Applications. LaMDA is a family of Transformer-based neural language models specialized for dialog, which have up to 137B parameters and are pre-trained on 1.56T words of public dialog data and web text. While model scaling alone can improve quality, it shows less improvements on safety and factual grounding. We demonstrate that fine-tuning with annotat…
▽ More
We present LaMDA: Language Models for Dialog Applications. LaMDA is a family of Transformer-based neural language models specialized for dialog, which have up to 137B parameters and are pre-trained on 1.56T words of public dialog data and web text. While model scaling alone can improve quality, it shows less improvements on safety and factual grounding. We demonstrate that fine-tuning with annotated data and enabling the model to consult external knowledge sources can lead to significant improvements towards the two key challenges of safety and factual grounding. The first challenge, safety, involves ensuring that the model's responses are consistent with a set of human values, such as preventing harmful suggestions and unfair bias. We quantify safety using a metric based on an illustrative set of human values, and we find that filtering candidate responses using a LaMDA classifier fine-tuned with a small amount of crowdworker-annotated data offers a promising approach to improving model safety. The second challenge, factual grounding, involves enabling the model to consult external knowledge sources, such as an information retrieval system, a language translator, and a calculator. We quantify factuality using a groundedness metric, and we find that our approach enables the model to generate responses grounded in known sources, rather than responses that merely sound plausible. Finally, we explore the use of LaMDA in the domains of education and content recommendations, and analyze their helpfulness and role consistency.
△ Less
Submitted 10 February, 2022; v1 submitted 20 January, 2022;
originally announced January 2022.
-
Google Crowdsourced Speech Corpora and Related Open-Source Resources for Low-Resource Languages and Dialects: An Overview
Authors:
Alena Butryna,
Shan-Hui Cathy Chu,
Isin Demirsahin,
Alexander Gutkin,
Linne Ha,
Fei He,
Martin Jansche,
Cibu Johny,
Anna Katanova,
Oddur Kjartansson,
Chenfang Li,
Tatiana Merkulova,
Yin May Oo,
Knot Pipatsrisawat,
Clara Rivera,
Supheakmungkol Sarin,
Pasindu de Silva,
Keshan Sodimana,
Richard Sproat,
Theeraphol Wattanavekin,
Jaka Aris Eko Wibawa
Abstract:
This paper presents an overview of a program designed to address the growing need for developing freely available speech resources for under-represented languages. At present we have released 38 datasets for building text-to-speech and automatic speech recognition applications for languages and dialects of South and Southeast Asia, Africa, Europe and South America. The paper describes the methodol…
▽ More
This paper presents an overview of a program designed to address the growing need for developing freely available speech resources for under-represented languages. At present we have released 38 datasets for building text-to-speech and automatic speech recognition applications for languages and dialects of South and Southeast Asia, Africa, Europe and South America. The paper describes the methodology used for developing such corpora and presents some of our findings that could benefit under-represented language communities.
△ Less
Submitted 13 October, 2020;
originally announced October 2020.
