-
Mitigating Translationese in Low-resource Languages: The Storyboard Approach
Authors:
Garry Kuwanto,
Eno-Abasi E. Urua,
Priscilla Amondi Amuok,
Shamsuddeen Hassan Muhammad,
Anuoluwapo Aremu,
Verrah Otiende,
Loice Emma Nanyanga,
Teresiah W. Nyoike,
Aniefon D. Akpan,
Nsima Ab Udouboh,
Idongesit Udeme Archibong,
Idara Effiong Moses,
Ifeoluwatayo A. Ige,
Benjamin Ajibade,
Olumide Benjamin Awokoya,
Idris Abdulmumin,
Saminu Mohammad Aliyu,
Ruqayya Nasir Iro,
Ibrahim Said Ahmad,
Deontae Smith,
Praise-EL Michaels,
David Ifeoluwa Adelani,
Derry Tanti Wijaya,
Anietie Andy
Abstract:
Low-resource languages often face challenges in acquiring high-quality language data due to the reliance on translation-based methods, which can introduce the translationese effect. This phenomenon results in translated sentences that lack fluency and naturalness in the target language. In this paper, we propose a novel approach for data collection by leveraging storyboards to elicit more fluent a…
▽ More
Low-resource languages often face challenges in acquiring high-quality language data due to the reliance on translation-based methods, which can introduce the translationese effect. This phenomenon results in translated sentences that lack fluency and naturalness in the target language. In this paper, we propose a novel approach for data collection by leveraging storyboards to elicit more fluent and natural sentences. Our method involves presenting native speakers with visual stimuli in the form of storyboards and collecting their descriptions without direct exposure to the source text. We conducted a comprehensive evaluation comparing our storyboard-based approach with traditional text translation-based methods in terms of accuracy and fluency. Human annotators and quantitative metrics were used to assess translation quality. The results indicate a preference for text translation in terms of accuracy, while our method demonstrates worse accuracy but better fluency in the language focused.
△ Less
Submitted 14 July, 2024;
originally announced July 2024.
-
BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages
Authors:
Junho Myung,
Nayeon Lee,
Yi Zhou,
Jiho Jin,
Rifki Afina Putri,
Dimosthenis Antypas,
Hsuvas Borkakoty,
Eunsu Kim,
Carla Perez-Almendros,
Abinew Ali Ayele,
Víctor Gutiérrez-Basulto,
Yazmín Ibáñez-García,
Hwaran Lee,
Shamsuddeen Hassan Muhammad,
Kiwoong Park,
Anar Sabuhi Rzayev,
Nina White,
Seid Muhie Yimam,
Mohammad Taher Pilehvar,
Nedjma Ousidhoum,
Jose Camacho-Collados,
Alice Oh
Abstract:
Large language models (LLMs) often lack culture-specific knowledge of daily life, especially across diverse regions and non-English languages. Existing benchmarks for evaluating LLMs' cultural sensitivities are limited to a single language or collected from online sources such as Wikipedia, which do not reflect the mundane everyday lifestyles of diverse regions. That is, information about the food…
▽ More
Large language models (LLMs) often lack culture-specific knowledge of daily life, especially across diverse regions and non-English languages. Existing benchmarks for evaluating LLMs' cultural sensitivities are limited to a single language or collected from online sources such as Wikipedia, which do not reflect the mundane everyday lifestyles of diverse regions. That is, information about the food people eat for their birthday celebrations, spices they typically use, musical instruments youngsters play, or the sports they practice in school is common cultural knowledge but uncommon in easily collected online sources, especially for underrepresented cultures. To address this issue, we introduce BLEnD, a hand-crafted benchmark designed to evaluate LLMs' everyday knowledge across diverse cultures and languages. BLEnD comprises 52.6k question-answer pairs from 16 countries/regions, in 13 different languages, including low-resource ones such as Amharic, Assamese, Azerbaijani, Hausa, and Sundanese. We construct the benchmark to include two formats of questions: short-answer and multiple-choice. We show that LLMs perform better for cultures that are highly represented online, with a maximum 57.34% difference in GPT-4, the best-performing model, in the short-answer format. For cultures represented by mid-to-high-resource languages, LLMs perform better in their local languages, but for cultures represented by low-resource languages, LLMs perform better in English than the local languages. We make our dataset publicly available at: https://github.com/nlee0212/BLEnD.
△ Less
Submitted 14 June, 2024;
originally announced June 2024.
-
IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models
Authors:
David Ifeoluwa Adelani,
Jessica Ojo,
Israel Abebe Azime,
Jian Yun Zhuang,
Jesujoba O. Alabi,
Xuanli He,
Millicent Ochieng,
Sara Hooker,
Andiswa Bukula,
En-Shiun Annie Lee,
Chiamaka Chukwuneke,
Happy Buzaaba,
Blessing Sibanda,
Godson Kalipe,
Jonathan Mukiibi,
Salomon Kabongo,
Foutse Yuehgoh,
Mmasibidi Setaka,
Lolwethu Ndolela,
Nkiruka Odu,
Rooweither Mabuya,
Shamsuddeen Hassan Muhammad,
Salomey Osei,
Sokhar Samb,
Tadesse Kebede Guge
, et al. (1 additional authors not shown)
Abstract:
Despite the widespread adoption of Large language models (LLMs), their remarkable capabilities remain limited to a few high-resource languages. Additionally, many low-resource languages (e.g. African languages) are often evaluated only on basic text classification tasks due to the lack of appropriate or comprehensive benchmarks outside of high-resource languages. In this paper, we introduce IrokoB…
▽ More
Despite the widespread adoption of Large language models (LLMs), their remarkable capabilities remain limited to a few high-resource languages. Additionally, many low-resource languages (e.g. African languages) are often evaluated only on basic text classification tasks due to the lack of appropriate or comprehensive benchmarks outside of high-resource languages. In this paper, we introduce IrokoBench -- a human-translated benchmark dataset for 16 typologically-diverse low-resource African languages covering three tasks: natural language inference~(AfriXNLI), mathematical reasoning~(AfriMGSM), and multi-choice knowledge-based QA~(AfriMMLU). We use IrokoBench to evaluate zero-shot, few-shot, and translate-test settings~(where test sets are translated into English) across 10 open and four proprietary LLMs. Our evaluation reveals a significant performance gap between high-resource languages~(such as English and French) and low-resource African languages. We observe a significant performance gap between open and proprietary models, with the highest performing open model, Aya-101 only at 58\% of the best-performing proprietary model GPT-4o performance. Machine translating the test set to English before evaluation helped to close the gap for larger models that are English-centric, like LLaMa 3 70B. These findings suggest that more efforts are needed to develop and adapt LLMs for African languages.
△ Less
Submitted 5 June, 2024;
originally announced June 2024.
-
Kreyòl-MT: Building MT for Latin American, Caribbean and Colonial African Creole Languages
Authors:
Nathaniel R. Robinson,
Raj Dabre,
Ammon Shurtz,
Rasul Dent,
Onenamiyi Onesi,
Claire Bizon Monroc,
Loïc Grobol,
Hasan Muhammad,
Ashi Garg,
Naome A. Etori,
Vijay Murari Tiyyala,
Olanrewaju Samuel,
Matthew Dean Stutzman,
Bismarck Bamfo Odoom,
Sanjeev Khudanpur,
Stephen D. Richardson,
Kenton Murray
Abstract:
A majority of language technologies are tailored for a small number of high-resource languages, while relatively many low-resource languages are neglected. One such group, Creole languages, have long been marginalized in academic study, though their speakers could benefit from machine translation (MT). These languages are predominantly used in much of Latin America, Africa and the Caribbean. We pr…
▽ More
A majority of language technologies are tailored for a small number of high-resource languages, while relatively many low-resource languages are neglected. One such group, Creole languages, have long been marginalized in academic study, though their speakers could benefit from machine translation (MT). These languages are predominantly used in much of Latin America, Africa and the Caribbean. We present the largest cumulative dataset to date for Creole language MT, including 14.5M unique Creole sentences with parallel translations -- 11.6M of which we release publicly, and the largest bitexts gathered to date for 41 languages -- the first ever for 21. In addition, we provide MT models supporting all 41 Creole languages in 172 translation directions. Given our diverse dataset, we produce a model for Creole language MT exposed to more genre diversity than ever before, which outperforms a genre-specific Creole MT model on its own benchmark for 26 of 34 translation directions.
△ Less
Submitted 13 May, 2024; v1 submitted 8 May, 2024;
originally announced May 2024.
-
SemEval-2024 Task 1: Semantic Textual Relatedness for African and Asian Languages
Authors:
Nedjma Ousidhoum,
Shamsuddeen Hassan Muhammad,
Mohamed Abdalla,
Idris Abdulmumin,
Ibrahim Said Ahmad,
Sanchit Ahuja,
Alham Fikri Aji,
Vladimir Araujo,
Meriem Beloucif,
Christine De Kock,
Oumaima Hourrane,
Manish Shrivastava,
Thamar Solorio,
Nirmal Surange,
Krishnapriya Vishnubhotla,
Seid Muhie Yimam,
Saif M. Mohammad
Abstract:
We present the first shared task on Semantic Textual Relatedness (STR). While earlier shared tasks primarily focused on semantic similarity, we instead investigate the broader phenomenon of semantic relatedness across 14 languages: Afrikaans, Algerian Arabic, Amharic, English, Hausa, Hindi, Indonesian, Kinyarwanda, Marathi, Moroccan Arabic, Modern Standard Arabic, Punjabi, Spanish, and Telugu. The…
▽ More
We present the first shared task on Semantic Textual Relatedness (STR). While earlier shared tasks primarily focused on semantic similarity, we instead investigate the broader phenomenon of semantic relatedness across 14 languages: Afrikaans, Algerian Arabic, Amharic, English, Hausa, Hindi, Indonesian, Kinyarwanda, Marathi, Moroccan Arabic, Modern Standard Arabic, Punjabi, Spanish, and Telugu. These languages originate from five distinct language families and are predominantly spoken in Africa and Asia -- regions characterised by the relatively limited availability of NLP resources. Each instance in the datasets is a sentence pair associated with a score that represents the degree of semantic textual relatedness between the two sentences. Participating systems were asked to rank sentence pairs by their closeness in meaning (i.e., their degree of semantic relatedness) in the 14 languages in three main tracks: (a) supervised, (b) unsupervised, and (c) crosslingual. The task attracted 163 participants. We received 70 submissions in total (across all tasks) from 51 different teams, and 38 system description papers. We report on the best-performing systems as well as the most common and the most effective approaches for the three different tracks.
△ Less
Submitted 17 April, 2024; v1 submitted 27 March, 2024;
originally announced March 2024.
-
SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languages
Authors:
Nedjma Ousidhoum,
Shamsuddeen Hassan Muhammad,
Mohamed Abdalla,
Idris Abdulmumin,
Ibrahim Said Ahmad,
Sanchit Ahuja,
Alham Fikri Aji,
Vladimir Araujo,
Abinew Ali Ayele,
Pavan Baswani,
Meriem Beloucif,
Chris Biemann,
Sofia Bourhim,
Christine De Kock,
Genet Shanko Dekebo,
Oumaima Hourrane,
Gopichand Kanumolu,
Lokesh Madasu,
Samuel Rutunda,
Manish Shrivastava,
Thamar Solorio,
Nirmal Surange,
Hailegnaw Getaneh Tilaye,
Krishnapriya Vishnubhotla,
Genta Winata
, et al. (2 additional authors not shown)
Abstract:
Exploring and quantifying semantic relatedness is central to representing language and holds significant implications across various NLP tasks. While earlier NLP research primarily focused on semantic similarity, often within the English language context, we instead investigate the broader phenomenon of semantic relatedness. In this paper, we present \textit{SemRel}, a new semantic relatedness dat…
▽ More
Exploring and quantifying semantic relatedness is central to representing language and holds significant implications across various NLP tasks. While earlier NLP research primarily focused on semantic similarity, often within the English language context, we instead investigate the broader phenomenon of semantic relatedness. In this paper, we present \textit{SemRel}, a new semantic relatedness dataset collection annotated by native speakers across 13 languages: \textit{Afrikaans, Algerian Arabic, Amharic, English, Hausa, Hindi, Indonesian, Kinyarwanda, Marathi, Moroccan Arabic, Modern Standard Arabic, Spanish,} and \textit{Telugu}. These languages originate from five distinct language families and are predominantly spoken in Africa and Asia -- regions characterised by a relatively limited availability of NLP resources. Each instance in the SemRel datasets is a sentence pair associated with a score that represents the degree of semantic textual relatedness between the two sentences. The scores are obtained using a comparative annotation framework. We describe the data collection and annotation processes, challenges when building the datasets, baseline experiments, and their impact and utility in NLP.
△ Less
Submitted 31 May, 2024; v1 submitted 13 February, 2024;
originally announced February 2024.
-
Analyzing COVID-19 Vaccination Sentiments in Nigerian Cyberspace: Insights from a Manually Annotated Twitter Dataset
Authors:
Ibrahim Said Ahmad,
Lukman Jibril Aliyu,
Abubakar Auwal Khalid,
Saminu Muhammad Aliyu,
Shamsuddeen Hassan Muhammad,
Idris Abdulmumin,
Bala Mairiga Abduljalil,
Bello Shehu Bello,
Amina Imam Abubakar
Abstract:
Numerous successes have been achieved in combating the COVID-19 pandemic, initially using various precautionary measures like lockdowns, social distancing, and the use of face masks. More recently, various vaccinations have been developed to aid in the prevention or reduction of the severity of the COVID-19 infection. Despite the effectiveness of the precautionary measures and the vaccines, there…
▽ More
Numerous successes have been achieved in combating the COVID-19 pandemic, initially using various precautionary measures like lockdowns, social distancing, and the use of face masks. More recently, various vaccinations have been developed to aid in the prevention or reduction of the severity of the COVID-19 infection. Despite the effectiveness of the precautionary measures and the vaccines, there are several controversies that are massively shared on social media platforms like Twitter. In this paper, we explore the use of state-of-the-art transformer-based language models to study people's acceptance of vaccines in Nigeria. We developed a novel dataset by crawling multi-lingual tweets using relevant hashtags and keywords. Our analysis and visualizations revealed that most tweets expressed neutral sentiments about COVID-19 vaccines, with some individuals expressing positive views, and there was no strong preference for specific vaccine types, although Moderna received slightly more positive sentiment. We also found out that fine-tuning a pre-trained LLM with an appropriate dataset can yield competitive results, even if the LLM was not initially pre-trained on the specific language of that dataset.
△ Less
Submitted 23 January, 2024;
originally announced January 2024.
-
Leveraging Closed-Access Multilingual Embedding for Automatic Sentence Alignment in Low Resource Languages
Authors:
Idris Abdulmumin,
Auwal Abubakar Khalid,
Shamsuddeen Hassan Muhammad,
Ibrahim Said Ahmad,
Lukman Jibril Aliyu,
Babangida Sani,
Bala Mairiga Abduljalil,
Sani Ahmad Hassan
Abstract:
The importance of qualitative parallel data in machine translation has long been determined but it has always been very difficult to obtain such in sufficient quantity for the majority of world languages, mainly because of the associated cost and also the lack of accessibility to these languages. Despite the potential for obtaining parallel datasets from online articles using automatic approaches,…
▽ More
The importance of qualitative parallel data in machine translation has long been determined but it has always been very difficult to obtain such in sufficient quantity for the majority of world languages, mainly because of the associated cost and also the lack of accessibility to these languages. Despite the potential for obtaining parallel datasets from online articles using automatic approaches, forensic investigations have found a lot of quality-related issues such as misalignment, and wrong language codes. In this work, we present a simple but qualitative parallel sentence aligner that carefully leveraged the closed-access Cohere multilingual embedding, a solution that ranked second in the just concluded #CoHereAIHack 2023 Challenge (see https://ai6lagos.devpost.com). The proposed approach achieved $94.96$ and $54.83$ f1 scores on FLORES and MAFAND-MT, compared to $3.64$ and $0.64$ of LASER respectively. Our method also achieved an improvement of more than 5 BLEU scores over LASER, when the resulting datasets were used with MAFAND-MT dataset to train translation models. Our code and data are available for research purposes here (https://github.com/abumafrim/Cohere-Align).
△ Less
Submitted 20 November, 2023;
originally announced November 2023.
-
AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages
Authors:
Jiayi Wang,
David Ifeoluwa Adelani,
Sweta Agrawal,
Marek Masiak,
Ricardo Rei,
Eleftheria Briakou,
Marine Carpuat,
Xuanli He,
Sofia Bourhim,
Andiswa Bukula,
Muhidin Mohamed,
Temitayo Olatoye,
Tosin Adewumi,
Hamam Mokayed,
Christine Mwase,
Wangui Kimotho,
Foutse Yuehgoh,
Anuoluwapo Aremu,
Jessica Ojo,
Shamsuddeen Hassan Muhammad,
Salomey Osei,
Abdul-Hakeem Omotayo,
Chiamaka Chukwuneke,
Perez Ogayo,
Oumaima Hourrane
, et al. (33 additional authors not shown)
Abstract:
Despite the recent progress on scaling multilingual machine translation (MT) to several under-resourced African languages, accurately measuring this progress remains challenging, since evaluation is often performed on n-gram matching metrics such as BLEU, which typically show a weaker correlation with human judgments. Learned metrics such as COMET have higher correlation; however, the lack of eval…
▽ More
Despite the recent progress on scaling multilingual machine translation (MT) to several under-resourced African languages, accurately measuring this progress remains challenging, since evaluation is often performed on n-gram matching metrics such as BLEU, which typically show a weaker correlation with human judgments. Learned metrics such as COMET have higher correlation; however, the lack of evaluation data with human ratings for under-resourced languages, complexity of annotation guidelines like Multidimensional Quality Metrics (MQM), and limited language coverage of multilingual encoders have hampered their applicability to African languages. In this paper, we address these challenges by creating high-quality human evaluation data with simplified MQM guidelines for error detection and direct assessment (DA) scoring for 13 typologically diverse African languages. Furthermore, we develop AfriCOMET: COMET evaluation metrics for African languages by leveraging DA data from well-resourced languages and an African-centric multilingual encoder (AfroXLM-R) to create the state-of-the-art MT evaluation metrics for African languages with respect to Spearman-rank correlation with human judgments (0.441).
△ Less
Submitted 23 April, 2024; v1 submitted 16 November, 2023;
originally announced November 2023.
-
NaijaRC: A Multi-choice Reading Comprehension Dataset for Nigerian Languages
Authors:
Anuoluwapo Aremu,
Jesujoba O. Alabi,
Daud Abolade,
Nkechinyere F. Aguobi,
Shamsuddeen Hassan Muhammad,
David Ifeoluwa Adelani
Abstract:
In this paper, we create NaijaRC: a new multi-choice Reading Comprehension dataset for three native Nigeria languages that is based on high-school reading comprehension examination. We provide baseline results by performing cross-lingual transfer using existing English RACE and Belebele training dataset based on a pre-trained encoder-only model. Additionally, we provide results by prompting large…
▽ More
In this paper, we create NaijaRC: a new multi-choice Reading Comprehension dataset for three native Nigeria languages that is based on high-school reading comprehension examination. We provide baseline results by performing cross-lingual transfer using existing English RACE and Belebele training dataset based on a pre-trained encoder-only model. Additionally, we provide results by prompting large language models (LLMs) like GPT-4.
△ Less
Submitted 19 May, 2024; v1 submitted 18 August, 2023;
originally announced August 2023.
-
HaVQA: A Dataset for Visual Question Answering and Multimodal Research in Hausa Language
Authors:
Shantipriya Parida,
Idris Abdulmumin,
Shamsuddeen Hassan Muhammad,
Aneesh Bose,
Guneet Singh Kohli,
Ibrahim Said Ahmad,
Ketan Kotwal,
Sayan Deb Sarkar,
Ondřej Bojar,
Habeebah Adamu Kakudi
Abstract:
This paper presents HaVQA, the first multimodal dataset for visual question-answering (VQA) tasks in the Hausa language. The dataset was created by manually translating 6,022 English question-answer pairs, which are associated with 1,555 unique images from the Visual Genome dataset. As a result, the dataset provides 12,044 gold standard English-Hausa parallel sentences that were translated in a fa…
▽ More
This paper presents HaVQA, the first multimodal dataset for visual question-answering (VQA) tasks in the Hausa language. The dataset was created by manually translating 6,022 English question-answer pairs, which are associated with 1,555 unique images from the Visual Genome dataset. As a result, the dataset provides 12,044 gold standard English-Hausa parallel sentences that were translated in a fashion that guarantees their semantic match with the corresponding visual information. We conducted several baseline experiments on the dataset, including visual question answering, visual question elicitation, text-only and multimodal machine translation.
△ Less
Submitted 28 May, 2023;
originally announced May 2023.
-
MasakhaPOS: Part-of-Speech Tagging for Typologically Diverse African Languages
Authors:
Cheikh M. Bamba Dione,
David Adelani,
Peter Nabende,
Jesujoba Alabi,
Thapelo Sindane,
Happy Buzaaba,
Shamsuddeen Hassan Muhammad,
Chris Chinenye Emezue,
Perez Ogayo,
Anuoluwapo Aremu,
Catherine Gitau,
Derguene Mbaye,
Jonathan Mukiibi,
Blessing Sibanda,
Bonaventure F. P. Dossou,
Andiswa Bukula,
Rooweither Mabuya,
Allahsera Auguste Tapo,
Edwin Munkoh-Buabeng,
victoire Memdjokam Koagne,
Fatoumata Ouoba Kabore,
Amelia Taylor,
Godson Kalipe,
Tebogo Macucwa,
Vukosi Marivate
, et al. (19 additional authors not shown)
Abstract:
In this paper, we present MasakhaPOS, the largest part-of-speech (POS) dataset for 20 typologically diverse African languages. We discuss the challenges in annotating POS for these languages using the UD (universal dependencies) guidelines. We conducted extensive POS baseline experiments using conditional random field and several multilingual pre-trained language models. We applied various cross-l…
▽ More
In this paper, we present MasakhaPOS, the largest part-of-speech (POS) dataset for 20 typologically diverse African languages. We discuss the challenges in annotating POS for these languages using the UD (universal dependencies) guidelines. We conducted extensive POS baseline experiments using conditional random field and several multilingual pre-trained language models. We applied various cross-lingual transfer models trained with data available in UD. Evaluating on the MasakhaPOS dataset, we show that choosing the best transfer language(s) in both single-source and multi-source setups greatly improves the POS tagging performance of the target languages, in particular when combined with cross-lingual parameter-efficient fine-tuning methods. Crucially, transferring knowledge from a language that matches the language family and morphosyntactic properties seems more effective for POS tagging in unseen languages.
△ Less
Submitted 23 May, 2023;
originally announced May 2023.
-
AfriQA: Cross-lingual Open-Retrieval Question Answering for African Languages
Authors:
Odunayo Ogundepo,
Tajuddeen R. Gwadabe,
Clara E. Rivera,
Jonathan H. Clark,
Sebastian Ruder,
David Ifeoluwa Adelani,
Bonaventure F. P. Dossou,
Abdou Aziz DIOP,
Claytone Sikasote,
Gilles Hacheme,
Happy Buzaaba,
Ignatius Ezeani,
Rooweither Mabuya,
Salomey Osei,
Chris Emezue,
Albert Njoroge Kahira,
Shamsuddeen H. Muhammad,
Akintunde Oladipo,
Abraham Toluwase Owodunni,
Atnafu Lambebo Tonja,
Iyanuoluwa Shode,
Akari Asai,
Tunde Oluwaseyi Ajayi,
Clemencia Siro,
Steven Arthur
, et al. (27 additional authors not shown)
Abstract:
African languages have far less in-language content available digitally, making it challenging for question answering systems to satisfy the information needs of users. Cross-lingual open-retrieval question answering (XOR QA) systems -- those that retrieve answer content from other languages while serving people in their native language -- offer a means of filling this gap. To this end, we create…
▽ More
African languages have far less in-language content available digitally, making it challenging for question answering systems to satisfy the information needs of users. Cross-lingual open-retrieval question answering (XOR QA) systems -- those that retrieve answer content from other languages while serving people in their native language -- offer a means of filling this gap. To this end, we create AfriQA, the first cross-lingual QA dataset with a focus on African languages. AfriQA includes 12,000+ XOR QA examples across 10 African languages. While previous datasets have focused primarily on languages where cross-lingual QA augments coverage from the target language, AfriQA focuses on languages where cross-lingual answer content is the only high-coverage source of answer content. Because of this, we argue that African languages are one of the most important and realistic use cases for XOR QA. Our experiments demonstrate the poor performance of automatic translation and multilingual retrieval methods. Overall, AfriQA proves challenging for state-of-the-art QA models. We hope that the dataset enables the development of more equitable QA technology.
△ Less
Submitted 11 May, 2023;
originally announced May 2023.
-
HausaNLP at SemEval-2023 Task 10: Transfer Learning, Synthetic Data and Side-Information for Multi-Level Sexism Classification
Authors:
Saminu Mohammad Aliyu,
Idris Abdulmumin,
Shamsuddeen Hassan Muhammad,
Ibrahim Said Ahmad,
Saheed Abdullahi Salahudeen,
Aliyu Yusuf,
Falalu Ibrahim Lawan
Abstract:
We present the findings of our participation in the SemEval-2023 Task 10: Explainable Detection of Online Sexism (EDOS) task, a shared task on offensive language (sexism) detection on English Gab and Reddit dataset. We investigated the effects of transferring two language models: XLM-T (sentiment classification) and HateBERT (same domain -- Reddit) for multi-level classification into Sexist or not…
▽ More
We present the findings of our participation in the SemEval-2023 Task 10: Explainable Detection of Online Sexism (EDOS) task, a shared task on offensive language (sexism) detection on English Gab and Reddit dataset. We investigated the effects of transferring two language models: XLM-T (sentiment classification) and HateBERT (same domain -- Reddit) for multi-level classification into Sexist or not Sexist, and other subsequent sub-classifications of the sexist data. We also use synthetic classification of unlabelled dataset and intermediary class information to maximize the performance of our models. We submitted a system in Task A, and it ranked 49th with F1-score of 0.82. This result showed to be competitive as it only under-performed the best system by 0.052% F1-score.
△ Less
Submitted 28 April, 2023;
originally announced May 2023.
-
MasakhaNEWS: News Topic Classification for African languages
Authors:
David Ifeoluwa Adelani,
Marek Masiak,
Israel Abebe Azime,
Jesujoba Alabi,
Atnafu Lambebo Tonja,
Christine Mwase,
Odunayo Ogundepo,
Bonaventure F. P. Dossou,
Akintunde Oladipo,
Doreen Nixdorf,
Chris Chinenye Emezue,
sana al-azzawi,
Blessing Sibanda,
Davis David,
Lolwethu Ndolela,
Jonathan Mukiibi,
Tunde Ajayi,
Tatiana Moteu,
Brian Odhiambo,
Abraham Owodunni,
Nnaemeka Obiefuna,
Muhidin Mohamed,
Shamsuddeen Hassan Muhammad,
Teshome Mulugeta Ababu,
Saheed Abdullahi Salahudeen
, et al. (40 additional authors not shown)
Abstract:
African languages are severely under-represented in NLP research due to lack of datasets covering several NLP tasks. While there are individual language specific datasets that are being expanded to different tasks, only a handful of NLP tasks (e.g. named entity recognition and machine translation) have standardized benchmark datasets covering several geographical and typologically-diverse African…
▽ More
African languages are severely under-represented in NLP research due to lack of datasets covering several NLP tasks. While there are individual language specific datasets that are being expanded to different tasks, only a handful of NLP tasks (e.g. named entity recognition and machine translation) have standardized benchmark datasets covering several geographical and typologically-diverse African languages. In this paper, we develop MasakhaNEWS -- a new benchmark dataset for news topic classification covering 16 languages widely spoken in Africa. We provide an evaluation of baseline models by training classical machine learning models and fine-tuning several language models. Furthermore, we explore several alternatives to full fine-tuning of language models that are better suited for zero-shot and few-shot learning such as cross-lingual parameter-efficient fine-tuning (like MAD-X), pattern exploiting training (PET), prompting language models (like ChatGPT), and prompt-free sentence transformer fine-tuning (SetFit and Cohere Embedding API). Our evaluation in zero-shot setting shows the potential of prompting ChatGPT for news topic classification in low-resource African languages, achieving an average performance of 70 F1 points without leveraging additional supervision like MAD-X. In few-shot setting, we show that with as little as 10 examples per label, we achieved more than 90\% (i.e. 86.0 F1 points) of the performance of full supervised training (92.6 F1 points) leveraging the PET approach.
△ Less
Submitted 20 September, 2023; v1 submitted 19 April, 2023;
originally announced April 2023.
-
SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)
Authors:
Shamsuddeen Hassan Muhammad,
Idris Abdulmumin,
Seid Muhie Yimam,
David Ifeoluwa Adelani,
Ibrahim Sa'id Ahmad,
Nedjma Ousidhoum,
Abinew Ayele,
Saif M. Mohammad,
Meriem Beloucif,
Sebastian Ruder
Abstract:
We present the first Africentric SemEval Shared task, Sentiment Analysis for African Languages (AfriSenti-SemEval) - The dataset is available at https://github.com/afrisenti-semeval/afrisent-semeval-2023. AfriSenti-SemEval is a sentiment classification challenge in 14 African languages: Amharic, Algerian Arabic, Hausa, Igbo, Kinyarwanda, Moroccan Arabic, Mozambican Portuguese, Nigerian Pidgin, Oro…
▽ More
We present the first Africentric SemEval Shared task, Sentiment Analysis for African Languages (AfriSenti-SemEval) - The dataset is available at https://github.com/afrisenti-semeval/afrisent-semeval-2023. AfriSenti-SemEval is a sentiment classification challenge in 14 African languages: Amharic, Algerian Arabic, Hausa, Igbo, Kinyarwanda, Moroccan Arabic, Mozambican Portuguese, Nigerian Pidgin, Oromo, Swahili, Tigrinya, Twi, Xitsonga, and Yorùbá (Muhammad et al., 2023), using data labeled with 3 sentiment classes. We present three subtasks: (1) Task A: monolingual classification, which received 44 submissions; (2) Task B: multilingual classification, which received 32 submissions; and (3) Task C: zero-shot classification, which received 34 submissions. The best performance for tasks A and B was achieved by NLNDE team with 71.31 and 75.06 weighted F1, respectively. UCAS-IIE-NLP achieved the best average score for task C with 58.15 weighted F1. We describe the various approaches adopted by the top 10 systems and their approaches.
△ Less
Submitted 1 May, 2023; v1 submitted 13 April, 2023;
originally announced April 2023.
-
AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages
Authors:
Shamsuddeen Hassan Muhammad,
Idris Abdulmumin,
Abinew Ali Ayele,
Nedjma Ousidhoum,
David Ifeoluwa Adelani,
Seid Muhie Yimam,
Ibrahim Sa'id Ahmad,
Meriem Beloucif,
Saif M. Mohammad,
Sebastian Ruder,
Oumaima Hourrane,
Pavel Brazdil,
Felermino Dário Mário António Ali,
Davis David,
Salomey Osei,
Bello Shehu Bello,
Falalu Ibrahim,
Tajuddeen Gwadabe,
Samuel Rutunda,
Tadesse Belay,
Wendimu Baye Messelle,
Hailu Beshada Balcha,
Sisay Adugna Chala,
Hagos Tesfahun Gebremichael,
Bernard Opoku
, et al. (1 additional authors not shown)
Abstract:
Africa is home to over 2,000 languages from more than six language families and has the highest linguistic diversity among all continents. These include 75 languages with at least one million speakers each. Yet, there is little NLP research conducted on African languages. Crucial to enabling such research is the availability of high-quality annotated datasets. In this paper, we introduce AfriSenti…
▽ More
Africa is home to over 2,000 languages from more than six language families and has the highest linguistic diversity among all continents. These include 75 languages with at least one million speakers each. Yet, there is little NLP research conducted on African languages. Crucial to enabling such research is the availability of high-quality annotated datasets. In this paper, we introduce AfriSenti, a sentiment analysis benchmark that contains a total of >110,000 tweets in 14 African languages (Amharic, Algerian Arabic, Hausa, Igbo, Kinyarwanda, Moroccan Arabic, Mozambican Portuguese, Nigerian Pidgin, Oromo, Swahili, Tigrinya, Twi, Xitsonga, and Yorùbá) from four language families. The tweets were annotated by native speakers and used in the AfriSenti-SemEval shared task (The AfriSenti Shared Task had over 200 participants. See website at https://afrisenti-semeval.github.io). We describe the data collection methodology, annotation process, and the challenges we dealt with when curating each dataset. We further report baseline experiments conducted on the different datasets and discuss their usefulness.
△ Less
Submitted 4 November, 2023; v1 submitted 17 February, 2023;
originally announced February 2023.
-
HERDPhobia: A Dataset for Hate Speech against Fulani in Nigeria
Authors:
Saminu Mohammad Aliyu,
Gregory Maksha Wajiga,
Muhammad Murtala,
Shamsuddeen Hassan Muhammad,
Idris Abdulmumin,
Ibrahim Said Ahmad
Abstract:
Social media platforms allow users to freely share their opinions about issues or anything they feel like. However, they also make it easier to spread hate and abusive content. The Fulani ethnic group has been the victim of this unfortunate phenomenon. This paper introduces the HERDPhobia - the first annotated hate speech dataset on Fulani herders in Nigeria - in three languages: English, Nigerian…
▽ More
Social media platforms allow users to freely share their opinions about issues or anything they feel like. However, they also make it easier to spread hate and abusive content. The Fulani ethnic group has been the victim of this unfortunate phenomenon. This paper introduces the HERDPhobia - the first annotated hate speech dataset on Fulani herders in Nigeria - in three languages: English, Nigerian-Pidgin, and Hausa. We present a benchmark experiment using pre-trained languages models to classify the tweets as either hateful or non-hateful. Our experiment shows that the XML-T model provides better performance with 99.83% weighted F1. We released the dataset at https://github.com/hausanlp/HERDPhobia for further research.
△ Less
Submitted 28 November, 2022;
originally announced November 2022.
-
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Authors:
BigScience Workshop,
:,
Teven Le Scao,
Angela Fan,
Christopher Akiki,
Ellie Pavlick,
Suzana Ilić,
Daniel Hesslow,
Roman Castagné,
Alexandra Sasha Luccioni,
François Yvon,
Matthias Gallé,
Jonathan Tow,
Alexander M. Rush,
Stella Biderman,
Albert Webson,
Pawan Sasanka Ammanamanchi,
Thomas Wang,
Benoît Sagot,
Niklas Muennighoff,
Albert Villanova del Moral,
Olatunji Ruwase,
Rachel Bawden,
Stas Bekman,
Angelina McMillan-Major
, et al. (369 additional authors not shown)
Abstract:
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access…
▽ More
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages (59 in total). We find that BLOOM achieves competitive performance on a wide variety of benchmarks, with stronger results after undergoing multitask prompted finetuning. To facilitate future research and applications using LLMs, we publicly release our models and code under the Responsible AI License.
△ Less
Submitted 27 June, 2023; v1 submitted 9 November, 2022;
originally announced November 2022.
-
MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition
Authors:
David Ifeoluwa Adelani,
Graham Neubig,
Sebastian Ruder,
Shruti Rijhwani,
Michael Beukman,
Chester Palen-Michel,
Constantine Lignos,
Jesujoba O. Alabi,
Shamsuddeen H. Muhammad,
Peter Nabende,
Cheikh M. Bamba Dione,
Andiswa Bukula,
Rooweither Mabuya,
Bonaventure F. P. Dossou,
Blessing Sibanda,
Happy Buzaaba,
Jonathan Mukiibi,
Godson Kalipe,
Derguene Mbaye,
Amelia Taylor,
Fatoumata Kabore,
Chris Chinenye Emezue,
Anuoluwapo Aremu,
Perez Ogayo,
Catherine Gitau
, et al. (20 additional authors not shown)
Abstract:
African languages are spoken by over a billion people, but are underrepresented in NLP research and development. The challenges impeding progress include the limited availability of annotated datasets, as well as a lack of understanding of the settings where current methods are effective. In this paper, we make progress towards solutions for these challenges, focusing on the task of named entity r…
▽ More
African languages are spoken by over a billion people, but are underrepresented in NLP research and development. The challenges impeding progress include the limited availability of annotated datasets, as well as a lack of understanding of the settings where current methods are effective. In this paper, we make progress towards solutions for these challenges, focusing on the task of named entity recognition (NER). We create the largest human-annotated NER dataset for 20 African languages, and we study the behavior of state-of-the-art cross-lingual transfer methods in an Africa-centric setting, demonstrating that the choice of source language significantly affects performance. We show that choosing the best transfer language improves zero-shot F1 scores by an average of 14 points across 20 languages compared to using English. Our results highlight the need for benchmark datasets and models that cover typologically-diverse African languages.
△ Less
Submitted 15 November, 2022; v1 submitted 22 October, 2022;
originally announced October 2022.
-
Separating Grains from the Chaff: Using Data Filtering to Improve Multilingual Translation for Low-Resourced African Languages
Authors:
Idris Abdulmumin,
Michael Beukman,
Jesujoba O. Alabi,
Chris Emezue,
Everlyn Asiko,
Tosin Adewumi,
Shamsuddeen Hassan Muhammad,
Mofetoluwa Adeyemi,
Oreen Yousuf,
Sahib Singh,
Tajuddeen Rabiu Gwadabe
Abstract:
We participated in the WMT 2022 Large-Scale Machine Translation Evaluation for the African Languages Shared Task. This work describes our approach, which is based on filtering the given noisy data using a sentence-pair classifier that was built by fine-tuning a pre-trained language model. To train the classifier, we obtain positive samples (i.e. high-quality parallel sentences) from a gold-standar…
▽ More
We participated in the WMT 2022 Large-Scale Machine Translation Evaluation for the African Languages Shared Task. This work describes our approach, which is based on filtering the given noisy data using a sentence-pair classifier that was built by fine-tuning a pre-trained language model. To train the classifier, we obtain positive samples (i.e. high-quality parallel sentences) from a gold-standard curated dataset and extract negative samples (i.e. low-quality parallel sentences) from automatically aligned parallel data by choosing sentences with low alignment scores. Our final machine translation model was then trained on filtered data, instead of the entire noisy dataset. We empirically validate our approach by evaluating on two common datasets and show that data filtering generally improves overall translation quality, in some cases even significantly.
△ Less
Submitted 20 October, 2022; v1 submitted 19 October, 2022;
originally announced October 2022.
-
Deep Sequence Models for Text Classification Tasks
Authors:
Saheed Salahudeen Abdullahi,
Sun Yiming,
Shamsuddeen Hassan Muhammad,
Abdulrasheed Mustapha,
Ahmad Muhammad Aminu,
Abdulkadir Abdullahi,
Musa Bello,
Saminu Mohammad Aliyu
Abstract:
The exponential growth of data generated on the Internet in the current information age is a driving force for the digital economy. Extraction of information is the major value in an accumulated big data. Big data dependency on statistical analysis and hand-engineered rules machine learning algorithms are overwhelmed with vast complexities inherent in human languages. Natural Language Processing (…
▽ More
The exponential growth of data generated on the Internet in the current information age is a driving force for the digital economy. Extraction of information is the major value in an accumulated big data. Big data dependency on statistical analysis and hand-engineered rules machine learning algorithms are overwhelmed with vast complexities inherent in human languages. Natural Language Processing (NLP) is equipping machines to understand these human diverse and complicated languages. Text Classification is an NLP task which automatically identifies patterns based on predefined or undefined labeled sets. Common text classification application includes information retrieval, modeling news topic, theme extraction, sentiment analysis, and spam detection. In texts, some sequences of words depend on the previous or next word sequences to make full meaning; this is a challenging dependency task that requires the machine to be able to store some previous important information to impact future meaning. Sequence models such as RNN, GRU, and LSTM is a breakthrough for tasks with long-range dependencies. As such, we applied these models to Binary and Multi-class classification. Results generated were excellent with most of the models performing within the range of 80% and 94%. However, this result is not exhaustive as we believe there is room for improvement if machines are to compete with humans.
△ Less
Submitted 18 July, 2022;
originally announced July 2022.
-
FontNet: Closing the gap to font designer performance in font synthesis
Authors:
Ammar Ul Hassan Muhammad,
Jaeyoung Choi
Abstract:
Font synthesis has been a very active topic in recent years because manual font design requires domain expertise and is a labor-intensive and time-consuming job. While remarkably successful, existing methods for font synthesis have major shortcomings; they require finetuning for unobserved font style with large reference images, the recent few-shot font synthesis methods are either designed for sp…
▽ More
Font synthesis has been a very active topic in recent years because manual font design requires domain expertise and is a labor-intensive and time-consuming job. While remarkably successful, existing methods for font synthesis have major shortcomings; they require finetuning for unobserved font style with large reference images, the recent few-shot font synthesis methods are either designed for specific language systems or they operate on low-resolution images which limits their use. In this paper, we tackle this font synthesis problem by learning the font style in the embedding space. To this end, we propose a model, called FontNet, that simultaneously learns to separate font styles in the embedding space where distances directly correspond to a measure of font similarity, and translates input images into the given observed or unobserved font style. Additionally, we design the network architecture and training procedure that can be adopted for any language system and can produce high-resolution font images. Thanks to this approach, our proposed method outperforms the existing state-of-the-art font generation methods on both qualitative and quantitative experiments.
△ Less
Submitted 13 May, 2022;
originally announced May 2022.
-
A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation
Authors:
David Ifeoluwa Adelani,
Jesujoba Oluwadara Alabi,
Angela Fan,
Julia Kreutzer,
Xiaoyu Shen,
Machel Reid,
Dana Ruiter,
Dietrich Klakow,
Peter Nabende,
Ernie Chang,
Tajuddeen Gwadabe,
Freshia Sackey,
Bonaventure F. P. Dossou,
Chris Chinenye Emezue,
Colin Leong,
Michael Beukman,
Shamsuddeen Hassan Muhammad,
Guyo Dub Jarso,
Oreen Yousuf,
Andre Niyongabo Rubungo,
Gilles Hacheme,
Eric Peter Wairagala,
Muhammad Umair Nasir,
Benjamin Ayoade Ajibade,
Tunde Oluwaseyi Ajayi
, et al. (20 additional authors not shown)
Abstract:
Recent advances in the pre-training of language models leverage large-scale datasets to create multilingual models. However, low-resource languages are mostly left out in these datasets. This is primarily because many widely spoken languages are not well represented on the web and therefore excluded from the large-scale crawls used to create datasets. Furthermore, downstream users of these models…
▽ More
Recent advances in the pre-training of language models leverage large-scale datasets to create multilingual models. However, low-resource languages are mostly left out in these datasets. This is primarily because many widely spoken languages are not well represented on the web and therefore excluded from the large-scale crawls used to create datasets. Furthermore, downstream users of these models are restricted to the selection of languages originally chosen for pre-training. This work investigates how to optimally leverage existing pre-trained models to create low-resource translation systems for 16 African languages. We focus on two questions: 1) How can pre-trained models be used for languages not included in the initial pre-training? and 2) How can the resulting translation models effectively transfer to new domains? To answer these questions, we create a new African news corpus covering 16 languages, of which eight languages are not part of any existing evaluation dataset. We demonstrate that the most effective strategy for transferring both to additional languages and to additional domains is to fine-tune large pre-trained models on small quantities of high-quality translation data.
△ Less
Submitted 22 August, 2022; v1 submitted 4 May, 2022;
originally announced May 2022.
-
Hausa Visual Genome: A Dataset for Multi-Modal English to Hausa Machine Translation
Authors:
Idris Abdulmumin,
Satya Ranjan Dash,
Musa Abdullahi Dawud,
Shantipriya Parida,
Shamsuddeen Hassan Muhammad,
Ibrahim Sa'id Ahmad,
Subhadarshi Panda,
Ondřej Bojar,
Bashir Shehu Galadanci,
Bello Shehu Bello
Abstract:
Multi-modal Machine Translation (MMT) enables the use of visual information to enhance the quality of translations. The visual information can serve as a valuable piece of context information to decrease the ambiguity of input sentences. Despite the increasing popularity of such a technique, good and sizeable datasets are scarce, limiting the full extent of their potential. Hausa, a Chadic languag…
▽ More
Multi-modal Machine Translation (MMT) enables the use of visual information to enhance the quality of translations. The visual information can serve as a valuable piece of context information to decrease the ambiguity of input sentences. Despite the increasing popularity of such a technique, good and sizeable datasets are scarce, limiting the full extent of their potential. Hausa, a Chadic language, is a member of the Afro-Asiatic language family. It is estimated that about 100 to 150 million people speak the language, with more than 80 million indigenous speakers. This is more than any of the other Chadic languages. Despite a large number of speakers, the Hausa language is considered low-resource in natural language processing (NLP). This is due to the absence of sufficient resources to implement most NLP tasks. While some datasets exist, they are either scarce, machine-generated, or in the religious domain. Therefore, there is a need to create training and evaluation data for implementing machine learning tasks and bridging the research gap in the language. This work presents the Hausa Visual Genome (HaVG), a dataset that contains the description of an image or a section within the image in Hausa and its equivalent in English. To prepare the dataset, we started by translating the English description of the images in the Hindi Visual Genome (HVG) into Hausa automatically. Afterward, the synthetic Hausa data was carefully post-edited considering the respective images. The dataset comprises 32,923 images and their descriptions that are divided into training, development, test, and challenge test set. The Hausa Visual Genome is the first dataset of its kind and can be used for Hausa-English machine translation, multi-modal research, and image description, among various other natural language processing and generation tasks.
△ Less
Submitted 6 May, 2022; v1 submitted 2 May, 2022;
originally announced May 2022.
-
NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysis
Authors:
Shamsuddeen Hassan Muhammad,
David Ifeoluwa Adelani,
Sebastian Ruder,
Ibrahim Said Ahmad,
Idris Abdulmumin,
Bello Shehu Bello,
Monojit Choudhury,
Chris Chinenye Emezue,
Saheed Salahudeen Abdullahi,
Anuoluwapo Aremu,
Alipio Jeorge,
Pavel Brazdil
Abstract:
Sentiment analysis is one of the most widely studied applications in NLP, but most work focuses on languages with large amounts of data. We introduce the first large-scale human-annotated Twitter sentiment dataset for the four most widely spoken languages in Nigeria (Hausa, Igbo, Nigerian-Pidgin, and Yorùbá ) consisting of around 30,000 annotated tweets per language (and 14,000 for Nigerian-Pidgin…
▽ More
Sentiment analysis is one of the most widely studied applications in NLP, but most work focuses on languages with large amounts of data. We introduce the first large-scale human-annotated Twitter sentiment dataset for the four most widely spoken languages in Nigeria (Hausa, Igbo, Nigerian-Pidgin, and Yorùbá ) consisting of around 30,000 annotated tweets per language (and 14,000 for Nigerian-Pidgin), including a significant fraction of code-mixed tweets. We propose text collection, filtering, processing and labeling methods that enable us to create datasets for these low-resource languages. We evaluate a rangeof pre-trained models and transfer strategies on the dataset. We find that language-specific models and language-adaptivefine-tuning generally perform best. We release the datasets, trained models, sentiment lexicons, and code to incentivizeresearch on sentiment analysis in under-represented languages.
△ Less
Submitted 18 June, 2022; v1 submitted 20 January, 2022;
originally announced January 2022.
-
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
Authors:
Julia Kreutzer,
Isaac Caswell,
Lisa Wang,
Ahsan Wahab,
Daan van Esch,
Nasanbayar Ulzii-Orshikh,
Allahsera Tapo,
Nishant Subramani,
Artem Sokolov,
Claytone Sikasote,
Monang Setyawan,
Supheakmungkol Sarin,
Sokhar Samb,
Benoît Sagot,
Clara Rivera,
Annette Rios,
Isabel Papadimitriou,
Salomey Osei,
Pedro Ortiz Suarez,
Iroro Orife,
Kelechi Ogueji,
Andre Niyongabo Rubungo,
Toan Q. Nguyen,
Mathias Müller,
André Müller
, et al. (27 additional authors not shown)
Abstract:
With the success of large-scale pre-training and multilingual modeling in Natural Language Processing (NLP), recent years have seen a proliferation of large, web-mined text datasets covering hundreds of languages. We manually audit the quality of 205 language-specific corpora released with five major public datasets (CCAligned, ParaCrawl, WikiMatrix, OSCAR, mC4). Lower-resource corpora have system…
▽ More
With the success of large-scale pre-training and multilingual modeling in Natural Language Processing (NLP), recent years have seen a proliferation of large, web-mined text datasets covering hundreds of languages. We manually audit the quality of 205 language-specific corpora released with five major public datasets (CCAligned, ParaCrawl, WikiMatrix, OSCAR, mC4). Lower-resource corpora have systematic issues: At least 15 corpora have no usable text, and a significant fraction contains less than 50% sentences of acceptable quality. In addition, many are mislabeled or use nonstandard/ambiguous language codes. We demonstrate that these issues are easy to detect even for non-proficient speakers, and supplement the human audit with automatic analyses. Finally, we recommend techniques to evaluate and improve multilingual corpora and discuss potential risks that come with low-quality data releases.
△ Less
Submitted 21 February, 2022; v1 submitted 22 March, 2021;
originally announced March 2021.
-
EPIC-Survival: End-to-end Part Inferred Clustering for Survival Analysis, Featuring Prognostic Stratification Boosting
Authors:
Hassan Muhammad,
Chensu Xie,
Carlie S. Sigel,
Michael Doukas,
Lindsay Alpert,
William R. Jarnagin,
Amber Simpson,
Thomas J. Fuchs
Abstract:
Histopathology-based survival modelling has two major hurdles. Firstly, a well-performing survival model has minimal clinical application if it does not contribute to the stratification of a cancer patient cohort into different risk groups, preferably driven by histologic morphologies. In the clinical setting, individuals are not given specific prognostic predictions, but are rather predicted to l…
▽ More
Histopathology-based survival modelling has two major hurdles. Firstly, a well-performing survival model has minimal clinical application if it does not contribute to the stratification of a cancer patient cohort into different risk groups, preferably driven by histologic morphologies. In the clinical setting, individuals are not given specific prognostic predictions, but are rather predicted to lie within a risk group which has a general survival trend. Thus, It is imperative that a survival model produces well-stratified risk groups. Secondly, until now, survival modelling was done in a two-stage approach (encoding and aggregation). The massive amount of pixels in digitized whole slide images were never utilized to their fullest extent due to technological constraints on data processing, forcing decoupled learning. EPIC-Survival bridges encoding and aggregation into an end-to-end survival modelling approach, while introducing stratification boosting to encourage the model to not only optimize ranking, but also to discriminate between risk groups. In this study we show that EPIC-Survival performs better than other approaches in modelling intrahepatic cholangiocarcinoma, a historically difficult cancer to model. Further, we show that stratification boosting improves further improves model performance, resulting in a concordance-index of 0.880 on a held-out test set. Finally, we were able to identify specific histologic differences, not commonly sought out in ICC, between low and high risk groups.
△ Less
Submitted 9 July, 2021; v1 submitted 26 January, 2021;
originally announced January 2021.
-
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages
Authors:
Wilhelmina Nekoto,
Vukosi Marivate,
Tshinondiwa Matsila,
Timi Fasubaa,
Tajudeen Kolawole,
Taiwo Fagbohungbe,
Solomon Oluwole Akinola,
Shamsuddeen Hassan Muhammad,
Salomon Kabongo,
Salomey Osei,
Sackey Freshia,
Rubungo Andre Niyongabo,
Ricky Macharm,
Perez Ogayo,
Orevaoghene Ahia,
Musie Meressa,
Mofe Adeyemi,
Masabata Mokgesi-Selinga,
Lawrence Okegbemi,
Laura Jane Martinus,
Kolawole Tajudeen,
Kevin Degila,
Kelechi Ogueji,
Kathleen Siminyu,
Julia Kreutzer
, et al. (23 additional authors not shown)
Abstract:
Research in NLP lacks geographic diversity, and the question of how NLP can be scaled to low-resourced languages has not yet been adequately solved. "Low-resourced"-ness is a complex problem going beyond data availability and reflects systemic problems in society. In this paper, we focus on the task of Machine Translation (MT), that plays a crucial role for information accessibility and communicat…
▽ More
Research in NLP lacks geographic diversity, and the question of how NLP can be scaled to low-resourced languages has not yet been adequately solved. "Low-resourced"-ness is a complex problem going beyond data availability and reflects systemic problems in society. In this paper, we focus on the task of Machine Translation (MT), that plays a crucial role for information accessibility and communication worldwide. Despite immense improvements in MT over the past decade, MT is centered around a few high-resourced languages. As MT researchers cannot solve the problem of low-resourcedness alone, we propose participatory research as a means to involve all necessary agents required in the MT development process. We demonstrate the feasibility and scalability of participatory research with a case study on MT for African languages. Its implementation leads to a collection of novel translation datasets, MT benchmarks for over 30 languages, with human evaluations for a third of them, and enables participants without formal training to make a unique scientific contribution. Benchmarks, models, data, code, and evaluation results are released under https://github.com/masakhane-io/masakhane-mt.
△ Less
Submitted 6 November, 2020; v1 submitted 5 October, 2020;
originally announced October 2020.
-
Towards Unsupervised Cancer Subtyping: Predicting Prognosis Using A Histologic Visual Dictionary
Authors:
Hassan Muhammad,
Carlie S. Sigel,
Gabriele Campanella,
Thomas Boerner,
Linda M. Pak,
Stefan Büttner,
Jan N. M. IJzermans,
Bas Groot Koerkamp,
Michael Doukas,
William R. Jarnagin,
Amber Simpson,
Thomas J. Fuchs
Abstract:
Unlike common cancers, such as those of the prostate and breast, tumor grading in rare cancers is difficult and largely undefined because of small sample sizes, the sheer volume of time needed to undertake on such a task, and the inherent difficulty of extracting human-observed patterns. One of the most challenging examples is intrahepatic cholangiocarcinoma (ICC), a primary liver cancer arising f…
▽ More
Unlike common cancers, such as those of the prostate and breast, tumor grading in rare cancers is difficult and largely undefined because of small sample sizes, the sheer volume of time needed to undertake on such a task, and the inherent difficulty of extracting human-observed patterns. One of the most challenging examples is intrahepatic cholangiocarcinoma (ICC), a primary liver cancer arising from the biliary system, for which there is well-recognized tumor heterogeneity and no grading paradigm or prognostic biomarkers. In this paper, we propose a new unsupervised deep convolutional autoencoder-based clustering model that groups together cellular and structural morphologies of tumor in 246 ICC digitized whole slides, based on visual similarity. From this visual dictionary of histologic patterns, we use the clusters as covariates to train Cox-proportional hazard survival models. In univariate analysis, three clusters were significantly associated with recurrence-free survival. Combinations of these clusters were significant in multivariate analysis. In a multivariate analysis of all clusters, five showed significance to recurrence-free survival, however the overall model was not measured to be significant. Finally, a pathologist assigned clinical terminology to the significant clusters in the visual dictionary and found evidence supporting the hypothesis that collagen-enriched fibrosis plays a role in disease severity. These results offer insight into the future of cancer subtyping and show that computational pathology can contribute to disease prognostication, especially in rare cancers.
△ Less
Submitted 12 March, 2019;
originally announced March 2019.
-
Mitochondria-based Renal Cell Carcinoma Subtyping: Learning from Deep vs. Flat Feature Representations
Authors:
Peter J. Schüffler,
Judy Sarungbam,
Hassan Muhammad,
Ed Reznik,
Satish K. Tickoo,
Thomas J. Fuchs
Abstract:
Accurate subtyping of renal cell carcinoma (RCC) is of crucial importance for understanding disease progression and for making informed treatment decisions. New discoveries of significant alterations to mitochondria between subtypes make immunohistochemical (IHC) staining based image classification an imperative. Until now, accurate quantification and subtyping was made impossible by huge IHC vari…
▽ More
Accurate subtyping of renal cell carcinoma (RCC) is of crucial importance for understanding disease progression and for making informed treatment decisions. New discoveries of significant alterations to mitochondria between subtypes make immunohistochemical (IHC) staining based image classification an imperative. Until now, accurate quantification and subtyping was made impossible by huge IHC variations, the absence of cell membrane staining for cytoplasm segmentation as well as the complete lack of systems for robust and reproducible image based classification. In this paper we present a comprehensive classification framework to overcome these challenges for tissue microarrays (TMA) of RCCs. We compare and evaluate models based on domain specific hand-crafted "flat"-features versus "deep" feature representations from various layers of a pre-trained convolutional neural network (CNN). The best model reaches a cross-validation accuracy of 89%, which demonstrates for the first time, that robust mitochondria-based subtyping of renal cancer is feasible
△ Less
Submitted 2 August, 2016;
originally announced August 2016.
-
The Numerical Control Design for a Pair of Dubins Vehicles
Authors:
Heru Tjahjana,
Iwan Pranoto,
Hari Muhammad,
J. Naiborhu,
Miswanto
Abstract:
In this paper, a model of a pair of Dubins vehicles is considered. The vehicles move from an initial position and orientation to final position and orientation. A long the motion, the two vehicles are not allowed to collide however the two vehicles cant to far each other. The optimal control of the vehicle is found using the Pontryagins Maximum Principle (PMP). This PMP leads to a Hamiltonian sy…
▽ More
In this paper, a model of a pair of Dubins vehicles is considered. The vehicles move from an initial position and orientation to final position and orientation. A long the motion, the two vehicles are not allowed to collide however the two vehicles cant to far each other. The optimal control of the vehicle is found using the Pontryagins Maximum Principle (PMP). This PMP leads to a Hamiltonian system consisting of a system of differential equation and its adjoint. The originally differential equation has initial and final condition but the adjoint system doesn't have one. The classical difficulty is solved numerically by the greatest gradient descent method. Some simulation results are presented in this paper.
△ Less
Submitted 30 April, 2008;
originally announced April 2008.
-
Effects of Leaders Position and Shape on Aerodynamic Performances of V Flight Formation
Authors:
H. P. Thien,
M. A. Moelyadi,
H. Muhammad
Abstract:
The influences of the leader in a group of V flight formation are dealt with. The investigation is focused on the effect of its position and shape on aerodynamics performances of a given V flight formation. Vortices generated the wing tip of the leader moves downstream forming a pair of opposite rotating line vortices. These vortices are generally undesirable because they create a downwash that…
▽ More
The influences of the leader in a group of V flight formation are dealt with. The investigation is focused on the effect of its position and shape on aerodynamics performances of a given V flight formation. Vortices generated the wing tip of the leader moves downstream forming a pair of opposite rotating line vortices. These vortices are generally undesirable because they create a downwash that increases the induced drag on leaders wing. However, this downwash is also accompanied by an upwash that can beneficial to the followers wing flying behind the leaders one, namely a favorable lift for the followers wing. How much contributions of the leaders wing to the followers wing in the V formation flight is determined by the strength of tip vortices generated by the leaders wing which is influenced by its position and shape including incidence angle, dihedral angle, aspect ratio and taper ratio. The prediction of aerodynamic performances of the V flight formation including lift, drag and moment coefficients is numerically performed by solving Navier Stokes equations with k e turbulence model. The computational domain is defined with multiblock topology to capture the complex geometry arrangement of the V flight formation.
△ Less
Submitted 24 April, 2008;
originally announced April 2008.
