-
Towards Modeling Learner Performance with Large Language Models
Authors:
Seyed Parsa Neshaei,
Richard Lee Davis,
Adam Hazimeh,
Bojan Lazarevski,
Pierre Dillenbourg,
Tanja Käser
Abstract:
Recent work exploring the capabilities of pre-trained large language models (LLMs) has demonstrated their ability to act as general pattern machines by completing complex token sequences representing a wide array of tasks, including time-series prediction and robot control. This paper investigates whether the pattern recognition and sequence modeling capabilities of LLMs can be extended to the dom…
▽ More
Recent work exploring the capabilities of pre-trained large language models (LLMs) has demonstrated their ability to act as general pattern machines by completing complex token sequences representing a wide array of tasks, including time-series prediction and robot control. This paper investigates whether the pattern recognition and sequence modeling capabilities of LLMs can be extended to the domain of knowledge tracing, a critical component in the development of intelligent tutoring systems (ITSs) that tailor educational experiences by predicting learner performance over time. In an empirical evaluation across multiple real-world datasets, we compare two approaches to using LLMs for this task, zero-shot prompting and model fine-tuning, with existing, non-LLM approaches to knowledge tracing. While LLM-based approaches do not achieve state-of-the-art performance, fine-tuned LLMs surpass the performance of naive baseline models and perform on par with standard Bayesian Knowledge Tracing approaches across multiple metrics. These findings suggest that the pattern recognition capabilities of LLMs can be used to model complex learning trajectories, opening a novel avenue for applying LLMs to educational contexts. The paper concludes with a discussion of the implications of these findings for future research, suggesting that further refinements and a deeper understanding of LLMs' predictive mechanisms could lead to enhanced performance in knowledge tracing tasks.
△ Less
Submitted 29 February, 2024;
originally announced March 2024.
-
A Design Space for Intelligent and Interactive Writing Assistants
Authors:
Mina Lee,
Katy Ilonka Gero,
John Joon Young Chung,
Simon Buckingham Shum,
Vipul Raheja,
Hua Shen,
Subhashini Venugopalan,
Thiemo Wambsganss,
David Zhou,
Emad A. Alghamdi,
Tal August,
Avinash Bhat,
Madiha Zahrah Choksi,
Senjuti Dutta,
Jin L. C. Guo,
Md Naimul Hoque,
Yewon Kim,
Simon Knight,
Seyed Parsa Neshaei,
Agnia Sergeyuk,
Antonette Shibani,
Disha Shrivastava,
Lila Shroff,
Jessi Stark,
Sarah Sterman
, et al. (11 additional authors not shown)
Abstract:
In our era of rapid technological advancement, the research landscape for writing assistants has become increasingly fragmented across various research communities. We seek to address this challenge by proposing a design space as a structured way to examine and explore the multidimensional space of intelligent and interactive writing assistants. Through a large community collaboration, we explore…
▽ More
In our era of rapid technological advancement, the research landscape for writing assistants has become increasingly fragmented across various research communities. We seek to address this challenge by proposing a design space as a structured way to examine and explore the multidimensional space of intelligent and interactive writing assistants. Through a large community collaboration, we explore five aspects of writing assistants: task, user, technology, interaction, and ecosystem. Within each aspect, we define dimensions (i.e., fundamental components of an aspect) and codes (i.e., potential options for each dimension) by systematically reviewing 115 papers. Our design space aims to offer researchers and designers a practical tool to navigate, comprehend, and compare the various possibilities of writing assistants, and aid in the envisioning and design of new writing assistants.
△ Less
Submitted 26 March, 2024; v1 submitted 21 March, 2024;
originally announced March 2024.
-
The Impact of Quantization on the Robustness of Transformer-based Text Classifiers
Authors:
Seyed Parsa Neshaei,
Yasaman Boreshban,
Gholamreza Ghassem-Sani,
Seyed Abolghasem Mirroshandel
Abstract:
Transformer-based models have made remarkable advancements in various NLP areas. Nevertheless, these models often exhibit vulnerabilities when confronted with adversarial attacks. In this paper, we explore the effect of quantization on the robustness of Transformer-based models. Quantization usually involves mapping a high-precision real number to a lower-precision value, aiming at reducing the si…
▽ More
Transformer-based models have made remarkable advancements in various NLP areas. Nevertheless, these models often exhibit vulnerabilities when confronted with adversarial attacks. In this paper, we explore the effect of quantization on the robustness of Transformer-based models. Quantization usually involves mapping a high-precision real number to a lower-precision value, aiming at reducing the size of the model at hand. To the best of our knowledge, this work is the first application of quantization on the robustness of NLP models. In our experiments, we evaluate the impact of quantization on BERT and DistilBERT models in text classification using SST-2, Emotion, and MR datasets. We also evaluate the performance of these models against TextFooler, PWWS, and PSO adversarial attacks. Our findings show that quantization significantly improves (by an average of 18.68%) the adversarial accuracy of the models. Furthermore, we compare the effect of quantization versus that of the adversarial training approach on robustness. Our experiments indicate that quantization increases the robustness of the model by 18.80% on average compared to adversarial training without imposing any extra computational overhead during training. Therefore, our results highlight the effectiveness of quantization in improving the robustness of NLP models.
△ Less
Submitted 8 March, 2024;
originally announced March 2024.
-
Unraveling Downstream Gender Bias from Large Language Models: A Study on AI Educational Writing Assistance
Authors:
Thiemo Wambsganss,
Xiaotian Su,
Vinitra Swamy,
Seyed Parsa Neshaei,
Roman Rietsche,
Tanja Käser
Abstract:
Large Language Models (LLMs) are increasingly utilized in educational tasks such as providing writing suggestions to students. Despite their potential, LLMs are known to harbor inherent biases which may negatively impact learners. Previous studies have investigated bias in models and data representations separately, neglecting the potential impact of LLM bias on human writing. In this paper, we in…
▽ More
Large Language Models (LLMs) are increasingly utilized in educational tasks such as providing writing suggestions to students. Despite their potential, LLMs are known to harbor inherent biases which may negatively impact learners. Previous studies have investigated bias in models and data representations separately, neglecting the potential impact of LLM bias on human writing. In this paper, we investigate how bias transfers through an AI writing support pipeline. We conduct a large-scale user study with 231 students writing business case peer reviews in German. Students are divided into five groups with different levels of writing support: one classroom group with feature-based suggestions and four groups recruited from Prolific -- a control group with no assistance, two groups with suggestions from fine-tuned GPT-2 and GPT-3 models, and one group with suggestions from pre-trained GPT-3.5. Using GenBit gender bias analysis, Word Embedding Association Tests (WEAT), and Sentence Embedding Association Test (SEAT) we evaluate the gender bias at various stages of the pipeline: in model embeddings, in suggestions generated by the models, and in reviews written by students. Our results demonstrate that there is no significant difference in gender bias between the resulting peer reviews of groups with and without LLM suggestions. Our research is therefore optimistic about the use of AI writing support in the classroom, showcasing a context where bias in LLMs does not transfer to students' responses.
△ Less
Submitted 6 November, 2023;
originally announced November 2023.
-
Visualizing Self-Regulated Learner Profiles in Dashboards: Design Insights from Teachers
Authors:
Paola Mejia-Domenzain,
Eva Laini,
Seyed Parsa Neshaei,
Thiemo Wambsganss,
Tanja Käser
Abstract:
Flipped Classrooms (FC) are a promising teaching strategy, where students engage with the learning material before attending face-to-face sessions. While pre-class activities are critical for course success, many students struggle to engage effectively in them due to inadequate of self-regulated learning (SRL) skills. Thus, tools enabling teachers to monitor students' SRL and provide personalized…
▽ More
Flipped Classrooms (FC) are a promising teaching strategy, where students engage with the learning material before attending face-to-face sessions. While pre-class activities are critical for course success, many students struggle to engage effectively in them due to inadequate of self-regulated learning (SRL) skills. Thus, tools enabling teachers to monitor students' SRL and provide personalized guidance have the potential to improve learning outcomes. However, existing dashboards mostly focus on aggregated information, disregarding recent work leveraging machine learning (ML) approaches that have identified comprehensive, multi-dimensional SRL behaviors. Unfortunately, the complexity of such findings makes them difficult to communicate and act on. In this paper, we follow a teacher-centered approach to study how to make thorough findings accessible to teachers. We design and implement FlippED, a dashboard for monitoring students' SRL behavior. We evaluate the usability and actionability of the tool in semi-structured interviews with ten university teachers. We find that communicating ML-based profiles spark a range of potential interventions for students and course modifications.
△ Less
Submitted 26 May, 2023;
originally announced May 2023.
