-
A Mapping Strategy for Interacting with Latent Audio Synthesis Using Artistic Materials
Authors:
Shuoyang Zheng,
Anna Xambó Sedó,
Nick Bryan-Kinns
Abstract:
This paper presents a mapping strategy for interacting with the latent spaces of generative AI models. Our approach involves using unsupervised feature learning to encode a human control space and mapping it to an audio synthesis model's latent space. To demonstrate how this mapping strategy can turn high-dimensional sensor data into control mechanisms of a deep generative model, we present a proo…
▽ More
This paper presents a mapping strategy for interacting with the latent spaces of generative AI models. Our approach involves using unsupervised feature learning to encode a human control space and mapping it to an audio synthesis model's latent space. To demonstrate how this mapping strategy can turn high-dimensional sensor data into control mechanisms of a deep generative model, we present a proof-of-concept system that uses visual sketches to control an audio synthesis model. We draw on emerging discourses in XAIxArts to discuss how this approach can contribute to XAI in artistic and creative contexts, we also discuss its current limitations and propose future research directions.
△ Less
Submitted 5 July, 2024;
originally announced July 2024.
-
Proceedings of The second international workshop on eXplainable AI for the Arts (XAIxArts)
Authors:
Nick Bryan-Kinns,
Corey Ford,
Shuoyang Zheng,
Helen Kennedy,
Alan Chamberlain,
Makayla Lewis,
Drew Hemment,
Zijin Li,
Qiong Wu,
Lanxi Xiao,
Gus Xia,
Jeba Rezwana,
Michael Clemens,
Gabriel Vigliensoni
Abstract:
This second international workshop on explainable AI for the Arts (XAIxArts) brought together a community of researchers in HCI, Interaction Design, AI, explainable AI (XAI), and digital arts to explore the role of XAI for the Arts. Workshop held at the 16th ACM Conference on Creativity and Cognition (C&C 2024), Chicago, USA.
This second international workshop on explainable AI for the Arts (XAIxArts) brought together a community of researchers in HCI, Interaction Design, AI, explainable AI (XAI), and digital arts to explore the role of XAI for the Arts. Workshop held at the 16th ACM Conference on Creativity and Cognition (C&C 2024), Chicago, USA.
△ Less
Submitted 9 July, 2024; v1 submitted 20 June, 2024;
originally announced June 2024.
-
Exploring Variational Auto-Encoder Architectures, Configurations, and Datasets for Generative Music Explainable AI
Authors:
Nick Bryan-Kinns,
Bingyuan Zhang,
Songyan Zhao,
Berker Banar
Abstract:
Generative AI models for music and the arts in general are increasingly complex and hard to understand. The field of eXplainable AI (XAI) seeks to make complex and opaque AI models such as neural networks more understandable to people. One approach to making generative AI models more understandable is to impose a small number of semantically meaningful attributes on generative AI models. This pape…
▽ More
Generative AI models for music and the arts in general are increasingly complex and hard to understand. The field of eXplainable AI (XAI) seeks to make complex and opaque AI models such as neural networks more understandable to people. One approach to making generative AI models more understandable is to impose a small number of semantically meaningful attributes on generative AI models. This paper contributes a systematic examination of the impact that different combinations of Variational Auto-Encoder models (MeasureVAE and AdversarialVAE), configurations of latent space in the AI model (from 4 to 256 latent dimensions), and training datasets (Irish folk, Turkish folk, Classical, and pop) have on music generation performance when 2 or 4 meaningful musical attributes are imposed on the generative model. To date there have been no systematic comparisons of such models at this level of combinatorial detail. Our findings show that MeasureVAE has better reconstruction performance than AdversarialVAE which has better musical attribute independence. Results demonstrate that MeasureVAE was able to generate music across music genres with interpretable musical dimensions of control, and performs best with low complexity music such a pop and rock. We recommend that a 32 or 64 latent dimensional space is optimal for 4 regularised dimensions when using MeasureVAE to generate music across genres. Our results are the first detailed comparisons of configurations of state-of-the-art generative AI models for music and can be used to help select and configure AI models, musical features, and datasets for more understandable generation of music.
△ Less
Submitted 14 November, 2023;
originally announced November 2023.
-
A Guide to Evaluating the Experience of Media and Arts Technology
Authors:
Nick Bryan-Kinns,
Courtney N. Reed
Abstract:
Evaluation is essential to understanding the value that digital creativity brings to people's experience, for example in terms of their enjoyment, creativity, and engagement. There is a substantial body of research on how to design and evaluate interactive arts and digital creativity applications. There is also extensive Human-Computer Interaction (HCI) literature on how to evaluate user interface…
▽ More
Evaluation is essential to understanding the value that digital creativity brings to people's experience, for example in terms of their enjoyment, creativity, and engagement. There is a substantial body of research on how to design and evaluate interactive arts and digital creativity applications. There is also extensive Human-Computer Interaction (HCI) literature on how to evaluate user interfaces and user experiences. However, it can be difficult for artists, practitioners, and researchers to navigate such a broad and disparate collection of materials when considering how to evaluate technology they create that is at the intersection of art and interaction. This chapter provides a guide to designing robust user studies of creative applications at the intersection of art, technology and interaction, which we refer to as Media and Arts Technology (MAT). We break MAT studies down into two main kinds: proof-of-concept and comparative studies. As MAT studies are exploratory in nature, their evaluation requires the collection and analysis of both qualitative data such as free text questionnaire responses, interviews, and observations, and also quantitative data such as questionnaires, number of interactions, and length of time spent interacting. This chapter draws on over 15 years of experience of designing and evaluating novel interactive systems to provide a concrete template on how to structure a study to evaluate MATs that is both rigorous and repeatable, and how to report study results that are publishable and accessible to a wide readership in art and science communities alike.
△ Less
Submitted 13 November, 2023;
originally announced November 2023.
-
AI (r)evolution -- where are we heading? Thoughts about the future of music and sound technologies in the era of deep learning
Authors:
Giovanni Bindi,
Nils Demerlé,
Rodrigo Diaz,
David Genova,
Aliénor Golvet,
Ben Hayes,
Jiawen Huang,
Lele Liu,
Vincent Martos,
Sarah Nabi,
Teresa Pelinski,
Lenny Renault,
Saurjya Sarkar,
Pedro Sarmento,
Cyrus Vahidi,
Lewis Wolstanholme,
Yixiao Zhang,
Axel Roebel,
Nick Bryan-Kinns,
Jean-Louis Giavitto,
Mathieu Barthet
Abstract:
Artificial Intelligence (AI) technologies such as deep learning are evolving very quickly bringing many changes to our everyday lives. To explore the future impact and potential of AI in the field of music and sound technologies a doctoral day was held between Queen Mary University of London (QMUL, UK) and Sciences et Technologies de la Musique et du Son (STMS, France). Prompt questions about curr…
▽ More
Artificial Intelligence (AI) technologies such as deep learning are evolving very quickly bringing many changes to our everyday lives. To explore the future impact and potential of AI in the field of music and sound technologies a doctoral day was held between Queen Mary University of London (QMUL, UK) and Sciences et Technologies de la Musique et du Son (STMS, France). Prompt questions about current trends in AI and music were generated by academics from QMUL and STMS. Students from the two institutions then debated these questions. This report presents a summary of the student debates on the topics of: Data, Impact, and the Environment; Responsible Innovation and Creative Practice; Creativity and Bias; and From Tools to the Singularity. The students represent the future generation of AI and music researchers. The academics represent the incumbent establishment. The student debates reported here capture visions, dreams, concerns, uncertainties, and contentious issues for the future of AI and music as the establishment is rightfully challenged by the next generation.
△ Less
Submitted 20 September, 2023;
originally announced October 2023.
-
Proceedings of The first international workshop on eXplainable AI for the Arts (XAIxArts)
Authors:
Nick Bryan-Kinns,
Corey Ford,
Alan Chamberlain,
Steven David Benford,
Helen Kennedy,
Zijin Li,
Wu Qiong,
Gus G. Xia,
Jeba Rezwana
Abstract:
This first international workshop on explainable AI for the Arts (XAIxArts) brought together a community of researchers in HCI, Interaction Design, AI, explainable AI (XAI), and digital arts to explore the role of XAI for the Arts.
Workshop held at the 15th ACM Conference on Creativity and Cognition (C&C 2023).
This first international workshop on explainable AI for the Arts (XAIxArts) brought together a community of researchers in HCI, Interaction Design, AI, explainable AI (XAI), and digital arts to explore the role of XAI for the Arts.
Workshop held at the 15th ACM Conference on Creativity and Cognition (C&C 2023).
△ Less
Submitted 10 October, 2023;
originally announced October 2023.
-
An Autoethnographic Exploration of XAI in Algorithmic Composition
Authors:
Ashley Noel-Hirst,
Nick Bryan-Kinns
Abstract:
Machine Learning models are capable of generating complex music across a range of genres from folk to classical music. However, current generative music AI models are typically difficult to understand and control in meaningful ways. Whilst research has started to explore how explainable AI (XAI) generative models might be created for music, no generative XAI models have been studied in music makin…
▽ More
Machine Learning models are capable of generating complex music across a range of genres from folk to classical music. However, current generative music AI models are typically difficult to understand and control in meaningful ways. Whilst research has started to explore how explainable AI (XAI) generative models might be created for music, no generative XAI models have been studied in music making practice. This paper introduces an autoethnographic study of the use of the MeasureVAE generative music XAI model with interpretable latent dimensions trained on Irish folk music. Findings suggest that the exploratory nature of the music-making workflow foregrounds musical features of the training dataset rather than features of the generative model itself. The appropriation of an XAI model within an iterative workflow highlights the potential of XAI models to form part of a richer and more complex workflow than they were initially designed for.
△ Less
Submitted 11 August, 2023;
originally announced August 2023.
-
Exploring XAI for the Arts: Explaining Latent Space in Generative Music
Authors:
Nick Bryan-Kinns,
Berker Banar,
Corey Ford,
Courtney N. Reed,
Yixiao Zhang,
Simon Colton,
Jack Armitage
Abstract:
Explainable AI has the potential to support more interactive and fluid co-creative AI systems which can creatively collaborate with people. To do this, creative AI models need to be amenable to debugging by offering eXplainable AI (XAI) features which are inspectable, understandable, and modifiable. However, currently there is very little XAI for the arts. In this work, we demonstrate how a latent…
▽ More
Explainable AI has the potential to support more interactive and fluid co-creative AI systems which can creatively collaborate with people. To do this, creative AI models need to be amenable to debugging by offering eXplainable AI (XAI) features which are inspectable, understandable, and modifiable. However, currently there is very little XAI for the arts. In this work, we demonstrate how a latent variable model for music generation can be made more explainable; specifically we extend MeasureVAE which generates measures of music. We increase the explainability of the model by: i) using latent space regularisation to force some specific dimensions of the latent space to map to meaningful musical attributes, ii) providing a user interface feedback loop to allow people to adjust dimensions of the latent space and observe the results of these changes in real-time, iii) providing a visualisation of the musical attributes in the latent space to help people understand and predict the effect of changes to latent space dimensions. We suggest that in doing so we bridge the gap between the latent space and the generated musical outcomes in a meaningful way which makes the model and its outputs more explainable and more debuggable.
△ Less
Submitted 10 August, 2023;
originally announced August 2023.
-
Extended Reality (XR) Remote Research: a Survey of Drawbacks and Opportunities
Authors:
Jack Ratcliffe,
Francesco Soave,
Nick Bryan-Kinns,
Laurissa Tokarchuk,
Ildar Farkhatdinov
Abstract:
Extended Reality (XR) technology - such as virtual and augmented reality - is now widely used in Human Computer Interaction (HCI), social science and psychology experimentation. However, these experiments are predominantly deployed in-lab with a co-present researcher. Remote experiments, without co-present researchers, have not flourished, despite the success of remote approaches for non-XR invest…
▽ More
Extended Reality (XR) technology - such as virtual and augmented reality - is now widely used in Human Computer Interaction (HCI), social science and psychology experimentation. However, these experiments are predominantly deployed in-lab with a co-present researcher. Remote experiments, without co-present researchers, have not flourished, despite the success of remote approaches for non-XR investigations. This paper summarises findings from a 30-item survey of 46 XR researchers to understand perceived limitations and benefits of remote XR experimentation. Our thematic analysis identifies concerns common with non-XR remote research, such as participant recruitment, as well as XR-specific issues, including safety and hardware variability. We identify potential positive affordances of XR technology, including leveraging data collection functionalities builtin to HMDs (e.g. hand, gaze tracking) and the portability and reproducibility of an experimental setting. We suggest that XR technology could be conceptualised as an interactive technology and a capable data-collection device suited for remote experimentation.
△ Less
Submitted 20 January, 2021;
originally announced January 2021.
