-
Towards Musically Informed Evaluation of Piano Transcription Models
Authors:
Patricia Hu,
Lukáš Samuel Marták,
Carlos Cancino-Chacón,
Gerhard Widmer
Abstract:
Automatic piano transcription models are typically evaluated using simple frame- or note-wise information retrieval (IR) metrics. Such benchmark metrics do not provide insights into the transcription quality of specific musical aspects such as articulation, dynamics, or rhythmic precision of the output, which are essential in the context of expressive performance analysis. Furthermore, in recent y…
▽ More
Automatic piano transcription models are typically evaluated using simple frame- or note-wise information retrieval (IR) metrics. Such benchmark metrics do not provide insights into the transcription quality of specific musical aspects such as articulation, dynamics, or rhythmic precision of the output, which are essential in the context of expressive performance analysis. Furthermore, in recent years, MAESTRO has become the de-facto training and evaluation dataset for such models. However, inference performance has been observed to deteriorate substantially when applied on out-of-distribution data, thereby questioning the suitability and reliability of transcribed outputs from such models for specific MIR tasks. In this work, we investigate the performance of three state-of-the-art piano transcription models in two experiments. In the first one, we propose a variety of musically informed evaluation metrics which, in contrast to the IR metrics, offer more detailed insight into the musical quality of the transcriptions. In the second experiment, we compare inference performance on real-world and perturbed audio recordings, and highlight musical dimensions which our metrics can help explain. Our experimental results highlight the weaknesses of existing piano transcription metrics and contribute to a more musically sound error analysis of transcription outputs.
△ Less
Submitted 12 June, 2024;
originally announced June 2024.
-
Are we describing the same sound? An analysis of word embedding spaces of expressive piano performance
Authors:
Silvan David Peter,
Shreyan Chowdhury,
Carlos Eduardo Cancino-Chacón,
Gerhard Widmer
Abstract:
Semantic embeddings play a crucial role in natural language-based information retrieval. Embedding models represent words and contexts as vectors whose spatial configuration is derived from the distribution of words in large text corpora. While such representations are generally very powerful, they might fail to account for fine-grained domain-specific nuances. In this article, we investigate this…
▽ More
Semantic embeddings play a crucial role in natural language-based information retrieval. Embedding models represent words and contexts as vectors whose spatial configuration is derived from the distribution of words in large text corpora. While such representations are generally very powerful, they might fail to account for fine-grained domain-specific nuances. In this article, we investigate this uncertainty for the domain of characterizations of expressive piano performance. Using a music research dataset of free text performance characterizations and a follow-up study sorting the annotations into clusters, we derive a ground truth for a domain-specific semantic similarity structure. We test five embedding models and their similarity structure for correspondence with the ground truth. We further assess the effects of contextualizing prompts, hubness reduction, cross-modal similarity, and k-means clustering. The quality of embedding models shows great variability with respect to this task; more general models perform better than domain-adapted ones and the best model configurations reach human-level agreement.
△ Less
Submitted 31 December, 2023;
originally announced January 2024.
-
Sounding Out Reconstruction Error-Based Evaluation of Generative Models of Expressive Performance
Authors:
Silvan David Peter,
Carlos Eduardo Cancino-Chacón,
Emmanouil Karystinaios,
Gerhard Widmer
Abstract:
Generative models of expressive piano performance are usually assessed by comparing their predictions to a reference human performance. A generative algorithm is taken to be better than competing ones if it produces performances that are closer to a human reference performance. However, expert human performers can (and do) interpret music in different ways, making for different possible references…
▽ More
Generative models of expressive piano performance are usually assessed by comparing their predictions to a reference human performance. A generative algorithm is taken to be better than competing ones if it produces performances that are closer to a human reference performance. However, expert human performers can (and do) interpret music in different ways, making for different possible references, and quantitative closeness is not necessarily aligned with perceptual similarity, raising concerns about the validity of this evaluation approach. In this work, we present a number of experiments that shed light on this problem. Using precisely measured high-quality performances of classical piano music, we carry out a listening test indicating that listeners can sometimes perceive subtle performance difference that go unnoticed under quantitative evaluation. We further present tests that indicate that such evaluation frameworks show a lot of variability in reliability and validity across different reference performances and pieces. We discuss these results and their implications for quantitative evaluation, and hope to foster a critical appreciation of the uncertainties involved in quantitative assessments of such performances within the wider music information retrieval (MIR) community.
△ Less
Submitted 31 December, 2023;
originally announced January 2024.
-
Symbolic Music Representations for Classification Tasks: A Systematic Evaluation
Authors:
Huan Zhang,
Emmanouil Karystinaios,
Simon Dixon,
Gerhard Widmer,
Carlos Eduardo Cancino-Chacón
Abstract:
Music Information Retrieval (MIR) has seen a recent surge in deep learning-based approaches, which often involve encoding symbolic music (i.e., music represented in terms of discrete note events) in an image-like or language like fashion. However, symbolic music is neither an image nor a sentence, and research in the symbolic domain lacks a comprehensive overview of the different available represe…
▽ More
Music Information Retrieval (MIR) has seen a recent surge in deep learning-based approaches, which often involve encoding symbolic music (i.e., music represented in terms of discrete note events) in an image-like or language like fashion. However, symbolic music is neither an image nor a sentence, and research in the symbolic domain lacks a comprehensive overview of the different available representations. In this paper, we investigate matrix (piano roll), sequence, and graph representations and their corresponding neural architectures, in combination with symbolic scores and performances on three piece-level classification tasks. We also introduce a novel graph representation for symbolic performances and explore the capability of graph representations in global classification tasks. Our systematic evaluation shows advantages and limitations of each input representation. Our results suggest that the graph representation, as the newest and least explored among the three approaches, exhibits promising performance, while being more light-weight in training.
△ Less
Submitted 10 September, 2023; v1 submitted 5 September, 2023;
originally announced September 2023.
-
The ACCompanion: Combining Reactivity, Robustness, and Musical Expressivity in an Automatic Piano Accompanist
Authors:
Carlos Cancino-Chacón,
Silvan Peter,
Patricia Hu,
Emmanouil Karystinaios,
Florian Henkel,
Francesco Foscarin,
Nimrod Varga,
Gerhard Widmer
Abstract:
This paper introduces the ACCompanion, an expressive accompaniment system. Similarly to a musician who accompanies a soloist playing a given musical piece, our system can produce a human-like rendition of the accompaniment part that follows the soloist's choices in terms of tempo, dynamics, and articulation. The ACCompanion works in the symbolic domain, i.e., it needs a musical instrument capable…
▽ More
This paper introduces the ACCompanion, an expressive accompaniment system. Similarly to a musician who accompanies a soloist playing a given musical piece, our system can produce a human-like rendition of the accompaniment part that follows the soloist's choices in terms of tempo, dynamics, and articulation. The ACCompanion works in the symbolic domain, i.e., it needs a musical instrument capable of producing and playing MIDI data, with explicitly encoded onset, offset, and pitch for each played note. We describe the components that go into such a system, from real-time score following and prediction to expressive performance generation and online adaptation to the expressive choices of the human player. Based on our experience with repeated live demonstrations in front of various audiences, we offer an analysis of the challenges of combining these components into a system that is highly reactive and precise, while still a reliable musical partner, robust to possible performance errors and responsive to expressive variations.
△ Less
Submitted 30 May, 2023; v1 submitted 24 April, 2023;
originally announced April 2023.
-
The match file format: Encoding Alignments between Scores and Performances
Authors:
Francesco Foscarin,
Emmanouil Karystinaios,
Silvan David Peter,
Carlos Cancino-Chacón,
Maarten Grachten,
Gerhard Widmer
Abstract:
This paper presents the specifications of match: a file format that extends a MIDI human performance with note-, beat-, and downbeat-level alignments to a corresponding musical score. This enables advanced analyses of the performance that are relevant for various tasks, such as expressive performance modeling, score following, music transcription, and performer classification. The match file inclu…
▽ More
This paper presents the specifications of match: a file format that extends a MIDI human performance with note-, beat-, and downbeat-level alignments to a corresponding musical score. This enables advanced analyses of the performance that are relevant for various tasks, such as expressive performance modeling, score following, music transcription, and performer classification. The match file includes a set of score-related descriptors that makes it usable also as a bare-bones score representation. For applications that require the use of structural score elements (e.g., voices, parts, beams, slurs), the match file can be easily combined with the symbolic score. To support the practical application of our work, we release a corrected and upgraded version of the Vienna4x22 dataset of scores and performances aligned with match files.
△ Less
Submitted 2 June, 2022;
originally announced June 2022.
-
Partitura: A Python Package for Symbolic Music Processing
Authors:
Carlos Cancino-Chacón,
Silvan David Peter,
Emmanouil Karystinaios,
Francesco Foscarin,
Maarten Grachten,
Gerhard Widmer
Abstract:
Partitura is a lightweight Python package for handling symbolic musical information. It provides easy access to features commonly used in music information retrieval tasks, like note arrays (lists of timed pitched events) and 2D piano roll matrices, as well as other score elements such as time and key signatures, performance directives, and repeat structures. Partitura can load musical scores (in…
▽ More
Partitura is a lightweight Python package for handling symbolic musical information. It provides easy access to features commonly used in music information retrieval tasks, like note arrays (lists of timed pitched events) and 2D piano roll matrices, as well as other score elements such as time and key signatures, performance directives, and repeat structures. Partitura can load musical scores (in MEI, MusicXML, Kern, and MIDI formats), MIDI performances, and score-to-performance alignments. The package includes some tools for music analysis, such as automatic pitch spelling, key signature identification, and voice separation. Partitura is an open-source project and is available at https://github.com/CPJKU/partitura/.
△ Less
Submitted 2 June, 2022;
originally announced June 2022.
-
Beyond synchronization: Body gestures and gaze direction in duo performance
Authors:
Laura Bishop,
Carlos Cancino-Chacón,
Werner Goebl
Abstract:
In this chapter, we focus on two main categories of visual interaction: body gestures and gaze direction. Our focus on body gestures is motivated by research showing that gesture patterns often change during joint action tasks to become more predictable (van der Wel et al., 2016). Moreover, coordination sometimes emerges between musicians at the level of body sway (Chang et al., 2017). Our focus o…
▽ More
In this chapter, we focus on two main categories of visual interaction: body gestures and gaze direction. Our focus on body gestures is motivated by research showing that gesture patterns often change during joint action tasks to become more predictable (van der Wel et al., 2016). Moreover, coordination sometimes emerges between musicians at the level of body sway (Chang et al., 2017). Our focus on gaze direction was motivated by the fact that gaze can serve simultaneously as a means of obtaining information about the world and as a means of communicating one's own attention and intent.
△ Less
Submitted 31 January, 2022;
originally announced January 2022.
-
partitura: A Python Package for Handling Symbolic Musical Data
Authors:
Maarten Grachten,
Carlos Cancino-Chacón,
Thassilo Gadermaier
Abstract:
This demo paper introduces partitura, a Python package for handling symbolic musical information. The principal aim of this package is to handle richly structured musical information as conveyed by modern staff music notation. It provides a much wider range of possibilities to deal with music than the more reductive (but very common) piano roll-oriented approach inspired by the MIDI standard. The…
▽ More
This demo paper introduces partitura, a Python package for handling symbolic musical information. The principal aim of this package is to handle richly structured musical information as conveyed by modern staff music notation. It provides a much wider range of possibilities to deal with music than the more reductive (but very common) piano roll-oriented approach inspired by the MIDI standard. The package is an open source project and is available on GitHub.
△ Less
Submitted 31 January, 2022;
originally announced January 2022.
-
On the Characterization of Expressive Performance in Classical Music: First Results of the Con Espressione Game
Authors:
Carlos Cancino-Chacón,
Silvan Peter,
Shreyan Chowdhury,
Anna Aljanaki,
Gerhard Widmer
Abstract:
A piece of music can be expressively performed, or interpreted, in a variety of ways. With the help of an online questionnaire, the Con Espressione Game, we collected some 1,500 descriptions of expressive character relating to 45 performances of 9 excerpts from classical piano pieces, played by different famous pianists. More specifically, listeners were asked to describe, using freely chosen word…
▽ More
A piece of music can be expressively performed, or interpreted, in a variety of ways. With the help of an online questionnaire, the Con Espressione Game, we collected some 1,500 descriptions of expressive character relating to 45 performances of 9 excerpts from classical piano pieces, played by different famous pianists. More specifically, listeners were asked to describe, using freely chosen words (preferably: adjectives), how they perceive the expressive character of the different performances. In this paper, we offer a first account of this new data resource for expressive performance research, and provide an exploratory analysis, addressing three main questions: (1) how similarly do different listeners describe a performance of a piece? (2) what are the main dimensions (or axes) for expressive character emerging from this?; and (3) how do measurable parameters of a performance (e.g., tempo, dynamics) and mid- and high-level features that can be predicted by machine learning models (e.g., articulation, arousal) relate to these expressive dimensions? The dataset that we publish along with this paper was enriched by adding hand-corrected score-to-performance alignments, as well as descriptive audio features such as tempo and dynamics curves.
△ Less
Submitted 5 August, 2020;
originally announced August 2020.
-
Real-time visualisation of fugue played by a string quartet
Authors:
Olivier Lartillot,
Carlos Cancino-Chacón,
Charles Brazier
Abstract:
We present a new system for real-time visualisation of music performance, focused for the moment on a fugue played by a string quartet. The basic principle is to offer a visual guide to better understand music using strategies that should be as engaging, accessible and effective as possible. The pitch curves related to the separate voices are drawn on a space whose temporal axis is normalised with…
▽ More
We present a new system for real-time visualisation of music performance, focused for the moment on a fugue played by a string quartet. The basic principle is to offer a visual guide to better understand music using strategies that should be as engaging, accessible and effective as possible. The pitch curves related to the separate voices are drawn on a space whose temporal axis is normalised with respect to metrical positions, and aligned vertically with respect to their thematic and motivic classification. Aspects related to tonality are represented as well. We describe the underlying technologies we have developed and the technical setting. In particular, the rhythmical and structural representation of the piece relies on real-time polyphonic audio-to-score alignment using online dynamic time warping. The visualisation will be presented at a concert of the Danish String Quartet, performing the last piece of The Art of Fugue by Johann Sebastian Bach.
△ Less
Submitted 17 June, 2020;
originally announced June 2020.
-
Moving to Communicate, Moving to Interact: Patterns of Body Motion in Musical Duo Performance
Authors:
Laura Bishop,
Carlos Cancino-Chacón,
Werner Goebl
Abstract:
Skilled ensemble musicians coordinate with high precision, even when improvising or interpreting loosely-defined notation. Successful coordination is supported primarily through shared attention to the musical output; however, musicians also interact visually, particularly when the musical timing is irregular. This study investigated the performance conditions that encourage visual signalling and…
▽ More
Skilled ensemble musicians coordinate with high precision, even when improvising or interpreting loosely-defined notation. Successful coordination is supported primarily through shared attention to the musical output; however, musicians also interact visually, particularly when the musical timing is irregular. This study investigated the performance conditions that encourage visual signalling and interaction between ensemble members. Piano and clarinet duos rehearsed a new piece as their body motion was recorded. Analyses of head movement showed that performers communicated gesturally following held notes. Gesture patterns became more consistent as duos rehearsed, though consistency dropped again during a final performance given under no-visual-contact conditions. Movements were smoother and interperformer coordination was stronger during irregularly-timed passages than elsewhere in the piece, suggesting heightened visual interaction. Performers moved more after rehearsing than before, and more when they could see each other than when visual contact was occluded. Periods of temporal instability and increased familiarity with the music and co-performer seem to encourage visual interaction, while specific communicative gestures are integrated into performance routines through rehearsal. We propose that visual interaction may support successful ensemble performance by affirming coordination throughout periods of temporal instability and serving as a social motivator to promote creative risk-taking.
△ Less
Submitted 20 November, 2019;
originally announced November 2019.
-
A Convolutional Approach to Melody Line Identification in Symbolic Scores
Authors:
Federico Simonetta,
Carlos Cancino-Chacón,
Stavros Ntalampiras,
Gerhard Widmer
Abstract:
In many musical traditions, the melody line is of primary significance in a piece. Human listeners can readily distinguish melodies from accompaniment; however, making this distinction given only the written score -- i.e. without listening to the music performed -- can be a difficult task. Solving this task is of great importance for both Music Information Retrieval and musicological applications.…
▽ More
In many musical traditions, the melody line is of primary significance in a piece. Human listeners can readily distinguish melodies from accompaniment; however, making this distinction given only the written score -- i.e. without listening to the music performed -- can be a difficult task. Solving this task is of great importance for both Music Information Retrieval and musicological applications. In this paper, we propose an automated approach to identifying the most salient melody line in a symbolic score. The backbone of the method consists of a convolutional neural network (CNN) estimating the probability that each note in the score (more precisely: each pixel in a piano roll encoding of the score) belongs to the melody line. We train and evaluate the method on various datasets, using manual annotations where available and solo instrument parts where not. We also propose a method to inspect the CNN and to analyze the influence exerted by notes on the prediction of other notes; this method can be applied whenever the output of a neural network has the same size as the input.
△ Less
Submitted 24 June, 2019;
originally announced June 2019.
-
User Curated Shaping of Expressive Performances
Authors:
Zhengshan Shi,
Carlos Cancino-Chacón,
Gerhard Widmer
Abstract:
Musicians produce individualized, expressive performances by manipulating parameters such as dynamics, tempo and articulation. This manipulation of expressive parameters is informed by elements of score information such as pitch, meter, and tempo and dynamics markings (among others). In this paper we present an interactive interface that gives users the opportunity to explore the relationship betw…
▽ More
Musicians produce individualized, expressive performances by manipulating parameters such as dynamics, tempo and articulation. This manipulation of expressive parameters is informed by elements of score information such as pitch, meter, and tempo and dynamics markings (among others). In this paper we present an interactive interface that gives users the opportunity to explore the relationship between structural elements of a score and expressive parameters. This interface draws on the basis function models, a data-driven framework for expressive performance. In this framework, expressive parameters are modeled as a function of score features, i.e., numerical encodings of specific aspects of a musical score, using neural networks. With the proposed interface, users are able to weight the contribution of individual score features and understand how an expressive performance is constructed.
△ Less
Submitted 14 June, 2019;
originally announced June 2019.
-
A Computational Study of the Role of Tonal Tension in Expressive Piano Performance
Authors:
Carlos Cancino-Chacón,
Maarten Grachten
Abstract:
Expressive variations of tempo and dynamics are an important aspect of music performances, involving a variety of underlying factors. Previous work has showed a relation between such expressive variations (in particular expressive tempo) and perceptual characteristics derived from the musical score, such as musical expectations, and perceived tension. In this work we use a computational approach t…
▽ More
Expressive variations of tempo and dynamics are an important aspect of music performances, involving a variety of underlying factors. Previous work has showed a relation between such expressive variations (in particular expressive tempo) and perceptual characteristics derived from the musical score, such as musical expectations, and perceived tension. In this work we use a computational approach to study the role of three measures of tonal tension proposed by Herremans and Chew (2016) in the prediction of expressive performances of classical piano music. These features capture tonal relationships of the music represented in Chew's spiral array model, a three dimensional representation of pitch classes, chords and keys constructed in such a way that spatial proximity represents close tonal relationships. We use non-linear sequential models (recurrent neural networks) to assess the contribution of these features to the prediction of expressive dynamics and expressive tempo using a dataset of Mozart piano sonatas performed by a professional concert pianist. Experiments of models trained with and without tonal tension features show that tonal tension helps predict change of tempo and dynamics more than absolute tempo and dynamics values. Furthermore, the improvement is stronger for dynamics than for tempo.
△ Less
Submitted 3 July, 2018;
originally announced July 2018.
-
The ACCompanion v0.1: An Expressive Accompaniment System
Authors:
Carlos Cancino-Chacón,
Martin Bonev,
Amaury Durand,
Maarten Grachten,
Andreas Arzt,
Laura Bishop,
Werner Goebl,
Gerhard Widmer
Abstract:
In this paper we present a preliminary version of the ACCompanion, an expressive accompaniment system for MIDI input. The system uses a probabilistic monophonic score follower to track the position of the soloist in the score, and a linear Gaussian model to compute tempo updates. The expressiveness of the system is powered by the Basis-Mixer, a state-of-the-art computational model of expressive mu…
▽ More
In this paper we present a preliminary version of the ACCompanion, an expressive accompaniment system for MIDI input. The system uses a probabilistic monophonic score follower to track the position of the soloist in the score, and a linear Gaussian model to compute tempo updates. The expressiveness of the system is powered by the Basis-Mixer, a state-of-the-art computational model of expressive music performance. The system allows for expressive dynamics, timing and articulation.
△ Less
Submitted 7 November, 2017;
originally announced November 2017.
-
What were you expecting? Using Expectancy Features to Predict Expressive Performances of Classical Piano Music
Authors:
Carlos Cancino-Chacón,
Maarten Grachten,
David R. W. Sears,
Gerhard Widmer
Abstract:
In this paper we present preliminary work examining the relationship between the formation of expectations and the realization of musical performances, paying particular attention to expressive tempo and dynamics. To compute features that reflect what a listener is expecting to hear, we employ a computational model of auditory expectation called the Information Dynamics of Music model (IDyOM). We…
▽ More
In this paper we present preliminary work examining the relationship between the formation of expectations and the realization of musical performances, paying particular attention to expressive tempo and dynamics. To compute features that reflect what a listener is expecting to hear, we employ a computational model of auditory expectation called the Information Dynamics of Music model (IDyOM). We then explore how well these expectancy features -- when combined with score descriptors using the Basis-Function modeling approach -- can predict expressive tempo and dynamics in a dataset of Mozart piano sonata performances. Our results suggest that using expectancy features significantly improves the predictions for tempo.
△ Less
Submitted 11 September, 2017;
originally announced September 2017.
-
From Bach to the Beatles: The simulation of human tonal expectation using ecologically-trained predictive models
Authors:
Carlos Cancino-Chacón,
Maarten Grachten,
Kat Agres
Abstract:
Tonal structure is in part conveyed by statistical regularities between musical events, and research has shown that computational models reflect tonal structure in music by capturing these regularities in schematic constructs like pitch histograms. Of the few studies that model the acquisition of perceptual learning from musical data, most have employed self-organizing models that learn a topology…
▽ More
Tonal structure is in part conveyed by statistical regularities between musical events, and research has shown that computational models reflect tonal structure in music by capturing these regularities in schematic constructs like pitch histograms. Of the few studies that model the acquisition of perceptual learning from musical data, most have employed self-organizing models that learn a topology of static descriptions of musical contexts. Also, the stimuli used to train these models are often symbolic rather than acoustically faithful representations of musical material. In this work we investigate whether sequential predictive models of musical memory (specifically, recurrent neural networks), trained on audio from commercial CD recordings, induce tonal knowledge in a similar manner to listeners (as shown in behavioral studies in music perception). Our experiments indicate that various types of recurrent neural networks produce musical expectations that clearly convey tonal structure. Furthermore, the results imply that although implicit knowledge of tonal structure is a necessary condition for accurate musical expectation, the most accurate predictive models also use other cues beyond the tonal structure of the musical context.
△ Less
Submitted 19 July, 2017;
originally announced July 2017.
-
Towards computer-assisted understanding of dynamics in symphonic music
Authors:
Maarten Grachten,
Carlos Eduardo Cancino-Chacón,
Thassilo Gadermaier,
Gerhard Widmer
Abstract:
Many people enjoy classical symphonic music. Its diverse instrumentation makes for a rich listening experience. This diversity adds to the conductor's expressive freedom to shape the sound according to their imagination. As a result, the same piece may sound quite differently from one conductor to another. Differences in interpretation may be noticeable subjectively to listeners, but they are some…
▽ More
Many people enjoy classical symphonic music. Its diverse instrumentation makes for a rich listening experience. This diversity adds to the conductor's expressive freedom to shape the sound according to their imagination. As a result, the same piece may sound quite differently from one conductor to another. Differences in interpretation may be noticeable subjectively to listeners, but they are sometimes hard to pinpoint, presumably because of the acoustic complexity of the sound. We describe a computational model that interprets dynamics---expressive loudness variations in performances---in terms of the musical score, highlighting differences between performances of the same piece. We demonstrate experimentally that the model has predictive power, and give examples of conductor ideosyncrasies found by using the model as an explanatory tool. Although the present model is still in active development, it may pave the road for a consumer-oriented companion to interactive classical music understanding.
△ Less
Submitted 13 December, 2016; v1 submitted 7 December, 2016;
originally announced December 2016.
