̽����ѡ

The Hum is a scalable audiovisual live performance instrument. It is a complete world full of sounding objects that interact with each other. It is based on a fictional story. The Hum currently contains around five thousand musical phrases. The sound dataset was then analysed with a bespoke set of descriptors, and the resulting representation was projected into 3D space using UMAP. The live performance involves exploring this 3D space both in terms of real-time visuals, and in a spatial audio setup. At its core the performance is the sonification of the various timbral and musical relationships of the sound dataset, expressed by various interactions triggered between neighboring sounds or using the whole dataset. The interactions may evoke associations of weather (blowing wind, rain or chain lightning). The structure of the performance follows a 3-stage plan, that is realized via improvisation. Thus the duration of the performance is flexible and can last between 15 to 30 minutes. The Hum is realized entirely in Max/MSP.

Parameter Mapping (PM) is probably the most used design approach in sonification. However, the relationship between a synthesizer’s input parameters and the perceptual distribution of its output sounds might not be proportional, limiting its ability to convey relationships within the source data in the sound. This study evaluates a basic Frequency Modulation (FM) synthesis module with perceptually motivated descriptors, measures of spectral energy distribution, and latent embeddings of pre-trained audio representation models. We demonstrate how these metrics do not indicate straightforward relationships between synthesis parameters and perceived sound. This is done using interactive audiovisual scatter plots—Synth Maps—that can be used to explore the sound distribution of the synthesizer and qualitatively evaluate how well the different representations align with human perception. Link to the code and the interactive Synth Maps are available.

The paper presents the Musical Gestures Toolbox (MGT) for Python, a collection of modules targeted at researchers working with video recordings. The toolbox includes video visualization techniques such as creating motion videos, motion history images, and motiongrams. These visualizations allow for studying video recordings from different temporal and spatial perspectives. The toolbox also includes basic computer vision methods, and it is designed to integrate well with audio analysis toolboxes. The MGT was initially developed to analyze music-related body motion (of musicians, dancers, and perceivers) but is equally helpful for other disciplines working with video recordings of humans, such as linguistics, pedagogy, psychology, and medicine.

Parameter Mapping (PM) is probably the most used design approach in sonification. However, the relationship between a synthesizer’s input parameters and the perceptual distribution of its output sounds might not be proportional, limiting its ability to convey relationships within the source data in the sound. This study evaluates a basic Frequency Modulation (FM) synthesis module with perceptually motivated descriptors, measures of spectral energy distribution, and latent embeddings of pre-trained audio representation models. We demonstrate how these metrics do not indicate straightforward relationships between synthesis parameters and perceived sound. This is done using interactive audiovisual scatter plots—Synth Maps—that can be used to explore the sound distribution of the synthesizer and qualitatively evaluate how well the different representations align with human perception. Link to the code and the interactive Synth Maps are available.

Parameter Mapping (PM) is probably the most used design approach in sonification. However, the relationship between a synthesizer’s input parameters and the perceptual distribution of its output sounds might not be proportional, limiting its ability to convey relationships within the source data in the sound. This study evaluates a basic Frequency Modulation (FM) synthesis module with perceptually motivated descriptors, measures of spectral energy distribution, and latent embeddings of pre-trained audio representation models. We demonstrate how these metrics do not indicate straightforward relationships between synthesis parameters and perceived sound. This is done using interactive audiovisual scatter plots—Synth Maps—that can be used to explore the sound distribution of the synthesizer and qualitatively evaluate how well the different representations align with human perception. Link to the code and the interactive Synth Maps are available.

This workshop is targeted at students and researchers working with video recordings. You will learn to use MG Toolbox, a Python package with numerous tools for visualizing and analyzing video recordings. This includes visualization techniques such as motion videos, motion history images, and motiongrams; techniques that, in different ways, allow for looking at video recordings from different temporal and spatial perspectives. It also includes some basic computer vision analysis, such as extracting quantity and centroid of motion, and using such features in analysis.MG Toolbox for Python is a collection of high-level modules for generating all of the above-mentioned visualizations and analyses. This toolbox was initially developed to analyze music-related body motion but is equally helpful for other disciplines working with video recordings of humans, such as linguistics, psychology, medicine, and educational sciences.

The Musical Gestures Toolbox for Python is a collection of tools for video visualization and video analysis.

Centre for Digital Life Norway (DLN) is excited to congratulate the team behind the project “The autophagic symphony – Unveiling the final rhythm” as winner of DLN’s RRI-inspired transdisciplinary side quest call.

The paper presents the Musical Gestures Toolbox (MGT) for Python, a collection of modules targeted at researchers working with video recordings. The toolbox includes video visualization techniques such as creating motion videos, motion history images, and motiongrams. These visualizations allow for studying video recordings from different temporal and spatial perspectives. The toolbox also includes basic computer vision methods, and it is designed to integrate well with audio analysis toolboxes. The MGT was initially developed to analyze music-related body motion (of musicians, dancers, and perceivers) but is equally helpful for other disciplines working with video recordings of humans, such as linguistics, pedagogy, psychology, and medicine.