̽����ѡ

Robots operating in the real world will experience a range of different environments and tasks. It is essential for the robot to have the ability to adapt to its surroundings to work efficiently in changing conditions. Evolutionary robotics aims to solve this by optimizing both the control and body (morphology) of a robot, allowing adaptation to internal, as well as external factors. Most work in this field has been done in physics simulators, which are relatively simple and not able to replicate the richness of interactions found in the real world. Solutions that rely on the complex interplay between control, body, and environment are therefore rarely found. In this paper, we rely solely on real-world evaluations and apply evolutionary search to yield combinations of morphology and control for our mechanically self-reconfiguring quadruped robot. We evolve solutions on two distinct physical surfaces and analyze the results in terms of both control and morphology. We then transition to two previously unseen surfaces to demonstrate the generality of our method. We find that the evolutionary search finds high-performing and diverse morphology-controller configurations by adapting both control and body to the different properties of the physical environments. We additionally find that morphology and control vary with statistical significance between the environments. Moreover, we observe that our method allows for morphology and control parameters to transfer to previously-unseen terrains, demonstrating the generality of our approach.

Robots are traditionally bound by a fixed morphology during their operational lifetime, which is limited to adapting only their control strategies. Here we present the first quadrupedal robot that can morphologically adapt to different environmental conditions in outdoor, unstructured environments. Our solution is rooted in embodied AI and comprises two components: (1) a robot that permits in situ morphological adaptation and (2) an adaptation algorithm that transitions between the most energy-efficient morphologies on the basis of the currently sensed terrain. We first build a model that describes how the robot morphology affects performance on selected terrains. We then test continuous adaptation on realistic outdoor terrain while allowing the robot to constantly update its model. We show that the robot exploits its training to effectively transition between different morphological configurations, exhibiting substantial performance improvements over a non-adaptive approach. The demonstrated benefits of real-world morphological adaptation demonstrate the potential for a new embodied way of incorporating adaptation into future robotic designs.

Evolutionary algorithms offer great promise for the automatic design of robot bodies, tailoring them to specific environments or tasks. Most research is done on simplified models or virtual robots in physics simulators, which do not capture the natural noise and richness of the real world. Very few of these virtual robots are built as physical robots, and the few that are will rarely be further improved in the actual environment they operate in, limiting the effectiveness of the automatic design process. We utilize our shape-shifting quadruped robot, which allows us to optimize the design in its real-world environment. The robot is able to change the length of its legs during operation, and is robust enough for complex experiments and tasks. We have co-evolved control and morphology in several different scenarios, and have seen that the algorithm is able to exploit the dynamic morphology solely through real-world experiments.

Machine-learning models of music often exist outside the worlds of musical performance practice and abstracted from the physical gestures of musicians. In this work, we consider how a recurrent neural network (RNN) model of simple music gestures may be integrated into a physical instrument so that predictions are sonically and physically entwined with the performer's actions. We introduce EMPI, an embodied musical prediction interface that simplifies musical interaction and prediction to just one dimension of continuous input and output. The predictive model is a mixture density RNN trained to estimate the performer's next physical input action and the time at which this will occur. Predictions are represented sonically through synthesized audio, and physically with a motorized output indicator. We use EMPI to investigate how performers understand and exploit different predictive models to make music through a controlled study of performances with different models and levels of physical feedback. We show that while performers often favor a model trained on human-sourced data, they find different musical affordances in models trained on synthetic, and even random, data. Physical representation of predictions seemed to affect the length of performances. This work contributes new understandings of how musicians use generative ML models in real-time performance backed up by experimental evidence. We argue that a constrained musical interface can expose the affordances of embodied predictive interactions.

The formal evaluation of new interfaces for musical expression (NIMEs) in their use by ensembles of musicians is a challenging problem in human-computer interaction (HCI). NIMEs are designed to support creative expressions that are often improvised and unexpected. In the collaborative setting of a musical ensemble, interactions are complex and it can be almost impossible to directly evaluate the impact of interface variations. The evaluation environment also needs to be carefully considered. In the wild, concert pressures and practicalities limit experimental control. In the laboratory, studies may not sufficiently reflect real-world usage to make their conclusions relevant. To address some of these issues, we propose a methodology of rehearsal-as-research to study free improvisation by ensembles of NIME performers. In this methodology, evaluation sessions are structured to mirror established practices for improvisation training and performance development. Such sessions can allow controlled, order-balanced studies with extensive data collection in the style of factorial HCI experiments while preserving the artistic setting of a rehearsal. Experiment design, questionnaires, and objective measures such as session duration will be discussed along with two case studies.

The structure and performance of neural networks are intimately connected, and by use of evolutionary algorithms, neural network structures optimally adapted to a given task can be explored. Guiding such neuroevolution with additional objectives related to network structure has been shown to improve performance in some cases, especially when modular neural networks are beneficial. However, apart from objectives aiming to make networks more modular, such structural objectives have not been widely explored. We propose two new structural objectives and test their ability to guide evolving neural networks on two problems which can benefit from decomposition into subtasks. The first structural objective guides evolution to align neural networks with a user-recommended decomposition pattern. Intuitively, this should be a powerful guiding target for problems where human users can easily identify a structure. The second structural objective guides evolution towards a population with a high diversity in decomposition patterns. This results in exploration of many different ways to decompose a problem, allowing evolution to find good decompositions faster. Tests on our target problems reveal that both methods perform well on a problem with a very clear and decomposable structure. However, on a problem where the optimal decomposition is less obvious, the structural diversity objective is found to outcompete other structural objectives—and this technique can even increase performance on problems without any decomposable structure at all.

The widespread adoption of mobile devices, such as smartphones and tablets, has made touchscreens a common interface for musical performance. While new mobile music instrument have been investigated from design and user experience perspectives, there is little examination of the performers’ musical outputs. In this work, we introduce a constrained touchscreen performance app, MicroJam, designed to enable collaboration between performers, and engage in a data-driven analysis of more than 1600 performances using the app. MicroJam constrains performances to five seconds, and emphasises frequent and casual music making through a social media-inspired interface. Performers collaborate by replying to performances, adding new musical layers that are played back at the same time. Our analysis shows that users tend to focus on the centre and diagonals of the touchscreen area, and tend to swirl or swipe rather than tap. We also observe that while long swipes dominate the visual appearance of performances, the majority of interactions are short with limited expressive possibilities. Our findings enhance our understanding of how users perform in touchscreen apps and could be applied in future app designs for social musical interaction.

We present an approach to analyzing the motion capture data of rowers using bivariate functional principal component analysis (bfPCA). The method has been applied on data from six elite rowers rowing on an ergometer. The analyses of the upper and lower body coordination during the rowing cycle revealed significant differ- ences between the rowers, even though the data was normalized to account for differences in body dimensions. We make an argument for the use of bfPCA and other functional data analysis methods for the quantitative evaluation and description of technique in sports.

RoboJam is a machine-learning system for generating music that assists users of a touchscreen music app by performing responses to their short improvisations. This system uses a recurrent artificial neural network to generate sequences of touchscreen interactions and absolute timings, rather than high-level musical notes. To accomplish this, RoboJam's network uses a mixture density layer to predict appropriate touch interaction locations in space and time. In this paper, we describe the design and implementation of RoboJam's network and how it has been integrated into a touchscreen music app. A preliminary evaluation analyses the system in terms of training, musical generation and user interaction.

We present an approach to analyzing the motion capture data ofrowers using bivariate functional principal component analysis(bfPCA). The method has been applied on data from six elite rowersrowing on an ergometer. The analyses of the upper and lower bodycoordination during the rowing cycle revealed significant differences between the rowers, even though the data was normalized toaccount for differences in body dimensions. We make an argumentfor the use of bfPCA and other functional data analysis methods forthe quantitative evaluation and description of technique in sports.

240862 259293 247697 288285 262762

247697 288285 262762

MicroJam is a mobile app for sharing tiny touch-screen performances. Mobile applications that streamline creativity and social interaction have enabled a very broad audience to develop their own creative practices. While these apps have been very successful in visual arts (particularly photography), the idea of social music-making has not had such a broad impact. MicroJam includes several novel performance concepts intended to engage the casual music maker and inspired by current trends in social creativity support tools. Touch-screen performances are limited to 5-seconds, instrument settings are posed as sonic "filters", and past performances are arranged as a timeline with replies and layers. These features of MicroJam encourage users not only to perform music more frequently, but to engage with others in impromptu ensemble music making.

We introduce a new self-contained and self-aware interface for musical expression where a recurrent neural network (RNN) is integrated into a physical instrument design. The system includes levers for physical input and output, a speaker system, and an integrated single-board computer. The RNN serves as an internal model of the user’s physical input, and predictions can replace or complement direct sonic and physical control by the user. We explore this device in terms of different interaction configurations and learned models according to frameworks of self-aware cyber-physical systems.

Evolutionary robotics has aimed to optimize robot control and morphology to produce better and more robust robots. Most previous research only addresses optimization of control, and does this only in simulation. We have developed a four-legged mammal-inspired robot that features a self-reconfiguring morphology. In this paper, we discuss the possibilities opened up by being able to efficiently do experiments on a changing morphology in the real world. We discuss present challenges for such a platform and potential experimental designs that could unlock new discoveries. Finally, we place our robot in its context within general developments in the field of evolutionary robotics, and consider what advances the future might hold.

The goal of this tutorial is to apply predictive machine learning models to human behaviour through a human computer interface. We will introduce participants to the key stages for developing predictive interaction in user-facing technologies: collecting and identifying data, applying machine learning models, and developing predictive interactions. Many of us are aware of recent advances in deep neural networks (DNNs) and other machine learning (ML) techniques; however, it is not always clear how we can apply these techniques in interactive and real-time applications. Apart from well-known examples such as image classification and speech recognition, what else can predictive ML models be used for? How can these computational intelligence techniques be deployed to help users? In this tutorial, we will show that ML models can be applied to many interactive applications to enhance users’ experience and engagement. We will demonstrate how sensor and user interaction data can be collected and investigated, modelled using classical ML and DNNs, and where predictions of these models can feed back into an interface. We will walk through these processes using live-coded demonstrations with Python code in Jupyter Notebooks so participants will be able to see our investigations live and take the example code home to apply in their own projects. Our demonstrations will be motivated from examples from our own research in creativity support tools, robotics, and modelling user behaviour. In creativity, we will show how streams of interaction data from a creative musical interface can be modelled with deep recurrent neural networks (RNNs). From this data, we can predict users’ future interactions, or the potential interactions of other users. This enables us to “fill in” parts of a tablet-based musical ensemble when other users are not available, or to continue a user’s composition with potential musical parts. In user behaviour, we will show how smartphone sensor data can be used to infer user contextual information such as physical activities. This contextual information can be used to trigger interactions in smart home or internet of things (IoT) environments, to help tune interactive applications to user’s needs, or to help track health data.

The goal of this tutorial is to apply predictive machine learning models to creative data. The focus of the tutorial will be recurrent neural networks (RNNs), deep learning models that can be used to generate sequential and temporal data. RNNs can be applied to many kinds of creative data including text and music. They can learn the long-range structure from a corpus of data and “create” new sequences by predicting one element at a time. When embedded in a creative interface, they can be used for “predictive interaction” where a human collaborates with, influences, and is influenced by a generative neural network. We will walk through the fundamental steps for training creative RNNs using live-coded demonstrations with Python code in Jupyter Notebooks. These steps are: collecting and cleaning data, building and training an RNN, and developing predictive interactions. We will also have live demonstrations and interactive live-hacking of our creative RNN systems! You’re welcome to bring a laptop with python to the tutorial and load up our code examples, or to follow along with us on the screen!

Automatic music generation is a compelling task where much recent progress has been made with deep learning models. But how these models can be integrated into interactive music systems; how can they encourage or enhance the music making of human users? Musical performance requires prediction to operate instruments, and perform in groups. Predictive models can help interactive systems to understand their temporal context, and ensemble behaviour. Deep learning can allow data-driven models with a long memory of past states. This process could be termed "predictive musical interaction", where a predictive model is embedded in a musical interface, assisting users by predicting unknown states of musical processes. I’ll discuss a framework for predictive musical interaction including examples from our lab, and consider how this work could be applied more broadly in HCI and robotics. This talk will cover material from this paper: https://arxiv.org/abs/1801.10492