Dynamic data representation model for musical cybernetics

Pauli Laine
Nokia Research Center, Itämerenkatu 11 – 13, 00180 Helsinki

1.Introduction
There have been attempts to model different musical styles and different aspects of human musical cognition using various generative methods. During last decades the artificial neuronal networks have been increasingly used to model cognitive musical behaviour. On the other hand the conventional artificial intelligence methods, like procedural and declarative description of musical knowledge, have been used to formulate musical rules (Balaban, Ebcioglu & Laske 1992). There have also been hybrid systems combining both symbolic and sub-symbolic knowledge representation (Camurri et al. 1995). These approaches have not resulted in simple and unified methodology for processing and representing musical time and time-based processes. 
Essential part of musical processing done in human mind is the feedback. If the feedback can be implemented in the generative model, the model would more closely resemble the music processing done by human brains. If the music generation is looked from the computational viewpoint, it can be seen that without feedback the power of the algorithms lies solely on the descriptive power of the used algorithm. When generating algorithm is combined with 'sensory' feedback self-emphasizing resonance may develop  (Laine 1998). It is assumed here that this resonance is essential part of musical thinking and which occurs also in musical imagery. Secondly I assume that musical time and rhythm is largely experienced and represented by using very low level pattern generator mechanisms in human brain. The MINN-model proposed by Laine (1997 and 1998) aimed to simulate this generation of cyclical repetitive patterns and rhythms. The model imitated biological pattern generators in which the time is not 'represented' but where the timed behaviour is direct result of the activity of the generator. Combining feedback to the MINN-model is not possible without dynamical representation of musical data. Dynamic representation means that there is no symbolic representation of notes and rhythms, but of processes, which can be perceived as notes and rhythms should emerge from the behaviour of the model.
The model proposed here aims to combine the methods of pattern generation (MINN-model) and feedback-system using dynamic musical data representation to testable musicological research tool. In general level the model makes possible to study new knowledge representation method, which is not declarative and which differs from procedural knowledge representation in that the timed behaviour of the system is part of its knowledge and intelligence (in a sense ‘rhythmic knowledge representation’). 
The dynamical music representation model is named NICAMUS (Neuronally inspired cellular automatons for music).  Model combines the computational theory of cellular automatons and their computer implementation to certain aspects of simulated neurons (similar connections is handled by Minsky 1972). The aim of the model is not to achieve stabile state of the network so as to represent some stored information, but to inspect the dynamics of the connection in each discrete time-slice (DTS). The dynamical time-series and musical event sequence transscribed from that is the result of the model. The NICAMUS-model gives possibilities to study the musical imagery by simulating dynamic representation of musical patterns and rhythms as well as the combined effect of pattern generator and feedback system.
The basic assumption behind the generative model is that inside the model the data that is transferred between modules consists of timed pulses. There is no separate pitch, chordal or rhythm representation. Only amount of the connections and the timing of the pulses going in these connections are varied. The purpose of this is to simulate the neural behaviour of the brain, which is generally assumed to use identical pulse strains for communication between neurons and neuron groups (Eckmiller 1991, 345).

2. The model 
NICAMUS-model consists of a network constructed from artificial neurons and the connections between them.The activity of the neurons, i.e. the generated pulses, are transferred to other neurons via the links. In each discrete time-slice the network is updated. Each successive link between neurons causes a delay of one timeslice in the progression of the neuronal pulse. Use of the delaying links is perhaps the biggest difference between the NICAMUS-model and the traditional cellular automaton connections. The minimum delay is one discrete time-slice (DTS). 
The network is initialized in the beginning of the experiment. In the initialization as many neurons as the user wants are created and creating the (in this setup random-) connections between the neurons. There is no hard-wired limit to the amount of the connections between the neurons. Each neuron can have arbitrary many input and output connections. After making of the connections and initialization of the resultlist the program then executes the functions assigned to each timestep as many times as the user have defined. It is possible to represent neuron groups or subnets using rows of aneurons. Separate rows can be assigned to different tasks in music generation system, like chordal representation, interval processing or kinematic (motor) processing. There can be also links between subnets, representing for example rhythmic information affecting the chords or dynamics.
Following operations are executed during each discrete time step (DTS): neurons are updated by calculating the result value using the instance-function and the values fetched from outcon-fields from input links. The result value is stored to outcon-field from where it goes to links. When all the neurons are updated the links are updated in turn by transferring the values from incon-fields to outcon-fields.  After updating the links the result of previous DTS are available in the successive DTS. After this the sum activation of the network (netact) is calculated by summing the outcon-value of all the neurons. Then the current situation of the network is shown using graphical representation.
The NICAMUS-network can receive feedback either from outside (like response from playing movements) or inside, from the other parts of the net itself. The feedback can be simply the activation levels of the other part of the network. It is also possible to connect the feedback either "serially" by connecting several inputs to some addition neuron, or "parallelly" by using several feedback connections.
Main purpose of the experimental setting presented here was to develop a neuronally inspired cellular automaton algorithm or network for music. The second purpose was to reveal the inherent music-processing properties of this kind of network. The network and its computer implementation work well in most situations. Even when initialized with random connections the network generates complicated "motion patterns" or noisy time-series. Only seldom the network is stuck to generate one value only. In the following some features of the generated products are discussed. The most common forms generated are periodical (period length 1 - N) patterns. 

Figure 1. The upper part of the figure shows the timeseries; the lower part the network size and activity. 
In the upper part of the figure 1 the time-series window is shown. In the lower part the momentarily situation (last DTS-of the series) is given as example of the network. The size of the circle shows the activity of the cell/neuron, the larger the circle is, the higher the activation value of the cell. One can see four rows with 12 cells each. These four rows are represented in the time-series window as four separate (yet synchronized) small time-series. The highest of the time-series shows short initial transient before stabilizing to repeating cycle.

Curriculum Vitae

Pauli Laine, PhD, University of Helsinki, 2000. Thesis subject "A Method for Generating Musical Motion Patterns"

Since early 1980's I have been interested in algorithmic composition and computer assisted musical style simulation.  I have tried numerous algorithms and approaches to gain understanding how the music is composed or improvised and how to simulate that. Lately my scientific work has been concentrated around two main themes: musical feedback and movement related compositional systems. I have experimented the movement related compositional systems with two series of experiments. The first one was the generation of musical patterns and rhythms using mutually inhibiting motion neurons (MINN) and the second one is finding the relations of finger movement and certain compositional practices in Baroque music.The musical feedback handles the control systems used by music player or composer to adjust the playing and composing parameters using delayed information. In the compositional domain modelling the feedback is complicated by the complexity of the data to be represented. The paper presented in this seminar tries to offer an practical solution for representing musical data in flexible manner.

Some publications:

Kohonen, Teuvo; Laine, Pauli; Tiits, Kalev; Torkkola, Kari 1991. A nonheuristic automatic composing method. In: Todd, Peter M.; Loy, Gareth, Music and Connectionism. Cambridge: MIT Press.
Laine, Pauli 1997. Generating Musical Patterns Using Mutually Inhibited Artificial Neurons. Proceedings of ICMC 1997, Thessaloniki.
Laine, Pauli 1998. Cybernetic perspective to Music Algorithms - The Control Feedback in Cognitive modelling. Proceedings of 5th International Conference of Musical Perception and Cognition, Seoul 1998.
Laine, Pauli; Lassfolk, Kai 1999. Feedback in Musical Computer Applications. International Computer Music Conference, Beijing 1999.
Laine, Pauli 1999. Motor Neuron Based Virtual Drummer. International Computer Music Conference, Beijing 1999.
Lemström, Kjell; Laine, Pauli 1998. Musical Information Retrieval Using Musical Paramaters 1998. Proceedings of ICMC 1998, Ann Arbor, USA.