Controllable Sound Environments

Sensors, actuators, microprocessors, and wireless connection facilities are increasingly being embedded into everyday objects. These can be augmented with sonic features that make the environment more enjoyable, social life more interesting, and personal life more relaxed and healthy.

Grandma and me

Hi, I'm a teenager - 15 years old - and I like to wake up late, at least on Sundays.But today I decided to spend a few hours with my grandmother, so I set my alarm clock bed for 9 a.m. First, the bed tries to wake me up gently, by vibrating and making purring noises. Even though I am not sleeping anymore, I like to wait until the bed gets more nervous, when it realises I am still lying on it, and starts making harsh rhythmic movements and squeaking. I love it!

I get up and go into the living room. It's a mess after last night's party. You know, mum and dad are on vacation, so ... Chairs are everywhere; chips and peanuts all over the floor. I play some of my favourite hip-hop music while I put things back in place and prepare for the day. Different MCs and DJs are embodied in different pieces of furniture. I know it sounds retro, but it is so cool to move DJ Grandmaster Flash while I am moving that old armchair around. While I move through the house, the music seamlessly follows me through the objects I pass. When I leave home, I put on my headset and keep riding the beat. The headset doesn't cut me off from the environment,though. When I catch sight of a strange bird singing, I look at it and put my hand to my ear. This gesture activates acoustic zooming, and I can appreciate the bird song in isolation. But then I am distracted by the sight of a friend of mine chatting to a girl. Instinctively, I steer my acoustic zoom towards them, but I realize she is wearing one of those new active jackets that can create an acoustic shield around you. I wonder what they are talking about.

I reach my grandma's house. She became almost deaf about five years ago, but she is really brave and decided to get an implant of powerful bionic ears. In recent years, she has become more and more worried about the bad things that could happen to her. That's why my dad bought her a new door. As we leave the house and she closes the door, she proudly explains that the complex sound of the lock tells her that the lights are switched off, the gas is turned off, the fish are fed, and the window in the living room has been left wide open. I'm sure she left it open on purpose, but we go back in and close it anyway.

We are going to the Fred Astaire club today. Grandma is wearing brand new Mike shoes. She feels much more confident about herself in these shoes, because they give her bionic ears some sonic feedback about equilibrium so she's not afraid of falling while she moves around. Before we go into the club she wants to check on her health, which is promptly sonified through her bionic ears. Everything sounds fine, so we go in just as the show's starting. There are a dozen over-eighties there, wearing Mike shoes, tap dancing, and clearly having a good time. Their subtle, gentle movements trigger a massive and diverse set of rhythms. Who knows - it may even be cooler than hip hop!

On the way back home, grandma tells me how different the town was when she was a child. There was even a working water mill. Fortunately, we can both enjoy its sound. Both my cheap headset and her bionic ear can induce selective silence and let the lost sounds of the town emerge from history. That makes her remember even more. It's fun being with grandma.

Musical Instruments for All

In 2020, many sound devices will have a general purpose computer in them and will include quite a number of real-time interaction capabilities, sensors and wireless communication. Basically, any sound-producing device will be able to behave like a personalised musical instrument. Music making will become pervasive and many new forms of communication with sound and music will become available. Music content will be inherently multimodal and music making available to everyone: music from all to all.

How I became a professional musician

A year ago I bought the new wearable mobile device from SMC Inc. With it I was able to listen to my favourite songs and interact with them in ways not possible with the previous generation of devices. Now I was able to change many aspects of the songs by gestural and vocal control. Some of my friends were really good at it, and I started to improve my skills by practising on my home multimedia system. This system includes Jeeves, a virtual musical assistant, which observes and analyses my body movement, my singing and my musical abilities in general. Jeeves teaches me how to express myself in the style of famous musicians, from The Beatles to cellist Yo-Yo Ma. After a period of training, I was ready to play and jam with other users over the Internet and get advice from more expert users or their virtual musical assistants. After a couple of months, I started to get a good reputation in the community. One day, a group of users asked me to join them in person at a discotheque. In that discotheque we were able to use our mobiles to plug into a music role in the overall show. People took all kinds of roles; some were projected as visual characters on the surrounding space and walls, some were projected into moving lighting, others, like me, were controlling the expressivity of the music being played. Since I was a beginner, I took a simple role as the controller of the timbral aspects of the drum. Another person took control of the drum sticks. We had to dance to coordinate the rhythm in this shared drum set with the other visual and musical roles. This experience felt much more physical and exciting than being at home with my multimedia system. In the discotheque I had the feeling of being part of a community and of real teamwork. The various haptic devices in my clothes heightened my aural and visual perception and interaction with friends.

I met these new friends many times; I practised a lot in discotheques and at home. I developed my own style and developed good skills in controlling virtual instruments, with Jeeves evolving with me and my community. I could control expression and lead my friends in improvisations and jam sessions. One day Jeeves asked me: "Could I please change my name to Madonna?; I feel that my background knowledge has changed". I realised that was true and changed her name. Today I am a professional musician: an MJ (Music Jockey). Madonna and I prepare the shows on the authoring system at home. I design the structure, framework, roles, and musical material to be used. In my shows, I sometimes improvise with acoustic instrument players. I collect data on my sound device by monitoring their movements and performance choices. I can also monitor and analyse all the events in the show and the behaviour of everyone involved in it, including the audience. My virtual musical assistant Madonna wants to change her name again. Now she would like to be called Karajan.

Personalized Musical Devices

Current portable mp3 players -despite their simplicity - have already radi- cally changed human music-listening behaviour. Now, the first web-based music information systems which provide contextual information about music simply by connecting already existing services (such as Wikipedia, CDDB, lastFM,etc.), without the utilisation of any musical expertise, are beginning to emerge. Based on current trends in SMC research, we predict that such systems are likely to further develop in the direction of multi-modal, interactive, open and adaptive systems that support both beginners and experts from different cultures in ac- cessing music and music-related information.

My new music friend

I take my expert music companion with me anywhere, anytime, because I love music. The companion doesn't just play music. It gives me a lot of other information about the music - from `practical' things such as transcriptions of instruments and harmonies, to animated visualisations of the structure of the music, contextual information such as style, historical and cultural relations, and the relationship of the piece to other, related pieces and styles.

My device is easy to use. I can talk to it, or I can shake it to show it the kinds of rhythms I like. It is aware of the music being played on radio stations and available in music databases world-wide, and it finds new music that I like in a particular situation. I can point it at music being played by a street band, and it will tell me what it is. It understands my intentions and learns my musical preferences. Sometimes it will surprise me, teaching me something new about music and my taste. And by the way, having had nano-sized loudspeakers (painlessly) implanted in my ears, I listen to my music without bothering with bulky headphones and earplugs.

My music companion also helps me out in social contexts. When I am desperately looking for a date, my companion alerts me there's a dance party around the corner for people with a similar interest in Brazilian music. When I get to it, my companion contacts the DJ system and sends it some of my favourite pieces (rare Brazilian stuff). The girl in the corner just goes "Wow".

My music companion is no longer an isolating device that runs playlists; it's a friend that enhances my musical abilities, reflects my personality and helps me to socialise.

Research

This section aims to identify the major research trends within which sound and music computing is to be situated. The focus is on trends in ICT, the cognitive sciences and the humanities. Given the broad scope of sound and music computing, this section devotes special attention to the rise and importance of a multidisciplinary research space that is the motor for innovation in society. References to this general research space can be found in several reports from the European Commission and the National Science foundation of the US (European Commission – New Instruments, 2004, European Commisison – Research Area, 2007, National Science Foundation, 2004). Below, we aim at identifying the specific research trends that are deemed to be relevant to sound and music computing. Each trend will be summarized by a short statement that aims at identifying the core issue that is relevant to future challenges of the sound and music computing research field.

Research Trend 1: Rapid progress in ICT

In recent decades, progress in sound and music computing has been driven by revolutionary developments in information technology. The transitions from analogue to digital data processing and from wired to wireless mobile data-communication have been key components in this development. The relentless rate of annual to biennial doubling of storage capacity, bandwidth and data-crunching computing power has been unprecedented in history, leading to fundamental transformation of all aspects of the production-processing-distribution-consumption chain of sound and music content. In no other field has an entire processing chain been digitalised and made available on broadband networks and mobile devices on such a massive scale. In this development, technological progress has had a direct empowering influence both on scientific knowledge and on end applications, which in turn have impacted on the development of new technologies. In the context of this development, a number of consequences for research can be identified (Hengeveld, 2000, ITRS Consortium, 2007, NEM Consortium, 2007, Microsoft Research Cambridge, 2006).

Statement 1: The rate of increase in storage capacity, bandwidth and data-crunching computing power is leading to fundamental transformations in all aspects of the music economical chain.

First, the increasing capacity of data storage and transfer supports the accumulation of, and easy access to, ever-larger volumes of data. A resulting benefit is better access to knowledge, such as online access to vintage publications, supplementary data, new publication formats and so on. This accessibility is empowering to the scientist. At the same time, like the invention of printing, it has an effect on the embodiment of knowledge itself, shifting the centre of gravity of knowledge from brain, to book and onward to database.

A second effect is the shift towards data intensive methodologies that involve gathering or compilation of large volumes of data. This allows a focus on data intensive phenomena, phenomena that are either intrinsically complex, or else accessible only as patterns within multiple or complex observations. In the field of music studies, an enquiry might call for the processing of a large library of musical scores or a large database of audio data. A few years ago such a quest might have remained untouched for lack of access to the data, or no room to store it in the computer, or no time to wait for the computer to give an answer. By the same token, a topic once respectable for its technical difficulty might suddenly become trivial. In ways such as these, information technology affects the focus of science.

A third consequence is the shift away from analytical and theoretical approaches towards a reliance on computer models and simulations. This approach, which can be observed in fields as diverse as pure mathematics (computational proofs), statistics (Montecarlo methods, bootstrap), biology (DNA sequence alignment), linguistics and speech engineering (data-driven methods), has engendered a degree of unease and debate (Seiden, 2001). Does a proof that only a computer can follow really contribute to our understanding? Similar unease met the invention of algorithms, infinity, or proof by induction. In similar vein, one can ask whether a drum machine can be qualified as a musician? Or whether 'jazz improvisation' by a computer is really a genuine improvisation?

A fourth consequence is the development of machine-embedded knowledge such as that gathered by machine-learning techniques. Arguably these techniques come closer to delivering the promises of intelligence than has the so-called Artificial Intelligence (AI) research itself. With them, intelligence is attained more by the clever use of tricks and devices in machines than by the artifice of man. At the confluence of statistical estimation techniques and neural network theory, machine learning harnesses the computer to compile and extract regularities from massive quantities of data. The knowledge thus obtained, usually impossible to describe to a human brain and useless without a computer, is nonetheless empowering for web search, spam filtering, or musical content indexing and retrieval. As models of brain processing, machine-learning techniques may eventually provide a bridge between information technology and neurosciences. Particularly relevant to music technology are new techniques of signal processing related to machine learning.

In summary, progress in information science and technology is fuelling a drive towards data- and computation-intensive approaches to knowledge acquisition and problem solving, particularly in domains relevant to sound and music computing. These have deep implications for the nature of scientific and technological knowledge and how it is brought to bear on our needs.

Statement 2: Information technology is profoundly reshaping the methodologies and scope of scientific inquiry and technological development.

Research Trend 2: Cognitive science: from musical mind to brain

Cognitive sciences (Wilson and Keil, 1999) focus on how humans interact with their environment, mostly from the viewpoint of perception and action. Developments in this research domain have had a huge impact on sound and music computing. In fact, studies on musical memory, learning and all activities related to music perception and action, such as extraction of high-level information from musical stimuli or gestural sound control, can be considered the basic constituents of sound and music computing applications.

The cognitive science of music (as practised in, for example, cognitive musicology, experimental music psychology or the neurosciences of music) has its focus on the semantic gap that exists between our daily meaningful experiences with sound and music on the one hand, and the encoded physical energy of sound and music on the other. When dealing with music, we call upon content and meaning, whereas the encoded physical energy is just a way of storing information in a technological device. How are the two connected? How can we access the encoded information by means of meaningful actions? Research in cognitive science aims at providing new insights into this semantic gap problem. Several different approaches to solving this problem can be distinguished.

A first research direction starts out from the premise that the human mind is embodied (Knoblich et al., 2006). Rather than trying to solve the semantic gap problem by looking at formal structures and higher-level or low-dimensional representational spaces, the relation between human meaning and encoded physical energy is here seen as being mediated by the human body. For example, if an ambiguous musical rhythm is presented, then it is assumed that the motor system of the human body engenders the anticipatory mechanisms (called emulation) that allow a disambiguated auditory perception of it. Action is here seen as a crucial component for auditory perception, with action and feedback mechanisms being considered at different processing levels, from feedback mechanisms in the auditory periphery (e.g. the role of outer hair cells in attenuation) to the role of intended actions in perception. The embodied viewpoint may revolutionise how we think about ICT development in that it calls for new technologies that mediate between the human mind and its musical environment, based on a multi-sensory approach to sound and music computing (Leman, 2007).

Statement 3: The embodied viewpoint calls for new technologies to mediate between the human mind and the environment.

A second research direction is concerned with the methodologies for acquiring knowledge about the semantic gap problem. In the last decade, these methodologies have been extended from behavioural to brain research. Knowledge about the brain is progressing rapidly and at multiple scales which include molecular, synaptic, cellular, cell assembly, and regional and functional anatomy as revealed by brain imaging. Today our tools include molecular biology techniques for probing the membrane and synaptic properties of neurons, physiological recording techniques to observe entire neuronal assemblies, non-invasive imaging techniques to probe activity within the human brain, computational tools to gather and process the resulting data, and theoretical tools to make sense of the complexity of what is observed. Some recent studies in neurophysiology include the use of awake preparations (often coupled with behavioural studies), multiple unit recordings, simultaneous invasive and non-invasive brain imaging techniques (to calibrate one with respect to the other), selective brain cooling, optical imaging and the coupling of one of these with genetic engineering or biochemical manipulations to probe specific stages in processing. Research in brain imaging includes the use of higher magnetic fields for structural and functional MRI (magnetic resonance imaging), increased numbers of channels in EEG (electroencephalography) or MEG (magnetoencephalography), simultaneous recording of fMRI and EEG, or EEG and MEG, and use of pre-surgical supradural or intracortical recordings from patients to obtain 'close up' snapshots of brain activity.

An important facilitating factor in these developments is progress in hardware and software techniques for handling and interpreting the massive data sets produced by brain imaging. In short, there is presently a rapid development of different technology-driven methodologies that provide new insights into how the brain is involved in the semantic gap problem.

Statement 4: New technology-driven methodologies are providing new insights into how the human brain processes sound and music.

A third major research effort, situated in theoretical neurosciences, is about the tight interaction between signal processing and machine learning techniques on the one hand, and models of neural processing on the other. A common goal is to find techniques that can harness the extreme complexity of relevant patterns in data (for example databases of environmental, speech or musical sounds) or the structures and mechanisms observed within the brain. The computer here is used as an aid to control a degree of complexity of which our brains cannot otherwise easily comprehend. One promising angle of enquiry is the use of data-driven methods to simulate the processing mechanisms (natural or artificial) under the drive of the data patterns that it is to process. This method can be used as an alternative or complement to more traditional engineering techniques.

The above developments lead to often rather wild speculations on the possible future benefits of neurosciences to computing. An example of such a hypothetical breakthrough might be the possibility of 'downloading' entire cognitive or perceptual processing mechanisms to software. This could result from a combination of progress in recording techniques, theoretical neurosciences and machine learning. Another hypothetical breakthrough (heralded by well-estabished cochlear implant technologies and recent experiments with animal models and impaired humans) could be the widespread development of brain-machine interfaces (BMI). This could result from a combination of progress in interface hardware (e.g. miniaturised electrode arrays), signal processing (to factor out 'noise') and machine learning (to translate between the different codes used by brain and machine). All this is likely to have a huge impact on the sound and music computing field. Examples are hearing aids (e.g. cochlear implants) that allow their users to listen to music at a high quality level, or an intracortical implant that would allow a quadriplegic to play the piano.

Statement 5: Cognitive sciences and neurosciences offer a rapidly expanding window on the human mind and brain, thereby providing new possibilities for solving the semantic gap problem.

Research Trend 3: From subjective experience to cultural content

Research in the humanities is focused on signification practices; that is, on how human beings make sense of their environment and give meaning to their lives. The humanities view this signification practice from a subjective and experiential point of view. Therefore, research of this kind includes anthropology, area studies, communications, cultural studies and media studies. The humanities not only provide insights into these aspects but also train people in the skills necessary for practitioners (e.g. in music playing, painting, film making). Traditionally, research methodologies in the humanities are based on analytic, descriptive, critical or even speculative and imitation approaches, although recent approaches also involve quantitative and empirical studies (e.g. Diamond, 1999; Foster, 1985; Tomasello, 1999). In the cultural and creative industries (KEA, 2006), the humanities can provide the content needed to develop a significant partnership between culture and technology.

Several research efforts in the humanities address this issue. A first approach has adopted the belief that subjective factors (related to gender, education and social and cultural background) play a central role in how people deal with technology. Humanities research may provide the necessary analysis of the role of subjective factors and the social and cultural contexts in which technological applications will function. Knowledge of these factors needs to be incorporated into music retrieval systems and interactive music systems.

Statement 6: Subjective factors play a central role in how people deal with technology in relation to sound and music.

A second research approach is concerned with what is sometimes called 'medialogy'; that is, an approach which combines technology and creativity to design new processes and tools for art, design and entertainment. It involves insight into the creative processes, thoughts and tools needed for media-productions and other arts to exist. Clearly, medialogy is at the crossroads of the human sciences, the creative arts and technology. As such, it is a central pillar of the creative industries.

A third research approach is concerned with the transformation of the cultural sector into the digital domain. This involves the digitalisation of a large part of our cultural heritage. From the humanities point of view, the preservation and archiving of cultural heritage poses huge challenges with respect to issues such as the authenticity of documents, flexible multi-language access and the provision of proper content descriptions of objects from multifarious cultures.

Statement 7: Technology, creative approaches to art, design and entertainment and the digitalisation of a large part of our cultural heritage stimulate each other.

A fourth key topic in the humanities concerns the role of the human body, embodiment, and corporeal skills in signification practices. Human skills, which often require intensive learning, have been studied and described for centuries from a humanistic point of view, often from entirely different cultural perspectives. Accordingly, the humanities provide a rich source of theories, concepts and traditions that are highly revealing and inspiring for new empirical studies and technological applications. An example is the Laban theory of effort (Laban and Lawrence, 1947), which provides a speculative theory but very valuable insight into choreography and expressive moving. This theory can be straightforwardly related to music perception, leading to the interesting approach of gesture-based music retrieval. Another example concerns the philosophical views on intentional behaviour of the human body and how this is currently being integrated into a neuroscientific approach to empathy and social cognition (Metzinger, 2003). The focus on the human body in artistic research is clearly connected with the empirical study of embodiment in cognitive science. In fact, it is thanks to the humanities (e.g. phenomenology, post-structuralism, post-modernism) that this topic has become a genuine research topic on the agenda of empirical sciences that deal with perception, action and the use of tools and technologies. Indeed, some aspects of embodiment, involving emotions and the gesture related to them, can be straightforwardly explored and used in technology-based artistic and cultural applications, even if our knowledge about these processes is limited.

In short, the humanities offer a very rich background from which the problem of the semantic gap can be addressed. Its focus on specific topics such as the human subject, embodiment and social and cultural interaction, along with its often descriptive analytic approach, is highly valuable from the perspective of content creation.

Statement 8: The humanities offer the cultural background and content for sound and music computing research.

Research Trend 4: The rise of multidisciplinary research

Scientific research is currently witnessing two opposing, though intimately related, approaches. On the one hand, it continues to differentiate into more and more specific and narrowly circumscribed sub-fields owing to the accelerating accumulation of ever more specific knowledge. At the same time, new multidisciplinary research fields are emerging within academia, for example in the life sciences, neurosciences and earth sciences. Understanding the complex phenomena facing mankind - from climate change to new epidemics to global economic and social developments - requires the integration of expertise from many fields. The growing importance of multidisciplinarity is being increasingly recognised in research funding agencies and educational organisations.

According to a report recently presented at the OECD Global Science Forum Workshop (National Institutes for Health [NIH], 2006) “[t]he increasing multidisciplinary nature of research [...] is an important overall trend in science policy. For example, during the past four years, the fraction of interdisciplinary research at the United States National Science Foundation has increased significantly”. The NIH Roadmap for Medical Research further states that “the traditional divisions [...] may in some instances impede the pace of scientific discovery”. In response to this, the NIH is establishing “a series of awards that make it easier for scientists to conduct interdisciplinary research”.

As early as the year 2000, The National Sciences and Engineering Research Council [NSERC] of Canada set up a special Advisory Group on Interdisciplinary Research (AGIR) with a mandate to study how interdisciplinary research could be better supported (NSERC, 2002). In 2003, the National Science Foundation [NSF] promoted a study on the convergence of technologies (NSF, 2003) which concluded that: “In the early decades of the 21st century, concentrated efforts can unify science based on the unity of nature, thereby advancing the combination of nanotechnology, biotechnology, information technology, and new technologies based in cognitive science”. Similarly, research funding institutions all over the world are beginning to recognise the need to give special attention to multidisciplinary research funding.

Of course, the fundamental importance of multidisciplinary research is also acknowledged by the European Commission. In the field of ICT, which is of direct relevance to sound and music computing, the “Future and Emerging Technologies” (FET) programme is explicitly targeted towards innovative, multidisciplinary work - in the chapter dedicated to FET, the ICT work programme of FP7 calls for “interdisciplinary explorations of new and alternative approaches towards future and emerging ICT-related technologies, aimed at a fundamental reconsideration of theoretical, methodological, technological and/or applicative paradigms in ICT”, one of the goals of FET being to “[help] new interdisciplinary research communities to establish themselves as bridgeheads for further competitive RTD” (ICT-FET Work Programme, 2007).

Sound and music computing is by definition an multidisciplinary field, ranging from the natural sciences like physics and acoustics through mathematics, statistics and computing, all the way to physiology, psychology and sociology. The global trend towards the recognition of multidisciplinarity should help sound and music computing establish itself more confidently as an encompassing discipline that studies a phenomenon of central relevance to humans in all its necessary breadth. In addition, the emergence of new multidisciplinary fields of research and application is producing new points of contact for sound and music computing.

A prime example of such contact is the current rise of the so-called creative industries (KEA, 2006). While the notion creative industries refers to a sector of the economy, its current upsurge (also in terms of public awareness) also leads to new opportunities for creative multidisciplinary research at the intersection of art, design and technology. Sound and music computing can and will play an important role here. The case of the creative industries also highlights once more - if that were needed - the close ties between scientific research and the arts (see also the Industrial Context section). Artistic visions coupled with creative application ideas are likely to drive sound and music computing research in more ways than can currently be envisioned, resulting in entirely new environments, devices and cultural services.

Statement 9: Multidisciplinary research is increasingly seen as a necessity and an asset, and special programmes for fostering and funding it are being developed. Sound and music computing can take advantage of this and should actively seek alliances with other disciplines, including the arts.

To sum up, in this section we have identified some major trends related to the rapid progress in ICT, the development of cognitive science and the advent of brain science, the role of human sciences in addressing the human subject and its action-related contexts, and the multidisciplinary nature of scientific research. Sound and music research is at the cutting edge of these trends. It is driven by these general trends in research and it plays an active role in pushing the most advanced stages of each of these developments.

References

Diamond, J. (1999). Guns, germs, and steel. The fates of human societies. New York: Norton & Comp.

European Commission – Research Area (2007). European research area. http://cordis.europa.eu/era/ .

European Commission – New Instruments (2004). Evaluation of the effectiveness of the New Instruments of Framework Programme VI. http://ica.cordis.lu/documents/documentlibrary/ADS0006763EN.pdf

Foster, H. (Ed.) (1895). Postmodern culture. London and Sydney: Pluto Press.

Hengeveld, P, Best, J.-P. , van Beumer, J. , Hooff, B. , van den Poot, H. , and R. de Westerveld (Eds.) (2000). Research trends in information and communication technology: Uncovering the agendas for the information age.Telematica Instituut, Enschede, The Netherlands.

ICT-FET Work Programme (2007). Future and Emerging Technologies. European commission. Information society and media. http://cordis.europa.eu/ist/fet/

ITRS Consortium (2007). The international technology roadmap for semiconductors. http://www.itrs.net/home.html.

KEA European Affairs [KEA] (2006). The economy of culture in Europe. Technical report. http://www.keanet.eu/Ecoculture/ecoculturepage.htm.

Laban, R., & Lawrence, F. C. (1947). Effort. London: Macdonald & Evans.

Leman, M. (2007). Embodied music cognition and mediation technology. Cambridge, MA: The MIT Press.

Metzinger, T. (2003). Being no one: The self-model theory of subjectivity. Cambridge, Mass.: MIT Press.

Microsoft Research Cambridge (2006). Towards 2020 Science. http://research.microsoft.com/towards2020science/.

National Institutes for Health [NIH] (2006). Interdisciplinary Research. http://nihroadmap.nih.gov/interdisciplinary.

National Science Foundation (2004). National Science Foundation Strategic Plan FY 2003-2008. http://www.nsf.gov/pubs/2004/nsf04201/FY2003-2008.pdf

Natural Sciences and Engineering Research Council [NSERC] (2002). First report of the advisory group on interdisciplinary research. http://www.nserc.ca/pubs/agir/AGIR_e_report.pdf.

NEM Consortium (2007). Networked and electronic media - European technology platform: Strategic research agenda. http://www.nem-initiative.org/.

Seiden, S. (2001). Can a computer proof be elegant? Communications of the ACM, 32: 111-114, 2001.

Tomasello, M. (1999). The cultural origins of human cognition. Cambridge: Harvard University Press.

Wilson, R.A. and Keil, F. (Eds.) (1999). The MIT Encyclopedia of the cognitive sciences (MITECS). Cambridge, Mass.: The MIT Press.

Education

The context of education of sound and music computing is quite complex, mainly due to its multidisciplinary nature and the consequent difficulty of fitting it into the traditional, discipline-oriented focus of most university level studies. There are almost no specific undergraduate degrees in sound and music computing and the possibilities for a specialisation in this field are centred at the graduate level, where multidisciplinarity is more common.

In Europe, the major developments in education are due to the so-called Bologna Declaration, which aims at creating a common space for higher education in Europe. Below, we identify the trends that are most relevant to sound and music computing. This will be done at each of the three university education levels, namely, Bachelor, Master's and PhD.

Educational Trend 1: The new European higher education area

The EU drive towards the creation of an open European higher education area (EHEA) is both a reaction to and a reinforcement of the profound changes which have occurred in recent years: universities are educating larger numbers of students, from a wider range of backgrounds and with a wider range of skills, on entry; students are more mobile, spending parts of their education in other countries. This drive, initiated with the Bologna Process (European Commission – The Bologna Process, 2007), is creating a framework that enables closer cooperation between higher education institutions in Europe, facilitates student and staff mobility and increases the attractiveness of European higher education in the world. In the following paragraphs, we discuss these trends and their significance for the design of new curricula in the field of sound and music computing.

Improving quality in the curricula is seen as one of the keys to greater recognition of qualifications across Europe, and this viewpoint is being taken by many universities as an opportunity to update and add more flexibility into their programmes (Reichert and Tauch, 2005). These changes will definitely be beneficial for multidisciplinary fields like sound and music computing; in fact many institutions explicitly praise the new freedom to design multidisciplinary Master's programmes, as well as programmes in emerging areas of science and knowledge. The wave of reform in European higher education seems to be going even further and deeper than the Bologna reforms themselves.

A second key ingredient in curricular reform is the link between higher education and employment. The Bologna Declaration particularly calls for undergraduate degrees to be relevant to the labour market. There is in general a growing push towards shorter study cycles, and many EU countries have already adopted the two-cycle qualification structure based on the Bachelor's and Master's distinction (Tauch, 2004). Employability is also seen as an important criterion in the design of new degrees in sound and music computing. The music/multimedia industry at large is in the middle of important changes and is trying to adapt to the new markets and exploring the potential of sound and music computing technologies (see the Industrial Context section). New curricula in sound and music computing have the opportunity to address these emerging labour markets.

A major recent change in higher education has been the increase in student mobility. A considerable part of overall mobility is supported through the EC's Erasmus/Socrates programme (European Commission – Socrates/Erasmus, 2007)), established in 1987, which seeks to reinforce the European dimension of higher education by encouraging transnational cooperation between universities and boosting European mobility. The figures for mobility reflect a steady improvement, but remain below what the Commission considers necessary (European Commission - Education and Training 2010, 2007). Moreover, the EU still attracts less talent than its competitors (European Commission – Lisbon, 2007). Sound and music computing research in Europe has a successful track record involving excellence spread over several centres which have gained world leadership through complementarity and coordination supported by EC funding. This excellence has to be exported to the higher education domain, in order to attract students, scholars and researchers from other world regions.

Statement 1: The creation of a common space of higher education in Europe give more possibilities for designing curricula in sound and music computing.

Educational Trend 2: Discipline oriented bachelor education

The tradition of bachelor (undergraduate) education is very much discipline oriented. A student has to choose a curriculum aimed at developing a number of specific competences in a particular discipline plus a few general academic and professional competences. However there are curricula in Europe that are more multidisciplinary or that allow a student a wider choice of itineraries, thus permitting the design of 'custom made' curricula. With respect to research, the involvement of bachelor students in such activities as a normal part of their curriculum is still very exceptional. Given that there are many academic disciplines integral to sound and music computing research, the education given in all the bachelor degrees supporting these disciplines is of interest to any future sound and music computing researcher. Thus a student wanting to become a sound and music computing researcher might choose a bachelor degree related to musicology, physics, computer science, electrical engineering, psychology, music composition, etc... Within most of the undergraduate programmes that support these disciplines, there are specific courses that might be of very great relevance. But in most cases it really depends on the professor responsible for the course and the special focus given to it. Figure 1 and 2 provide some indicative data about content areas in courses and curricula. These data were gathered in a survey by the S2S2-consortium and will be updated and expanded in the future. The content areas for education in sound and music computing include systematic musicology, auditory and music perception-action, auditory and music cognition, music acoustics, audio signal processing, hardware and software, sound modelling, sound analysis and coding, music information processing, music performance, multimodal interfaces, sound design and auditory display, and application areas.

Statement 2: Numerous paths, embedded in different well-established undergraduate degrees, can be designed to approach a multidisciplinary field such as sound and music computing.

In the context of sound and music computing, the music conservatories are a special case of higher education institutions. Traditionally, they have a strong professional orientation and thus might not provide the necessary background for a musician wanting to follow a research career. This situation has been slowly changing, due both to the transformations taking place in the music profession and also, in Europe, to the inclusion of the conservatories into EHEA (European Association of Conservatoires, 2005). Slowly, the conservatories are converging with the university system. It is now becoming quite common for a conservatory to offer a degree with a strong technological component. There are, for example, conservatory degrees in sound recording, tonmeister, sonology, music technology, electro-acoustic music, etc... Most of these degrees remain professionally oriented but very much related to sound and music computing. Conservatories are also slowly incorporating the idea of research as one of their institutional aims and are designing curricula which are closer to the university model.

Statement 3: New conservatory degrees are a model for professionally oriented undergraduate curricula in sound and music computing.

Apart from the traditional university degrees and the case of the music conservatories, there are quite a number of multidisciplinary undergraduate programmes related to sound and music computing, especially in the US and Great Britain. In the Anglo-Saxon system, it is much easier for universities to establish multidisciplinary programmes or even to allow student-centred curricula with individual academic pathways. However, there is an ongoing discussion among academics and researchers about the type of undergraduate education that is best suited to the preparation for a research career in a multidisciplinary field like sound and music computing. Should it be a strongly discipline oriented undergraduate degree or a multidisciplinary programme?

The adoption of a common system of credits, such as the ECTS system, plus the existence of funding programs like Erasmus to support mobility have had a big impact on undergraduate education too. They have led students to become familiar with other approaches to a given field and have given them the opportunity to take courses not offered in their home university. The Erasmus programme has also facilitated the creation of networks of universities with complementary undergraduate degrees in a given discipline, so that experiences among faculty members can be shared and the curricula opportunities for students are widened. Due to the variety of disciplines and methodological approaches involved in the sound and music computing field, it is not easy to find educational institutions with an extensive coverage of all of them. It is thus very useful for a bachelor student wanting to get a wider view of the field to take courses in different centres.

Statement 4: Bachelor (undergraduate) degrees with multidisciplinary contents encourage student mobility.

Educational Trend 3: Multidisciplinary studies at Master level

The objective of a Master programme is academic or professional. The academic Master serves as the bridge between undergraduate programmes, which are mainly based on courses, and PhD studies, which are mainly based on research. These Master degrees are generally developed by universities that take advantage of existing research strengths. Therefore, the Master programmes tend to reflect the research focus of university departments and faculty. Universities have a large degree of autonomy in setting up and modifying their Master programmes, much more so than at undergraduate level. These programmes can therefore be more easily adapted to universities' educational and research strategies. Research Masters used to be common in Great Britain but rare in continental Europe. But as part of the Bologna process, most European universities are now integrating PhD courses into Master's programmes and creating new Master degrees (Reichert and Tauch, 2005). Many programmes continue the traditional discipline oriented focus, thus offering a clear continuity from undergraduate studies, but they tend to have a greater degree of flexibility. The students have a greater choice of optional courses and, since the research thesis is a major part of the programme, they are able to work independently under the supervision of a tutor.

Statement 5: It is becoming easier for university faculties and research groups to introduce a student enrolled in a Master programme into any given research field.

In the last few years there has been a proliferation of multidisciplinary Master programmes. Many of the key current research issues require multidisciplinary approaches and researchers need to be trained appropriately. Multidisciplinary education requires collaboration between institutions and thus there is a clear trend toward promoting it. Collaborations between departments of the same university, between universities of the same country and between universities of different countries are becoming commonplace.

At the European level, the Erasmus Mundus (European Commission - Erasmus Mundus, 2007) is a co-operation and mobility programme which supports European top-quality Master courses and enhances the visibility and attractiveness of European higher education in third countries. It also provides EU-funded scholarships for third country nationals participating in these Master Courses, as well as scholarships for EU-nationals studying at partner universities throughout the world.

It is a challenge for music institutions to offer musicians, in addition to instrumental training and practice, a reflective environment that nourishes innovation and creativity paired with the extension of knowledge and artistic understanding (European Association of Conservatoires, 2005). It becomes equally interesting when attempts are made to bridge the gap between theoretical research and musical practice. A great effort is being made by the European conservatories to develop Master programmes and PhD studies and thus to incorporate research into their educational and institutional aims. It might take some time before this happens.

Statement 6: The multidisciplinary nature of sound and music computing research can find the right educational framework at the Master level.

Educational Trend 4: The professionalisation of PhD studies

Doctoral studies have traditionally been based on independent research undertaken by the doctoral candidate who draws upon the advice and guidance of a supervisor, supposedly on the model of a Master/apprentice relationship. This type of arrangement has long been the norm. For non-traditional fields like sound and music computing, it has had the advantage that a student is able to do a PhD just by finding an appropriate faculty member with sufficient knowledge of the chosen topic and a willingness to guide and support the student through the programme.

However, as a result of the changing context, PhD studies have recently come under scrutiny. Among the new challenges faced by universities in relation to doctoral education, it is worth mentioning the following desiderata (Sadlak, 2004):

PhD students doing multidisciplinary research are more diverse than their disciplinary counterparts. They may have any one of a wide range of subject backgrounds and may sometimes have followed more than one educational pathway. The background of students doing research in sound and music computing ranges from music to mathematics, from psychology to electrical engineering. What they have in common is the aim of bridging disciplines to develop new and multidisciplinary knowledge. There is general agreement (Metzger and Zare, 1999) that this type of multidisciplinary research should conform to the following:

Statement 7: The traditional model of a Master/apprentice relationship in PhD studies is evolving in a much more complex education environment, especially for multidisciplinary fields like sound and music computing.

The need for more structured PhD studies in Europe and the relevance of such studies to the Bologna Process have been highlighted repeatedly in recent years. In particular, joint PhD programmes can be amongst the most attractive features of the EHEA. But for the time being, interested students are still confronted with a variety of national and institutional structures that are not easily comparable.

Statement 8: Joint sound and music computing PhD programmes at the EU level can be built by exploiting excellence spread over several centres with complementary competencies.

There is a clearly growing trend towards the professionalisation of PhD studies, involving the inclusion of coursework and training in transferable skills aimed at facilitating the flow of doctoral students into the wider job market. Students are becoming employed researchers within well-structured research groups and funded within well-focused research projects. This increases the pressure to have money for PhD programs. Within this context, PhD students represent major academic and financial investments and contribute to much of the original research in universities. The role of supervisors seems key to the success or failure of multidisciplinary PhD projects (Fry et al., 2004). There is clear evidence that the disciplinary background, interest and motivation of the supervisor have much influence on research outcomes, both in terms of its quality and also whether PhD studies are completed on time (or at all).

However the added-value of a PhD for employment outside the areas of research in universities, research institutes and R&D functions in industry remains somewhat limited. Central and East European countries especially, as well as South European countries, experience a continuing lack of interest on the part of employers outside the academy in hiring PhDs. The situation is almost reversed in the US, where a significant and ever growing number of PhDs are attracted to private sector employment in which remuneration is higher than in the academy (Sadlak, 2004).

Statement 9: multidisciplinary PhD programmes avoid a focus which is too narrow and provide a broad spectrum of knowledge that also qualifies their graduates for careers outside the academy.

To sum up, the above trend analysis shows that the European educational system is in full development at all levels from Bachelor to Master and PhD. Furthermore, there is a willingness to further integrate educational systems from art and science. These developments will have a huge impact on the sound and music research field.

References

European Commission - Education and training 2010 (2007). Diverse systems, shared goals. http://europa.eu.int/comm/education/policies/2010/et_2010_en.html.

European Commission - Erasmus Mundus (2007). http://ec.europa.eu/education/programmes/mundus/index_en.html.

European Commission – Lisbon (2007). Progress towards the Lisbon objectives in education and training, 2005. http://europa.eu.int/comm/education/policies/2010/doc/progressreport05.pdf.

European Commission - Socrates/Erasmus (2007). The European Community programme in the field of higher education. http://ec.europa.eu/education/programmes/socrates/erasmus/erasmus_en.html.

European Commission - The Bologna Process (2007). Towards the European higher education Area. http://ec.europa.eu/education/policies/educ/bologna/bologna_en.html.

Fry, G.,Tress, B. and G. Tress (2004). PhD students and integrative research. In Proc. Frontis workshop from landscape research to landscape planning: Aspects of integration, education and application, Wageningen, The Netherlands, June 2004.

Metzger, N. and R. Zare (1999). Science policy: Interdisciplinary research: From belief to reality. Science, 283: 642-643.

Reichert, S. and Tauch, C. (2005). Trends IV: European universities implementing Bologna. http://www.eua.be/eua/jsp/en/upload/TrendsIV_FINAL.1117012084971.pdf.

Sadlak, J. (2004). Doctoral studies and qualifications in Europe and the United States: Status and prospects. Technical report, UNESCO-CEPES.

Tauch, C. (2004). Almost half-time in the Bologna Process. Where do we stand? European Journal of Education, 39(3): 275-288.

Industrial

Sound and music computing has always been an applied research field quite close to the music industry, thus close to the industries that create, perform, promote and preserve music. These industries involve: composers; performers and ensembles; publishers, record producers, manufacturers, labels and distributors; managers and agents; instrument makers; and some others. But right now sound and music computing technologies have a much broader impact and are present in most of the industries that sit at the nexus of cultural, entertainment, leisure and fast moving consumer goods.

A recent study of the economic impact of the cultural and creative sector in Europe (KEA European Affairs [KEA], 2006) revealed that the annual turnover of the sector (€654 billion in 2003) is larger than that of the motor industry or even ICT Manufacturers. This sector, of which music industry forms a major part, contributed 2.6% of EU GDP in 2003, slightly more than the contribution of the chemicals, rubber and plastic products industries combined. The sector's growth in 1999-2003 was 12.3% higher than that of the general economy and in 2004, about 5.8 million people worked in it, equating to 3.1% of the total employed population in the EU25. In view of the European Council's Lisbon agreement of March 2000, that the EU by 2010 should become “the most competitive and dynamic knowledge-based economy in the world, capable of sustainable economic growth with more and better jobs and greater social cohesion”, reinforced coordination of activities and policies impacting on the cultural and creative sector within the EU should be given a high priority.

Given this context, it is clear that the industries that relate to sound and music computing are in the middle of important changes and most are trying to adapt to the new markets and exploring the potential of these technologies. From the writings and presentations of industry experts, we can identify seven major trends.

Industrial Trend 1: Towards a knowledge-based economy

Modern economies are increasingly based on the production, distribution and use of knowledge and information. Knowledge is now recognised as the driver of productivity and economic growth. From the OECD report (OECD, 2005) it is clear that this long-term trend towards a knowledge-based economy is continuing. Science, technology and innovation have become key contributors to economic growth in both advanced and developing economies. Investment in knowledge (comprising expenditure on R&D, software and higher education) in the OECD area reached around 5.2% of GDP in 2001, compared to around 6.9% for investment in machinery and equipment. The share of knowledge-based market services is continuing to rise and now accounts for over 20% of OECD value added. The share of high and medium-high technology manufacturing fell to about 7.5% of total OECD value added in 2002, compared to about 8.5% in 2000.

Statement 1: Music related activities are part of the new knowledge economy and they should take advantage of the continuing growth of this sector.

Industrial Trend 2: A global economy

Economies have expanded beyond national borders. Production in particular has been expanded by multinational corporations to many countries around the world. The global economy includes the globalisation of production, markets, finance, communications and the labour force.

From the OECD report (OECD, 2005) we learn that this is not a new phenomenon per se, but that it has become more pervasive and driven mainly by the use of information and communication technologies (ICT). In the knowledge economy, information circulates at the international level through trade in goods and services, direct investment and technology flows and the movement of people. According to the American National Science Board (2006) the globalisation of R&D, S&T, and S&E labour markets is continuing. Countries are seeking competitive advantage by building indigenous S&T infrastructures, attracting foreign investments and importing foreign talent. The location of S&E employment is becoming more internationally diverse and those who are employed in S&E have become more internationally mobile.

Statement 2: Both the production and consumption of music related goods is now globalised and international cooperation is more important than ever.

Industrial Trend 3: The development of the ICT sector

In the final decade of the twentieth century, the almost simultaneous arrival of mobile phones and the Internet not only changed the face of communications but also gave impetus to dramatic economic growth. We now speak of the Information and Communication Technologies (ICT) sector to refer to the agglomeration of the communications sector, including telecommunications providers and the information technology sector, which ranges from small software development firms to multinational hardware and software producers.

According to the i2010 report (i2010 - European Information Society, 2007), ICT accounts for a quarter of EU GDP growth and 40% of productivity growth. The digital convergence of the information society and media services, networks and devices is finally becoming an everyday reality: ICT will become smarter, smaller, safer, faster, always connected and easier to use, with content moving to three-dimensional multimedia formats. It has been pointed out (Saracco, 2002) that any economic indicator ties together progress and communications infrastructure, and that the dissemination and progress of culture go hand in hand with the possibility of interacting and sharing ideas, thus putting telecommunications at centre stage.

The American National Science Board (2006) reports that the number of industrial researchers has grown along with rapidly increasing industrial R&D expenditures. Across OECD member nations, employment of researchers by industry has grown at about twice the rate of total industrial employment. For the OECD as a whole, the full-time equivalent number of researchers more than doubled in the two decades from 1981 to 2001, from just below 1 million to almost 2.3 million. Over the same period, the number of researchers in the United States rose from 0.5 million to nearly 1.1 million.

According to the KEA report (KEA, 2006) the ICT sector is central to European growth and competitiveness and has been identified as a pillar of the European Lisbon Strategy. It accounts for 5.3% of EU GDP and 3.4% of total employment in Europe. In the period 2002-2003 it was responsible for more than a quarter both of productivity growth and of the total European R&D effort. Darlin (2006) predicts that flat-screen televisions will get bigger and that MP3 players and cell phones will get smaller. And almost everything will get cheaper. But the biggest trend expected is that these machines will communicate with one another.

According to the OECD (OECD – Digital Music, 2005), digital music and other digital content are also drivers of global technology markets, both to consumer electronics manufacturers and PC vendors. The increase in revenues from hardware in the PC and consumer electronics branch resulting from the availability of online music, authorised or not, is potentially greater than the revenues currently generated by paid music streaming or downloads.

Statement 3: The growth of the ICT sector and the innovations coming out of it will be the main driving forces for the music related industries.

Industrial Trend 4: The interdependence of the cultural & creative sector and ICT

The cultural and creative sector generates significant economic performance in other non-cultural sectors, thereby indirectly contributing to economic activity and development, and in particular in the ICT sector. Culture contributes directly to the economy by providing products for consumption, namely the cultural goods and services embodied in books, films, music sound recordings, concerts, etc. But the recent growth of the creative media, according to KEA (2006), is due to the growing diffusion and importance of the Internet. The impact of this development on media consumption has been huge in recent years and will be the major factor for this sector in the future. At the same time, creative content is a key driver of ICT uptake. The consultancy firm PriceWaterhouseCoopers estimates that spending on ICT- related content will account for 12% of the total increase in global entertainment and media spending until the year 2009 (see KEA, 2006). Accordingly, the development of new technology depends to a large extent on the attractiveness of content and the new networks are no exception. The development of mobile telephony and networks is based on the availability of attractive value-added services that will incorporate creative content, to which the sound and music computing may contribute.

However, the KEA report also predicts that the roll out of broadband and the digitisation of production processes will require significant investment for the creative industries to adapt, as well as changes in its management practices. Some industries (notably music) have to go through aggressive cost restructuring programmes and are experiencing consolidation through mergers. Without a strong music, film, video, TV and game industry in Europe, the ICT sector will be the hostage of content providers established in Asia or North America.

Statement 4: Content is a major driver of ICT development.

Industrial Trend 5: New models of exploitation of content

The new ICT technologies have opened up new possibilities for the exploitation of music. Traditionally there have been two distribution channels for media content, namely, physical distribution and analogue broadcasting (radio, TV). Now we also have IP/Internet, Mobile communications (UMTS), Digital TV and Radio.

The OECD report on digital music (OECD - Digital Music, 2005) identifies that network convergence and widespread diffusion of high-speed broadband have shifted attention towards broadband content and applications that promise new business opportunities, growth and employment. Digital content and digital delivery of content and information are becoming increasingly ubiquitous, driven by the increasing technological capabilities and performance of delivery platforms, the rapid uptake of broadband technologies - with 2004 identified as a breakthrough year for broadband penetration in OECD countries - innovative creation and use of content and improved performance of hardware and software.

Through a combination of new technologies, new business relationships and innovative service offers to consumers, the market is developing rapidly in order to realise the potential of online music. Saracco (2002) predicts that in ten years' time nearly all communications (over 90% of it) will be using fixed networks, while most people will be under the impression they are using mobile networks. He observes that in the coming years we are going to see a tremendous increase in communicating entities, be they applications or objects. The amount of communication directly involving humans will keep growing but at a slower pace, fuelled mostly by the dissemination of telecommunications in developing countries.

According to OECD – Digital Economy (2006), users are becoming increasingly active, in such a way we are entering a participatory culture not of consumers but of users. Users are increasingly active and want to express themselves. This is highly relevant to a field such as sound and music computing, which is closely linked to creation and expression.

Statement 5: Interactive broadband networks are revolutionising the way music is distributed and consumed.

Industrial Trend 6: New forms of Intellectual Property protection

Concerning intellectual property protection, there are traditionally two extreme positions to be defended, namely, absolute control of a creation or complete release of the rights to it. However, until recently, there was no easy way to make explicit the rights that an author gives in relation to a creation. Creative Commons (http://creativecommons.org/) is the first example of a system that offers flexible protection of intellectual rights.

Kusek and Leonhard (2005) claim that the issue of protecting intellectual property goes far beyond music and audio technologies. Nevertheless, the crisis started in the music industry. Already, music recording industry revenues are down sharply, despite an overall increase in the distribution of music. The financial crisis has caused music labels to become cautious and conservative, investing in proven artists, with less support available for new and experimental musicians. Kusek and Leonhard note that the breakdown of copyright protection is even starting to impact on musical instruments. Synthesisers, samplers, mixers and audio processors can all be emulated in software. For example they estimate that at least 90% of the copies of Reason, one of the emulation software leaders, are pirated.

Commenting on OECD – Digital Economy (2006) they note the existence of sharp disagreement as to whether intellectual property rights (IPR) currently strike the right balance. There are three points of view: some believe that interest- group pressure has led to excessive protection; some adopt an intermediate position, believing that recent court cases such as Grokster have clarified secondary liability and that this has been sufficient to clarify the IPR situation; a third group maintain that levels of protection and enforcement are still insufficient and should be strengthened. That same OECD report proposes a tentative work agenda that might address the following needs: first, putting intellectual property in its proper place, that is, balancing private incentive versus the public good; second, achieving new digital rights definitions which integrate old rights (e.g. fair and legitimate use) and new rights (e.g. access to orphaned and out-of- print works); finally, accommodating new models of production and distribution (Open Source, Open Format, Open Access).

According to KEA (2006), the main beneficiaries in Europe of the digital revolution have been the telecom operators acting as Internet service providers. Broadband access spending has risen very rapidly. This growth is largely due to the availability of free content. Indeed, 95% of music downloads today, for example, are unpaid for.

Statement 6: New models of the control and use of intellectual property rights are impacting on the music industry and opening up new possibilities for the protection and dissemination of music content.

Industrial Trend 7: Revolution in the music business

The whole music business is going through a major revolution, the main cause of which is the development and expansion of the ICT sector. According to OECD – Digital Music (2005) the rise of online music has resulted in product and process innovation, the entry of new players and new opportunities for music consumption and revenues, involving different forms of disintermediation, and the continued strong role of some traditional market participants (especially the record labels). In the new digital model, artists, majors and publishers have so far retained their creative roles related to the development of sound recordings.

Direct sales from artist to consumer or the building of an artist's career purely through the online medium are still rare. Nevertheless, the Internet allows for new forms of advertising and possibilities that lower the entry barriers for artistic creation and music distribution. According to Kusek and Leonhard (2005), ever since the invention of electricity, music and technology have worked hand-in-hand, and technology continues to catapult music to unprecedented heights. Today, the Internet and other digital networks, despite all the legal wrangling, have made music bigger than ever before. Within ten to fifteen years, Kusek and Leonhard claim, the “Music Like Water” business model will make the industry two or three times larger than it is today. They imagine a world where music flows all around us, like water or electricity, and where access to music becomes a kind of "utility". Not for free per se, but certainly for what feels like free. Along the same line, Kurzweil (2003) claims music technology is about to be radically transformed. Communication bandwidths, the shrinking size of technology, our knowledge of the human brain and human knowledge in general are all accelerating. Music will remain the communication of human emotion and insight through sound from musicians to their audience, but the concepts and process of music will be transformed once again.

Statement 7: The possibilities of the ICT technologies are completely reshaping the music business.

To sum up, the identified trends show a rapid development towards a knowledge-based and global economy, with a major role of the ICT sector. Reports indicate a growing interest in the mutual dependency of the cultural and creative sector and ICT, which leads to new roles of content exploitation and dealing with intellectual property issues. All these developments accompany the revolution that is currently taking place in the music business. The sound and music computing field is expected to play a crucial role in these developments.

References

OECD. OECD Science, Technology and Industry Scoreboard - Towards a knowledge-based economy. Organisation for Economic Co-operation and Development. http://hermia.sourceoecd.org/vl=849972/cl=36/nw=1/rpsv/scoreboard/, 2005.

Information Society and Media DG. i2010 - A European Information Society for growth and employment. http://europa.eu.int/information_society/eeurope/i2010/index_en.htm.

R. Saracco. Is there a future for Telecommunications? http://www.singaren.net.sg/library/presentations/21nov02_1.pdf, 2002.

National Science Board. Science and Engineering Indicators 2006. Technical report, National Science Foundation, 2006. http://www.nsf.gov/statistics/seind06/.

D. Kusek and G. Leonhard. The Future of Music: Manifesto for the Digital Music Revolution. Omnibus Press, 2005. http://www.futureofmusicbook.com/.

Ray Kurzweil. The Future of Music in the Age of Spiritual Machines. Lecture to the AES 2003. http://www.kurzweilai.net/meme/frame.html?main=/articles/art0597.html.

D. Darlin. Data, music, video: Raising a curtain on future gadgetry. New York Times, January 2006. http://tinyurl.com/njg3f.

OECD. OECD Report on Digital Music: Opportunities and Challenges. Technical report, 2005.

OECD. The Future Digital Economy: Digital Content Creation, Distribution and Access. Technical report, 2006. http://www.oecd.org/dataoecd/54/35/36854745.pdf.

KEA European Affairs. The Economy of Culture in Europe. Technical report. http://www.keanet.eu/Ecoculture/ecoculturepage.htm.

Creative Commons. http://creativecommons.org/.

Social and Cultural

This section is about the social and cultural context in which music appears and the way in which sound and music computing is related to it. Indeed, music is an important aspect of all human cultures (Merriam, 1964). Musical activity involves a mental context of values and goals, as well as an institutional context of societal organisations and structures, and relates to all kinds of interactions with other humans, with nature and with material objects and machines.

Musical activity is, moreover, explorative, creative, and innovative, and can focus on expression (via art and music works), the acquisition of knowledge (via music science and research) or the development of tools to act (via music technology and industry). Besides all this, music is also meant to provide new experience, to give sense and meaning to life, to console and to promote social coherence and personal identity in and over very diverse social and ethnic groups (Hargreaves and North, 1999). Rooted in the biology of every human being (Wallin et al., 2000), music is a core occupation of our technological society.

The KEA (2006) study on the cultural and creative industries in Europe reveals that the expansion of the ICT sector depends to a large extent on the attractiveness of cultural content. Music has thereby been identified as one of the most vibrant cultural industries with a flourishing music research component embedded within a particular social and cultural context. According to this study, cultural activities can be stimulated by both bottom-up, grass-roots initiatives and also the top-down initiatives of administrations and institutes. These social and cultural strategies are beneficial to the economic environment because they:

Below, some particular features of the current socio-cultural context are described. These provide a background against which we can better understand trends and open problems related to sound and music computing research.

Socio-Cultural Trend 1: Transgression and uncertainty

Classical views hold that the socio-cultural context is largely shaped by developments in science/technology, whose authority, values and practices permeate all dimensions of society and culture. However, more recent views (Nowotny et al., 2001) hold that, owing to the growth of complexity, unpredictability and irregularity in both science and society, this one-way influence has given way to the mutual influencing, or even transgression, of science/technology and society/culture, as well as of university, industry and government (Etzkowitz and Leydesdorff, 2000).

The inherent generation of uncertainties (often resulting from the quest for innovation) yield different research practices, which are reflected in an increasing number of different directions in which technology could be explored and exploited. Which directions are selected may be strongly driven by the dynamics of innovation and by economic rationality. However, as this dynamics cannot be entirely planned, there is a need for values and goals which allow for uncertainty.

In the context of EU research policy, the European Commission - Europe 2010 (2005) has defined strategic objectives which draw upon solidarity and security. These objectives are based on concepts such as a friendly business environment, the embracing of change, economic and social cohesion, responsibility for common values, justice and risk management. This approach can be adopted as a basic framework for the social and cultural values and goals of sound and music computing research. It implies, among other things:

Statement 1: The uncertainty that is inherent in sound and music computing research should be guided by the specification of social and cultural values and goals.

Socio-Cultural Trend 2: Beyond the logic of economic rationality

Socio-cultural values and goals may guide the development of sound and music computing research by bringing forward certain requests. For example, while music information retrieval research has excelled in developing tools for common mainstream commercial (popular) music, it has to a large extent neglected more culturally interesting musical expressions, such as classical music and music of other non-Western cultures. Clearly, in such a situation, stakeholders in the social and cultural domains (such as governments, universities, cultural institutions) may require sound and music computing research to develop technology beyond the logic of pure economic rationality and require the development of music information retrieval tools for all kinds of music.

The reason for doing this could be that apart from commercial music, society feels that a broad spectrum of music traditions has a high social and cultural value. Thus, a diverse set of different applications in music information retrieval, interactive systems, education, archiving and entertainment, which form important components for the future eCulture (the electronic environment in which culture is produced, distributed and consumed) should be developed. If society and culture require that this broad spectrum should be taken into account in research, then support should compensate for biases induced by economic rationality. Often, the required socially and cultural valuable developments are supported by government and other institutions.

It is not excluded that support for these areas may boost very innovative technologies which, once a critical mass has been achieved, can then be taken up again in a logic of economic rationality. The European Commission is a strong player in defining the societal values with respect to scientific research.

Statement 2: The (EU) government should inject its support for research at the frontiers of economic rationality.

Socio-Cultural Trend 3: Local specialisation and global integration

In Europe, research in sound and music computing shows a trend towards local specialisation and global integration. Research in sound and music computing is typically done in small dynamic institutions, which are often specialised in small niche areas (such as ethnomusicology, cognitive musicology, data processing or music synthesis). Thanks to collaboration, these small research units can become quite powerful when complementary competences are organized as a broader European network of research units. Over the past decennium, such networks have been entirely based on competition and shifting alliances.

Statement 3: Local specialisation and global integration offers a competitive environment for sound and music computing research.

The multidisciplinary orientation suits the object of research, which is in itself very broad, covering issues in signal processing as well as in symbolic handling of musical information. This multidisciplinary orientation is situated within an economic rationality of production, distribution and consumption, a social rationality involving diverse players such as musicians, organisers, the mass media and the music industry, and a cultural rationality involving contexts related to high culture, low culture, cross-culture and interculture.

Statement 4: Research should be grounded in a multidisciplinary basis because that is the best guarantee for its embedding in the economic, social and cultural reality of our post-industrial society.

Socio-Cultural Trend 4: A neo-evolutionary research model

Given the broad context in which audio and music manifest themselves, sound and music computing research strategies are characterised by emergence rather than planning. This emergence, moreover, is driven by creativity and innovation. Hence it is difficult to predict what may be successful and what not. Sound and music computing's scientific paradigm is therefore close to a neo-evolutionary model (Leydesdorff and Meyer, 2003), in which elaborate systems of peer review, assessment and evaluation leave room for strategies of variation to be pursued by smaller laboratories in different alliances.

Statement 5: Sound and music computing research is strongly driven by innovation, albeit in a context of emergence rather than planning.

In this model, risk analysis is needed to consider the possible implications of research. After all, science and technology do not automatically lead to the best possible world. In developing them, it is necessary to calculate the risks, to keep an eye on the volatile and ambiguous dynamics. The co-evolution of the socio-cultural context and the scientific/technological context implies that an analysis of values and goals should become an integral part of the development of sound and music computing (Nowotny et al., 2001). The best guarantee to cope with unpredictable outcomes, or uncertainties initiated by innovation, is to allow society and culture to speak back to science and technology, hence the importance of reflection, the development of a code of ethics, the concern for democratic access and several other values that should be taken into account.

Statement 6: Democratic access, reflection and a code of ethics should form an integral part of sound and music computing research.

Socio-Cultural Trend 5: Innovation through artistic creation

Creation and innovation form the motor of sound and music computing research. Most interestingly, they are strongly driven by the context of artistic application. In that respect, it is of interest to mention that content-based music technology has roots in the particular cultural rationality of the 1950s (Born, 1995, Leman, 2005). That rationality, heavily supported by European governments of the time, led to novel developments in electronic music, of which interactive multimedia is a recent outcome. In contrast, audio-recording technology had already begun by the early 20th Century and was driven by the logic of economic rationality and the free market (Pichevin, 1997).

The trend of allying content-based music technology to economic rationality is new. But it is reasonable to assume that artistic creation remains a major factor in maintaining the former's innovative character. There are at least two reasons why art is likely to continue to contribute innovating challenges to sound and music computing research:

In short, the context of art application results in a constant drive towards human-friendly and expressive technologies of mediation. Artistic and creative research is an important source for innovation and as a producer of content, it can really push the development of ICT (KEA, 2006).

Statement 7: Sound and music computing research should include artistic creation because the latter is a major driving force for innovation, including innovation in music technology.

In the 1950 and 1960s, numerous small music research laboratories played an important role in the development of content-based music technologies (Leman, 2005). Their original focus on electronic music production has now been extended to multi-media art production. This distinctive European approach, based on small but very innovative and specialised art centres connected through electronic networks, offers a unique and rich context for innovation in music/multimedia technology. Participative technologies involving all players in the cultural domain (developers, distributors, consumers, users and artists) can contribute to the formation of a space for eCulture. This space is closely connected to research/science and technology/industry.

Statement 8: eCulture draws on a platform of participation in culture and on a culture of participation.

Socio-Cultural Trend 6: Focus on the user

The socio-cultural context definitely calls for more attention to the user and the human factor in the practice of music technology. Sound and music computing research is characterised by its potential for use and hence by a strong willingness to respond to signals from society and culture. Indeed, the development of music technology should take into account a context of application and focus on different categories of users, the design of appropriate mediation technologies and the pursuit of personalised approaches.

The user can no longer be considered passive, as one that merely registers what is given as stimulus. Instead, the user is an active consumer, which implies a transgression from the domain of pure consumption into that of production and distribution. The active consumer is also a producer and distributor of music, and therefore an active contributor to what happens with music. Being an active consumer implies participation in the whole chain of production, distribution and consumption, forming part of a network of participating users.

Statement 9: Sound and music computing research should take into account the context of application, in which the active user/consumer occupies a central place.

Socio-Cultural Trend 7: Ethics in research

Ethics pertains to what is morally right and wrong. In view of the growing impact of technology, this perspective needs to be addressed in sound and music computing research. The impact manifests itself in various aspects of our social and cultural life. Examples are the personal integrity of subjects involved in experiments and exchange of data, the safeguarding of the rights of those who have invested in producing valuable content, the right to democratic access to information and so on. It is clear that new developments in sound and music computing research should take this context of implication into account. For example, issues of IPR ownership can be a significant barrier to the conducting of large and ambitious research projects, and the new concepts being developed around this issue may therefore be of critical value.

Sensor technologies are another sensitive issue. They may infringe the personal integrity of subjects and therefore the privacy and confidentiality of information. The conceptual and philosophical implications regarding human responsibility in contexts of application need consideration in sound and music computing research.

Statement 10: Sound and music computing research should take into account the context of implication, assessing risks and ethical implications.

To sum up, the social and cultural context has been identified to be an important driver for sound and music computing. Due to the complexity, unpredictability and irregularity in both science and society, decisions in research are often characterized by a fundamental uncertainty. More and more, this uncertainty is solved by a logic of economic rationality. Research then goes where economy requires it. However, social and cultural values are important and call for a transgressive approach to science and society, that is, an approach in which a mutual interchange between science and society is possible. This may result in governmental support for research fields that focus on important social and cultural values that go beyond the logic of economic rationality. Society may also support the creation of research spaces, and support local specialisation and global integration of the many small research units, thereby supporting a neo-evolutionary research model. The social and cultural context is all about the values that really concern to our lifestyle and that, in a democratic society, contributes to what is considered the highest good for all. Music is one such phenomenon that contributes to human well-being. It fosters creative activity, expression and social interaction. Through artistic creation, innovation is possible and contribution to culture is renewed. Society also requires more attention to the role of the user of ICT and there is an important ethical aspect related to modern sound and music computing research applications.

References

G. Born. Rationalizing culture IRCAM, Boulez, and the institutionalization of the Musical Avant-Garde. University of California Press, 1995.

H. Etzkowitz and L. Leydesdorff. The dynamics of innovation: from national systems and "mode 2" to a triple helix of university-industry-government relations. Research Policy, 29(2):109-123, 2000.

D. J. Hargreaves and A. C. North. The functions of music in everyday life: Redefining the social in music psychology. Psychology of Music, 27(1):71-83, 1999.

M. Leman. Musical creativity research. In J.C. Kaufman and J. Baer, editors, Creativity Across Domains: Faces of the Muse, pages 103-122. Lawrence Erlbaum, Mahwah, NJ, 2005.

L. Leydesdorff and M. Meyer. The triple helix of university-industry-government relations. Scientometrics, 58(2):191-203, 2003.

A. Merriam. The Anthropology of Music. Northwestern University Press, 1964.

H. Nowotny, P. Scott, and M. Gibbons. Rethinking Science: Knowledge and the Public in an Age of Uncertainty. Polity Press, 2001.

A. Pichevin. Le disque à l'heure d'Internet, l'industrie de la musique et les nouvelles technologies de diffusion. L. Harmattan, 1997.

N. L. Wallin, B. Merker, and S. Brown. The origins of music. MIT Press, 2000.

European Commission. Europe 2010 : A Partnership for European Renewal Prosperity, Solidarity and Security. Strategic Objectives 2005-2009. http://europa.eu.int/eur-lex/lex/LexUriServ/site/en/com/2005/com2005_0012en01.pdf.

Sound

In this section we review the research on sound that is being carried out within the boundaries identified in the definition of the field. From a sound to sense point of view, we include the analysis, understanding and description of all musical and non-musical sounds except speech. Then, in the sense to sound direction, we include the research that is more related to sound synthesis and processing.

Sound Description and Understanding

One of the basic aims of SMC research is to understand the different facets of sound from a computational point of view, or by using computational means and models. We want to understand and model not only the properties of sound waves but also the mechanisms of their generation, transmission and perception by humans. Even more, we want to understand sound as the basic communication channel for music and a fundamental element in our interaction with the environment. Sound serves as one of the main signals for human communication, and its understanding and description requires a notably multidisciplinary approach.

Traditionally, the main interest of SMC researchers has been musical sounds and thus the understanding of the sound generated by musical instruments and the specific transmission and perception mechanisms involved in the music communication chain. In recent years, this focus has been broadened and there is currently an increased interest in non-musical sounds and aspects of communication beyond music. A number of the methodologies and technologies developed for music are starting to be used for human communication and interaction through sound in general (e.g., ecological sounds) and there is increasing cross-fertilisation between the various sound-related disciplines.

There has been a great deal of research work on the analysis and description of sound by means of signal processing techniques, extracting features at different abstraction levels and developing source-specific and application-dependent technologies. Most of the current research in this domain starts from frequency domain techniques as a step towards developing sound models that might be used for recognition, retrieval, or synthesis applications. Other approaches consider sparse atomic signal representations such as matching pursuit, the analytical counterpart to granular synthesis [Sturm et al. 2006].

Also of importance has been the study of sound-producing physical objects. The aim of such study is to understand the acoustic characteristics of musical instruments and other physical objects which produce sounds relevant to human communication. Its main application has been the development of physical models of these objects for synthesis applications [Smith, 2006; Välimäki, et al., 2006; Rocchesso & Fontana, 2003], so that the user can produce sound by interacting with the models in a physically meaningful way.

However, beyond the physical aspect, sound is a communication channel that carries information. We are therefore interested in identifying and representing this information. Signal processing techniques can only go so far in extracting the meaningful content of a sound. Thus, in the past few years there has been an exponential increase in research activity which aims to generate semantic descriptions automatically from audio signals. Statistical Modelling, Machine Learning, Music Theory and Web Mining technologies have been used to raise the semantic level of sound descriptors. MPEG-7 [Kim et al., 2005] has been created to establish a framework for effective management of multimedia materials, standardising the description of sources, perceptual aspects and other relevant descriptors of a sound or any multimedia asset.