Computational Methods for Portraying Emotion in Generative Music Composition by Alison Mattek An Undergraduate Thesis. University of Miami Department of Music Engineering May 2010 2   Table of Contents 1. Introduction ....................................................................................................... 4 2. A Background on Music and Emotion ............................................................ 6 2.1 Antiquity: Pythagoras, Plato ................................................................. 6 2.2 The Middle Ages: Boethius .................................................................. 7 2.3 Baroque: Doctrine of Affections ........................................................... 8 2.4 The Age of Enlightenment ..................................................................... 9 2.5 19th Century Romanticism ..................................................................... 11 2.6 20th Century: Leonard A. Meyer ............................................................ 13 2.7 Conclusion ............................................................................................. 15 3. Algorithmic Composition in the Late Twentieth Century ............................. 16 3.1 John Cage ............................................................................................... 17 3.2 Lajaren Hiller & Leonard Isaacson ........................................................ 19 3.3 Gottfried Michael Koenig ...................................................................... 22 3.4 Iannis Xenakis ........................................................................................ 23 3.5 Conclusion ............................................................................................. 25 4. Computer-Aided Algorithmic Composition .................................................... 27 3   4.1 Modeling Traditional, Non-Algorithmic Processes ............................... 27 4.2 Modeling New, Original Compositional Procedures ............................. 28 4.3 Selecting Algorithms from Extra-Musical Disciplines .......................... 29 4.4 Conclusion ............................................................................................. 32 5. Algorithms for Emotional Expression ............................................................. 33 5.1 KTH System .......................................................................................... 34 5.2 Computational Music Emotion Rule System (CMERS) ....................... 35 5.3 Conclusion ............................................................................................. 38 6. Theory of Emotional Conveyance in Generative Composition ..................... 40 7. Implementation .................................................................................................. 43 8. Listening Tests .................................................................................................... 47 9. Results ................................................................................................................. 49 10. Discussion ......................................................................................................... 58 11. Future Work ..................................................................................................... 60 12. Citations ............................................................................................................ 61 4   1. Introduction The art of music can be examined through two fundamental perspectives: psychology and mathematics. Music, as a form of emotional expression and therapy, is deeply rooted in the field of psychology. However, music is also an extraordinary physical phenomenon that results from the organization of numbers (frequencies) and possesses many underlying mathematical properties. Composers and music theorists alike have reflected on both of these perspectives. This study examines the aesthetic tendencies of algorithmic computer music compared to traditionally composed Western music. The Western music tradition is deeply rooted in emotional expression of the composer and manipulating the affective response of the listener. Conversely, algorithmic computer music promotes an aesthetic approach that emphasizes the mathematical properties of music. This is mainly due to the fact that computers are extremely capable of producing and analyzing numbers and are lacking when it comes to analyzing subjective ideas such as ineffable emotion. This paper is organized as follows. First, history of the correlation between Western music composition and affective response is documented in the first section. Following this, I discuss the introduction of computers to the field of composition along with the forms and aesthetics of computer-assisted algorithmic composition. This background creates a foundation for identifying the distinction between traditionally composed Western music and algorithmically composed computer music. Next, I address current research that link algorithmic composition and affective response. The emotional gap between traditional Western music and computer music is becoming 5   increasingly more narrow due to such research. Specifically, empirical studies that link specific musical features to correlating affective responses have been applied to music performance algorithms. These performance algorithms have successfully produced desired affective responses in listeners. These algorithms are discussed in detail and provide a framework for applying the same principles in an attempt to convey emotion in computer assisted algorithmic composition. Finally, this study proposes a theory for expressing emotion in computer assisted algorithmic composition. Sixteen, twenty-second excerpts are algorithmically generated and later judged by listeners in a two-dimensional emotional space. The results indicate that it is possible to manipulate specific musical attributes through algorithms in order to convey a desired emotion. 6   2. A Background on Music and Emotion Discussion on the relationship between music and emotions has existed since the philosophical treaties of antiquity. In Emotion and Meaning in Music (1956), Leonard A. Meyer tells us “from Plato down to the most recent discussions of aesthetics and the meaning of music, philosophers and critics have, with few exceptions, affirmed their belief in the ability of music to evoke emotional responses in listeners (p. 6).” This section will investigate how the Western tradition’s approach to the relationship between emotion and music developed over time. 2.1 Antiquity: Pythagoras, Plato The mathematical nature of music was first discovered by the studies of Pythagoras, who found that musical pitch is dependent on the velocity of a moving object (i.e., a plucked string). Larger objects (or longer strings) vibrate with a lower frequency, thus projecting a lower pitched sound. Conversely, smaller objects (or shorter strings) vibrate with a higher frequency and project higher pitched sounds. The ancient Greek philosophers emphasized the importance of order and reason; thus, these numerical properties of music resonated with their values. However, Greek philosophers admitted that music was not a strict science by any means. The philosopher Plato considered music to be a powerful and dangerous art form capable of causing great emotional unrest. In Republic X, Plato discusses how music “feeds and waters the passions instead of drying them up; she lets them rule, although they ought to be controlled, if mankind are ever to increase in happiness and virtue.” Plato feared that the emotional character of music was more powerful than its numerical and rational 7   character, and consequently would mislead people into believing that affect was more important than reason. Plato’s student, Aristotle, also addresses the emotional consequences of music. Aristotle is more tolerant of the emotional effect of music than his teacher, and he admits that the emotional release catalyzed by music can be both pleasant and valuable (Bowman 1998, p. 53). Because affective responses to music varied depending on the style, Aristotle argued that only certain types of music were appropriate. He hypothesizes that invoking pleasurable affective responses is highly dependent on the choice of musical mode: “Some modes, like the mixolydian, make people sad and grave, while others can ‘enfeeble the mind.’ The phrygian ‘inspires enthusiasm.’ And the dorian, which ‘produces a moderate and settled temper,’ is also the ‘manliest’ (Bowman 1998, p. 55).” Aristotle’s speculation on the relationship between musical mode and emotional response has been reinforced by many empirical studies (Livingstone 2005, 2010). 2.2 The Middle Ages: Boethius Boethius was the prominent music theorist of the Middle Ages, and he is responsible for translating the philosophy of the Greeks for the Latin world. Like the Greeks, Boethius was concerned with the mathematical proportions that existed in music. He even dismissed the major third as a consonant interval, simply owing to its ratio, 81:64, being too inelegant (Bonds 2006). Boethius also understood that music influenced the listener emotionally, and consequently he believed it influenced the listener’s virtue and morality as well (Bowman 1998, p. 64). In his work The Fundamentals of Music, 8   Boethius describes the variety of affective responses to music: “[M]usic is associated not only with speculation but with morality as well. For nothing is more characteristic of human nature than to be soothed by pleasant modes and disturbed by their opposites” (Boethius tr. 1989, p. 2). Boethius encourages the study of modes that are “temperate, simple, and masculine, rather than effeminate, violent, or fickle. (Bowman 1998, p. 64)” Boethius’s examination of the relationship to mode and affective response mirrored that of the Greek philosophers that came before him. The ideas of modes associated with moods carried over into the philosophy of Baroque musicians and theorists. However, in the Baroque era, most of the Greek modes became obsolete, and only the Ionian (major) and Aeolian (minor) modes remained in common use. 2.3 Baroque: Doctrine of Affections Baroque composer and theorist, Nicola Vicentino wrote in his treatise Ancient Music Adapted to Modern Practice (1989) that “the composer’s sole obligation is to animate the words, and, with harmony, to represent their passions—now harsh, now sweet, now cheerful, now sad—in accordance with their subject matter.” Various ideas regarding the relationship between music and emotions were discussed during the Baroque Era. These ideas are often referred to as the “doctrine of affections.” Rene Descartes attempted to codify the human affections in The Passions of the Soul (1649). The six basic emotions that he identified were: wonder, love, hate, desire, joy, and sadness. Composers were expected to induce these emotions through their music. Bowman in Philosophical Perspectives on Music (1998) describes the doctrine: 9   “The doctrine of affections, which maintained music’s capacity to portray particular emotions... assuming as it did the existence of logically discernable congruence between tonal and emotional patterns that could be systematically exploited by composer and appreciated by listener.” (p.73) Johan Sebastian Bach often exploited these patterns in his church music. In his chorale prelude Durch Adams Fall, in which Adam falls from a state of innocence to a state of sin, the “falling” is depicted by the seventh in the bass: “since it was a fall into sin and since sin was conventionally represented by chromaticism, Bach made the sevenths not diatonic, but diminished” (Bukofzer 1947, p. 283). Extra-musical meanings such as this were often implied by Bach and other Baroque composers. Although these meanings may now come across as artificial, they are founded on the principle that there is a correlative relationship between harmonic complexity and affective response. Essentially, complex harmony is associated with negative affect (i.e. Adam falling into sin), and conversely, simple, diatonic harmonies are associated with positive affect. 2.4 The Age of Enlightenment The Age of Enlightenment brought forth a rise in empiricism. The empiricists sought truth in direct experience rather than in unrestrained speculation. This emphasis on subjective experience as opposed to objective reasoning led to the conclusion that the musical experience was more psychological than logical. Philosopher Immanuel Kant discussed thoughts on music and its relationship to affect in his Critique of Judgment (1790). Kant considered music to be the lowest of the arts due to its “fleeting nature”. The fleeting nature of music is itself representative of 10   emotions, which are also temporary. According to Kant, because music appeals so much to the senses (emotions), it in turn cannot appeal to the mind (reason). Kant believed the purpose of music was nothing more than “giving an expression to...the dominant affection in the piece (Kant 1790, p. 194).” Kant also did not neglect to mention his belief in the correlation between specific musical features and affective response, concluding that music instigates affective reaction by an “accord of the sensations,” which derives from the imperceptible “numerical relations of the vibrations of the air (Kant 1790, p.194).” However, he did not reflect on which specific numerical relations influenced correlated affective responses. Composers of the Classical Era did not attempt to evoke specific emotional responses in the way Baroque composers did with the doctrine of affections. The empiricist perspective that truth could be derived through the direct experience of nature infiltrated into the practice of music composition. Because there was an emphasis on nature rather than abstract speculation, composers attempted to write “natural” sounding music. This goal reflects the extensive use of diatonic melodies and harmonies, whereas chromaticism was used sparingly. As was the theory in the doctrine of affections, chromaticism was associated with negative affect. Classical music and its diatonic harmonies can be associated with a positive affective response according to recent day studies, such as David Campbell’s The Mozart Effect (1997). One of the conclusions in this study is that the music of Mozart is able to reduce tension in the listener in addition to enhancing verbal ability and special-temporal reasoning. It is possible that temporarily enhanced skills is connected to the individual experiencing a positive mood, according to research in positive psychology (Csikszentmihalyi 1998).