Processing of Musical and Vocal Emotions through Cochlear Implants By Duha G. Ahmed, MBBS Department of Otolaryngology- Head and Neck Surgery McGill University, Montreal April, 2017 A thesis submitted to McGill University in partial fulfillment of the requirements of the degree of Master of Science © Duha G. Ahmed 2017 TABLE OF CONTENTS TABLE OF CONTENTS ................................................................................................................................ ii ABSTRACT ................................................................................................................................................ iv RÉSUMÉ .................................................................................................................................................... v ACKNOWLEDGMENTS ............................................................................................................................. vi CONTRIBUTION OF AUTHORS ................................................................................................................ vii CHAPTER 1: Thesis Introduction .............................................................................................................. 8 1.1 Introduction ........................................................................................................................................ 8 1.2 Emotional perception ......................................................................................................................... 8 1.3 Auditory emotions ............................................................................................................................. 8 1.4 Neuro-imaging studies of emotional processing .............................................................................. 9 1.5 Cochlear implants users’ perception of auditory emotions ........................................................... 11 1.6 Brain mechanisms of emotion processing in cochlear implants users ........................................... 14 1.7 Thesis Rationale ............................................................................................................................... 15 CHAPTER 2: Recognition of Musical and Vocal Emotions through Cochlear Implants Simulation (Manuscript 1) ........................................................................................................................................ 16 2.1. Introduction ..................................................................................................................................... 18 2.2 Methods ............................................................................................................................................ 22 2.2.1 Participants .................................................................................................................................... 22 2.2.2 Stimuli ............................................................................................................................................ 22 2.2.3 Procedure ...................................................................................................................................... 22 2.2.4 Analysis .......................................................................................................................................... 23 2.3. Results ............................................................................................................................................. 25 2.3.1 Accuracy......................................................................................................................................... 25 2.3.2 Arousal and Valence Ratings ........................................................................................................ 27 2.3.3 Confidence judgments .................................................................................................................. 30 2.3.4 Correlation of emotional judgments with acoustical features .................................................... 31 2.3.4.1 Accuracy ..................................................................................................................................... 31 2.3.4.2 Arousal ........................................................................................................................................ 31 2.3.4.3 Valence ....................................................................................................................................... 32 2.4. Discussion ........................................................................................................................................ 33 ii 2.5. Acknowledgement and funding sources ........................................................................................ 36 2.6 Connecting text ................................................................................................................................ 36 CHAPTER 3: Neural Processing of Musical and Vocal Emotions through Cochlear Implants (Manuscript 2) ........................................................................................................................................ 37 3.1. Introduction ..................................................................................................................................... 39 3.2. Methods .......................................................................................................................................... 41 3.2.1. Participants ................................................................................................................................... 41 3.2.2. Stimuli ........................................................................................................................................... 41 3.2.3. Task and procedure ...................................................................................................................... 42 3.2.4. Neurophysiological recording ...................................................................................................... 42 3.2.5. Processing and artifact rejection ................................................................................................. 43 3.2.6. Statistical Analysis ........................................................................................................................ 44 3.3. Results ............................................................................................................................................. 45 3.3.1. Behavioral results ......................................................................................................................... 45 3.3.2. ERP results .................................................................................................................................... 45 3.4. Discussion ........................................................................................................................................ 52 3.4.1. The effect of cochlear implant simulation .................................................................................. 52 3.4.2. Comparing vocal and musical emotional processing .................................................................. 54 3.5. Conclusion ....................................................................................................................................... 55 CHAPTER 4: General Discussion & Conclusions ..................................................................................... 56 4.1 General discussion............................................................................................................................ 56 4.2 General conclusion ........................................................................................................................... 57 REFERENCES ............................................................................................................................................ 59 ABBREVIATIONS ..................................................................................................................................... 72 LIST OF TABLES AND FIGURES ................................................................................................................ 73 iii ABSTRACT Cochlear implants (CI) partially restore hearing in the deaf. However, the ability to recognize emotions in speech and music is limited due to the implant’s technological limitations and the impaired neural pathways that developed after sensorineural hearing loss. This leads to developmental and socioeconomic problems for CI-users and thus a decrease in quality of life. Behavioural and neural correlates of this deficit are not yet well established. This thesis aims to characterize the effect of CIs on auditory emotion perception and, for the first time, to directly compare vocal and musical emotion perception through a CI-simulator. The thesis investigated the ability of normal hearing individuals to perceive basic emotions in CI-simulated vocal and musical sounds, using a behavioural task and electroencephalography (EEG). In the behavioural study, the perception of musical and vocal emotions was impaired in the CI-simulated condition. Perception was correlated with timbral acoustic cues. In the EEG study, the averaged event-related potentials’ components had reduced amplitudes and delayed latency as early as 50 milliseconds in the CI-simulated condition. Using this previously validated neuro-behavioural approach with CI-users can further enhance our knowledge and prove the importance of timbral acoustical cues for emotion recognition. It can lead to developing new processing strategies that capitalize on these cues, leading to better perception of auditory emotions and thus improving the quality of life for CI-users. iv RÉSUMÉ Les implants cochléaires (IC) restaurent partiellement l'ouïe chez les sourds. Cependant, la capacité à reconnaître les émotions dans la parole et la musique est limitée en raison des limites technologiques de l'implant et de la voie neurale altérée après perte d'audition neurosensorielle. Ceci conduit à des problèmes développementaux et socio-économiques pour les utilisateurs de IC et donc à une diminution de la qualité de vie. Les corrélats comportementaux et neuronaux de ce déficit ne sont pas encore bien établis. Cette thèse vise à caractériser l'effet des IC sur la perception des émotions auditives et, pour la première fois, à comparer directement la perception de l'émotion vocale et musicale à travers un simulateur IC. L'étude a investigué la capacité des individu avec une ouïe normale à percevoir les émotions fondamentales dans les sons vocaux et musicaux simulés par IC, en utilisant une tâche comportementale et l'électroencéphalographie (EEG). Dans l'étude comportementale, la perception des émotions musicales et vocales a été altérée dans l'état simulé par IC. La perception était corrélée avec les indices timbrales acoustiques. Dans l'étude EEG, les composantes des potentiels moyens liés à l'événement avaient des amplitudes réduites et une latence retardée dès 50 millisecondes dans les conditions simulées par IC. En utilisant cette approche neuro-comportementale validée précédemment avec les utilisateurs de IC, nous pouvons améliorer nos connaissances et prouver l'importance des signaux acoustiques timbrales pour la reconnaissance des émotions. Cela peut conduire à développer de nouvelles stratégies de traitement qui capitalisent sur ces indices, conduire à une meilleure perception des émotions auditives et ainsi améliorer la qualité de vie des utilisateurs de IC. v ACKNOWLEDGMENTS I would first like to thank my thesis supervisors Dr. Alexandre Lehmann and Dr. Anthony Zeitouni, for their support and guidance. I would also like to thank my colleagues and friends at BRAMS for their help and support. And a special thanks to Mihaela Felezeu for her help with EEG. Finally, I must express my very profound gratitude to my mother and siblings for providing me with unfailing support and continuous encouragement throughout my years of study and through the process of researching and writing this thesis. This accomplishment would not have been possible without them. Thank you. Duha Ahmed vi CONTRIBUTION OF AUTHORS This thesis comprises 2 manuscripts. The first manuscript (Chapter 2), entitled, “Recognition of Musical and Vocal Emotions through Cochlear Implants Simulation” is by S Paquette, D Ahmed G., I Peretz, and Alexandre Lehmann, and was submitted to the Journal of Hearing Research. The work was originally conceived by A Lehmann, with contributions from S Paquette, D Ahmed, and I Peretz. All data was collected only by D Ahmed; S Paquette wrote the first draft of the manuscript; S Paquette, D Ahmed, and A Lehmann analyzed the data and interpreted the results; finally, D Ahmed, A Lehmann, S Paquette, and I Peretz critically reviewed and revised the manuscript. The second manuscript (Chapter 3), entitled, “Neural Processing of Musical and Vocal Emotions through Cochlear Implants” is by D Ahmed, S Paquette, A Zeitouni, and A Lehmann, and was submitted to the Journal of Clinical EEG & Neuroscience. The work was originally conceived by A Lehmann, with contributions from D Ahmed, S Paquette, and A Zeitouni. All data was collected only by D Ahmed; D Ahmed wrote the first draft of the manuscript; D Ahmed, S Paquette, and A Lehmann analyzed the data and interpreted the results; finally, D Ahmed, S Paquette, A Lehmann, Anthony Zeitouni, critically reviewed and revised the manuscript. vii CHAPTER 1: Thesis Introduction 1.1 Introduction Cochlear implants (CI) are surgically implanted devices that partially restore hearing abilities in those with severe to profound hearing loss. However, the capacity to recognize emotions in speech and music is limited, because understanding both vocal and musical expressions require the perception of specific acoustic cues (Juslin and Laukka, 2003). CI users’ perception of many of these cues is severely impoverished due to the limitations of the electrical signal transmitted by the implant and due to the impaired neural-pathway post-sensorineural hearing loss (Nakata et al., 2012; Wang et al., 2013; Volkova et al., 2013). Perceiving emotions is essential for social interaction and development. Its deficit leads to miscommunication, and eventually, it might lead to depression and a general decrease in the quality of life of cochlear implant users (Wiefferink et al., 2012). 1.2 Emotional perception An important element of social interaction is communicating emotions (Ekman, 1992). It helps us to empathize and to rejoice with others, and to alert us to sarcasm or hazards. The communication of emotions occurs through vocal sounds of verbal & nonverbal expressions (Scherer, 1986) (e.g. growls, screams, and laughter), facial expressions, postures and gestures (Ekman and Friesen, 1969). It has been proposed that there is a group of basic emotions from which all other emotions are derived (Ekman, 1992). Basic emotions include fear, anger, sadness, enjoyment, disgust and surprise. They are discrete, innate, universal and easily conveyed and perceived through facial, verbal and nonverbal expressions. 1.3 Auditory emotions Neuropsychological research indicates that both music and speech use acoustical-cue transmission to relay their messages through changes in pitch (fundamental frequency (F0)), intensity and duration (Juslin, 1998; Scherer, 2003). Moreover, Juslin and Lukka (2003) have 8 suggested the presence of emotion-specific patterns of cues across voice and music for the three major cues (speech rate/tempo, vocal intensity/sound level, and high-frequency energy). Each of these cues is neither necessary nor sufficient, but the more cues used, the more reliable the communication is (Juslin, 2000). Some musical features were found to be processed through the same pathways as speech (e.g., timbre); while others elicited some unique neurophysiological responses (e.g., tonality) (Patel and Peretz, 1997; Peretz, 2002). For these reasons, a close relationship has been always suggested between the processing of these two means of emotional communication. The previous literature suggests that the recognition of basic emotions in auditory stimuli is dependent on the transmitted acoustical cues for both voice and music. The parameters of these acoustical cues changes for each basic emotion in a way that makes them specific and recognizable for that emotion. 1.4 Neuro-imaging studies of emotional processing How the brain processes vocal emotions can be investigated in two ways, using two modalities. One is “the when”, which is the time course of decoding the acoustical cues. It is studied using electroencephalography (EEG) by measuring evoked event-related potentials (ERPs) and time- frequency analysis (TF). The second is “the where”, which are the anatomical regions of the brain that are involved in the processing of emotional perception. These are studied using brain lesions and neuroimaging employing functional magnetic radio imaging (fMRI). EEG measures brain activity, and when averaged, it can be used to obtain ERPs, which are the neural responses associated with specific events. Recorded ERP waveforms consist of a sequence of positive and negative voltage deflections, which are called peaks, waves, or components. These waves are sometimes named after their peak’s position (N1, P2, N2, and P3) or their latency (e.g., P50 is a positive wave at 50 ms from stimulus onset, and N100 is a negative wave at 100 ms…) (Luck, 2014). TF measures brain oscillations at various frequencies (theta, alpha, beta, and gamma). Brain oscillations have been found to be a powerful measur- 9 to analyze the cognitive processes related to emotion processing (Başar et al., 1999; Krause, 2003). Using ERPs, Schirmer and Kotz (2006) have described a model of three stages for the processing of emotionally-relevant acoustic cues in vocal stimuli. The first stage is the sensory processing stage. It occurs in a pathway that runs from the ears to the auditory cortex. In this phase, the frequency and intensity of the stimulus modulate the amplitude of N1 (occurs 100 ms after the onset of a stimulus). The second stage of processing is the integration of emotionally important acoustic cues and extracting a significant emotion. It happens 200 ms (P200) after stimulus onset, in a pathway between the auditory cortex to the lateral superior temporal gyrus (STG) and the superior temporal sulcus (STS). Moreover, linguistic auditory objects processing was found to be localized in the left hemisphere’s STS and processing of paralinguistic aspects of vocal speech (speaker gender, age, prosody, emotional state) is lateralized to the right middle and anterior STS. The third stage is cognition, which is the integration of the vocal (the extracted emotion) with the verbal (speech semantics processing). Semantics processing occurs at 400 ms from word onset with activation of the left inferior frontal gyrus. Many scientists in their pursuit to explain auditory decoding have adopted this model. However, in contrast to Schirmer and Kotz’s findings, Iredale et al. (2013) reviewed the neural correlates of emotional prosody, and they found N1 to have a greater amplitude in response to words spoken in emotional prosody compared to neutral prosody. This might suggest that emotional prosody processing starts at an earlier stage (N1) than what was previously suggested (P200). Furthermore, Liu et al. (2012) investigated the processing of nonverbal vocal emotions and found that it starts as early as 50 ms (increased P50 amplitudes). Their findings demonstrate that the response to nonverbal emotional vocalization is an automatic and rapid process. Pell at al. (2015) further examined this topic and compared the processing of speech prosody (anger, happiness, and sadness) and nonverbal vocalizations. N100 amplitude was significantly reduced and P200 was significantly larger for vocalizations than speech. This suggests that nonverbal vocalizations are prioritized over speech due to their basic nature and are preferentially processed as early as N100 (Jessen and Kotz, 2011; Paulmann et al., 2013). In 10