▪ Degree’s Final Project ▪ Computer Engineering An Experiment on Melody Training for Amateur Musical Instrument Players Author: Ainhize Goenaga Director: Manuel Graña Degree’s Final Project Ainhize Goenaga ii Degree’s Final Project Ainhize Goenaga Acknowledgements I would like to thank my director, Manuel Graña, for lending me a hand whenever I had problems, for teaching me the steps and a proper way to develop a project with such dimensions and for keeping me motivated all the time. I would also like to thank Leire Ozaeta, for helping me with everything she could, mainly the NAO robot part of my project (even if, in the end I had to leave the robot apart). Another important person I would like to thank is Borja Fernández, who introduced me to Manuel Graña and for his help whenever my director wasn’t available. Last, but not least, I would like to thank my family and my friends, for supporting me and for believing in me. iii Degree’s Final Project Ainhize Goenaga iv Degree’s Final Project Ainhize Goenaga Abstract The long term goal of the research started in this project is to develop systems able to assist people training to sing or hum melodies, or play them on musical instruments. The current project will focus on single instrument training for short monophonic melodies. The interaction between the user and a fully developed application would go as follows: The user selects a melody from the database and the application reproduces it. The human plays the melody with an instrument while the application records it, so, when finished; it processes the recording to give its approval or sings again the melody if it hasn’t been well reproduced. Materials: computational experiments are carried out on the recordings of a well known melody played on two instruments (piano, flute), performed by a custom laptop computer microphone, in order to test robustness against rich harmonics interferences when trying to extract the melody. Methods: Programming has been carried out in Python, using available open source resources: NumPy, SciPy, MatPlotLib, PyAudio. The computational process steps are the following: signal windowing, Fourier transform, transform spectre downsampling, fundamental melody frequency extraction, pattern comparison by correlation. v Degree’s Final Project Ainhize Goenaga Resumen El objetivo a largo plazo de este proyecto de investigación es crear sistemas capaces de entrenar personas para cantar o tocar melodías en instrumentos musicales. El proyecto actual se centrará en el entrenamiento de un único instrumento a la vez con melodías monofónicas cortas. La interacción entre el usuario y la aplicación desarrollada por completo será la siguiente: El usuario elige una melodía de la base de datos, y la aplicación la reproduce. El usuario toca la melodía con un instrumento musical mientras que la aplicación lo graba. Cuando el usuario termina de reproducir la melodía, la aplicación procesará lo grabado para dar su aprobación o volverá a cantar la melodía si ha habido fallos. Materiales: trabajaremos con una melodía conocida que será grabada con el micrófono de un ordenador portátil. Se probará con 2 instrumentos musicales diferentes (piano, flauta) para probar la solidez del algoritmo cuando aparecen armónicos indeseados en el momento de procesar la melodía. Métodos: La programación de la aplicación se hará en Python, y se utilizarán recursos de código abierto tales como: NumPy, SciPy, MatPlotLib, PyAudio. Los pasos del algoritmo son los siguientes: enventanado de la señal, transformada de Fourier, diezmado del espectro, extracción de la frecuencia fundamental de la melodía, comparación de patrones utilizando la correlación. vi Degree’s Final Project Ainhize Goenaga Laburpena Ikerketa honen luzerako helburua pertsonak kantatu eta musika instrumentuak jotzen trebatzen laguntzeko sistemak garatzea da. Proiektu hau instrumentu bakarreko melodia monofonikotan zentratuko da. Zeharo garatutako produktu eta erabiltzailearen arteko elkarrekintza ondorengoa izango da: Erabiltzaileak melodia bat hautatuko du datu-basetik, eta aplikazioak erreproduzitu egingo du. Erabiltzaileak entzundakoa instrumentu batekin joko du aplikazioa grabatzen dagoen bitartean. Erabiltzaileak bukatzean, aplikazioak grabatutakoa prozesatuko du emaitzaren onespena emango du edo, gaizki eginez gero, berriro kantatuko du. Materialak: Ordenagailu eramangarri bateko mikrofonotik grabatutako musika instrumentu batekin jotako melodia ezagun batekin arituko gara lanean, melodia bi instrumentu desberdinekin joko da (pianoa, txirula) sendotasuna frogatzeko, melodia prozesaketan ager daitezkeen harmoniko interferentziak direla eta. Metodoak: Aplikazioaren programazioa Python lengoaian egingo da, eskura dauden kode irekiko baliabideak erabiliz: NumPy, SciPy, MatPlotLib, PyAudio. Prozesuaren pausoak ondorengoak dira: seinalearen leihoketa, Fourier-en transformatua, transformatutako espektroaren hamarrena hartu (downsampling), melodiatik oinarrizko frekuentzia lortu, patroien konparaketa korrelazioaren bitartez. vii Degree’s Final Project Ainhize Goenaga viii Degree’s Final Project Ainhize Goenaga Index Introduction .............................................................................................................................. 1 1. Motivation ...................................................................................................... 1 2. Objectives ...................................................................................................... 1 3. Structure of the Memory ................................................................................ 1 Project Management ................................................................................................................ 3 1. Work Breakdown Structure ........................................................................... 3 2. Time Estimation ............................................................................................. 3 3. Gantt Chart ..................................................................................................... 4 4. Risk Management .......................................................................................... 6 5. Monitoring and Control ................................................................................. 6 5.1 Communication ......................................................................................... 6 5.2 Deviations ................................................................................................. 6 Introduction to Pitch Estimation ............................................................................................ 7 1. Definitions...................................................................................................... 7 1.1 Pitch .......................................................................................................... 7 1.2 Musical Tone ............................................................................................ 7 1.3 Octave ....................................................................................................... 7 1.4 Melody ...................................................................................................... 8 1.5 Polyphony ................................................................................................. 8 1.6 Harmonics ................................................................................................. 9 1.7 MIR (Music Information Retrieval) ......................................................... 9 2. Mathematics ................................................................................................. 10 2.1 Fourier Transform ................................................................................... 10 2.2 Downsample / Decimation ..................................................................... 11 Pitch Estimation Algorithms ................................................................................................. 12 1. Frequency Domain Algorithms ................................................................... 12 1.1 HPS (Harmonic Product Spectrum) ....................................................... 12 1.2 Cepstrum Analysis .................................................................................. 13 1.3 Parabolic Interpolation............................................................................ 14 2. Time Domain Algorithms ............................................................................ 14 2.1 ZCR (Zero-crossing Rate) ...................................................................... 14 2.2 Autocorrelation ....................................................................................... 14 ix Degree’s Final Project Ainhize Goenaga 3. Frequency Domain vs. Time Domain .......................................................... 15 Work Environment ................................................................................................................ 17 1. Hardware ...................................................................................................... 17 2. Software ....................................................................................................... 17 2.1 NumPy vs. SciPy .................................................................................... 18 3. Decisions ...................................................................................................... 18 3.1 Python ..................................................................................................... 18 3.2 Speech Recognition ................................................................................ 19 3.3 Monophonic melody ............................................................................... 19 3.4 Harmonic Product Spectrum (HPS) ....................................................... 19 3.5 Hanning Window .................................................................................... 19 Design ...................................................................................................................................... 21 1. Interaction .................................................................................................... 21 2 Sound Process ............................................................................................... 22 3. Comparison .................................................................................................. 23 Implementation ...................................................................................................................... 24 1. Sound Processing ......................................................................................... 24 1.1 Read_File ................................................................................................ 24 1.2 HPS_Algorithm ...................................................................................... 25 1.3 Create_File .............................................................................................. 26 1.4 Correlation .............................................................................................. 27 2. Interaction .................................................................................................... 28 2.1 Song_Choice ........................................................................................... 28 2.2 Sing_Song ............................................................................................... 28 2.3 Listen_Song ............................................................................................ 29 2.4 Compare_Files ........................................................................................ 30 3. Synthesized Song ......................................................................................... 30 3.1 note .......................................................................................................... 30 3.2 main ........................................................................................................ 30 4. Tuner ............................................................................................................ 31 4.1 Record ..................................................................................................... 31 4.2 Convert_To_Note ................................................................................... 32 Testing ..................................................................................................................................... 33 x