Donald P. Pazel Music Representation and Transformation in Software Structure and Algorithms in Python Music Representation and Transformation in Software Donald P. Pazel Music Representation and Transformation in Software Structure and Algorithms in Python Donald P. Pazel Fishkill, NY, USA ISBN 978-3-030-97471-8 ISBN 978-3-030-97472-5 (eBook) https://doi.org/10.1007/978-3-030-97472-5 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland Preface Introduction Suppose you have an interest in programming computer music and need a starting point. Perhaps you have tried various music applications or tools, such as mixers, production systems, or notation systems, but feel a need to dig deeper, to be more hands-on and work with music fundamentals. Perhaps you have familiarity with concepts of musical notes, keys, chords, scores, and other music elements, but you don’t know where to begin in designing and programming about them, let alone translating them to MIDI®1. You want to be able to put all those elements to work through programming, but you need direction to get to that next killer music app that you have in mind. This book guides you to that sweet spot of developing and using your programming skills to design and build systems about music. However, this book does more than provide that kind of instruction. This book takes the reader on a journey through music concepts in an organized approach that develops music essentials, from the concepts of tone, pitch, and time, through notes, intervals, chords, and scores while at the same time interpreting these elements as software artifacts. As these many concepts are covered, we do so with close attention to the organization of and relationships amongst these concepts and reflect those as Python classes and objects, and so learn about music from a software design viewpoint. By this process, we approach this design in a logical and structured way. The objective for doing this is to make music theory elements programmatically accessible. The result- ing software should allow you to be able to code in the language of music, letting you to think primarily in musical concepts about your programming objectives, while leaving various music calculations, such as key or chord construction, transparent. That is the point behind the software discussion on music representation in the first part of this book. The second part of the book is devoted to music transformation algorithms. These transforms comprise an interesting application of music representation for modifying music through key change, melodic reflection, and much more. This part of the book is more academic and has more of a research feel. However, the connection to programming prominently remains with discussions grounded in 1 MIDI is a registered trademark of the Association of Manufacturers of Electronic Instruments (AMEI). v Preface implementation details and software structure considerations. We hope the reader finds inspiration in this section for novel applications or advanced interests in the wide area of computer music research. The Origin of this Book The origin of this book has its roots in the late 90's. At that time, I had the honor of being a member of a computer music research team at IBM®2 TJ Watson Research Center in New York. This department should not be confused with being a club of musicians that jammed after work. Rather, the Computer Music Center (CMC) focused on MIDI-based composition software and real-time music modification as full-time work in the Mathematics Department of this esteemed IBM research center. The depart- ment was technically and principally led by computer music veterans Danny Oppenheim and Jim Wright. During my tenure there we constructed MusicSketcher [1], a prototype application that demon- strated a wide range of music transformational capability, including dynamic chord replacement, and harmony preserving dynamic note shifts, and other capabilities. These transformations went by the name Smart Harmony [2]. My work experiences in this department served as an introduction to com- puter music, MIDI, and various music tools. The novelty and creativity behind the work there was very impressive, and I found the talks, works, and efforts there stimulating in ideas. Many years and many different careers later, along with finally having freedom of time, I decided to take a fresh look at a few ideas in computer music research, but now in the context of some 20 to 30 years of technological change. The prior work with which I was familiar was preserved in papers and memory. This forced me to think through from scratch the foundations for music representations and transformational capabilities. Additionally, instead of using C/C++ as we had years earlier, I now have access to Java or Python, computer languages with which I have become more adept through my pro- fessional industrial experiences. Computing hardware also improved considerable over time. Further, VST®3 digital audio provided advanced capability in MIDI rendering and provided access to rich sound samples which are currently plentiful in the market. I considered building on an idea for music collaboration that I thought about while I was in the CMC long ago [3]. The idea was to allow multiple people to log into a group-based music session, choose instruments, insert riffs or melody lines, and dynamically modify them in real time collaboratively. As I moved forward on this idea, the effort took on many technical challenges. Firstly, and most im- portantly, I needed a rich flexible music representation with functionality that translates into MIDI and music performance dynamics. While I thought this was my main concern, I quickly learned that there are many other critical features outside of music to address. As this project was designed to have a web-based front end and significantly, real-time collaborative interaction, it required a coordinating server as well. After much effort, I achieved a prototypical result that was very close to what I wanted. Multiple players could engage in music performance through a web interface as a shared experience, much in the way I envisioned. A tune is composed on a shared (web-based) screen using selectable multi-measure music fragments stitched together. The players take on different instrument roles, say one on drums, another guitar, etc. Once the playback started, each player controls their instrument’s pitch (while automatically maintaining harmony) and volume levels. The tempo and harmony (chords) are also adjusted in real time through a special “conductor” role. A server takes care of all the interac- tivity and music dynamics, and ensures the players are relatively synchronized to a unified experience. As these features became demonstrable, the key question became "Where to go from here?" A question that remains unsettled still. Even as I contemplated the work’s uncertain future, there remained a nag- ging consideration that I had reluctantly been putting off and needed to consider. 2 IBM is a registered trademark of International Business Machines. 3 VST is a trademark of Steinberg Media Technologies, GmbH. vi Prerequisites and Approach The problem which with I struggled was that this project was simply too massive for one person to contain within reasonable time. Although initially I focused on building a proper music representation, I quickly became absorbed with server architecture details, distributed playback algorithms, client syn- chronization, web interfaces (JavaScript), and general maintenance. Quickly, the core music represen- tation became brittle and sketchy with the load of work. What started as a music project, became more of a “soup to nuts” web application project, a terribly interesting one, but one that left scarce time and energy for addressing the increasingly insufficient core music representative and transformative issues on which I wanted and needed to spend much more time and energy. I took a break from the project for reflection, to reexamine the work that had been done, determine how to improve upon it, and see to where that circumspection leads. As I realized early on, music represen- tation is a deep and interesting software modeling exercise in itself. The intuitive concepts of music require rigorous definition. The closer I looked, it seemed that although we are generally adept with many music concepts like intervals, scales, chords, and the like, our understanding of these are some- what superficial and at times not so well structured, if not incomplete, and rough around the edges. It’s like the many skills we adopt from rote learning. We work with certain skills daily but don’t think deeply about them nor even understand their foundations with much clarity. Looking at the music representation I had developed, I found needless complexity that seemed to come more from lack of foundation (from admittedly a rush to get that component finished as an expense for other needs) than say sloppy code. Yet another important aspect surfaced, that music representation comes with a set of algorithms that are in fact, genuinely interesting. The computation of scales, computing interval end notes, and similar, are interesting topics, that in a thought-provoking way made me question how as a musician I had done those things, and if there are better ways to think about those topics, especially, when designing soft- ware to manipulate them. As an example, the notion of time in music is interesting, that along with the varied temporal representations comprises an interesting study, including the conversions of one tem- poral representation to another. It is interesting to think about the properties of a musical note, to ques- tion why we limit to specific durations, and pitches. These are just a few examples of what came to light. It speaks to the larger issue of questioning our understanding of music, what restrictions might be too rigid, and if anything can be done to lessen those restrictions. I never returned to that initial distributed application. Instead, I decided to focus on music representa- tion, the related foundational algorithms, and eventually to music transformational issues, from the perspective of a software engineer and to some degree, a mathematician. This turn of focus resulted in a sizeable body of study and code that is detailed in this book. The result is a far more fulfilling adven- ture than I could have imagined. This experience is what led me to writing this book. After years of working solely on music represen- tation, and years looking at how to build transformational algorithms with that representation, I thought I would share that experience here in hopes of motivating others to carry on and perfect that effort and inspire people towards efforts in computer music. Prerequisites and Approach Readers are expected to have basic programming skills, and some familiarity with Python. This in- cludes familiarity with computational statements, logical structures such as if and loop (for/while) statements, and function or procedure definitions with calling protocols, including parameter passing. Also, a familiarity with the concepts of list and dictionary (or maps), embodied as Python lists and maps or dicts, is also assumed. Unlike many topical introductory books with claims to programming or a specific programming language training, this book assumes basic programming skills in Python. If needed, there are many excellent books for learning Python such as [4] [5] or online references such as [6]. v ii Preface The reader should have some loosely defined notions of musical elements. That is, the reader should have notions about what notes, intervals and chords are. However, the book is built around introducing all these concepts in a methodical structured way. So, unlike the programming prerequisites, the music prerequisites are less strict. Object-Oriented design skills are not required but are a critical part of the book’s teachings. This book places a heavy emphasis on learning object-oriented (O-O or OO) design and thinking. Designing in an object-oriented manner forces one to step back from a problem space, and pull apart the concepts and their dependencies, and make judgments on how to approach a programming task. It forces devel- opers to: • code a software model that clearly maps to the problem domain’s conceptual model, and in a sense, “talks in its terms”. • reflect relationships and dependencies directly in the software model that are relevant in the do- main’s conceptual model. • keep in mind efficiencies and inefficiencies, or limitations of programming artifacts used in the software model. • stretch for generality where needed, but within reasonable bounds. What the reader should realize is that like with so many software endeavors, the software design pro- cess never ends. This is particularly so with the music domain, with, for example, the introduction of different instruments, different styles of music, and so forth, each introducing their own conceptual needs. We do not claim to present in this book the best possible music representation but do believe this model goes a long way on generality and will change over time to be more inclusive, and to intro- duce the reader to software design in the object-oriented style. Since music many times involves calculations, especially involving dynamics over time, some mathe- matics is involved. The reader should be able to understand simple equations for calculations and sim- ilar and be capable of translating these into code. What this Book is Not About As important as describing what this book is about, is mentioning what it is not about. Importantly, this is not a book about music notation, nor how to build music notation software. Music notation for sure is important and related to the topic of this book. Notation concerns the visual and textual communi- cation of music content and instruction for performance. Consequently, notation incorporates a map- ping of music concepts to text/symbol and/or visa-versa. This book’s concern is with embodying music concepts as software model objects, something more abstract that could be useful in building a music notation system but is not one in and of itself. By the way, there are numerous exemplar music notation systems on the market: Dorico®4, Sibelius®5, Notion®6, and Finale®7 to name a few. As a warning to more ambitious readers, building a music notation editor is a herculean effort. Entire teams of devel- opers and testers are actively devoted to such endeavors with extensive time and resource commitments by a few companies. Secondly, this is not a book about building MIDI rendering software. MIDI and some of its important aspects and translation of music models to MIDI are described in detail here and provide a useful 4 Dorico is a registered trademark of Steinberg Media Technologies GMbH. 5 Sibelius is a registered trademark of Avid Technology Europe LTD. 6 Notion is a registered trademark of Presonus Expansion, L.L.C. 7 Finale is a registered trademark of MakeMusic, Inc. viii Intended Audience educational introduction for readers interested in learning about MIDI. However, construction of MIDI rendering software is a topic unto itself and is outside the scope of this book. Intended Audience A wide variety of readership is envisioned for this book. Primarily it is for students to accelerate their software development capabilities in designing and building music software. With that, we offer the following thoughts for the varied reader constituency: Students Students should take seriously the technical prerequisites. It is important to be able to program proce- durally in some programming language. If you already know Python, fine. If not and you pick up programming languages easily, fine. However, knowing the basics of Python is the expectation of this book. The book also delivers an introduction to object-oriented principles and how to program to them in Python. The book applies object-oriented principles to varied music elements and accelerates in depth rapidly. The algorithms, even in the representation section at times can be complex and intricate, and in some cases are expressed in mathematical formulae which the user should be able to understand and translate to code. In a few instances, advanced computer science concepts are discussed, for example, red-black trees and constraint engines. Students need not understand the implementations of these concepts in detail, but should concentrate in understanding the features they provide, how to use them, their interfaces, and chiefly their strengths and limitations. The more intrigued student may want to invest more time on theory and implementation details in these areas. While the primary focus for entry-level students should be on the representation part of the book, the transformation part could be of great interest and well-worth reading. It comprises a combination of practical applications of music representation and an introduction to constraint management. Students, if not guided by an instructor, should read through the transformations and at least understand what they contribute musically, and understand them technically as best can. Reading through the many examples should help in providing this understanding. For representation chapters, the end of chapter questions are mostly for provoking thought about the range in complexity of what is presented. The student is encouraged to read through the questions and select a few (if not assigned by the instructor) to think through as best can. For the transformation chapters, each chapter ends with a “final thoughts” section which highlights both positive and less than positive aspects of the chapter’s topic. It is meant to be thought provoking, and fuel for further efforts. Instructors Instructors can consider using this book for a one semester course on computer music programming. The representation part could easily occupy a semester. For classes comprised of students with lower- level programming skills, the instructor could augment the course with a robust Python review. That said, it is recommended to spend a good deal of time on object-oriented concepts and how to code Python to an object-oriented methodology. After that, the book can be followed chapter to chapter with the idea of introducing new music concepts in succession, along with representation and coding prin- ciples. If the semester has more time to offer, one can consider introducing topics from the transformation section, with at least a guide through the transformations and examples. The constraint engine itself can be technically demanding for students, and the instructor can defer on its implementation details, and instead highlight its interface and usage. The constraint engine is an interesting topic, and advanced ix Preface students may want to consider augmenting or working with it as an advanced project in this or other domains. Practitioners Practitioners may come to this book for a variety of reasons, and in a variety of ways. If you want to learn how to approach object-oriented programming with computer music as a domain, then a straight- forward read is recommended. Otherwise, treating it as a reference text, picking and choosing chapters in the representation section is useful. As for the transformation section, there are many topics from which to choose. A few of the transforms make use of the constraint engine, but many don’t. It is a matter of looking through the transform types to determine which topics might be of interest. Experienced Researchers Experienced researchers may generally find the representation section less interesting than the trans- formation section, except perhaps for the later chapters of the representation section concerning varia- ble dynamics and harmony representation. We would expect this audience to be more interested in the transformation section of the book, on which there are a wide range of topics to study and expand upon. Regarding the Musical Examples The reader will find an abundance of musical examples. While music written on a page is not as satis- fying as listening to it, there are several ways to listen to them. • Use the score to midi conversion code discussed in this book to generate MIDI files and play the examples on a MIDI player of choice. • Although not covered in this book, there is a primitive VST host referenced in the code base that can be used with Python interface code in the code base. • The author found copying generated music to a notation application with playback is very practical and useful, and allows for quickly changing an example’s dynamics, tempo, etc. • Hands-on playing examples on a musical instrument. The author is aware that many books of this type go through great effort to provide support for MIDI playback. The decision was made to invest effort on the main ideas of this book on music representation and transformation, even at the expense of not providing what would at best be a poorly functional MIDI playback software package that would only detract from the learning experience. About the Software As a bonus this book is associated with an accessible software package that reflects the work of this text. This software can be found at: https://github.com/dpazel/music_rep This software package reflects the class hierarchy and details described throughout this text. It is cov- ered by MIT license with no guarantees nor warrantees. x