Faust audio DSP language in the Web Stephane LETZ and Sarah DENOUX and Yann ORLAREY and Dominique FOBER GRAME 11, cours de Verdun (GENSOUL) 69002 LYON, FRANCE, {letz, sdenoux, orlarey, fober}@grame.fr Abstract API, section 5 presents the different approaches experimentedtodeployFaustDSPprogramson WiththeadventofbothHTML5andtheWebAudio API (a high-level JavaScript API for audio process- the Web. Section 6 exposes some use cases, and ingandsynthesis)interestingaudioapplicationscan finally some results and benchmarks are given nowbedevelopedfortheWeb. TheWebAudioAPI in section 6.1. offers a set of fast predefined audio nodes as well as customizable ScriptProcessor node, allowing devel- 2 Programming audio in the Web opers to add their own javascript audio processing 2.1 Web Audio API code. Several projects are developing abstractions on The Web Audio API [12] specification describes topoftheWebAudioAPItoextenditscapabilities, a high-level JavaScript API for processing and andoffermorecomplexunitgenerators,DSPeffects synthesizing audio in Web applications. The libraries, or adapted syntax. This paper brings an- conception model is based on an audio routing otherapproachbasedontheuseoftheFaustaudio graph, where a number of AudioNode objects DSPlanguagetodevelopadditionalnodestobeused are connected together to program the global as basic audio DSP blocks in the Web Audio graph. audio computation. Differentmethodshavebeenexplored: goingfrom an experimental version that embeds the complete The actual processing is executed in the un- Faust native compilation chain (based on libfaust derlying implementation 1 for native nodes, but + LLVM) in the browser, to more portable solu- direct JavaScript processing and synthesis is tions using JavaScript or the much more efficient also supported using the ScriptProcessorNode. asm.js version. Embedding the Faust compiler it- self as a pure JavaScript library (produced using 2.2 Native nodes Emscripten) will also be described. Theinitialideaofthespecificationistogivede- The advantages and issues of each approach will velopers a set of highly optimized native nodes, be discussed and some benchmarks will be given. implementing the commonly needed functions: Keywords playing buffers, filtering, panning, convolution etc. The nodes are connected to create an au- WebAudioAPI,Faust,DomainSpecificLanguage, dio graph, to be processed by the underlying DSP, real-time audio real-time rendering layer. 1 Introduction 2.3 JavaScript ScriptProcessorNode This paper demonstrates how an efficient com- The ScriptProcessorNode interface allows the pilation chain from Faust to the Web Audio generation, processing, or analyzing of audio API can be done, allowing the available Faust using JavaScript. It is an AudioNode audio- programs and libraries to be immediately used processing module that is linked to two buffers, in a browser. one containing the input audio data, one con- Section 2 describes the Web Audio API and taining the processed output audio data. howitcanbeextendedandtargetedbyDomain An event, implementing the AudioPro- Specific Languages. Section 3 describes the cessingEventinterface,issenttotheobjecteach Faust language and its mechanisms to be de- timetheinputbuffercontainsnewdata,andthe ployedonalargevarietyofplatforms. Section4 event handler terminates when it has filled the exposes the compilation chain and the multiple output buffer with data. target languages available from a unique DSP specification. In the context of the Web Audio 1typically optimized assembly or C/C++ code This is the hook given to developers to add C++ programs3 taking care of generating the newlowlevelDSPprocessingcapabilitiestothe most efficient code. The compiler also offers system. various options to control the generated code, including options to do fully automatic paral- 2.4 Programming over the Web Audio lelization and to take advantage of multicore API architectures. VariousJavaScriptDSPlibrariesormusicallan- FromasyntacticpointofviewFaustisatex- guages, have been developed over the years ([4], tual language, but nevertheless block-diagram [6], [8], [10]) to extend, abstract and empower oriented. It actually combines two approaches: the capabilities of the official API. They offer functional programming and algebraic block- users a richer set of audio DSP algorithms and diagrams. Thekeyideaistoviewblock-diagram sound models to be directly used in JavaScript construction as function composition. For that code. purpose, Faust relies on a block-diagram alge- bra offivecompositionoperations(: , ˜ <: :>) When following this path, developments have [1]. to be restarted from scratch, or by adapting Here is an example of how to write a noise already written code (often in more real-time generator in Faust: friendlylanguageslikeC/C++)intoJavaScript. An interesting alternative has recently been random = +(12345)˜∗(1103515245); developed by the Csound team [11]: by using noise = random/2147483647.0; the C/C++ to JavaScript Emscripten [3] com- process = noise ∗ vslider(”Volume” ,0 ,0 ,1 ,0.1); piler, the complete C written Csound runtime and DSP language (so including a large number 3.1 Language deployment of sound opcodes and DSP algorithms) is now available in the context of the Web Audio API. Being a specification language, the Faust code UsinganautomaticC/C++toJavaScriptcom- tellsnothingabouttheaudiodriversortheGUI pilation chain opens interesting possibilities to toolkit to be used. It is the role of the architec- ease the deployment of well-known and mature ture file to describe how to relate the DSP code code base on the Web. to the external world. Additional generic code is added to connect the DSP computation it- 3 FAUST language description self to audio inputs/outputs, and to control pa- rameters, which could be buttons, sliders, num Faust [Functional Audio Stream] [1] [2] is a entries etc. in a standard user interface, or any functional, synchronous, domain-specific pro- kindofcontrolusingaremoteprotocollikeOSC gramming language specifically designed for or HTTP. real-time signal processing and synthesis. A ThisapproachallowsasingleFaustprogram unique feature of Faust, compared to other ex- to be easily deployed to a large variety of audio istingmusiclanguageslikeMax2,PureData,Su- standards (Max-MSP externals, PD externals, percollider, etc., is that programs are not inter- VST plugins, CoreAudio or JACK standalone preted, but fully compiled. Faust provides a applications, etc.). high-level alternative to hand-written C/C++ to implement efficient sample-level DSP algo- 4 FAUST compilation chain rithms. 4.1 Static compilation chain One can think of Faust as a specification The current version of the Faust compiler language. It aims at providing the user with (faust1) produces DSP code as a C++ class, to an adequate notation to describe signal pro- beinsertedinanarchitecturefile. Theresulting cessors from a mathematical point of view. file isfinally compiledwith a regularC++ com- This specification is free, as much as possible, pilertoobtainanexecutableprogramorplug-in from implementation details. It is the role of (Figure 1). the Faust compiler to automatically provide The produced application is structured as the best possible implementation. The com- shown in Figure 2. The DSP becomes an audio piler translates Faust programs into equivalent computationmodulelinkedtotheuserinterface and the audio driver. 2thegenobjectaddedinMax6nowcreatescompiled code from a patch-like representation, using the same 3Infaust1,faust2branchallowstocompilefordiffer- LLVM based technology ent languages Figure 1: Steps of Faust compilation chain Figure 3: Faust2 compilation chain It can, for instance, convert a C or C++ source file into LLVM IR code. Domain-specific languages like Faust can easily target the LLVM IR. This has been done by developing a special LLVM IR backend in Figure 2: Faust application structure the Faust compiler. 4.4 Dynamic compilation chain The complete chain goes from the DSP source 4.2 Multiple backends code, compiled in LLVM IR using the LLVM Faust2 development branch uses an intermedi- back-end,tofinallyproducetheexecutablecode ateFIRrepresentation(FaustImperativeRep- using the LLVM JIT [5]. All steps take place in resentation), which can be translated to several memory, getting rid of the classical file based output languages. approaches. Pointers to executable functions The FIR language describes the computation can be retrieved from the resulting LLVM mod- performed on the samples in a generic manner. ule and the code directly called with the appro- It contains primitives to read and write vari- priate parameters (Figure 4). ables and arrays, do arithmetic operations, and define the necessary control structures (for and while loops, if structure etc.). The language of signals (internal to the Faust compiler) is now compiled in FIR intermediate language. Togeneratevariousoutputlanguages, several Figure 4: libfaust + LLVM dynamic compila- backends have been developed: for C, C++, tion chain Java, JavaScript, asm.js, and LLVM IR (Figure 3). The native LLVM based compilation chain is particularly interesting: it provides direct 4.5 FAUST compiler as a library compilation of a DSP source into executable In the faust2 development branch, the Faust code in memory, bypassing the external com- compiler has been packaged as a library called piler requirement. libfaust, published with an associated API [5] 4.3 LLVM thatimitatestheconceptoforiented-objectlan- LLVM (formerly Low Level Virtual Machine) is guages, like C++: a compiler infrastructure, designed for compile- • given a Faust source code (as a file or a time, link-time, run-time optimization of pro- string), calling the createDSPFactory func- grams written in arbitrary programming lan- tion runs the compilation chain (Faust + guages. Executable code is produced dynami- LLVM JIT) and generates the “prototype” cally using a “Just In Time” compiler from a of the class, as a llvm-dsp-factory pointer. specific code representation, called LLVM IR. Clang, the “LLVM native” C/C++/Objective- • next, the createDSPInstance function, cor- C compiler is a front-end for LLVM Compiler. responding to the new className of C++, instantiates a llvm-dsp pointer, to be acti- Asanativenode,itcanbeusedlikeanyother vated and controlled through its interface, regular native node and connected to other and connected to the audio drivers. nodes in the graph: dsp.connect(context. destination ); Having the compiler available as a library opens new interesting possibilities explored in The Faust source code usually contains the FaustLive [9] application. DSP source code an abstract description of its user interface, canbecompiledontheflyandwillrunatnative described in terms of buttons, sliders, bar- speed. graphs..., to be “interpreted” and displayed by an actual user interface builder component. 5 Using FAUST compiler in the Web This user interface can be obtained as a JSON We have tested and implemented two different description, that can be decoded to implement methods to use the Faust compilation chain in the UI themselves to control the node’s param- the Web: eters: • the first one consists in embedding the lib- var json = dsp. json (); faust + LLVM native compilation chain di- InternalcontrolparametersoftheDSPcanbe rectly in the browser. Starting from the retrieved as a list of AudioParams, to be used Faust DSP source, a native WebAudio like regular ones: nodewillbecompiled onthefly, tobeused like any regular native node. The set of all var num params control parameters will be exposed as We- = dsp.numberOfAudioParams(); bAudio AudioParams objects. var audio param = dsp.getAudioParam(0); • an alternative and more portable method audio param. setValue (0.5); purely stays at JavaScript level, using Insteadofdirectlyaccessingthegivenparam- asm.js and Emscripten. Starting from the Faust DSP source, a highly optimized eter, another possibility is to use the follow- ing generic function, taking a complete access asm.js based ScripProcessor node will be “path” to the parameter, and a given value: produced. The set of all control param- eters will be exposed to control the DSP dsp.setAudioParamValue(”/wet” ,0.5); node. Embedding the Faust compiler in a browser Both approaches have advantages and issues is quite efficient, since the native executable that will be explained in detail in the following code runs in the real-time audio thread that sections. computes the audio graph rendering. But more general deployment and acceptance would re- 5.1 Native FAUST DSP Web Audio quire convincing the Web Audio community to node embed a DSL language for audio processing in Embedding the libfaust + LLVM compilation all browsers. chain has been experimented by “hacking” the WebKit open-source browser and by plugging 5.2 Compiling to JavaScript the Faust compiler in its Web Audio sub- More portable solutions have to use the project. ScriptProcessorNode node, directly producing A new native C++ FaustNode (sub-class of JavaScript code to be executed in the node. base class AudioNode) has been added to the set of native Web Audio nodes4. This node 5.2.1 JavaScript backend takes the DSP source code as a string parame- A pure JavaScript backend has been added to ter, compiles it on the fly to native executable Faust in 2012 to produce standard JavaScript code, and activates it: code. The DSP class definition is then wrapped with a generic JavaScript file in order to get a var dsp fully working Web Audio ScriptProcessorNode. = context.createFaustNode(code); 5.2.2 Results 4This work was done in summer 2012 with the gen- Two main problems have been discovered with erous help of Chris Rogers, working at Google at that time. this approach: • for some of its computations, the Faust compiler relies on pure 32 bits integral mathematical operations. Since JavaScript stores numbers as floating-point values ac- cording to the IEEE-754 Standard, this Figure 5: Faust to asm.js (using Emscripten) kind of computation can’t produce the ex- static compilation chain pectedresult. Thus, someDSPeffects(like noise generation that uses a wrapping 32 bits integer division) do not work correctly. 5.3.2 Developing a direct asm.js backend • since standard JavaScript is not really suited to implement fast DSP code, the A pure asm.js backend has been added to the generated program is significantly slower faust2 branch, bypassing the Emscripten com- compared to the native C/C++ or LLVM pilation chain (Figure 6). versions. The resulting audio nodes are us- The backend produces the asm.js module as able only when the programmed DSP code well as some additional helper JavaScript func- is simple enough, but more demanding al- tions, to be wrapped by generic JavaScript to gorithms (like physical models) can usually become a completely usable Web Audio node. not be used. Heap memory code to be used with the asm.js module, and connection with compiled helper 5.3 Compiling to asm.js JavaScript functions is managed by the wrapping code. Started in 2011 to facilitate the port of large C/C++ code base in JavaScript, Mozilla de- velopers have started the Emscripten compiler project, based on LLVM technology, that gen- erates JavaScript from C/C++ code. Figure6: Fausttoasm.js(usingFIRbackend) Later on, they designed asm.js, a completely static compilation chain typed subset of JavaScript, statically compil- able, garbage-collection free, that can be highly optimized by the compilation chain embedded A new DSP instance is created using the fol- in recent Web browsers. It is then possible to lowing code, taking the Web Audio context and reach performances similar to pure native code5 a given “buffer size” as parameters: Mainly designed to manipulate simple types var dsp like floating point or integer numbers, asm.js = faust . karplus(context , buffer size ); language is particularly of interest for audio code. Two successive developments have been TheuserinterfacecanbeobtainedasaJSON carried out with this approach. description, that can be decoded to implement the UI themselves to control the node’s param- 5.3.1 Using Emscripten compiler eters: Starting from the Faust DSP generated C++ var json = dsp. json (); class, the Emscripten compiler translates it to JavaScript. Additional wrapping JavaScript The instance can be used with the following code connects the Emscripten runtime mem- code: ory manager and makes the generated code be- dsp. start (); come a ScriptProcessorNode node to be used in dsp.connect(context. destination ); the audio graph (Figure 5). This method has dsp.setValue(path to control , val ); been successfully developed and demonstrated by Myles Boris [7]. 5.4 Embedding the JavaScript FAUST Although this approach performs rather well, compiler in the browser it requires the Emscripten tool chain to be in- Thanks to the Emscripten compiler, the Faust stalled on the user machine. A more integrated compiler itself can be compiled to asm.js system has been later on developed. JavaScript. This has been done by compil- ing the libfaust C++ library to the libfaust.js 5In the best cases, asm.js code is said to be only 2 JavaScript library (Figure 7), that exports a or 3 times slower than pure native code, see http:// kripken.github.io/mloc_emscripten_talk unique entry point: 6 Use cases Using the previously explained technologies, three different use cases have been experi- mented: Figure 7: Compiling C++ libfaust to libfaust.js with Emscripten • compiling self-contained ready to use Web Audio nodes (see section 6.1) • using Faust static compilation chain to • createAsmCDSPFactoryFromString(...) produce HTML pages with DSP code (see allows to create a DSP factory from a section 6.2) given DSP program as a source string and a set of compilations parameters, • using the Faust dynamic compilation uses the asm.js backend, and produces the chain to directly program DSP in the Web complete asm.js module and additional (see section 6.3). pure JavaScript methods as a string. 6.1 Programming Web Audio nodes • then calling JavaScript “eval” function on with FAUST this string compiles it in the browser. The Self contained ready to use Web Audio nodes dynamicallycreatedasm.jsmoduleandad- can be produced using the faust2asmjs script, ditional pure JavaScript methods (Figure using the static compilation chain explained in 8) can then be used. section5.2. ThescriptbasicallycallstheFaust compiler targeting the asm.js backend with the appropriate architecture file, that wraps the producedcodewithgenericJavaScripttobeus- able in the Web Audio API context (Figure 9). Figure 8: libfaust.js + asm.js dynamic compila- tion chain This internal code in then wrapped with ad- ditional JavaScript code. A DSP “factory” will be created from the DSP source code with the following code: var factory = faust .createDSPFactory(code); Figure 9: faust2amsjs and faust2webaudioasm compilation chains AfullyworkingDSP“instance”asaWebAu- dio node is then created with the code: var dsp 6.2 Deploying FAUST DSP examples = faust .createDSPInstance(factory , in the Web context , Using the faust2webaudioasm script, a DSP buf size ); source file can be compiled to a self-contained TheuserinterfacecanberetrievedasaJSON ready to run HTML page (Figure 10), using description: thestaticcompilationchain(seesection5.2and Figure 9). var json = dsp. json (); The Faust compiler targeting the asm.js The instance can be used with the following backend with the appropriate architecture file code: is called. The asm.js + JavaScript WebAu- dio node is then wrapped in a more complex dsp. start (); HTMLcodetemplate,andthefinalHTMLpage dsp.connect(context. destination ); dsp.setValue(path to control , val ); is obtained. Adding the -links parameter to the scriptmakestheHTMLpagealsocontainslinks to the original DSP textual file, as well as the Since the various presented methods could block-diagram SVG representation. not be developed in a same browser, we had to Thus it becomes quite simple to publish DSP use two different ones. Native version is tested algorithms, helping it wider usage of the Faust inthe“hacked”WebKitapplication, JavaScript DSL approach. and asm.js using Firefox version 32.0.3. Effect native JavaScript asm.js cubic distortion 6.0 % 45 % 28 % harpe 2.7 % 50 % 8 % kisanaWD 4 % over 100% 14 % Table 1: Global CPU use of the application tested on a MacBook Pro 2,3 GHz Even with this limited testing method, some interesting results emerge. The native chain (based on libfaust + LLVM) is clearly the Figure10: ExampleofSVGbaseduserinterface fastest, the asm.js based one is usable in a lot generated from the JSON description of real world use cases. The JavaScript ver- sion performs poorly, and is even not usable be- cause of CPU overuse in a lot of examples (like 6.3 Programming DSP in the Web “kisanaWD” here). Having the Faust compiler itself as a library in 7.2 Known issues and perspective the browser opens interesting capabilities: Although the previously described develop- • “light” Faust IDE allowing users to test ments show some promising results, they are the language can be easily developed on still several issues to be solved: the Web, completing the more full featured FaustLive application [9]. • code for pure JavaScript and asm.js gener- • combining existing DSP sources published ated nodes is executed in the main thread. as HTML pages, to create new DSP pro- Soitmaysufferfrominterferenceswiththe grams to be directly tested and used in the UI computation or possibly garbage collec- Web, or possibly exported to any native tion. Moreover latency is added since an platform supported by the FaustWeb ex- additionalbufferisusedintheaudiochain. ternal compilation service. This has been Thus real-time guaranties may not be met demonstrated by Sarah Denoux [13]. typically resulting in audio glitches 6. 7 Tests and benchmarks • a specific problem has been discovered Thethreepreviouslydescribedapproacheshave when audio computation produces “denor- been tested on a 4 cores MacBook Pro 2,3 GHz. mal”floatvalues: onIntelprocessors, CPU performances degrade a lot 7. 7.1 Benchmarks The Web Audio API is still a fresh specifica- • on the contrary, the “native” version is tion. Its implementation in different browsers much more stable, has less latency since on different platforms is not always complete the computation is done in the real-time or stable. Comparing the previously described thread with no added buffer, but is much approaches has been quite challenging, mainly more difficult to deploy and maintain 8. becauseofslightdifferencesofbehaviororinter- action with the underlying operating system. 6A possible solution to this problem by moving the The proposed benchmarks have been done by ScriptProcessorNode code in audio worker threads has simplycomparingtheapplicationCPUusewith been recently discussed in the W3C Audio working list, some heavy Faust programs, using the ”Activ- see http://webaudio.github.io/web-audio-api 7Theproblemhasbeenreportedandshouldbesolved ity Monitor” tool included in OSX. Three dif- at the JavaScript language definition level. ferent DSP programs have been tested. 8A port in Firefox is in progress. 8 Conclusion languages and applications, pages 301–312. The Faust audio DSP language can now be ACM , 2011. used to easily develop new audio nodes in the [4] H. Choi, J.Berger, “Waax: Web Audio API Web Audio model, and use them in an audio extension”, In Proceedings of the Thirteenth graph. Complete HTML pages with a working New Interfaces for Musical Expression Con- userinterfacecanalsobegenerated. Havingthe ference., 2013. dynamic compilation chain (either in native or [5] S. Letz, Y. Orlarey and D. Fober, “Com- pure JavaScript mode) directly available in the ment embarquer le compilateur Faust dans browser is also interesting to further explore. vosapplications?”,Journeesd’Informatique Even if the Web Audio approach starts to Musicale, 2013. mature, there are still some problematic issues, forinstancefloatsamplesdenormalizationprob- [6] C. Roberts, G. Wakefield, and M. Wright, lem,ornonreal-timeguarantieswhilerendering “TheWebBrowserasSynthesizerandInter- the ScriptProcessorNode JavaScript code. face”.NewInterfacesforMusicalExpression The recent discussion on the Audio Workers conference (NIME), 2013. model opens perspectives for a better render- [7] M. Borins, “From Faust to Web Audio: ing scheme. Basically the JavaScipt audio code Compiling Faust to JavaScript using Em- will be moved to the real-time audio thread, scripten”, Linux Audio Conference, 2014. and communications to get/set parameter val- ues will be done from/to the main thread. [8] C. Clark, A. Tindale, “Flocking: a frame- It remains to be tested how the compilation work for declarative music-making on the of DSP to Web Audio nodes from a high-level Web”, International Computer Music Con- DSLlanguagelikeFaustorCsoundwillbenefit ference, 2014. from it. [9] S. Denoux, S. Letz, Y. Orlarey and D. Fober, “FAUSTLIVE Just-In-Time Faust Acknowledgments Compiler... and much more”, Linux Audio This work has been done under the FEEVER Conference, 2014. project [ANR-13-BS02-0008] supported by the “Agence Nationale pour la Recherche”. [10] J. Kalliokoski, “audiolib.js, a powerful toolkit for audio written in JS”, https: References //github.com/jussi-kalliokoski/ [1] Y. Orlarey, D. Fober, and S. Letz, “Syntac- audiolib.js/ tical and semantical aspects of Faust”, Soft [11] V. Lazzarini, E. Costello, S. Yi and J. Computing, 8(9), 2004, pp. 623–632. Fitch, “Csound on the Web”, Linux Audio [2] S. Letz, Y. Orlarey and D. Fober, “Work Conference, 2014. Stealing Scheduler for Automatic Paral- [12] WebAudioAPI reference descrip- lelization in Faust”, Linux Audio Confer- tion, http://webaudio.github.io/ ence, 2010. web-audio-api/ [3] A. Zakai, “Emscripten: an LLVM to [13] S. Denoux, Y. Orlarey, S. Letz, and D. JavaScript compiler”, In Proceedings of the Fober, “Compose with Faust in the Web”, ACM international conference companion Web Audio Conference, IRCAM & Mozilla on Object oriented programming systems Paris, France 2015.