Introduction

Last time, I wrote about my failed attempt at creating an audio engine for my website. We’ve gone about all the trouble I went into, including finding a potential bug in the Faust compiler, and the unavoidable abandonment of that project. Yet, undefeated, I tried one more time.

In this post I will focus on how I managed, finally, to create a decent real-time interactive audio synthesis engine for my website. This solution ended up being a lot simpler than the previous one, and fitted the ambiance I was going for.

Last attempt fallout

Before trying to create a new audio project from scratch, it was important to understand what went wrong. This helped me to tackle the issues (or avoid them entirely) and have new ideas. This is the quick list I compiled:

  1. In my case, creating a song is a bad idea. It’s hard to fit the feeling I want;
  2. It was harder to match instruments than I expected. Even for simpler instruments, programming them from scratch to sound like the ones from DAWs is tricky;
  3. Because of that, it is best to do the bulk of the work already in the language, and not from a DAW;
  4. Looping the same track (or track segments) alone is not diverse enough.

I came full circle to one of my initial visions. One of the first ideas I had was to have an ambient or abstract soundtrack running in the background. From there, I made the mistake of consolidating to a more well-defined track. That was the idea I initially pursued (which I explained here). I was keen on not making that mistake again, so I started thinking about how I would approach this project from an ambient music standpoint.

Second attempt: A more reasonable approach

Generative music

After thinking a bit and playing around with my DAW, it became obvious I was going to create generative music. Generative music is a genre of music that is algorithmically generated and encompasses some kind of randomness in it. This term was coined by Brian Eno, during his pioneering audio experiments in the 90s. Due to its nature, it is non-repeating and always evolving. These traits make it very attractive to this project. This type of music is also highly connected to live coding, which is a performing art in which the musician creates live music using code. Some of the audio synthesis languages I wrote about in the last post are particularly designed with this kind of application in mind. For example, SuperCollider was used in one of my favorite live coding acts, performed by two members of the Benoît and the Mandelbrots band, in a German TEDx:

Forming the new plan

But back to this project. I wanted to do something simpler and play with the piece sonically. I continued to play around in my DAW, and had an idea. I created eight tone generators, altered some of their configurations slightly from one another, and gave them the notes to form a Am7add4 chord (A minor seventh add 4) but in a scattered and mirrored configuration. By playing with the mixing of those oscillators, I got interesting sounds. I was pleased with that, and the job was now to implement it in Faust. Note that I did not create a song in the DAW and tried to replicate it afterward, I used the DAW to brainstorm an idea to later implement in Faust.

Creating the audio engine

On the premise of trying not to over-extend this post, I won’t go into too much detail about how everything is done, but I’ll give a general idea of the architecture of the DSP. On another note, as I mentioned in the previous post, to develop this Faust project, I am using their WebIDE. Since I am now dealing with a small unordered amount of notes, I can ditch the step-sequencer, removing a previous pain point. Consequently, I also did not use the MIDI python script I wrote for the previous project.

I created a basic synth voice (using only a sine oscillator for now) and started working on the note-triggering part. The eight oscillators are constantly running, but their volume is controlled by ADSR envelope generators (allows the control of the volume shape). These envelope generators default at the closed position, so without external triggers, these oscillators are muted. I wanted to trigger these envelopes independently, and randomly. To do that, I generated 8 white noise sources, one for each synth voice. Then, I configured a reading frequency of the noise source to each of the oscillators, using an auxiliary low-frequency oscillator (LFO). When the random value is read, it is checked if it is bigger than a set constant. This allows me to control the probability of the note being played. Both these controls allow me to alter the timings and probabilities of each oscillator.

This proof-of-work setup worked nicely! It’s been a smooth sailing project, how odd. Well, right about now I started to play a little bit more with the oscillators. I changed the output wave to a mixture of square and sawtooth waves. I also added an output low-pass filter to each voice. Each voice contains an additional square wave that oscillates at a perfect fifth above the voice’s center frequency. This voice is one of the first user controls I added, and serves to enrich the harmonics of the base note.

The voices seem completed, but I knew it could sound much better. As it stands, the sound is dry and surgical. Not good for ambiance, but I knew just how to fix it using my favorite audio magic sauce: a reverb. A reverb does the same effect as playing a sound in a big echoey room, where you can control an assortment of the characteristics of that room. I love reverbs. I think they are the part of the digital audio chain that gives the most character and organicness to a digital audio source. They are very diverse, and there is no correct_ way of implementing one. In my opinion, reverb creation is an art and opens the gates of creativity and sonic diversity in music. Fortunately for me, the Faust standard libraries contain a reverbs library, with a collection of nice reverbs. After playing around with them, and doing A/B testing on my project, I settled on a reverb called JVerb. This reverb is modeled on the vintage early-digital reverbs from Lexicon (a very important pioneering company in the digital reverberation market) and Alesis. I played around with it and dialed in the final reverb values. To tame the loudness of the sound, I added a compressor and dialed it to just intervene slightly in the top end of the dB range. The main part of the project was now over.

Creating user interactivity

Adding interactivity to the project was very easy. Faust allows interaction outside of the program using buttons and sliders. Here’s an example:

I planned to use this feature to tie in the screen coordinates of the mouse hover. But first, I needed to decide on which attributes I was going to modulate with the interactivity.

I managed to create the following eight modulation elements:

  1. Master detune (in half-steps);
  2. Synth voice volume envelope release time;
  3. Synth voice volume envelope hold time;
  4. Synth voice low-pass filter cutoff frequency;
  5. Synth voice perfect fifth auxiliary oscillator volume;
  6. Reverb delay-line modulation depth (chorusing effect);
  7. Reverb delay-line modulation frequency (chorusing effect);
  8. Reverb T60 (reverberation audio decay time of 60dB).

Every modulation variable has its set ranges and offset.

Unfortunately, I did not come up with a practical way of modulating those eight values independently. So, I combined them into only three Faust interaction variables, two for the screen coordinates and one for the screen click. After playing around, I converged into this separation:

  1. Screen X (modx) controls modulation 4, 5 and 6;
  2. Screen Y (mody) controls modulation 2, 3, 7 and 8;
  3. Screen click is left to control the master detune value.

Finally, I also added an output gain/volume control (gain input slider). In reality, this does not control directly the output gain but rather the input gain of the reverb. This allows for a more organic fade-out instead of an abrupt cut.

At last, the audio engine is done. Now it’s time to compile it to WebAssembly and integrate it with the homepage (and Hugo).

Final steps

Packaging up the solution

To integrate the project with the website, first, we need to compile the Faust DSP project (named home.dsp) to WebAssembly. Faust has multiple targets within WebAssembly, but the two most relevant ones are:

  1. webaudiowasm: Outputs a WebAssembly file and an HTML file. Provides the default Faust UI, and the scripts are embedded in the HTML;

  2. wasmjs: Outputs a WebAssembly file and a javascript file. No UI is provided, the user must create and call the main DSP of the javascript file.

The first option is useful to understand how Faust functions within a Web context but is not suited to be integrated into another website. The second option however suits our needs, since we want to integrate the DSP into another website. We can compile the DSP to this target by a terminal (using the faust2wasm command) or in the WebIDE (clicking on the truck icon, setting the platform to web and then the architecture to wasmjs).

After compilation, we are left with two files: home.js and home.wasm.

With the help of this paper, some reverse engineering and trial-and-error, I was able to understand how to update the slider control values in the Faust WebAssembly platform.

The only modification we need to do in the Faust compiler-generated code is to alter the dspFile variable in the home.dsp file to satisfy the location I choose. Then, I built a top-level script that contains two functions: one to start the audio engine, and another to stop it.

When the audio engine is started, the script creates event listeners to handle mouse moves and mouse clicks. I also later added listeners to touch start and touch move events, so the engine responds to mobile browser interactions. For each mouse move (or screen swipe), the script divides the mouse coordinates by the size of the window. This value is then fed to the audio engine (through the controls modX and modY). For a better user experience, I inverted the Y coordinate, since screen coordinates start from the top-left corner of the screen (as opposed to the more common bottom-left coordinate system used in basic math). The mouse clicks are handled a little bit differently. When the mouse is clicked, the script selects between the numbers -1 and 1. It then adds this selected value to the current master detune value. This means that in every click, the center frequency of the oscillators rises or falls one semitone. If, by chance, the offset reaches more than the absolute value of 4, the master detune value resets to zero.

When it is required to stop the audio engine (by calling the top-level stop function), the reverb input gain of the DSP is set to zero (through the gain control slider), and the audio context is killed and closed five seconds after. These five seconds leave the DSP enough time to reverberate the rest of the sound, acting as a natural fade-out.

Integration with Hugo

The only step left is to make the homepage call the two top-level functions, and we are golden! Unfortunately, since we are dealing with the main page and not a content page like this one, we can’t simply inject a script into the page content (this is a limitation of Hugo).

Of course, we can work around that restriction by extending the Hugo theme to include it forcefully, so that’s what I did. By overriding the partial layout HTML file of the main page, I added the top-level script and some glue logic. Essentially, the glue logic creates a button and connects it to the audio engine start top-level function, and creates another one to stop the engine.

I had some performance worries about adding this audio engine to the website, but it seems well optimized and does not consume excessive computational resources. I also tested it on my phone and had favorable results. Also, the page can be visited without starting (or loading) the audio engine. This is because the audio context is only created upon button click.

Demo

Well, check my home page!

Just joking, since you came this far, I’ll allow you to play with two versions of the project:

  1. The version used in the webpage, with only two combined controls (modX,modY) and the separated master detune (+gain);

  2. An unlocked control version, with the eight modulation control sliders (+gain).

Since Faust’s default UI is ugly and its code seems bloated, I prepared a module that runs the Faust modules and exposes a more decent UI. Feel free to put some good headphones and play around. You can even play both engines at the same time! If you do, I recommend setting their mdt to -4 and 3 for their center frequencies to make a perfect fifth (-4 and 1 also sounds really cool).

Regular version


Unlocked version


Closing thoughts

This project took much longer than I expected. Not only did I have to develop two different projects until reaching something I liked, but I also had to deal with arguably rare language compiler issues that set me back for quite a while. In hindsight, I should have given up on the first project much before I did, and start the reevaluation phase earlier. Having to scratch everything and starting again was discouraging, but I am glad I pushed on because I’m very happy with the results. In the future, I might enhance the engine even more, or fine-tune and polish some modulation variables and range. But, for now, this will do.

Next post we are going to abandon audio, and we are going to explore how I created the render engine of my homepage. Thanks for reading!

TL;DR

Last time I failed to create a good audio engine for my homepage, but I tried again and succeeded. Instead of focusing on creating and implementing a song, I simplified the project and created a user-controllable ambient soundscape in Faust. After this, it was easy to port it to my homepage. You can play with two fun demos I created based on the algorithm used on the homepage in the demo section of this post.