Introduction

Last post, I wrote about how I ended up creating this website. We went over the reason for its existence and the process from the selection of the framework, to the final foundation and how it is deployed. After selecting Hugo (Framework) and PaperMod (Base Theme), I was not completely satisfied with its aesthetics - especially the home page. To solve this, I set out to add interactive media to the home page: an audio engine and a render engine.

This post was originally supposed to explain the first failed attempt at interactive audio synthesis on my homepage and then the successful attempt. Yet, after writing the first system, the post itself was already very long. So, if you want to read about the second (and successful) system, you’ll have to wait for my next post.

So, without further ado, let me explain the problems I faced, how I overcame them and how I, after all of this, scrapped the whole thing. This adventure also includes the discovery of a sneaky bug in the compiler of a language called Faust.

First attempt: The naive plan

The first idea to enhance the homepage was to create a simple song to loop in the background of the homepage. This, however, is rather monotonous and would most likely become boring and static after a handful of loops. So, I quickly changed the plan to have a song that is synthesized in real-time that evolves, in some fashion. With this goal in mind, I now needed to compose a little song and translate it into code, in some way.

I love audio and have worked with it on a healthy amount of projects, so finding a way to synthesize audio through code was rather easy. One of the most recent projects I created in this field was my master’s thesis. Essentially, in that thesis, I created a new hardware description language for digital signal processing and audio synthesis, with an almost fully-compliant compiler to VHDL. This language/project was called RTFSS (Real-Time FPGA Sound Synthesis) and surely will have its own dedicated post somewhere in the future.

During my research of the state-of-the-art of my thesis, I investigated the current audio synthesis engines. The most popular (open) ones are SuperCollider, Pure Data, CSound, ChucK and Faust. I tested them all at the time, and while they all have advantages and disadvantages, over time I grew rather fond of Faust! My language ended up sharing some of the fundamental concepts with Faust, and I was eager to learn more about it. So, naturally, given the nature of this project, I selected Faust for this task.

Using Faust

When I started to think about this audio sub-project, my mind immediately latched to Faust, and for a good reason. As I recalled, Faust had a very diverse set of compilation targets, including WebAssembly (which can be natively run on a browser)! Not only that, Faust has decent documentation, has diverse standard libraries and a very good Web IDE. So, it was a very good fit and a no-brainer for me.

In brief, Faust (Functional AUdio STream) is a functional programming language for real-time signal processing. It allows the description of digital signal processing (DSP) algorithms. The language is based on a block-diagram algebra that allows the textual description of composition operations between functional blocks. After the compilation of a Faust DSP, (depending on the compiled platform) it can be run to process signal (or in our case, audio) in real-time.

A simple processor that generates a mono 440Hz sine wave oscillator can be implemented like this:

import("stdfaust.lib");
process = 440 : os.osc;

Which yields the following block diagram:

The process statement is equivalent to the main function in a C program. A constant with the value “440” is piped (composition operation :) into an instance of the osc block (from the os library). The program looks pretty simple but, under the hood, the osc block is doing all the hard work of generating the oscillator (creating a wave-table, phasor, …) from basic primitives.

Although the language seems easy, the kind of syntax it uses can get very tricky to write in and to read when a lot of parallelism or odd routing is required. I encourage the reader to research more into Faust, learn its basics and play a bit with it in the Web IDE. From this point on, I’ll share some more complex code snippets and will assume you know at least the basics of the language. Note that even considering the bad things I’ll write in the next sections, it is still a very cool language and I’m sure the issues I had with it might be resolved in the future. Also note that I’m a beginner in Faust, so I might be missing something important while making some of my complaints.

Troubles with Faust

After running a quick test to see how Faust worked with WebAssembly, I composed a demo for a little 16-ish second song loop in a DAW (Digital Audio Workstation). The task now was to convert this song into a Faust DSP. To convert the track, even before creating the instruments (audio voices), I had to create a step-sequencer to play/change the notes at the correct time and a way to extract the notes (and durations) from my DAW into that new step-sequencer.

The first task I tackled was creating a good step-sequencer. A step-sequencer is a device that outputs a sequence of values with a set pace. Often, when the sequence reaches the end, the sequencer loops over. My use of a step-sequencer in this application is to store the melody notes (and their timings). To represent the notes, I’ll use their MIDI representation. In short, this means that every note is represented by a 7-bit value, where the A4 note is represented by value 69 and higher notes have higher values.

I think that maybe Faust was not designed to perform this, and it was NOT very happy when I tried to implement it. The step-sequencing I initially wrote was based on the os.phasor function, which receives a table size and a frequency. This is intended to be used with wavetables. If all sequences in your step-sequencer are the same size, using this might be a decent hack since, again, you are pushing the need of implementing the harder stuff to the standard library. However, if the sequences are of different lengths, the sequencers get out of sync very easily, even if you adjust for the frequency.

Alright, I’ll implement it my way:

roller(roll) = int(ba.pulse_countup_loop(roll:(_-1,!),1)) : rdtable(roll);

This function receives an argument roll of type waveform (essentially a tuple of an array’s size and its values). It also receives a pulse that sets the beat of the sequencer. Pretty simple, pulse_countup_loop receives a pulse and counts up from zero up to the roll’s length minus one, and then loops back to zero. This value is fed to the rdtable primitive that reads a waveform at a given input position. DO NOT use this, it caused me a lot of headaches, and you’ll see now why.

Step-sequencer seems done, it’s time to test it. After testing it alone (seeing its outputs in the virtual oscilloscope), I quickly implemented the instrument voice signalpath and connected everything together. Everything worked fine… until the loop reset. When the loop resets, the program would output silence without any warning or error and stay like that indefinitely. The next (perhaps) hours were very frustrating. I thoroughly analyzed my code and tested it in different manners. Every piece worked alone (and looked well written), but their pairing would break. Not good, I started to investigate if any of the standard functions I used were faulty. First, I thought it was the os.square function (square wave oscillator similar to os.osc). After implementing my own, the problem was seemingly fixed. However, after adding a filter (like a fi.resonlp) to the chain, a RuntimeError would pop up in the JavaScript console complaining of an invalid conversion to an integer. Finally, some valuable information. Until here, I was debugging blindly, it almost felt like dealing with sneaky bugs when designing digital hardware.

This lead was key for discovering the problem. That conversion error can only be triggered in a handful of situations, and the one I was setting my money on was trying to convert the special values of a floating-point variable (NaN, plus or minus infinity, …) into an integer. Most of these special numbers are absorbent. This means that arithmetic operations with those numbers will yield a special number (for example 1+NaN=NaN). However, one unaffected operation, at least by some of these, is the numeric comparison. Using them I discovered where a special value would appear. This is how I found that the step-sequencer I created tends to output garbage during a very short period while looping over, perhaps just a sample’s worth of time. This value would then be used by rdtable to fetch an out-of-bounds memory position (outside the array). One might think that this would, in the best of cases, generate an audio glitch caused by the wrong note being played for a very small fraction of time. Oh no, something much more interesting happens 🙂. Somewhere in the instrument voice pipeline, the MIDI note gets converted to a frequency value (ba.midikey2hz). This is performed by the following equation:

$$ f=440\cdot 2^{(n-69)/12} $$

Get this, the most likely big value outputted by the rdtable, courtesy of the shoddy index value provided by the loop-over of the ba.pulse_countup_loop block, travels the signal path until it reaches the exponentiation of this formula. This big exponent causes the floating calculation to trip and output plus infinity. A quick test in the Javascript terminal of my installation of Firefox, shows that the max exponent (with base 2) that does not yield infinity is 1023 (too round of a number to be a coincidence, I think). The point is, the garbage value does not need to be too big to ruin the program, which further confirms my theory. Even more interesting, if we avoid using ba.pulse_countup_loop and implement our own, the issue remains. So, it does look like a deeper problem in the compiler’s implementation than a bug in the standard library.

This is very ugly. This should be fixed ASAP and in the meantime Faust developers should provide some kind of protection or warning on the matter. I plan on opening an issue in their GitHub to explain the problem. I also think I’ll explore the source code to find the culprit and fix it (perhaps a topic for a future post 🤔). I hacked my way around this issue by using the min and max primitives to clamp the output values of my loop counter. Using my off-brand version of the ba.pulse_countup_loop called safe_pulse_loop, the code looks like this:

safe_pulse_loop(n) = (_,_:+)~(_,n:%) : min(n-1) : max(0) : int;
roller(roll) = _ : safe_pulse_loop(roll:(_,!)) : rdtable(roll);

Which results in the following diagram (when using a 128 step-sequencer):

That was quite the adventure, but now it’s working smoothly.

Exporting MIDI to populate the step-sequencer

Ok, the worst part is over, I promise 🙏.

I needed a way to extract the notes from my DAW to place them into the Faust DSP. Doing it by hand would be torture, so I turned to a solution based on exported MIDI. Almost every DAW allows to export the project into MIDI, so we only need to dissect that MIDI file and extract what we want. Fortunately, there is a Python library that already deals with the MIDI protocol: Mido. Using this library, I created a basic Python script that reads a MIDI file and prints out the notes in Faust waveform:

import mido
import sys

mid = mido.MidiFile(sys.argv[1])
demul_factor=4 #1=whole note, 2=half note, 3=quarter note, 4=eighth note
comp_start=1

tpb=mid.ticks_per_beat
i = 0
for track in mid.tracks:
	note=[0]
	gate=[0]
	trig=[0]
	for msg in track:
		#Handle delay
		delay_time=int(msg.time*(1<<(demul_factor-1))/tpb+0.5)
		note+=note[-1:]*delay_time
		gate+=gate[-1:]*delay_time
		trig+=[0]*delay_time
		
		#Handle notes
		if msg.type=='note_on':
			note[-1]=msg.note
			gate[-1]=1
			trig[-1]=1
		elif msg.type=='note_off':
			note[-1]=msg.note
			gate[-1]=0
			trig[-1]=0

	if len(note)==1 and note[0]==0:
		continue

	#clip sequence
	note=note[:-1]
	gate=gate[:-1]
	trig=trig[:-1]

	#complete beat
	note+=[0]*(-len(note)%(1<<(demul_factor-1)))
	gate+=[0]*(-len(gate)%(1<<(demul_factor-1)))
	trig+=[0]*(-len(trig)%(1<<(demul_factor-1)))

	#rotate lists
	note=note[-comp_start:]+note[:-comp_start]
	gate=gate[-comp_start:]+gate[:-comp_start]
	trig=trig[-comp_start:]+trig[:-comp_start]

	print('//',len(note))
	print('roll',i,' = ','waveform{',','.join(map(str,note)),'};',sep='')
	print('trig',i,' = ','waveform{',','.join(map(str,trig)),'};',sep='')
	print('gate',i,' = ','waveform{',','.join(map(str,gate)),'};',sep='')
	print()
	i+=1

This script handles multiple channel MIDI files, and through the comp_start variable you can adjust the precision of the step-sequencer. Among other operations, the script adjusts the size of the sequences for them to match.

Putting it all together and giving up

Having populated the step-sequencer with the output of that Python script, it was now time to join everything. To facilitate the instrument creation part, I only used basic synthesizers in my audio demo. Nevertheless, I struggled to get the sound I wanted. The notes, timing and some of the timbre were correct, but the song sounded duller than in the original demo.

It was at this time that I realized it would sound bad in the webpage and that I skewed a lot from my vision of this project. After all this work and time spent, this was a major disappointment. Even by adding the interactivity I initially planned, it was not interesting enough (at least to my ears). Even worse, I wasn’t completely satisfied with the music demo I composed from the beginning, and now after listing to it on loop for a couple of hours while debugging and developing this solution, I was sick of it.

Second attempt…? Next post

Frustrated, I deemed this sub-project hopeless and unfixable. I now had three options in my hands:

Add it to the website anyways;
Dump this idea altogether;
Start from scratch and carve another path.

Since I rather have a silent page than an annoying one, the first option was immediately discarded. After investing all this time, the second option also seemed unreasonable because I believe I was capable of doing something better. So yeah, the third option it is 🙂. Not everything is lost, at least the Python MIDI script might be useful for someone else trying to attempt the same thing as me.

As I noted in the introduction, I was planning on using this post to talk about the new system. But, given the sheer size that this post already has, I might as well talk about this in a new post.

So yeah, next post we’ll talk about the current and much simpler solution I adapted. Thanks for reading.

Bonus Faust snippet

While doing this project, I had to create some cool little Faust snippets to perform miscellaneous tasks. This is one example:

to_pulse = 1-_: ba.countdown(2) : (_,1) : ==;
rand_ofs(trig) = ((_,no.noise : select2(trig : to_pulse))~ _);

Which creates the following diagrams:

This circuit acts like a Sample and Hold circuit over a white noise source. Essentially what this means is that for every pulse in the trig input of the rand_ofs function, it outputs a random value between $-1$ and $1$. I used this as a way of user interaction to shift the center frequencies of the oscillators.

TL;DR

I wanted to add interactive audio to my homepage. To create it, I decided to use the Faust language. Faust allows the creation of real-time digital signal processing modules and can be added to webpages easily. Faust was not made to play songs, but I made it do it anyways. It was not happy with that and threw me into hours of debugging and hack creation. After fixing everything and configuring the music in it, I was disappointed with the result. After all that work, I decided to scrap everything and start over. Next post I will write about the good solution.

Introduction#

First attempt: The naive plan#

Using Faust#

Troubles with Faust#

Exporting MIDI to populate the step-sequencer#

Putting it all together and giving up#

Second attempt…? Next post#

Bonus Faust snippet#

TL;DR#