Anthony Mattox Interaction Design + Digital Art

Although Processing does not have the ability to process audio on it’s own there are a few libraries which can be used to add such features. I’ve been experimenting with the ESS library but you can find others from the processing libraries page. Gathering Data from live or recorded sound can be a powerful tool for artists and designers in creating interactive applications, audio visualizers, music videos, or anything else that could involve an interaction with sound. Getting the data is fairly straight forward and the script returns an array of data updated each frame. This demonstration with display the data very simply, however, the numbers could be applied to any attribute in any system to produce different effects.

Here are the steps to creating a simple script to get the spectrum data from a microphone into a processing sketch. Using pre-recorded sound is similar. Once the data is in the sketch it can be manipulated and used to generate graphics or effect other aspects of a program including complex forms in 3d space using the OpenGL or P3D libraries.

The first thing to do is import the ESS library into your sketch, making all it’s classes and functions available to you. import krister.Ess.*; The import command needs to be at the beginning of your script and the library needs to be in the library folder of Processing. Any other libraries you want to use can be put in the same place.

Next we need to create our FFT object. FFT myfft; The first “FFT” denotes the class and “myfft” is a variable for our new instance and can be whatever you like. An FFT (Fast Fourier Transform) is a mathematical algorithm which breaks down a sequence of numbers into a series of frequencies. The output will be the peak amplitude for each frequency band. This will allow us to create a visual representation of the audio spectrum.

Defining a few more variables brings us here:

import krister.Ess.*;

FFT myfft;
AudioInput myinput;
int bufferSize=512;

“myinput” is an instance of the input class which will get audio from a microphone. Again it can be whatever you want, but remember that processing is a Case Sensitive language. “bufferSize” is a variable we will use later to to tell our FFT how many frequency bands to process.

Now That we have created our objects we can move on two our setup function. The setup function is run once when the program is run. Here we define all the information about our programs environment and the initial parameters for objects.

void setup() {
  size(532,400);
  frameRate(30);
  background(255);
  noStroke();
  fill(0);

  Ess.start(this);
  myinput=new AudioInput(bufferSize);
  myfft=new FFT(bufferSize*2);
  myinput.start();

  myfft.damp(.3);
  myfft.equalizer(true);
  myfft.limits(.005,.05);
}

The first chunk should be simple enough if you are familiar with processing. Here we declare the size of the program, the frame rate, and display colors. In second we define our audio objects using the “bufferSize” variable we defined earlier and tell ESS and our input to start running. Notice how all these actions are run just once at the startup of the program, although some, like the stroke and fill colors could be redefined as the program is running. The last chunk effects how the FFT looks but is not necessary. Limits will effect the scale of the output, equalizer smooths out the spectrum and damping effects how quickly the output of one band will move making it look smoother or rougher.

void draw() {
  background(255);
  for (int i=0; i<bufferSize;i++) {
    rect(i+10,390,1,myfft.spectrum[i]*-400);
  }
}

To make the program run we need to define our “draw” function. This will run continuously while the program is open. The background is drawn each frame so what was drawn previously is erased. The for each point in the spectrum (“i” represents the index of the value) we draw a rectangle at point (i+10,390) with width 1 pixel and height the spectrum value. The value is scaled negative to draw upwards.

import krister.Ess.*;

FFT myfft;
AudioInput myinput;
int bufferSize=512;

void setup() {
  size(532,400);
  frameRate(30);
  background(255);
  noStroke();
  fill(0);

  Ess.start(this);
  myinput=new AudioInput(bufferSize);
  myfft=new FFT(bufferSize*2);
  myinput.start();

  myfft.damp(.3);
  myfft.equalizer(true);
  myfft.limits(.005,.05);
}

void draw() {
  background(255);
  for (int i=0; i<bufferSize;i++) {
    rect(i+10,390,1,myfft.spectrum[i]*-400);
  }
}

public void audioInputData(AudioInput theInput) {
  myfft.getSpectrum(myinput);
}

Finall we define one more function to link the input to the FFT and we have our complete and functional script. This script renders the audio spectrum very simply but any graphics can be added within the for loop using myfft.spectrum[i]to adjust any parameter. If you have any syntax questions the Processing reference is a fantastic resource. Or leave a comment if you have more questions.