Subscribe now

Building a Theremin with Web Audio [05.13.2016]

In today's episode, we'll build a Theremin with the Web Audio API. To do this, we'll start off by setting up the audio graph on the html side and we'll interact with our Elm app via ports. Let's get started.

Project

I've got a basic project set up already in the resources, and tagged with before_episode_010.2. It's just a basic counter application with our typical project structure in place.

Let's start out by modifying the html to build out our audio graph that we'll interact with, and just get some audio playing in the browser. We'll make an oscillator and a gain node, wire the output of the oscillator up to the input of the gain node, and wire the output of the gain node up to the speakers, which are our audio context's destination.

      // create web audio api context
      var audioCtx = new (window.AudioContext || window.webkitAudioContext)();
      // create Oscillator and gain node
      var oscillator = audioCtx.createOscillator();
      var gainNode = audioCtx.createGain();
      // connect oscillator to gain node to speakers
      oscillator.connect(gainNode);
      gainNode.connect(audioCtx.destination);
      // We'll set the initial oscillator frequency to 3000 Hertz.
      var initialFreq = 3000;
      // We'll set the initial volume to 0.001
      var initialVol = 0.001;
      // and we'll set some options for the oscillator
      oscillator.type = 'square';
      oscillator.frequency.value = initialFreq;
      oscillator.detune.value = 100;
      oscillator.start(0);
      // set options for the gain node
      gainNode.gain.value = initialVol;

      var app = Elm.Main.fullscreen();

Now we'll build the project, serve it, and open it up:

./build
servedir

(visit http://localhost:9091)

OK, I'm not sure if you can hear that annoying whining, but I sure can. We're successfully outputting audio via our audio graph, but at present it's not controllable from Elm. Let's modify the gain so that elm can control it, by subscribing to a port:

      app.ports.gain.subscribe(function(value){
        gainNode.gain.value = value;
      });

OK, so now if we had a port called gain we could send it values, and they would set the value of our gain node - ultimately, the volume. Let's hook our counter up to that.

First, our gain is a float, so we'll modify our Model to be a float as well for now:

type alias Model = Float
model = 0.001

We'll make a gain port:

-- We have to make our module a port module
port module Main exposing (..)
-- ...

-- Now we'll add the outbound port
port gain : Float -> Cmd msg

We'll want to send the value out that port each time we modify our model, so we'll use a Cmd to do that.

-- Also, when our Cmd succeeds we want it to send in a NoOp action, so we'll
-- add that to our union type
type Msg
    = Increment
    | Decrement
    | NoOp


update : Msg -> Model -> ( Model, Cmd Msg )
update msg model =
  -- We'll modify our increments to be a lot smaller since otherwise we'll hit
  -- max gain immediately.
    case msg of
        Increment ->
            ( model + 0.001, gain model )

        Decrement ->
            ( model - 0.001, gain model )

        NoOp ->
            ( model, Cmd.none )

Now if we build the app, we can control the volume with our counter controls.

Next, we'd like to wire up the frequency for the same sort of control. We'll start by modifying our javascript to assume we'll have an object sent with both a gainValue and a frequencyValue field on it. We're also calling this port audio rather than gain now:

      app.ports.audio.subscribe(function(model){
        gainNode.gain.value = model.gainValue;
        oscillator.frequency.value = model.frequencyValue;
      });

Next, we'll modify our Model to have those fields:

type alias Model =
    { gainValue : Float
    , frequencyValue : Float
    }


model =
    { gainValue = 0.001
    , frequencyValue = 3000
    }

Our gain port is now really an audio port:

update : Msg -> Model -> ( Model, Cmd Msg )
update msg model =
    case msg of
        Increment ->
            ( model + 0.001, audio model )

        Decrement ->
            ( model - 0.001, audio model )

        NoOp ->
            ( model, Cmd.none )


port audio : Model -> Cmd msg

We'll now have messages for incrementing and decrementing Gain and Frequency separately:

type Msg
    = IncrementGain
    | DecrementGain
    | IncrementFrequency
    | DecrementFrequency
    | NoOp

update : Msg -> Model -> ( Model, Cmd Msg )
update msg model =
    case msg of
        IncrementGain ->
            ( { model | gainValue = model.gainValue + 0.001 }, audio model )

        DecrementGain ->
            ( { model | gainValue = model.gainValue - 0.001 }, audio model )

        IncrementFrequency ->
            ( { model | frequencyValue = model.frequencyValue + 100 }, audio model )

        DecrementFrequency ->
            ( { model | frequencyValue = model.frequencyValue - 100 }, audio model )

        NoOp ->
            ( model, Cmd.none )

And we need the view to represent each of these values distinctly:

import Html exposing (Html, div, button, text, h1, section)

view : Model -> Html Msg
view model =
    div []
        [ section []
            [ h1 [] [ text "Gain" ]
            , button [ onClick DecrementGain ] [ text "-" ]
            , div [] [ text (toString model.gainValue) ]
            , button [ onClick IncrementGain ] [ text "+" ]
            ]
        , section []
            [ h1 [] [ text "Frequency" ]
            , button [ onClick DecrementFrequency ] [ text "-" ]
            , div [] [ text (toString model.frequencyValue) ]
            , button [ onClick IncrementFrequency ] [ text "+" ]
            ]
        ]

If we build it again, we can see that we now have control over both the frequency and the gain from Elm through this port. Now it's fairly trivial to add a Msg for the Mouse that allows us to control both values through mouse position.

Before we can do that, though, we want to know the window dimensions. This is because we want to use the mouse's percentage across the X and Y axes as our controls for gain and frequency.

First, we'll install the Window package:

elm package install elm-lang/window

Now, we can use Window.resizes as a subscription to get the window dimensions sent to us as they change. We can use Window.size to get the window size on app initialization, so we'll always have the size as long as our app is running.

First, we'll track it in our model:

import Window

type alias Model =
    { gainValue : Float
    , frequencyValue : Float
    , windowWidth : Int
    , windowHeight : Int
    }

model =
    { gainValue = 0.001
    , frequencyValue = 3000
    , windowWidth = 100
    , windowHeight = 100
    }

Next, we want to get all the window resizes in a subscription, and update our model:

type Msg
    -- ...
    | UpdateDimensions { width : Int, height : Int }


subscriptions : Model -> Sub Msg
subscriptions model =
    Window.resizes UpdateDimensions


update : Msg -> Model -> (Model, Cmd Msg)
update msg model =
    case msg of
      -- ...
        UpdateDimensions { width, height } ->
            ( { model | windowWidth = width, windowHeight = height }, audio model )
      -- ...

Now let's view the dimensions in the view:

view : Model -> Html Msg
view model =
    div []
        [ section []
            [ h1 [] [ text "Gain" ]
            , button [ onClick DecrementGain ] [ text "-" ]
            , div [] [ text (toString model.gainValue) ]
            , button [ onClick IncrementGain ] [ text "+" ]
            ]
        , section []
            [ h1 [] [ text "Frequency" ]
            , button [ onClick DecrementFrequency ] [ text "-" ]
            , div [] [ text (toString model.frequencyValue) ]
            , button [ onClick IncrementFrequency ] [ text "+" ]
            ]
        , section []
            [ h1 [] [ text "Dimensions" ]
            , div [] [ text (toString model.windowHeight) ]
            , div [] [ text (toString model.windowWidth) ]
            ]
        ]

If we build it and refresh it, you'll see that our window dimensions update just fine, but they don't get loaded correctly on initialization. We'll need to update init to ask for the initial window dimensions:

getInitialWindowSize : Cmd Msg
getInitialWindowSize =
    Task.perform (\_ -> NoOp) UpdateDimensions Window.size


main =
    App.program
        { init = ( model, getInitialWindowSize )
        , view = view
        , update = update
        , subscriptions = subscriptions
        }

OK, so if we build it and refresh, you can see that we get an immediate update for our window dimensions now. Now that that's in place, we can move on to the mouse bits, which are far less tricky.

First, we'll need to install Mouse:

elm package install elm-lang/mouse

Then we'll import it:

import Mouse

Next, we'll add a message to handle mouse updates:

type Msg
    -- ...
    | UpdateMouse ( Int, Int )


update : Msg -> Model -> (Model, Cmd Msg)
update msg model =
  let
    -- ...
  in
    case msg of
        -- ...
        UpdateMouse ( x, y ) ->
            let
                -- gain is the percentage you are across the screen, from left to right, mapped from 0 to 0.03
                newGain =
                    ((toFloat x) / (toFloat model.windowWidth)) * 0.03

                -- frequency is the percentage you are vertically down the screen, mapped from 0 to 6000
                newFrequency =
                    ((toFloat y) / (toFloat model.windowHeight)) * 6000.0

                newModel =
                    { model | frequencyValue = newFrequency, gainValue = newGain }
            in
                ( newModel, audio newModel )
        -- ...

So here we've set a range that we're mapping our mouse percentage across, and updating our model. Of course, we have to get these UpdateMouse messages into the event loop still. We'll just add a subscription:

subscriptions : Model -> Sub Msg
subscriptions model =
    Sub.batch
        [ Window.resizes UpdateDimensions
        , Mouse.moves (\{ x, y } -> UpdateMouse ( x, y ))
        ]

OK, with that all wired up, let's try it out. Build it and refresh the page...and we have a pretty fun little audiotoy here.

There's a tiny bug here in that we aren't actually sending the immediately updated value to the port, since we update it in our response but we send the original model across. In practice it doesn't matter too much, but the correct code would look as follows:

update : Msg -> Model -> ( Model, Cmd Msg )
update msg model =
    case msg of
        IncrementGain ->
            let
                newModel =
                    { model | gainValue = model.gainValue + 0.001 }
            in
                ( newModel, audio newModel )

        DecrementGain ->
            let
                newModel =
                    { model | gainValue = model.gainValue - 0.001 }
            in
                ( newModel, audio newModel )

        IncrementFrequency ->
            let
                newModel =
                    { model | frequencyValue = model.frequencyValue + 100 }
            in
                ( newModel, audio newModel )

        DecrementFrequency ->
            let
                newModel =
                    { model | frequencyValue = model.frequencyValue - 100 }
            in
                ( newModel, audio newModel )

        UpdateDimensions { width, height } ->
            let
                newModel =
                    { model | windowWidth = width, windowHeight = height }
            in
                ( newModel, audio newModel )

        UpdateMouse ( x, y ) ->
            let
                -- gain is the percentage you are across the screen, from left to right, mapped from 0 to 0.03
                newGain =
                    ((toFloat x) / (toFloat model.windowWidth)) * 0.03

                -- frequency is the percentage you are vertically down the screen, mapped from 0 to 6000
                newFrequency =
                    ((toFloat y) / (toFloat model.windowHeight)) * 6000.0

                newModel =
                    { model | frequencyValue = newFrequency, gainValue = newGain }
            in
                ( newModel, audio newModel )

        NoOp ->
            ( model, Cmd.none )

Summary

So that's a fairly quick overview of integrating Elm into the Web Audio API via ports. It's a pretty straightforward setup on the JavaScript side once you know how this Web Platform API works, and then it's just a matter of subscribing to a port and sending data across to control the objects. That's all for now. See you soon!

Resources