[260] NIFs

Building a Native Implemented Function and corresponding Elixir module, writing the Makefile, and integrating it into mix.

Subscribe now

NIFs [08.10.2016]

NIF stands for Native Implemented Function. These are functions that look like normal erlang functions, but are implemented in native code. We can build one, so let's do that!

Project

We'll start a new project:

mix new nif_playground
cd nif_playground

Basics

Now we want to write a basic C program. Let's just do that first. We'll make a src dir that our C files will live in.

mkdir src

Before we move on to the C program, let's write a quick test for our Elixir module.

vim test/hello_test.exs

We'll add an ints function that calls a C function that just returns an integer:

defmodule HelloTest do
  use ExUnit.Case

  test "ints" do
    assert 1 = Hello.ints()
  end
end

Now let's write the C program:

vim src/hello.c

We're going to start off with a simple function that takes zero arguments and returns an integer.

/* We'll include the erl_nif headers provided by erlang */
#include "erl_nif.h"

/* This function returns a term.

   The first argument to the function is an ErlNifEnv, which is an environment
   that can host erlang terms.  I won't go into these at all but I've linked to
   the docs in the resources section.

   The second argument is an integer with the number of arguments
*/
static ERL_NIF_TERM
ints(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[]) {
  /* We can just use enif_make_int to turn an int into an erlang nif term */
  return enif_make_int(env, 1);
}

/* Let's define the array of ErlNifFunc beforehand.
   This is just an array of 4-element structs that have the name of the function
   in erlang, the arity of the function, the name of the corresponding C
   function, and a flag specifying whether it's a dirty nif or not.
*/
static ErlNifFunc
nif_funcs[] = {
  /* {erl_function_name, erl_function_arity, c_function, flags} */
  {"ints", 0, ints, 0} /* flags 0 indicates a non-dirty NIF */
};

/* Finally, we call ERL_NIF_INIT, which is a macro, with our Erlang module name,
   the list of function mappings, and 4 pointers to functions: load, reload, upgrade,
   and unload.  The docs specify these functions in detail, but our examples are
   simple so we can ignore them for now.
*/
*/
ERL_NIF_INIT(Elixir.Hello, nif_funcs, NULL, NULL, NULL, NULL)

Next, we'll compile it:

mkdir priv # Need the dir it ends up in
# This works on OS X
cc -g -O3 -ansi -pedantic -Wall -Wextra -Wno-unused-parameter -undefined dynamic_lookup -dynamiclib -I/Users/jadams/.asdf/installs/erlang/19.0/lib/erlang/erts-8.0/include -Isrc -shared  -o priv/hello.so src/hello.c
# This works on my linux box
cc -g -O3 -ansi -pedantic -Wall -Wextra -Wno-unused-parameter -fPIC -I/home/jadams/.asdf/installs/erlang/19.0/lib/erlang/erts-8.0/include -Isrc -shared  -o priv/hello.so src/hello.c

OK, we'll deal with a Makefile later, but for now we can just move on to the Elixir module.

vim lib/hello.ex
# Our nif references Elixir.Hello, which implies our module is named Hello.
defmodule Hello do
  # The @on_load module attribute specifies a function to run when the VM loads
  # this module.  We'll use that to load our nif.
  @on_load :init

  def init do
    # Loading the nif just implies calling `:erlang.load_nif` on the path to the
    # library.  This will load and link the dynamic library.  We have to turn
    # the string path we get from `Application.app_dir` into a charlist since we're
    # passing it to an Erlang function.
    path = Application.app_dir(:nif_playground, "priv/hello") |> String.to_char_list
    :ok = :erlang.load_nif(path, 0)
  end

  # Then we define a version of our nif function that will exit with
  # `:nif_not_loaded`.  This function will be replaced by the specified C
  # function when the nif is loaded, so if we hadn't loaded the nif this
  # function call would exit.
  def ints() do
    exit(:nif_not_loaded)
  end
end

Let's see if that works:

mix test

And it does. OK, so this is the basics of NIFs. Let's move on to a Makefile and some mix integration.

Makefile

We don't want to type that cc command constantly, so of course we want to specify a Makefile. I've cobbled one together from a few different projects and my own tweaks that I'm fairly happy with so far. I'll paste it in and we can talk about it:

CFLAGS = -g -O3 -ansi -pedantic -Wall -Wextra -Wno-unused-parameter -undefined dynamic_lookup -dynamiclib

ERLANG_PATH = $(shell erl -eval 'io:format("~s", [lists:concat([code:root_dir(), "/erts-", erlang:system_info(version), "/include"])])' -s init stop -noshell)
CFLAGS += -I$(ERLANG_PATH)
CFLAGS += -Isrc


.PHONY: all clean

all: priv/hello.so

priv/hello.so: src/hello.c
	$(CC) $(CFLAGS) -shared $(LDFLAGS) -o $@ src/hello.c

clean:
	$(RM) -r priv/hello.so*

The dynamic_lookup bit was necessary to successfully build on OS X. Other than that, these are just normal CFLAGS. Next, we extract the ERLANG_PATH by shelling out to erl and evaluating some code that finds our erlang installation's root dir, appends the erts version directory, and appends the include directory. We then add that to our include paths, as well as our src directory. We'll add a .PHONY for all and clean, which just tells make that those are always tasks, and to ignore files ith the same name. Then our all task depends on priv/hello.so. Our priv/hello.so depends on src/hello.c and produces the corresponding .so file with our C compiler. Finally, clean knows how to remove the files this compilation generates.

So now we can execute these tasks to test it out:

make clean
make

Great. But we'd really like the necessary NIF to be compiled alongside our project, since we have a module that depends on it.

Mix integration

Mix has support for make as a compiler. Let's modify our mix.exs to add that compiler and add two mix tasks for make:

defmodule NifPlayground.Mixfile do
  use Mix.Project

  def project do
    [app: :nif_playground,
     version: "0.1.0",
     elixir: "~> 1.3",
     build_embedded: Mix.env == :prod,
     start_permanent: Mix.env == :prod,
     compilers: [:make, :elixir, :app], # <-- Add the make compiler
     aliases: aliases(),
     deps: deps()]
  end

  defp aliases do
    # Make `mix clean` also run `make clean`
    [clean: ["clean.make", "clean"]]
  end

  # ...
end

# The make compiler is expected to live at `Mix.Tasks.Compile.Make`, to be run
# as `mix compile.make`
defmodule Mix.Tasks.Compile.Make do
  @shortdoc "Compiles helper in src"

  def run(_) do
    # We just run `make`
    {result, _error_code} = System.cmd("make", [], stderr_to_stdout: true)
    Mix.shell.info result

    :ok
  end
end

# We also add a cleaner for `make`, will be available as `mix clean.make`.  This
# is what we added to our alias for clean.
defmodule Mix.Tasks.Clean.Make do
  @shortdoc "Cleans helper in src"

  def run(_) do
    {result, _error_code} = System.cmd("make", ['clean'], stderr_to_stdout: true)
    Mix.shell.info result

    :ok
  end
end

Alright, now when we compile our project it makes the requisite library automatically, and cleans it as well as expected. Let's clean it and just run our tests:

mix clean
mix test

Slightly more complex example

OK, so our example right now is a zero-arity function. Let's take in 2 integer arguments and add them together, returning an integer. We'll add a test:

vim test/hello_test.exs
defmodule HelloTest do
  use ExUnit.Case

  # ...
  test "add" do
    assert 3 = Hello.add(1, 2)
  end
end

OK, if we run it there's no function. Let's define the function in Elixir and run it again:

vim lib/hello.ex
defmodule Hello do
  # ...
  def add(a, b) do
    exit(:nif_not_loaded)
  end
end

If we run it now, of course, it exits with nif_not_loaded. Now we'll define it in our C file:

vim src/hello.c
/* ... */
static ERL_NIF_TERM
add(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[]) {
  int a, b;

  /* We'll make sure we can get an int from the expected int arguments, and if we
     can't we'll return a badarg.
  */
  if (!enif_get_int(env, argv[0], &a)) {
    return enif_make_badarg(env);
  }
  if (!enif_get_int(env, argv[1], &b)) {
    return enif_make_badarg(env);
  }
  return enif_make_int(env, a+b);
}

/* And we'll add it to our nif_funcs */
static ErlNifFunc
nif_funcs[] = {
  /* {erl_function_name, erl_function_arity, c_function, flags} */
  {"ints", 0, ints, 0}, /* flags 0 indicates a non-dirty NIF */
  { "add", 2,  add, 0}
};

ERL_NIF_INIT(Elixir.Hello, nif_funcs, NULL, NULL, NULL, NULL)

OK, run clean and then run the tests:

mix clean
mix test

And they pass.

Caveats

I can't really justify an episode teaching how to create NIFs without specifically calling out that there are explicit requirements that your NIF needs to pay attention to. If your NIF is not guaranteed to complete in 1ms or less, you will need to flag it as dirty. There are a lot of additional things you should concern yourself with, so if you intend to create a NIF do yourself a favor and read the documentation on them in detail. It's not very long, and reading the Erlang docs is always a good use of time.

Summary

In today's episode we looked at interfacing our Elixir code with C programs as NIFs. We also saw how to build a quick Makefile for it, and tie it all into our mix tasks to give us first-class support for building and cleaning them along with the rest of our code. If you build a NIF after watching this episode, I'd love to hear some details, so let me know. See you soon!

Resources