inaka

Latest blog entries

/
SpellingCI: No more spelling mistakes in your markdown flies!

Feb 14 2017 : Felipe Ripoll

/
Fast reverse geocoding with offline-geocoder

Do you need a blazing fast reverse geocoder? Enter offline-geocoder!

Jan 18 2017 : Roberto Romero

/
Using Jayme to connect to the new MongooseIM REST services

MongooseIM has RESTful services!! Here I show how you can use them in an iOS application.

Dec 13 2016 : Sergio Abraham

/
20 Questions, or Maybe a Few More

20 Questions, or Maybe a Few More

Nov 16 2016 : Stephanie Goldner

/
The Power of Meeting People

Because conferences and meetups are not just about the technical stuff.

Nov 01 2016 : Pablo Villar

/
Finding the right partner for your app build

Sharing some light on how it is to partner with us.

Oct 27 2016 : Inaka

/
Just Play my Sound

How to easily play a sound in Android

Oct 25 2016 : Giaquinta Emiliano

/
Opening our Guidelines to the World

We're publishing our work guidelines for the world to see.

Oct 13 2016 : Brujo Benavides

/
Using NIFs: the easy way

Using niffy to simplify working with NIFs on Erlang

Oct 05 2016 : Hernan Rivas Acosta

/
Function Naming In Swift 3

How to write clear function signatures, yet expressive, while following Swift 3 API design guidelines.

Sep 16 2016 : Pablo Villar

/
Jenkins automated tests for Rails

How to automatically trigger rails tests with a Jenkins job

Sep 14 2016 : Demian Sciessere

/
Erlang REST Server Stack

A description of our usual stack for building REST servers in Erlang

Sep 06 2016 : Brujo Benavides

/
Replacing JSON when talking to Erlang

Using Erlang's External Term Format

Aug 17 2016 : Hernan Rivas Acosta

/
Gadget + Lewis = Android Lint CI

Integrating our Android linter with Github's pull requests

Aug 04 2016 : Fernando Ramirez and Euen Lopez

/
Passwordless login with phoenix

Introducing how to implement passwordless login with phoenix framework

Jul 27 2016 : Thiago Borges

/
Beam Olympics

Our newest game to test your Beam Skills

Jul 14 2016 : Brujo Benavides

/
Otec

Three Open Source Projects, one App

Jun 28 2016 : Andrés Gerace

/
CredoCI

Running credo checks for elixir code on your github pull requests

Jun 16 2016 : Alejandro Mataloni

/
Thoughts on rebar3

Thoughts on rebar3

Jun 08 2016 : Hernán Rivas Acosta

/
7 Heuristics for Development

What we've learned from Hernán Wilkinson at our Tech Day

May 31 2016 : Brujo Benavides

/
See all Inaka's blog posts >>

/
Using NIFs: the easy way

A photo of Hernan Rivas Acosta wrote this on October 05, 2016 under c, erlang, nif, performance .

The joys of native code

Sometimes, just using Erlang is not enough. Perhaps we really need extra performance in some places (jiffy is a good example), or we need to take advantage of libraries written in other languages (like crypto does). For those cases, Erlang offers us two tools, Ports and NIFs. Now, both choices have their pros and cons, but today we are focusing on NIFs.

So what's a NIF? In a nutshell, it's a "natively implemented function". As we all know, Erlang compiles to bitcode, which can then be run by the BEAM (Bogdan/Björn's Erlang Abstract Machine), Erlang's equivalent of Java's JVM. But, of course, code in a VM can only run so fast; which is exactly why we would want to use native functions. Sometimes we just need to sacrifice the advantages and guarantees of Erlang code for that extra performance (or because we are not crazy enough to rewrite OpenSSH).

And what guarantees are we leaving behind by choosing to implement some of our code as a NIF? Well, many of them. First of all, concurrency and responsiveness: The Erlang schedulers are unable to break down C code in parts, meaning executing any NIF for more than a millisecond can make the whole VM (or, to be more precise, an entire scheduler) unresponsive. And yes, Erlang 19 introduced some improvements here, but it's still not ideal. Secondly, safety: You can actually crash the entire VM with a poorly implemented NIF. And I don't mean just crashing a process, we are Erlangers and don't need to worry about that, I'm talking about bringing down the whole VM with something as silly as a segmentation fault. And finally, simplicity: Not only will our codebase be further divided into more files and our compilation process will get way more complicated than it has to, but we will also have something way harder to debug in our hands.

Writing a NIF, the hard way

And writing a NIF is on itself a complicated task, you can't expect to just plug in some C code in Erlang and have it work. You need to know what's in the erl_nif.h header that Erlang provides and how to compile and then link it from Erlang. Not really a trivial thing, and we are not even considering the risks associated with doing it wrong.

Let's see an example, a truly small C file that defines a function that returns the square of a number:

#include "erl_nif.h"

static ERL_NIF_TERM square_nif(
  ErlNifEnv* env,
  int argc,
  const ERL_NIF_TERM argv[])
{
  int a;
  if (!enif_get_int(env, argv[0], &a))
    return enif_make_badarg(env);
  return enif_make_int(env, a * a);
}

static ErlNifFunc niffuncs[] =
  square;

ERL_NIF_INIT(example, niffuncs,
             NULL, NULL, NULL,
             NULL)

Yeah, it's not obscure, but it's complicated. Notice how we have to retrieve the arguments using the enif_get_* functions, how we can't just return an int, and how we need to add some metadata for Erlang to understand it.

And that's not even considering what we need to do on the erlang side: We have to load the NIF and implement the mock functions that will be replaced at runtime with the C code.

-module(example).
-export([square/1]).
-on_load(init/0).

init() ->
  ok = erlang:load_nif("./example_nif", 0).

square(_X) ->
  throw(nif_not_loaded).

And of course, don't forget about compiling the C code with the right flags before using it.

So, summarizing:

  • Hard to write.
  • Hard to compile and link.
  • Divides the codebase into more separated files.
  • Can easily bring down the VM if you make a mistake.

So, it stands to reason that before writing a NIF, you need to ask yourself if it's worth doing it. You should really weight in the pros and cons before taking the decision.

But what if we could make the first 3 problems disappear? That might tip the balance towards using them more often, right? What if by compiling the following module we can just make it work?

-module(example).
-niffy([nat_log/1]).
-export([nat_log/1]).

nat_log(_A) ->
  "#include <math.h>
   double nat_log(double a)
   {
     // this will return the natural log
     double logA = log(a);
     // return logB;
     return logA;
   }".

You gotta admit, that would be pretty awesome.

Well, that's exactly what niffy allows you to do! Just inline your C code and let niffy do the rest! No configuration required*!

* (note: some configuration required)

So, how does it work?

Short answer:

Long answer:

Niffy is a parse transform. You add the -niffy attribute to specify which functions you want "niffified" (I'm working on the terminology) and then add it as a parse_transform in your erl_opts just like you do with lager. Niffy will then find the functions you included on the -niffy attribute, take the C code from them, replace their code with throws to nif_not_loaded, build and compile the C file, add an on_load function to the module (if none was present) and then add the nif loading code to it. Also, you don't even need to write C in the "NIF style"! Just write regular C code and it will work, niffy will add all the extra boilerplate and enif_get_*, and enif_make_* calls for you.

But I need feature X and it's not available

Well, niffy does a bit more stuff than what the above example implies. To achieve this, instead of a list of functions, you can have a map with multiple properties where you can set the compilation flags, specify if a function is cpu or io bound to support dirty schedulers (available since Erlang 19) and add some headers you want included in the generated file. Also, niffy looks for some properties in your erl_opts. For example, you can set some compile flags you might need for all your files (here I added some MacOS specific flags and my include path):

{niffy_options, [% The default compiler is gcc
                 {compiler, "clang"},
                 {flags,
                  ["-Werror",
                   "-flat_namespace",
                   "-undefined suppress",
                   "-I/usr/lib/erlang/
                    erts-5.8.1/include"]},
                 % Where the generated c files
                 {c_dir, "_generated/c_src"},
                 {strict_types, false}]}

How are you handling atoms, strings and binaries?

Well, that's a long answer, and I'm still working on it so I would suggest going to the example project, and in particular this file, for reference. In short, you specify const char* and let niffy take care of the malloc, the strcpy and freeing the memory, but there are some other things to keep in mind, like how to differenciate between strings and binaries.

But how could I return a tuple or a more complex erlang term?

Oh, in that case, you want to handle the return type yourself. Simply set ERL_NIF_TERM as the return type of your function and niffy won't modify your return statements. Just keep in mind that the Erlang environment variable will be called env.

And there is also an analogous mechanism for handling the parameters yourself. Just set (ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) as your function parameters; niffy will then assume you know what you are doing and it will not generate code to parse the parameters.

But I already read the above and I still need feature X and it's not available

Ok, then one last stop. You can see an even longer example of usage here, which should cover every single feature niffy has (it also serves as the test for the library).

But, in the case you still can't find what you need, please open an issue or feel free to send a PR! I'll be the first to admit the parse transform is kinda messy (you can't call functions in other modules from a PT, so the god-module look is inevitable), but it should be pretty straight forward and easy to understand/modify.

Is this actually production ready?

Actually yes, specially for numeric types, just make sure to check the generated C code by hand when you make a release and add plenty of test cases for the niffified functions. At the very least, be as careful when using niffy as you would be when using NIFs the hard way.