inaka

Latest blog entries

/
The Art of Writing a Blogpost

The Art of Writing a Blogpost

Mar 09 2017 : Matias Vera

/
SpellingCI: No more spelling mistakes in your markdown flies!

Feb 14 2017 : Felipe Ripoll

/
Fast reverse geocoding with offline-geocoder

Do you need a blazing fast reverse geocoder? Enter offline-geocoder!

Jan 18 2017 : Roberto Romero

/
Using Jayme to connect to the new MongooseIM REST services

MongooseIM has RESTful services!! Here I show how you can use them in an iOS application.

Dec 13 2016 : Sergio Abraham

/
20 Questions, or Maybe a Few More

20 Questions, or Maybe a Few More

Nov 16 2016 : Stephanie Goldner

/
The Power of Meeting People

Because conferences and meetups are not just about the technical stuff.

Nov 01 2016 : Pablo Villar

/
Finding the right partner for your app build

Sharing some light on how it is to partner with us.

Oct 27 2016 : Inaka

/
Just Play my Sound

How to easily play a sound in Android

Oct 25 2016 : Giaquinta Emiliano

/
Opening our Guidelines to the World

We're publishing our work guidelines for the world to see.

Oct 13 2016 : Brujo Benavides

/
Using NIFs: the easy way

Using niffy to simplify working with NIFs on Erlang

Oct 05 2016 : Hernan Rivas Acosta

/
Function Naming In Swift 3

How to write clear function signatures, yet expressive, while following Swift 3 API design guidelines.

Sep 16 2016 : Pablo Villar

/
Jenkins automated tests for Rails

How to automatically trigger rails tests with a Jenkins job

Sep 14 2016 : Demian Sciessere

/
Erlang REST Server Stack

A description of our usual stack for building REST servers in Erlang

Sep 06 2016 : Brujo Benavides

/
Replacing JSON when talking to Erlang

Using Erlang's External Term Format

Aug 17 2016 : Hernan Rivas Acosta

/
Gadget + Lewis = Android Lint CI

Integrating our Android linter with Github's pull requests

Aug 04 2016 : Fernando Ramirez and Euen Lopez

/
Passwordless login with phoenix

Introducing how to implement passwordless login with phoenix framework

Jul 27 2016 : Thiago Borges

/
Beam Olympics

Our newest game to test your Beam Skills

Jul 14 2016 : Brujo Benavides

/
Otec

Three Open Source Projects, one App

Jun 28 2016 : Andrés Gerace

/
CredoCI

Running credo checks for elixir code on your github pull requests

Jun 16 2016 : Alejandro Mataloni

/
Thoughts on rebar3

Thoughts on rebar3

Jun 08 2016 : Hernán Rivas Acosta

/
See all Inaka's blog posts >>

/
Canillita (Your First Erlang Server) - V2

Harenson Henao wrote this on January 04, 2016 under cowboy, cowboy-swagger, cowboy-trails, dev, erlang, lasse, restful, sse, sumo_db, sumo_rest, swagger, trails .

Introduction

Let's start this new year with a big bang!

Do you remember when you learned how to create your first web server with Erlang? Well that's so 2013! The libraries we used back then had been updated, new libraries had been created; even Erlang itself had been improved... a lot. So, let's do it again! In this post I will show you how to create a very basic, but yet useful RESTful server using some widely known Erlang libraries. It will not be enough to teach you how to program in Erlang and I won't dive into the core aspects of the language itself. For that you can always learn you some erlang for great good! ;). On the other hand, if you're an experienced Erlang programmer and you need a RESTful server with SSE capabilities for your application, you may use this example as a starting point to build your system.

Scope

What's in this article

These are the components, protocols and features that I'll use and show in this article. Each one comes with a link where you can find more information about them.

  • SSE: a technology from where a browser gets automatic updates from a server via HTTP connection
  • cowboy: the ultimate server for the modern Web, written in Erlang with support for Websocket, SPDY and more
  • cowboy-trails: Some improvements over Cowboy Routes
  • cowboy-swagger: Swagger integration for Cowboy (built on trails)
  • sumo_db: a very simple persistance layer capable of interacting with different db's, while offering a consistent api to your code
  • sumo_rest: Generic Cowboy handlers to work with Sumo
  • lasse: Server-Sent Event (SSE) handler for Cowboy
  • mixer: Mix in functions from other modules

In 2013 CanillitaV1 was still using SSE, Cowboy and sumo_db. In this new version we will be introducing some new libraries: cowboy-trails, cowboy-swagger, sumo_rest, lasse and mixer. All will be explained later in this blog post.

What's not in this article

You will not find the following stuff here:

  • HTTP authentication, QueryString and many other things -- you can easily add these to your RESTful server using Cowboy
  • Complex persistency operations -- with sumo_db you can do much more than what I did here
  • Complex swagger structures -- for those you have to check both swagger and cowboy-swagger docs.
  • Tests -- Although the project in github was built with TDD, I will not go into detail on how that works (We have multiple posts about it on our blog already)

The Application

Canillita is a very basic pubsub server.

For this article I am using the same idea behind CanillitaV1 but I will be adding more routes and I will use more open-source libraries we have created so far. You can see the source code here.

Since this version of Canillita was built with cowboy-swagger, full interactive documentation is provided with the project itself and you can see it after the server is started. To start the server you have to:

# Create release
$ make rel
# Run server
$ _rel/canillita/bin/canillita console

After that go to http://localhost:4892/api-docs and you will see the API documentation.

canillitaV2 is implemented as a simple RESTful server with 3 URLs that provides eight (8) endpoints:

POST /newspapers

This endpoint accepts a JSON object with the parameters name and description and returns:

  • 201 Created if everything went okay and the object was created
  • 400 Bad Request if the JSON object is malformed
  • 409 Conflict if there is another newspaper with the same name

Example

curl -vX POST \
"http://localhost:4892/newspapers" \
--header "Content-Type: application/json" \
--header "Accept: application/json" \
-d "{\"name\": \"newspaper1\",
     \"description\": \"description1\"}"
> POST /newspapers HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:4892
> Content-Type: application/json
> Accept: application/json
> Content-Length: 53
>
* upload completely sent off: 53 out of 53 bytes
< HTTP/1.1 201 Created
* Server Cowboy is not blacklisted
< server: Cowboy
< date: Thu, 17 Dec 2015 18:56:04 GMT
< content-length: 122
< content-type: application/json
< location: /newspapersnewspaper1
<

GET /newspapers

This endpoint returns a list of JSON objects with all the newspapers created so far.

Example

curl -vX GET \
"http://localhost:4892/newspapers" \
--header "Accept: application/json"
> GET /newspapers HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:4892
> Accept: application/json
>
< HTTP/1.1 200 OK
* Server Cowboy is not blacklisted
< server: Cowboy
< date: Thu, 17 Dec 2015 19:18:05 GMT
< content-length: 124
< content-type: application/json
<
[{"updated_at":"2015-12-17T18:56:05Z",
  "name":"newspaper1",
  "description":"description1",
  "created_at":"2015-12-17T18:56:05Z"}]

PUT /newspapers/:id

This endpoint accepts a JSON object with the parameter description, updates the resource with the given :id or creates a new one if it doesn't exists and returns:

  • 200 OK if a newspaper with that id exists and it was successfully updated
  • 201 Created if a newspaper was created

Note: the newspaper's name acts as the id for this endpoint.

Example

curl -vX PUT \
"http://localhost:4892/newspapers/newspaper1" \
--header "Content-Type: application/json" \
--header "Accept: application/json" \
-d "{\"description\":
     \"description modified\"}"
> PUT /newspapers/newspaper1 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:4892
> Content-Type: application/json
> Accept: application/json
> Content-Length: 39
>
* upload completely sent off: 39 out of 39 bytes
< HTTP/1.1 200 OK
* Server Cowboy is not blacklisted
< server: Cowboy
< date: Thu, 17 Dec 2015 19:38:08 GMT
< content-length: 130
< content-type: application/json
<

GET /newspapers/:id

This endpoint returns:

  • 200 OK and the newspaper identified by the given id if it exists
  • 404 Not Found if it doesn't exists

Example

curl -vX GET \
"http://localhost:4892/newspapers/newspaper1" \
--header "Accept: application/json"
> GET /newspapers/newspaper1 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:4892
> Accept: application/json
>
< HTTP/1.1 200 OK
* Server Cowboy is not blacklisted
< server: Cowboy
< date: Thu, 17 Dec 2015 19:45:01 GMT
< content-length: 130
< content-type: application/json
<
{"updated_at":"2015-12-17T19:38:08Z",
 "name":"newspaper1",
 "description":"description modified",
 "created_at":"2015-12-17T18:56:05Z"}

DELETE /newspapers/:id

This endopoint deletes from the db the newspaper identified with the given id and returns 204 No Content.

Example

curl -vX DELETE \
"http://localhost:4892/newspapers/newspaper1" \
--header "Accept: application/json"
> DELETE /newspapers/newspaper1 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:4892
> Accept: application/json
>
< HTTP/1.1 204 No Content
* Server Cowboy is not blacklisted
< server: Cowboy
< date: Thu, 17 Dec 2015 19:48:10 GMT
< content-length: 0
< content-type: application/json
<

POST /newspapers/:name/news

This endpoint accepts a JSON object with the parameters title and body set and returns:

  • 201 Created if everything went ok
  • 400 Bad Request if malformed JSON is provided
  • 404 Not Found if there is not a newspaper with the given :name

Example

curl -vX POST \
"http://localhost:4892/newspapers/newspaper1/news" \
--header "Content-Type: application/json" \
--header "Accept: application/json" \
-d "{\"title\":\"title1\", \"body\":\"body1\"}"
> POST /newspapers/newspaper1/news HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:4892
> Content-Type: application/json
> Accept: application/json
> Content-Length: 36
>
* upload completely sent off: 36 out of 36 bytes
< HTTP/1.1 201 Created
* Server Cowboy is not blacklisted
< server: Cowboy
< date: Thu, 17 Dec 2015 20:06:25 GMT
< content-length: 143
< content-type: application/json
< location: /newspapers/newspaper1/news/87a803ed
<

GET /newspapers/:name/news/:id

This endpoint returns 200 OK and the newsitem identified by the given id that belongs to the newspaper with name :name. If no matching newsitem is found it returns 404 Not Found.

Example

curl -vX GET "http://localhost:4892/newspapers/newspaper1/news/87a803ed-f50e-4606-9c6b-77eac048f0ec" \
--header "Accept: application/json"
> GET /newspapers/newspaper1/news/87a803ed HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:4892
> Accept: application/json
>
< HTTP/1.1 200 OK
* Server Cowboy is not blacklisted
< server: Cowboy
< date: Fri, 18 Dec 2015 02:41:17 GMT
< content-length: 143
< content-type: application/json
<
{"title":"title1",
 "newspaper_name":"newspaper1",
 "id":"87a803ed-f50e-4606-9c6b-77eac048f0ec",
 "created_at":"2015-12-17T20:06:26Z",
 "body":"body1"}

GET /news

This endpoint provides an SSE connection that starts by replaying all the news on the db or the ones after the given last-event-id header (if provided). Then, it lets you stay connected to automatically get the newer ones posted using the POST /newspapers/:name/news/ endpoint.

News format

Following the SSE standard, the fields should be interpreted as follows:

  • id is the id of the newsitem
  • event is the newspaper's name
  • data is composed by two lines: title and body

Example without last-event-id header

curl -vX GET "http://localhost:4892/news" \
--header "Accept: application/json"
> GET /news HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:4892
> Accept: application/json
>
< HTTP/1.1 200 OK
< transfer-encoding: chunked
* Server Cowboy is not blacklisted
< server: Cowboy
< date: Fri, 18 Dec 2015 02:46:39 GMT
< content-type: text/event-stream
< cache-control: no-cache
<
id: 87a803ed-f50e-4606-9c6b-77eac048f0ec
event: newspaper1
data: title1
data: body1

id: 7b161ab2-7257-4506-bf81-2fa1298e72f4
event: newspaper1
data: newsitem2
data: body2

…

Example including last-event-id header

In this example the first event is skipped because we are asking the server for the events after that one. Check last-event-id header and compare it with the last example.

curl -vX GET "http://localhost:4892/news" \
--header "Accept: application/json" \
--header "last-event-id: 87a803ed"
> GET /news HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:4892
> Accept: application/json
> last-event-id: 87a803ed
>
< HTTP/1.1 200 OK
< transfer-encoding: chunked
* Server Cowboy is not blacklisted
< server: Cowboy
< date: Fri, 18 Dec 2015 02:49:30 GMT
< content-type: text/event-stream
< cache-control: no-cache
<
id: 7b161ab2
event: newspaper1
data: newsitem2
data: body2

...

How it is done

canillita uses erlang.mk as its building tool.

  • Files like Makefile and erlang.mk are used to compile and run the application
  • The test folder contains all the tests for the app
    • test.config: It's an optional file that is only required because the configuration of our libraries (and Canillita itself) is too complex to be just passed in from command line and config files are Erlang's default method to deal with that.
  • And of course we have src, which contains the code. In it, you will see:
    • canillita.app.src: the application description file.
    • canillita.erl: the main application module.
    • canillita_newspapers.erl: the newspapers model.
    • canillita_newspapers_handler.erl: a newspapers HTTP handler. It manages POST and GET requests to our /newspapers endpoint.
    • canillita_single_newspaper_handler.erl: a newspaper HTTP handler. It manages GET, PUT and DELETE requests to our /newspapers/:id endpoint.
    • canillita_newspapers_repo.erl: the newspapers repository. We will put our sumo_db calls for canillita_newspapers table here.
    • canillita_newsitems.erl: the newsitems model.
    • canillita_newsitems_handler.erl: a newsitems HTTP handler. It manages POST requests to our /newspapers/:name/news endpoint.
    • canillita_single_newsitem_handler.erl: a newsitem HTTP handler. It manages GET requests to our /newspapers/:name/news/:id endpoint.
    • canillita_newsitems_repo.erl: the newsitems repository. We will put our sumo_db calls for canillita_newsitems table here.
    • canillita_news_handler.erl: the news HTTP handler. It manages GET requests to our /news endpoint.
    • canillita_newsitems_events_handler.erl: Sumo events handler. Processes and notifies listeners about sumo_db events.

Every project that uses sumo, ends up implementing a lousy version of the repository pattern. This one is not the exception. Therefore canillita_newspapers_repo and canillita_newsitems_repo (our repositories) contains all the business logic for Canillita.

Let's Code!!! (allthethings)

Initial setup

The very first thing you need to do is to create a folder for your project, something like: mkdir ~/projects/canillita and get into it cd ~/projects/canillita.

Since we are going to use erlang.mk as our building tool for this project, we "install" it by executing:

$ wget https://raw.githubusercontent.com/ninenines/erlang.mk/master/erlang.mk

Now we have to add the required configuration (test.config) for the different applications our canillita depends on:

Swagger: we just defined the minimun required properties

{ cowboy_swagger
, [ { global_spec
  , #{ swagger => "2.0"
     , info => #{title => "Canillita Test API"}
     }
  }
]
}

Mnesia: It is a term-oriented (i.e. NoSQL) database that is distributed with Erlang/OTP. We've chosen Mnesia as our backend, so we just enabled debug on it (not a requirement, but a nice thing to have on development environments).

{ mnesia
, [{debug, true}]
}

SumoDB: sumo_db's Mnesia backend/store is really easy to set up, we just need to tell sumo (using the stores key) that we want to use canillita_store_mnesia store with sumo_db's sumo_store_mnesia backend, and that's all.

Canillita will have just 2 models: canillita_newspapers and canillita_newsitems and we specify them in the docs key. We will store them both on Mnesia.

In the events key we add the model we want to listen events for (canillita_newsitems) and we associate it with the event's manager. This piece of config is used by Sumo to start an events manager in its own supervision tree with the name we've chosen (canillita_newsitems_events_manager) so we can later find and register a handler for it. Later we will see why we are listening for newsitems events.

{ sumo_db
, [ {wpool_opts, [{overrun_warning, 100}]}
, {log_queries, true}
, {query_timeout, 30000}
, {storage_backends, []}
, { stores
  , [ { canillita_store_mnesia
      , sumo_store_mnesia
      , [{workers, 10}]
      }
    ]
  }
, { docs
  , [ { canillita_newspapers
      , canillita_store_mnesia
      }
    , { canillita_newsitems
      , canillita_store_mnesia
      }
    ]
  }
, { events
  , [ { canillita_newsitems
      , canillita_newsitems_events_manager
      }
    ]
  }
]
}

Since this is a typical Erlang application, we need to create the application description file. This file is composed by a single Erlang tuple that, along with the applicaton name, includes some high level parameters for the app, like its description and vsn. It also includes three very important things:

  • mod which describes the starting point of our application (in our case, the module canillita)
  • applications which lists the other applications that should be started for canillita to run, in our case:
    • kernel and stdlib: default apps used by all Erlang applications
    • lager: the logging framework
    • sasl: System Architecture Support Libraries
    • sumo_rest: Generic cowboy handlers to work with sumo_db
    • lasse: Server-Sent-Event (SSE) handler for Cowboy
  • start_phases which is a list of start phases and corresponding start arguments for the application, in our case:
    • create_schema: creates our persistency schema (and in fact, sumo:create_schema() just updates it so if the schema is already created, it just stays as-is).
    • start_cowboy_listeners: starts our HTTP server.
    • start_canillita_events_management: sets up sumo_db events handler and creates a new group of processes using pg2, which is the simplest possible library (although not the most efficient one) for that. This group will eventually include all the users connected to the SSE endpoint.

Note: cowboy, cowboy-trails, cowboy-swagger and sumo_db applications are not being started by Canillita because they get started by sumo_rest, you can see it in sumo_rest's application description file.

{application, canillita, [
  {description,
    "Canillita - your first Erlang server!"},
  {vsn, "2.0.0"},
  {modules, []},
  {registered, []},
  {applications, [
      kernel,
      stdlib,
      lager,
      sasl,
      sumo_rest,
      lasse
  ]},
  {mod, {canillita, []}},
  {env, []},
  {start_phases , [
    {create_schema, []},
    {start_cowboy_listeners, []},
    {start_canillita_events_management, []}
  ]}
]}.

In order to make sure we have our deps downloaded and compiled each time we want to compile or run canillita, we need to add a Makefile that looks as follows:

PROJECT = canillita

CONFIG ?= test/test.config

DEPS       = sumo_rest lasse katana
DEPS       += swagger sumo_db trails lager
SHELL_DEPS = sync
TEST_DEPS  = shotgun mixer
LOCAL_DEPS = tools compiler syntax_tools
LOCAL_DEPS += common_test inets test_server
LOCAL_DEPS += dialyzer wx

dep_sumo_rest = git https://github.com/inaka/sumo_rest.git 0.1.1
dep_lasse = git https://github.com/inaka/lasse.git 1.0.1
dep_sync = git https://github.com/rustyio/sync.git 9c78e7b
dep_katana = git https://github.com/inaka/erlang-katana.git 07efe94
dep_shotgun = git https://github.com/inaka/shotgun.git 0.1.12
dep_mixer = git https://github.com/inaka/mixer.git 0.1.4
dep_swagger = git https://github.com/inaka/cowboy-swagger.git 0.1.0
dep_sumo_db = git https://github.com/inaka/sumo_db.git f8a3689
dep_trails = git https://github.com/inaka/cowboy-trails.git 0.1.0
dep_lager = git https://github.com/basho/lager.git 3.0.2

include erlang.mk

DIALYZER_DIRS := ebin/ test/
DIALYZER_OPTS := --verbose --statistics
DIALYZER_OPTS += -Wunmatched_returns

ERLC_OPTS := +debug_info +'{parse_transform, lager_transform}'
TEST_ERLC_OPTS += +debug_info +'{parse_transform, lager_transform}'
CT_OPTS = -cover test/canillita.coverspec
CT_OPTS += -erl_args -config ${CONFIG}

SHELL_OPTS = -s sync -config ${CONFIG}

For a complete description of the things included in erlang.mk Makefiles, check out its documentation

Application module

Every Erlang application has an application module, which is identifiable because it adheres to the application behaviour, and also because it's the one listed in the mod attribute of the .app.src file. It usually carries the same name as the app itself. In our case, that module is canillita. The module includes the following functions:

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% ADMIN API
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%% @doc Starts the Application
start() ->
  {ok, _} =
    application:ensure_all_started(canillita).

%% @doc Stops the Application
stop() -> ok = application:stop(canillita).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% BEHAVIOUR CALLBACKS
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

start(_Type, _Args) -> {ok, self()}.

stop(_State) ->
  gen_event:delete_handler(
    canillita_newsitems_events_manager,
    canillita_newsitems_events_handler, []),
  ok.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% START PHASES
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%% @private
start_phase(create_schema, _StartType, []) ->
  _ = application:stop(mnesia),
  Node = node(),
  case mnesia:create_schema([Node]) of
    ok -> ok;
    {error, {Node, {already_exists, Node}}} ->
      ok
  end,
  {ok, _} =
    application:ensure_all_started(mnesia),
  % Create persistency schema
  sumo:create_schema();
start_phase(
  start_cowboy_listeners, _StartType, []) ->
  Handlers =
    [ canillita_newspapers_handler
    , canillita_single_newspaper_handler
    , canillita_newsitems_handler
    , canillita_single_newsitem_handler
    , canillita_news_handler
    , cowboy_swagger_handler
    ],
  % Get the trails for each handler
  Routes = trails:trails(Handlers),
  % Store them so Cowboy is able to get them
  trails:store(Routes),
  % Set server routes
  Dispatch =
    trails:single_host_compile(Routes),
  % Set the options for the TCP layer
  TransOpts = [{port, 4892}],
  % Set the options for the HTTP layer
  ProtoOpts =
    [{env, [ {dispatch, Dispatch}
           , {compress, true}
           ]}],
  % Start Cowboy HTTP server
  case cowboy:start_http(
    canillita_server, 1, TransOpts,
    ProtoOpts) of
    {ok, _} -> ok;
    {error, {already_started, _}} -> ok
  end;
start_phase(
  start_canillita_events_management,
  _StartType, []) ->
  % Set the handler for processing events
  ok = gen_event:add_handler(
    canillita_newsitems_events_manager,
    canillita_newsitems_events_handler, []),
  % Create pg2 group to hold news listeners
  pg2:create(canillita_listeners).

Those in the ADMIN API section are the external API for the system. start/0 starts the application using one of the latest additiions to the application module (ensure_all_started/1) and stop/0 stops the server.

Those in the BEHAVIOUR CALLBACKS section are the behaviour implementation functions that are called when the application is started or stopped.

Those in the START PHASES section are the ones that start all the stuff necessary for the application to work as expected. They are called after the application is started.

Note: What is done in each start phase is described in the comments within the code.

…and that's it! Now we have our server running. But to process requests we have to add some handlers for the specified endpoints.

Handlers

The functions required by cowboy for all handlers, in general, are pretty straightforward. Almost all of them receive 2 parameters (a request and a state) and are expected to return a tuple with 3 elements: a response, a request and a state.

Erlang, being a functional language, doesn't allow us to keep state of any kind in global or local variables as OOP languages would. So, our state must travel through functions, getting into them in the form of parameters and going out of them in the function result. That's why all these functions receive a description of the HTTP request and the internal state of your handler as parameters and expect new versions of them as part of the responses.

We have 2 types of handlers here: HTTP handlers and events handler.

HTTP Handlers

Since we are implementing trails_handler behaviour in our HTTP handlers, we will be defining the API routes within each handler. Also because we are using cowboy-swagger we will also add some metadata to the routes in order to get a nice documentation for our API.

Every HTTP handler in our canillita will implement a trails_handler behaviour so each one of them will define and export a trails/0 function that returns the list of the trails it is responsible for.

For our HTTP handlers we will be using mixer + sumo_rest that allow us to include into our handlers some generic Cowboy handlers that works with sumo_db.

Using mixer we save time and reuse code by "including" code from other modules into our handler. sumo_rest exposes two (2) types of handlers that we can use to mix in code: sr_entities_handler and sr_single_entity_handler.

sr_entities_handler exported functions:

  • allowed_methods/2
  • announce_req/2
  • content_types_accepted/2
  • content_types_provided/2
  • handle_get/2
  • handle_post/2
  • handle_post/3
  • init/3
  • module_info/0
  • module_info/1
  • resource_exists/2
  • rest_init/2

We mix in these functions into canillita_newspapers_handler and canillita_newsitems_handler, we do that by adding the following piece of code into our handler:

We need those functions in our handlers because our handlers are cowboy_rest handler implementations and all those functions are optional callbacks for that behaviour.

-include_lib("mixer/include/mixer.hrl").
-mixin([{ sr_entities_handler
        , [ init/3
          , rest_init/2
          , allowed_methods/2
          , resource_exists/2
          , content_types_accepted/2
          , content_types_provided/2
          , handle_get/2
          , handle_post/2
          ]
        }]).

canillita_newspapers_handler

POST|GET /newspapers handler

trails() ->
  RequestBody =
    #{ name => <<"request body">>
     , in => body
     , description => <<"request body">>
     , required => true
     },
  Metadata =
    #{ get =>
       #{ tags => ["newspapers"]
        , description =>
            "Returns the list of newspapers"
        , produces => ["application/json"]
        }
     , post =>
       # { tags => ["newspapers"]
         , description => "Creates a newspaper"
         , consumes => ["application/json"]
         , produces => ["application/json"]
         , parameters => [RequestBody]
         }
     },
  Path = "/newspapers",
  Options =
    #{ path => Path
     , model => canillita_newspapers
     },
  [trails:trail(
      Path, ?MODULE, Options, Metadata)].

canillita_newsitems_handler

POST /newspapers/:name/news/ handler

trails() ->
  NewspaperName =
    #{ name => name
     , in => path
     , description => <<"Newspaper name">>
     , required => true
     , type => string
     },
  RequestBody =
    #{ name => <<"request body">>
     , in => body
     , description => <<"request body">>
     , required => true
     },
  Metadata =
    #{ post =>
       #{ tags => ["newsitems"]
        , description => "Creates a news item"
        , consumes => ["application/json"]
        , produces => ["application/json"]
        , parameters =>
            [NewspaperName, RequestBody]
        }
     },
  Path = "/newspapers/:name/news",
  Options =
    #{ path => Path
     , model => canillita_newsitems
     },
  [trails:trail(
      Path, ?MODULE, Options, Metadata)].

NOTE: by the time this article was written sumo_rest was not using the parameters in the PATH, so for canillita_newsitems_handler we did not need to use handle_post/2 function from the mixin and defined it ourselves in order to get the :name parameter in the Path when creating a newsitem; also we need to check that the given newspaper name does exists otherwise we return 404 Not Found. Here is how our handle_post/2 looks:

handle_post(Req, State) ->
  try
    {ok, Body, Req1} = cowboy_req:body(Req),
    Json = sr_json:decode(Body),
    {NewspaperName, _Req} =
      cowboy_req:binding(name, Req),
    % Checks that the newspaper does exists
    case canillita_newspapers_repo:exists(
          NewspaperName) of
      true ->
        case canillita_newsitems:from_json(
              NewspaperName, Json) of
          {error, Reason} ->
            Req2 =
              cowboy_req:set_resp_body(
                sr_json:error(Reason), Req1),
            {false, Req2, State};
          {ok, Entity} ->
            handle_post(Entity, Req1, State)
        end;
      false ->
        cowboy_req:reply(404, Req),
        {halt, Req, State}
    end
  catch
    _:badjson ->
      Req3 =
        cowboy_req:set_resp_body(
          sr_json:error(
            <<"Malformed JSON request">>),
            Req),
      {false, Req3, State}
  end.

We do use sr_entities_handler:handle_post/3 which is not part of cowboy_rest but is a helper function provided by sumo_rest specifically for this kind of scenarios where you need external info to build your entities.

sr_single_entity_handler exported functions:

  • allowed_methods/2
  • announce_req/2
  • content_types_accepted/2
  • content_types_provided/2
  • delete_resource/2
  • handle_get/2
  • handle_patch/2
  • handle_put/2
  • init/3
  • module_info/0
  • module_info/1
  • resource_exists/2
  • rest_init/2

We mix in these functions into canillita_single_newspaper_handler and canillita_single_newsitem_handler, we do that by adding the following piece of code into our handler:

-include_lib("mixer/include/mixer.hrl").
-mixin([{ sr_single_entity_handler
        , [ init/3
          , rest_init/2
          , allowed_methods/2
          , resource_exists/2
          , content_types_accepted/2
          , content_types_provided/2
          , handle_get/2
          , handle_put/2
          , delete_resource/2
          ]
        }]).

canillita_single_newspaper_handler

GET|PUT|DELETE /newspapers/:id handler

trails() ->
  RequestBody =
    #{ name => <<"request body">>
     , in => body
     , description =>
        <<"request body (as json)">>
     , required => true
     },
  Id =
    #{ name => id
     , in => path
     , description => <<"Newspaper key">>
     , required => true
     , type => string
     },
  Metadata =
    #{ get =>
       #{ tags => ["newspapers"]
        , description => "Returns a newspaper"
        , produces => ["application/json"]
        , parameters => [Id]
        }
     , put =>
       #{ tags => ["newspapers"]
        , description =>
            "Updates or creates a newspaper"
        , consumes => ["application/json"]
        , produces => ["application/json"]
        , parameters => [RequestBody, Id]
        }
     , delete =>
       #{ tags => ["newspapers"]
        , description => "Deletes a newspaper"
        , parameters => [Id]
        }
     },
  Path = "/newspapers/:id",
  Options =
    #{ path => Path
     , model => canillita_newspapers
     },
  [trails:trail(
      Path, ?MODULE, Options, Metadata)].

canillita_single_newsitem_handler

GET /newspapers/:name/news/:id handler

trails() ->
  NewspaperName =
    #{ name => name
     , in => path
     , description => <<"Newspaper name">>
     , required => true
     , type => string
     },
  NewsItemId =
    #{ name => id
     , in => path
     , descripcion => <<"News item id">>
     , required => true
     , type => string
     },
  Metadata =
    #{ get =>
       #{ tags => ["newsitems"]
        , description => "Return a newsitem"
        , produces => ["application/json"]
        , parameters =>
            [NewspaperName, NewsItemId]
        }
     },
  Path = "/newspapers/:name/news/:id",
  Options =
    #{ path => Path
     , model => canillita_newsitems
     },
  [trails:trail(
    Path, ?MODULE, Options, Metadata)].

Here we need to do something similar to what we did with canillita_newsitems_handler but for resource_exists/2, we need not to include it in the mixin but define it here instead:

resource_exists(Req, State) ->
  #{opts := #{model := _Model}, id := Id} =
    State,
  {NewspaperName, Req2} =
    cowboy_req:binding(name, Req),
  case canillita_newsitems_repo:fetch(
        NewspaperName, Id) of
    notfound ->
      {false, Req2, State};
    Entity ->
      {true, Req2, State#{entity => Entity}}
  end.

canillita_news_handler

GET /news handler

trails() ->
  RFC = "http://www.w3.org/TR/eventsource/",
  Metadata =
    #{ get =>
       #{ tags => ["news"]
        , summary =>
          "WARNING: Do not try to use this"
          " endpoint from this page."
          " Swagger doesn't understand SSE"
        , description =>
          "Opens an [SSE](" ++ RFC ++ ")"
          " connection to retrieve updates"
        , externalDocs =>
          #{ description => "RFC"
           , url => RFC
           }
        , produces => ["application/json"]
        }
     },
  Path = "/news",
  Options =
    #{ module => ?MODULE
     , init_args => #{path => Path}
     },
  [trails:trail(
    Path, lasse_handler, Options, Metadata)].

By the time this article was written there was an issue still opened for Swagger specification to add support for SSE into it. Therefore, because SSE was still not supported in swagger, we decided to add the warning in the summary. Users will still be able to hit the endpoint, but it will block the browser for them instead of returning anything.

This particular handler also implements a lasse_handler behaviour so we need to define its callbacks. We implement that behaviour here because this is our SSE handler.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% API
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%% @doc sends an event to all the listeners
notify(Event) ->
  lists:foreach(
    fun(Listener) ->
      lasse_handler:notify(Listener, Event)
    end,
    pg2:get_members(canillita_listeners)
   ).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% lasse_handler callbacks
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%% @doc Will be called upon initialization of
%%      the handler, receiving the value of the
%%      "last-event-id" header if there is one
%%      or 'undefined' otherwise.
init(_InitArgs, LastEventId, Req) ->
  Req1 =
    sr_entities_handler:announce_req(Req, #{}),
  NewsItems =
    canillita_newsitems_repo:fetch_since(
      LastEventId),
  News =
    [ canillita_newsitems:to_sse(NewsItem)
    || NewsItem <- NewsItems
    ],
  ok = pg2:join(canillita_listeners, self()),
  {ok, Req1, News, #{}}.

%% @doc Receives and processes in-band messages
%%      sent through lasse_handler:notify/2
handle_notify(NewsItem, State) ->
  Event = canillita_newsitems:to_sse(NewsItem),
  {send, Event, State}.

%% @doc Receives and processes out-of-band
%%      messages sent directly to the
%%      handler's process.
handle_info(Info, State) ->
  _ =
    lager:notice(
      "~p received at ~p", [Info, State]),
  {nosend, State}.

%% @doc If there's a problem while sending a
%%      chunk to the client, this function will
%%      be called after which the handler will
%%      terminate.
handle_error(Event, Error, State) ->
  _ =
    lager:warning(
      "Couldn't send ~p in ~p: ~p",
      [Event, State, Error]),
  State.

%% @doc This function will be called before
%%      terminating the handler, its return
%%      value is ignored.
terminate(Reason, _Req, _State) ->
  _ =
    lager:notice(
      "Terminating news: ~p", [Reason]),
  ok.

Check out notify/1. That's the function that delivers events to the clients that are connected and listening to GET /news through SSE. This function basically goes over the list of members of the canillita_listeners pg2 group. Every time lasse_handler:notify/2 is called it sends a message to the process identified with the given Pid with the tuple {message, Msg} (the given Msg) and this message is then processed by the Cowboy loop handler info/3 callback which in turn, ends up calling canillita_news_handler:handle_notify/2 and then sending whatever we return to the client as SSE events.

Events Handler

There is just one event handler in canillita. This handler is used to notify the listeners in canillita_listeners pg2 group every time a newsitem is created.

canillita_newsitems_events_handler

Here we are implementing a gen_event behaviour. So we need to add its callbacks and export them.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% gen_event functions.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
init([]) ->
  {ok, []}.

handle_info(_Info, State) ->
  {ok, State}.

handle_call(_Request, State) ->
  {ok, not_implemented, State}.

handle_event(
  {canillita_newsitems, created, [Entity]},
  State) ->
  canillita_news_handler:notify(Entity),
  {ok, State};
handle_event(_Event, State) ->
  {ok, State}.

code_change(_OldVsn, State, _Extra) ->
  {ok, State}.

terminate(_Arg, _State) ->
  ok.

The important thing in this handler is the handle_event/2 function, that's the one that processes the events generated by sumo_db and calls our canillita_news_handler:notify/1 in order to notify our listeners. In our case we are just interested to notify the listeners whenever a new newsitem is created.

Models

We have just 2 models in canillita: canillita_newspapers and canillita_newsitems. For our models we are going to implement sumo_rest_doc and sumo_doc behaviours so we need to write and export its callbacks. Our models do not include business logic, they're just descriptions of data types and provide abstraction for those data types.

sumo_rest abstracts sumo_db interaction, so we don't need to do a lot of things to create, save, update or delete Newspapers or NewsItems in our application because sumo_rest does most of that work. The only exception for our project are the NewsItems because sumo_rest doesn't let you use the parameters in the PATH yet.

sumo_doc requires us to add the schema, sleep and wakeup functions. Since we'll use maps for our internal representation (just like sumo_db does), they're trivial:

sumo_schema() ->
  sumo:new_schema(
    ?MODULE,
    [ sumo:new_field(
        name, string, [id, unique])
    , sumo:new_field(
        description, string, [not_null])
    , sumo:new_field(
        created_at, datetime, [not_null])
    , sumo:new_field(
        updated_at, datetime, [not_null])
    ]).

%% @doc Convert to sumo's representation
sumo_sleep(Newspaper) -> Newspaper.

%% @doc Convert to system representation
sumo_wakeup(Newspaper) -> Newspaper.

sumo_rest_doc on the other hand requires functions to convert to and from json (which should also validate user input):

%% @doc Convert a newspaper to json.
to_json(Newspaper) ->
  #{ name => maps:get(name, Newspaper)
   , description =>
      maps:get(description, Newspaper)
   , created_at =>
      sr_json:encode_date(
        maps:get(created_at, Newspaper))
   , updated_at =>
      sr_json:encode_date(
        maps:get(updated_at, Newspaper))
   }.

%% @doc Convert a newspaper from json.
from_json(Json) ->
  Now =
    sr_json:encode_date(
      calendar:universal_time()),
  try
    { ok
    , #{ name => maps:get(<<"name">>, Json)
       , description =>
          maps:get(<<"description">>, Json)
       , created_at =>
          sr_json:decode_date(
            maps:get(
              <<"created_at">>, Json, Now))
       , updated_at =>
          sr_json:decode_date(
            maps:get(
              <<"updated_at">>, Json, Now))
       }
    }
  catch
    _: {badkey, Key} ->
      {error,
       <<"missing field: ", Key/binary>>}
  end.

We also need to provide an update function for PUT and PATCH:

update(Newspaper, Json) ->
  try
    NewDescription =
      maps:get(<<"description">>, Json),
    UpdatedNewspaper =
      Newspaper#{
        description := NewDescription,
        updated_at := calendar:universal_time()
        },
    {ok, UpdatedNewspaper}
  catch
    _:{badkey, Key} ->
      {error,
       <<"missing field: ", Key/binary>>}
  end.

For Sumo Rest to provide URLs to the callers, we need to specify the location URL:

%% @doc Specify the uri part that uniquely
%%      identifies a Newspaper.
location(Newspaper, Path) ->
  iolist_to_binary([Path, name(Newspaper)]).

To let Sumo Rest avoid duplicate keys (and return 409 Conflict in that case), we provide the optional callback id/1:

%% @doc Optional callback id/1 to let sumo_rest
%%      avoid duplicated keys
%%      and return `409 Conflict` in that case
id(Newspaper) -> name(Newspaper).

The rest of the functions in the module are just helpers, particularly useful for our tests.

new(Name, Description) ->
  Now = calendar:universal_time(),
  #{ name         => Name
   , description  => Description
   , created_at   => Now
   , updated_at   => Now
   }.

name(#{name := Name}) -> Name.

description(#{description := Description}) ->
  Description.

updated_at(#{updated_at := UpdatedAt}) ->
  UpdatedAt.

That's our canillita_newspapers model. It's almost the same for canillita_newsitems model but with 2 differences:

1) we don't need to define the id/1 callback because every newsitem id is unique and automatically generated on its creation, therefore there will never be duplicates,

2) for canillita_newsitems model we need to add a helper to return the SSE representation of a newsitem used by canillita_newsitems_events_handler handler.

%% @doc Convert a newspaper to SSE.
to_sse(NewsItem) ->
  #{ id => maps:get(id, NewsItem)
   , event =>
      maps:get(newspaper_name, NewsItem)
   , data => iolist_to_binary(
              [ maps:get(title, NewsItem)
              , "\n"
              , maps:get(body, NewsItem)
              ])
   }.

Repositories

Here at Inaka we use model repositories to hold our business logic, in particular we put there all the functions that deal with persistence through sumo_db stores. In canillita we have 2 of them, one for Newspapers (canillita_newspapers_repo) and the other one for NewsItems (canillita_newsitems_repo).

Newspaper Repository

There is just one function in our Newspapers repository, it's exists/1 and it's used in canillita_newsitems_handler:handle_post/2 to verify that the given newspaper does exists before creating a newsitem:

%% @doc Checks existence of a newspaper
exists(NewspaperName) ->
  notfound /=
    sumo:find(
      canillita_newspapers, NewspaperName).

NewsItems Repository

For this repository we have 2 exported functions:

fetch/2, used by canillita_single_newsitem_handler:resource_exists/2 to return the newsitem identified with the given Id and NewspaperName:

%% @doc Returns the newsitem that matches the
%%      given newspaper_name and id (if any).
fetch(NewspaperName, Id) ->
  Conditions =
    [ {id, Id}
    , {newspaper_name, NewspaperName}
    ],
  sumo:find_one(
    canillita_newsitems, Conditions).

fetch_since/2, used by canillita_news_handler:init/3 lasse_handler callback to return just the events after the given last-event-id:

%% @doc returns all the news after the given
%%      event-id or all the news if not
%%      event-id provided.
fetch_since(undefined) ->
  fetch_all();
fetch_since(LastEventId) ->
  #{created_at := CreatedAt} =
    fetch(LastEventId),
  fetch_all(CreatedAt).

And the other ones are for internal use of the repository:

%% @doc returns the newsitem with the given id.
fetch(Id) ->
  sumo:find(canillita_newsitems, Id).

%% @doc returns all the newsitems stored so far
fetch_all() ->
  sumo:find_all(canillita_newsitems).

%% @doc returns elements created after a point
fetch_all(CreatedAt) ->
  Conditions = [{created_at, '>', CreatedAt}],
  sumo:find_by(
    canillita_newsitems, Conditions).

CHANGELOG

For those of you keeping track, since our initial blog post from 2013, we have changed multiple things. In a nutshell, these are the new libraries that help us build RESTful APIs with the same confidence but even less code:

  • cowboy-trails: it allows us to define the routes in each cowboy handler thanks to its trails_handler behaviour. And it also let us add some metadata to our routes.
  • cowboy-swagger: it let us use the metadata from the routes to create a really nice and useful documentation for our API.
  • sumo_rest: adds the ability to create generic Cowboy handlers to work with sumo_db.
  • lasse: makes SSE server-side implementation really easy with the use of its callbacks.
  • mixer: allow us to mix in functions from other modules so we can save time and code to get the same results by "including" code defined in another modules into our module.

And by the way, this time we developed the project using TDD and Meta-Testing and we let gadget review all of its PRs. That gave us a lot of confidence and also helped a lot with the definition of the scope of this blog post and the project in general.

Conclusion

Simple as that, if properly configured, we now have an SSE / RESTful server capable of handling thousands of listeners in one node. And it benefits from all the virtues of Erlang, like:

  • hot code swapping: you can make changes on the code and compile them just running make:all([load]) in the console without turning the server off
  • easy multi-node scalability: pg2 is multi-node aware so adding new nodes is just a matter of turning them on and connecting them
  • easy deployability: Thanks to erlang.mk and relx you can easily build a release for your whole system and deploy it wherever you like without worrying about installed Erlang/OTP versions, dependent apps, VM versions, boot scripts, control mechanisms. It's all included within the release itself.

And if you're a newcomer to Erlang, you can see that you don't need to learn a lot of stuff before you start working on your first real-life project. This one, for instance, has little if any code that is Erlang-intensive. So thanks to the many libraries that are already out there, you can start with simple projects like this one and satisfy some basic but important requirements in a complete way.

Join us on hipchat if you want to talk about all these with us live. And tell us about your experiences building this kind of apps and leave general questions in the comments below.