inaka

Latest blog entries

/
The Art of Writing a Blogpost

The Art of Writing a Blogpost

Mar 09 2017 : Matias Vera

/
SpellingCI: No more spelling mistakes in your markdown flies!

Feb 14 2017 : Felipe Ripoll

/
Fast reverse geocoding with offline-geocoder

Do you need a blazing fast reverse geocoder? Enter offline-geocoder!

Jan 18 2017 : Roberto Romero

/
Using Jayme to connect to the new MongooseIM REST services

MongooseIM has RESTful services!! Here I show how you can use them in an iOS application.

Dec 13 2016 : Sergio Abraham

/
20 Questions, or Maybe a Few More

20 Questions, or Maybe a Few More

Nov 16 2016 : Stephanie Goldner

/
The Power of Meeting People

Because conferences and meetups are not just about the technical stuff.

Nov 01 2016 : Pablo Villar

/
Finding the right partner for your app build

Sharing some light on how it is to partner with us.

Oct 27 2016 : Inaka

/
Just Play my Sound

How to easily play a sound in Android

Oct 25 2016 : Giaquinta Emiliano

/
Opening our Guidelines to the World

We're publishing our work guidelines for the world to see.

Oct 13 2016 : Brujo Benavides

/
Using NIFs: the easy way

Using niffy to simplify working with NIFs on Erlang

Oct 05 2016 : Hernan Rivas Acosta

/
Function Naming In Swift 3

How to write clear function signatures, yet expressive, while following Swift 3 API design guidelines.

Sep 16 2016 : Pablo Villar

/
Jenkins automated tests for Rails

How to automatically trigger rails tests with a Jenkins job

Sep 14 2016 : Demian Sciessere

/
Erlang REST Server Stack

A description of our usual stack for building REST servers in Erlang

Sep 06 2016 : Brujo Benavides

/
Replacing JSON when talking to Erlang

Using Erlang's External Term Format

Aug 17 2016 : Hernan Rivas Acosta

/
Gadget + Lewis = Android Lint CI

Integrating our Android linter with Github's pull requests

Aug 04 2016 : Fernando Ramirez and Euen Lopez

/
Passwordless login with phoenix

Introducing how to implement passwordless login with phoenix framework

Jul 27 2016 : Thiago Borges

/
Beam Olympics

Our newest game to test your Beam Skills

Jul 14 2016 : Brujo Benavides

/
Otec

Three Open Source Projects, one App

Jun 28 2016 : Andrés Gerace

/
CredoCI

Running credo checks for elixir code on your github pull requests

Jun 16 2016 : Alejandro Mataloni

/
Thoughts on rebar3

Thoughts on rebar3

Jun 08 2016 : Hernán Rivas Acosta

/
See all Inaka's blog posts >>

/
Shotgun: HTTP client for Server-sent Events

Juan Facorro wrote this on October 20, 2014 under erlang, http, inaka, shotgun .

Shotgun in its most basic form is just an HTTP client implemented on top of Gun, a very flexible library to perform requests over SPDY, HTTP and HTTPS. Though powerful Gun forces you to take care of each message that involves receiving the response, which is not that convenient after a couple of times you write the code to handle these messages between processes. Although there are a bunch of existing HTTP client libraries for Erlang, Shotgun is not yet another HTTP client. It was originally conceived to support and gracefully handle Server-sent Events in a way that is simple for the user and doesn't involve handling message passing.

Server-sent Events

What exactly is SSE? It is just a way for the server to push events through an HTTP connection to a client. For this to work the response should be a chunked response and its content-type header must be text/event-stream. Once the connection is established, the server can send at any time little chunks of information as a stream to the client, who can then parse this information based on the SSE specification.

Events can have any of the following fields (any other field is ignored):

  • event: specifies the event's type.
  • data: the content of each of these fields make up each of the lines of the event's data.
  • id: specified the event's id.
  • retry: reconnection time in millisecond in case the connection is lost.

Here's how a bunch of events would look like in the stream sent form the server:

id: 17
event: start
data: brace yourself, events are coming

id: 18
event: info
data: important information
data: more important information

id: 19
event: end
data: no more events

As you can see, every field ends with a new line and every event ends with two new lines. For more information on what SSE is all about check out this article and the current specs.

Using Shotgun

Now that we (kinda) know what Server-sent Events are all about let's use Shotgun to perform some traditional requests first and then receive some SSE later. On both cases we need to open a connection to the server, not surprisingly this is what the open/2 function is for.

Note: To try out the code examples just clone Shotgun's github repository and run make all shell in your command line.

{ok, Conn} = shotgun:open("facebook.com", 80).

Then we can either use one of the functions available to do a request with a specific HTTP method (get/[2, 3, 4], post/5, head/4, etc.) or use the generic request/6 function and specifiy the HTTP method as a first argument.

{ok, Response} = shotgun:get(Conn, "/").
%% Result.
{ok,#{headers => [{<<"location">>,
                   <<"http://www.facebook.com/unsupportedbrowser">>},
                  {<<"content-type">>,<<"text/html; charset=utf-8">>},
                  {<<"x-fb-debug">>,
                   <<"avEhxgcr/zayShjZ0D9g8a8MjSz4RTGJZqzqBx9XJ"...>>},
                  {<<"date">>,<<"Mon, 20 Oct 2014 18:07:57 GMT">>},
                  {<<"connection">>,<<"keep-alive">>},
                  {<<"content-length">>,<<"0">>}],
      status_code => 302}}

Or the equivalent using request/6:

Headers = #{},
Body = "",
Options = #{},
{ok, _} = shotgun:request(Conn,
                          get,
                          "/",
                          Headers,
                          Body,
                          Options).

When you are done with all the requests you wanted to do, you should close/1 the connection.

shotgun:close(Conn).

Consuming Chunks or Events

So far so good, now it's time to talk about consuming SSE with Shotgun.

Note: The code examples shown below assume you have the following example application running: ping-pong. Just clone this repo, go to /examples/ping_pong and run make run.

A basic request to start receiving a chunked response only needs to specify in its options that it expects an asynchronous response.

{ok, Conn} =  shotgun:open("localhost", 8080),
Headers = #{},
Options = #{async => true},
{ok, Ref} = shotgun:get(Conn,
                        "/pong",
                        Headers,
                        Options).

Once the request is performed, the chunks or events that have been received so far can be retrieved with the events/1 function, which returns a list of these items. Each item is either a single binary chunk or the complete contents of an event, which one depends on the async_mode option that we will look at next. Subsequent calls to events/1 will not include items already returned.

shotgun:events(Conn).
%% Result
[]
%% ... the server sends some chunks
shotgun:events(Conn).
%% Result
[{nofin,#Ref<0.0.2.120054>,<<"data: pong\n\n">>},
 {nofin,#Ref<0.0.2.120054>,<<"data: pong\n\n">>},
 {nofin,#Ref<0.0.2.120054>,<<"data: pong\n\n">>},
 {nofin,#Ref<0.0.2.120054>,<<"data: pong\n\n">>}]
%% Let's call events again
shotgun:events(Conn).
%% No new events arrived, so nothing is returned
[]

binary vs sse

Because the server is not required to send a server-sent event in each chunk that it emits, it is necessary to have Shotgun parse the chunks received, detecting the start and end of each event. This is why the async_mode option is available, its two possible values are binary (the default one) and sse. The latter tells Shotgun to buffer all chunks until it can detect the end of a server-sent event and only then add it to the items queue. The former just adds every chunk received regardless of what its contents are.

Both modes are equivalent in terms of how you get the information sent from the server (using the events/1 function), but not in terms of how many items are returned and the contents of each one of these.

HandleEvent callback

Instead of queueing up each chunk or event and retrieving them using the events/1 function, you can specify a callback function with the handle_event option. The value provided should be a function that receives three arguments:

  • IsFin: indicates whether the server has finished sending a response or not. Its possible values are fin or nofin.
  • Ref: the same reference returned by the call to perform the asynchronous request.
  • Data: can be either a binary chunk or a the binary content of an event.

If a handle_event function is provided, all items will be processed by calling this function and they will not be added to the items queue that's returned by events/1.

F = fun(_, _, X) -> io:format("~p~n", [X]) end,
{ok, Conn} =  shotgun:open("localhost", 8080),
Headers = #{},
Options = #{async => true,
            async_mode => sse,
            handle_event => F},
{ok, Ref} = shotgun:get(Conn, "/pong", Headers, Options).

The previous code will print each event in the console immediately after it is received. We made sure that whatever it prints is an event by specifying the sse value in the option async_mode.

Parsing SSE

Typically the information you want from an event is not the actual raw binary content but the value for each of its fields. That's why Shotgun includes a parse_event/1 function that eliminates the pain of having to deal with the parsing of each SSE. It just expects a valid SSE in binary format and returns a map with each field that is present.

EventBin = <<"data: pong\ndata: pong\n"
             "data: more pong\n"
             "id: 42\nevent: ping">>,
shotgun:parse_event(EventBin).
%% Result
#{data => [<<"pong">>,<<"pong">>,<<"more pong">>],
  event => <<"ping">>,
  id => <<"42">>}

Conclusion

Server-sent Events are a nice and simple way of having server push notifications without dealing with other more complicated technologies.

Shotgun is a tool that (apart from being a standard HTTP client) allows you to handle SSE in a simple way while also trying to provide enough flexibility to accomodate your needs.

If you have any suggestions, questions or comments please drop us a line at our public open source HipChat room here or open an issue in Shotgun's repo.