Latest blog entries
The Art of Writing a Blogpost
Mar 09 2017 : Matias Vera
Feb 14 2017 : Felipe Ripoll
Do you need a blazing fast reverse geocoder? Enter offline-geocoder!
Jan 18 2017 : Roberto Romero
MongooseIM has RESTful services!! Here I show how you can use them in an iOS application.
Dec 13 2016 : Sergio Abraham
20 Questions, or Maybe a Few More
Nov 16 2016 : Stephanie Goldner
Because conferences and meetups are not just about the technical stuff.
Nov 01 2016 : Pablo Villar
Sharing some light on how it is to partner with us.
Oct 27 2016 : Inaka
How to easily play a sound in Android
Oct 25 2016 : Giaquinta Emiliano
We're publishing our work guidelines for the world to see.
Oct 13 2016 : Brujo Benavides
Using niffy to simplify working with NIFs on Erlang
Oct 05 2016 : Hernan Rivas Acosta
How to write clear function signatures, yet expressive, while following Swift 3 API design guidelines.
Sep 16 2016 : Pablo Villar
How to automatically trigger rails tests with a Jenkins job
Sep 14 2016 : Demian Sciessere
A description of our usual stack for building REST servers in Erlang
Sep 06 2016 : Brujo Benavides
Using Erlang's External Term Format
Aug 17 2016 : Hernan Rivas Acosta
Integrating our Android linter with Github's pull requests
Aug 04 2016 : Fernando Ramirez and Euen Lopez
Introducing how to implement passwordless login with phoenix framework
Jul 27 2016 : Thiago Borges
Our newest game to test your Beam Skills
Jul 14 2016 : Brujo Benavides
Three Open Source Projects, one App
Jun 28 2016 : Andrés Gerace
Running credo checks for elixir code on your github pull requests
Jun 16 2016 : Alejandro Mataloni
Thoughts on rebar3
Jun 08 2016 : Hernán Rivas Acosta
(Not) failing with Node.js
Some time ago, I was tasked with a quite interesting work: to mix old and new, PostgreSQL and Node.js, into an application, a backend for an app, providing a RESTful API. Being a 100% Rails developer, I felt lost without a beloved ORM betweeen the hostile SQL wasteland and my nice tidy code. Since I never used an ORM in Node, I did a quick google search and found a promising ORM named Sequelize. I quickly installed it, tried it and found that it worked as expected. Not as cute, not as easy as ActiveRecord, but it did its job, namely isolate me of the burdens of SQL generation.
Fast-forward a few weeks and we were production-ready. It was load testing time. Client requirements were 1,250 concurrent requests. I was using Node, so of course, I felt confident. Node can handle thousands of connections per second, as that first demo taught us all. So it will handle that puny 1,250 concurrent requests.
In our first load test we managed to handle the incredible amount of 150 concurrent requests per core. Not 1000, not 500 but only 150, with the CPU begging for mercy at a constant 100% load, and the database server perfectly handling the load. That was wrong. We were allowed to use more than one core, but even doing that trick we will need more than eight cores to handle the required amount of users. Not acceptable.
We then did some profiling, with the help of the --prof option in node, perf, and node.js flame graphs for the profiling part, and locust.io and ab for the scale testing part. With the profiling results, we had some clues. Of course, the problem was in the code, as server's CPU was constantly at 100% while the database server sat idly. Another thing we found is that the culprit was in Sequelize. It was high level, but also quite unoptimized.
Looking for alternatives, we found an ORM known by its speed, although not so high level, named node-orm2. We started load testing, disabling all endpoints and converting them to node-orm2, and then enabling them, one by one. Picture appeared promising. We got a 100% speed-up, and we were happy as a clam. Then, with our confidence in unprecedent high levels, we decided to fully switch to node-orm2. And we were smacked to the ground after finding that, after adding all endpoints, we got a speed-up of 5% at best.
Time to switch. Tired of ORMs which promised everything and delivered nothing, we switched to bare node-postgres, which is a package providing only database connection. We also used a repository pattern, which is a layer designed to remove querying from the code. We moved all the code related with data querying to group of prototypes, one for each table, containing all the necessary functions to fetch the data. Such functions will only return pure data, as hashes.
Returning only pure data was great improvement, as we discovered that objects with many functions were quite difficult for node to handle. This is an inherent advantage of the repository pattern.
After that, we found we could handle the impressive amount of 350 concurrent requests. It was not awesome, but at least it was an improvement of 233%. That means we could achieve 1,250 concurrent requests by having a machine with four cores.
All of this taught three lessons for us. The first of all is an old lesson which people keep rediscovering: Node.js is not a silver bullet. It's fast for some scenarios, but it's not always fast. Second one, is do not get over-excited by a partial result. When we partially switched to node-orm2 we found a giant improvement, only to become hugely disappointed when we finally switched everything. And last of all, and I think it's the most important lesson, is: never, ever use an ORM which is not well tested and optimized, and backed by a lot of benchmarks and real use. ORMs are nice and good to have, but they are too a resource hog if they are not really optimized. It's better to not to have any ORM than having a bad ORM.