Power up your node.js REST API

Building micro services can be very exciting. The ease and speed of creating them using node.js has changed the way devs think. From requirement analysis to continuous integration, micro services can be developed within weeks and operate quite impressively.
Using node.js you can create very robust apps with asynchronous calls by design. Still though there are some times where you may add up a little more in that and make your application even stronger and harder to collapse on tough situations like an unpredictable request that destroys the process or too much concurrent requests or maybe a dos attack from a hacker.
With this post I want to present some node modules that will help you overcome problems like these.

Module 1) cluster. This module comes in handy when you want to overcome memory limitations and processing power when using a single process app. Apart from that, will give you the ability to respawn dead node processes (like pm2 does).
Useful links:
http://www.sitepoint.com/how-to-create-a-node-js-cluster-for-speeding-up-your-apps/
https://nodejs.org/api/cluster.html

Module 2) express.  It is one of the best modules for creating HTTP endpoints. Some people prefer other modules like restify – and this is quite ok. Just use the one you think it fits more to you. Just keep in mind that in terms of speed express might be the best one.

Module 3) ddos. This module will give you some basic dos protection against malicious users. Probably you will never need it but again why risking it? Note though that ddos module will only protect you against simple dos attacks. If you also want protection against ddos attacks you will need something more. Personally, I believe that this is the work of the load balancer. Those that using Haproxy read this: http://blog.haproxy.com/2012/02/27/use-a-load-balancer-as-a-first-row-of-defense-against-ddos/ .
Useful links:
https://github.com/rook2pawn/node-ddos

Module 4) toobusy-js. This is one of the many implementations of the original toobusy module. What it does? When receiving an http request instead of trying to execute it, it first checks the load of the process. If the process is overburdened it returns a 503 response telling that the server is busy at the moment. So instead of having too many failures with timeouts you will return on some requests service unavailable fast and serve the rest normally.
Useful links:
https://www.npmjs.com/package/toobusy-js
https://hacks.mozilla.org/2013/01/building-a-node-js-server-that-wont-melt-a-node-js-holiday-season-part-5/

Module 5) bunyan. Yes it is a logger but who can live without a logger. Just be very cautious with production environments to only log what is necessary. Also bunyan’s log rotation using cluster is buggy so you should better avoid it. You should anyway let the operating system do the job with logrotate.

For your convenience I have created a github repo here with a template project which uses all the previous modules. Feel free to clone it and use it as your as a skeleton for your new project
Have fun!

Useful links:

https://github.com/mdagis/api-template

Install Node.js on EC2 Linux AMI

Recently I wanted to install  node.js on ec2 linux machine and to my surprise this task turned to be a little more tedious than I thought it would be. The reason for that is mainly because there are many ways to do it each of them with their pros and cons. The only alternative that I found easier to follow was to run the command: "sudo yum install nodejs npm --enablerepo=epel" . Unfortunately the epel repository has a very old version of node so I had to stick with a better option.

In this topic I am going to cover the option to download the binaries and just add the node directory to path.

The steps:

1. Go to https://nodejs.org/dist/ and select the version of node.js that you want. Normally you will need the latest. When I did the installation i used https://nodejs.org/dist/v4.2.1/node-v4.2.1-linux-x64.tar.gz .

2. After selecting the url that you want go to your machine and create a folder to have your node installation (be sure that the owner of the folder is ec2-user, if not change it with chown). In my case I created /opt/ . Then download your package with the command: wget https://nodejs.org/dist/v4.2.1/node-v4.2.1-linux-x64.tar.gz

3. Uncompress the file with command: tar -xvzf node-v4.2.1-linux-x64.tar.gz  and then navigate to created folder. In my case it was node-v4.2.1-linux-x64

4. The last step now is to add the node bin directory to path. To do so just create a new file in /etc/profile.d/node.sh . Edit the file and add the line: pathmunge /opt/node-v4.2.1-linux-x64/bin/ . Save the file and then run: source /etc/profile to refresh profile and you are ready.


This should normally be enough to have your node installation up and running.

How to survive from Node.js and JavaScript callbacks

Many people coming from Java ecosystem to Node.js (like I did) are facing problems to adjust with the new platform philosophy since the basic principles of JavaScript itself are way different from the other well-known Object Oriented Languages.

Personally I think the biggest problem that I had was to understand the nature of functions in JavaScript and subsequently the way callbacks behave.

The truth is the first time I read about them I thought that I understood them. Unfortunately understanding them was not enough because all those years of experience turned into a barrier for me.

I will try to present the conceptual problem that I had in the beginning and also what I did in order to overcome it.


THE PROBLEM


Let’s say that you want to make a simple program that does 3 steps.

Step1. It reads a file.

Step 2. It accepts a prompt from console with username and password.

Step 3. It sends an email connecting to the mail server using the given credentials and having as a body the file contents read from step 1.


Normally with a conventional OO language you would create 3 functions each one doing one step and call them all three from the body of a third with the order 1, 2, 3.

Most programming languages are doing things synchronously by default which means that step 2 will be executed after 1 and 3 after 2.

This is a sample of something we could do in Java. For the sake of the example please ignore the static and the lack of OO architecture.

public static void main(String[] args) {
        readfile();
        getCredentials();
        sendMail();
}

    public static void readfile(){}

    public static void getCredentials(){}

    public static void sendMail(){}


Unfortunately (and fortunately) the following work flow would not work in Node.js. When your code reaches the step 2 the result of step 1 will not be ready and on step 3 the result of step 2 will not be ready either. The reason is that most modules of node.js work asynchronously and that requires a more delicate approach.


THE SOLUTION


As a matter of fact there are 3 solutions.

A) Using callbacks properly

B) Using nested callbacks

C) Using Promises.

I will skip solution B and C because B is not a decent solution since the readability of the code would be terrible and C because it is a different topic.

I will try to focus on solution (A).


As we saw on the previous code we had 3 functions doing the work and one more calling those 3 with a sequence.

In Node.js you will only need the fourth function to trigger the Step 1 and passing as a parameter the function that knows what to do after Step 1 ends.

Note that the fourth function - let’s call it trigger function – will not do anything more than calling step 1. The callback function 1 will be responsible to acquire the result of step 1 and trigger step 2 passing a new callback to step 2. Call back function 2 will be called by step 2 after finishing and that will call step 3.


Here is a diagram showing the procedure:

Here is some ample code:


function trigger(){
    readfile(callBack1);
}

function callBack1(){
    getCredentials(callBack2);
}

function callBack2(){
    sendMail();
}

// Step 1

function readfile(nextStep) {
    fs.readfile(path, function (error, text) {
        nextStep();
    });
}

// Step 2

function getCredentials(nextStep) {
    prompt.getCredentials(value, function (error, result) {
        nextStep();
    });
}

// Step 3

function sendMail() {
    transporter.sendMail(mailOptions, function (error, info) {
        //done
    });
}

Implement Simple Websocket with Node.js

It's been almost two months since I wrote anything in here and before moving forward in topics I will try to add something more with the web socket technology.

As I have written in the last topic here I 'd like to present a very simple way to create Websocket server. This was implemented with Java EE7.

This time I wanted to try to use Node.js and to my surprise things got much easier.

The only thing I had to do (apart from installing Node.js) was to install the module ws with the command:

npm install ws

That's it! Now the following code is a sample of doing the whole thing:

... and in case you haven't read the previous post here is the html5 client to test it:

JavaEE7 Websocket Example

In this tutorial I will try to present a very easy way to create a minimal Websocket Application using Java EE7.

The specific web application will have a working  Websocket Server deployable on every Java EE7 container plus a basic HTML5 page that will act as a client just to test your work.

Let's get started then. The first thing that you will have to do is to create a web application. I used Netbeans IDE but you can do exactly the same thing using your IDE of preference or even just using maven and a text editor.

For those that use Netbeans go to -> New Project -> Maven -> Web Application. Give a name to your application and Select Java EE 7 Web.

The whole Websocket server will consist of 4 classes, the Message Class, the Encoder Class, the Decoder Class and the ServerEndpoint Class.

Here is the sample code for all 4 classes plus the HTML client:

RESTful API testing tools comparison

Tool name	Description	Testing	CI integration	Total
SOAP UI	3	2	3	3
REST-assured	5	1	2	3
Postman	3	2	5	4
frisby.js	5	4	2	4
RAML	1	4	5	4
Runscope Radar	3	3	5	3
Dredd	1	3	2	2

Read the analysis here

... or read all 6 chapters here

HTML Barcode Scanner

Have you ever tried to type in a long number on your cell phone or simply enter the number of your membership card into a web application?

This might be a time consuming and error prone task which can be avoided by using barcodes.
This is nothing new. Many solutions exist for reading barcodes with a regular camera, like zxing, but they require a native platform such as Android or iOS.

Building your own application in two different platforms its not the easiest thing to do because it requires a lot of effort to develop and maintain in parallel. Of course there are cross platform mobile SDKs that give you the flexibility - and some of them are quite exciting like Phonegap/Cordova.

The problem is that you will need to learn and invest on a new API which might lead to unfortunate results.

If let's say you have build and invest a lot of know-how and source code in a technology that will not evolve, then your company will be exposed to a big risk.

If we had to gather up all the requirements needed in order to build an application that uses the device camera to scan barcodes, we would probably come up with the following aspects:

- It must be able to use the mobile's/tablet's camera to scan barcodes quickly enough.

- It must be developed in one platform (at least the 95% of it) in order to avoid duplicate work and maintenance.

- It must be developed with a widely approved technology that minimizes the possibility to be discontinued or having everlasting bugs.

After thoroughly examining the related technologies and the requirements that I had, I concluded on 2 different solutions and each of them had it's own strengths and weaknesses.

Solution A. Pure HTML5.

This is a very promising solution since it is all implemented in HTML. The HTML5 specification gives to the developer a very powerful toolset that can even gain access to the device camera. Hopefully there are people that have took advantage of it and have already implemented a javascript library that actually does the work.

The best implementation I have come across so far is made by Christoph Oberhofer and it is named quaggaJS. The specific library although it feels a bit immature  works quite impressive on specific devices and browsers. Those that are interested may have a look on http://serratus.github.io/quaggaJS/ which is the original web site. You can also read a nice article in Mozilla about it here

https://hacks.mozilla.org/2014/12/quaggajs-building-a-barcode-scanner-for-the-web/ .


After testing it for a while on android I realized that it still has some issues important enough to be a barrier for production environment.
Some of them are:

- It fails to use camera's autofocus on some browsers. On Android I managed to make it work in Firefox but I had no luck with chrome.

-  It's a bit slow on scan. The main requirement of the project is to make it easier to scan than typing. If it takes more time to scan the people will prefer to type.

Solution B. HTML & Native app hybrid.

This solution is a bit hard to grasp the first time you hear about but it makes sense after a while. The basic idea is to write the whole application business logic in pure HTML and leave only the scanning part for the native app. Then in order to activate the scan ability of you phone you can click a link of a URL. The specific url is hooked by a native android/ios application where it's only scope is to scan and return the value to the caller.

If you don't know how to open a native app from browser in android you can read here whereas  those that want to do in ios should read here.

The tricky part here is returning the scanned value back to browser since there is an obvious isolation between them. Unfortunately the cookies cannot help you since the native application's cookies cannot be accessed by the caller browser. So what is left?

The solution that I though was to have browser open the app and at the same time passing a unique token code. Then, after the native app scans the barcode it will make an http post to a web service passing as parameter the token and the value scanned. After that, the mobile app will close automatically and return focus to the caller browser. At the same time the browser should poll the same web service and actual ask it if there is available any value with the specific token.

Here is a diagram describing the process flow.

With this technique, when the value arrives from the mobile app to the web service, the web application will learn about it and display on GUI.

You might think that this is too complicated but we have actually managed to implement a full operational proof of concept with a colleague of mine in a few hours.

Here is a commercial app that more or less does the same.