Building microservices architecture with Node.js and Moleculer

How do we classify IT projects?

There are many different ways of dividing IT projects and the classification process should be done at the very beginning, preferably before a developer or architect work starts. Projects can be long (up to several years) or short (up to a few months), they can be made for a startup where the fastest possible entry into the MVP phase helps see how the product performs or for an enterprise client where time and flexibility matters less than stability, scalability, high reliability and well-chosen system architecture allowing for the long-term development by often dozens of programmers.

Another criterion for the division of the IT project is the number of team members, they can be one or two or maybe a dozen or even several dozen. If many programmers are involved in the project, it happens that not everyone works in the same location, it can be a group of freelancers working remotely or several groups of developers working in different locations and even time zones. You probably wonder what all this IT project qualifying process has in common with the title of the article. Let me explain - microservices. 

What are microservices?

Microservices are simply… services. “Micro” in its name doesn’t mean it has to contain a small piece of code lines or it takes little time to write it. The size of each of the services depends mainly on the specification of the problem that a given project solves - it is important that the service operates within one context and fulfill only one role.

The number of programmers or teams has no influence on it. One team/programmer can deal with several services, it is only important to try to develop individual microservice only by one team to avoid all possible complications similar to those found in monolith (if it is required it probably means that this service should be broken down into smaller ones). For example, microservices for an online store can be responsible for:

• user account management, orders, products, recommendations, reports for sellers - singled out here according to the business context;
• sending e-mails or push notifications - separated from others due to the specific operational functionality.

What do you need to build microservices architecture?

Personally, I think that only the teams that are aware of the consequences should set to work with microservices and only when their choice is dictated by a detailed analysis of the project, which should meet at least some of the following features:

• the team has at least one experienced architect familiar with terms such as Event sourcing, CQRS, Saga or Bounded Context;
• the team has at least one experienced dev-ops, who can cope with the maintenance of few or even several dozen microservices and will be able to manage them;
• has an additional initial budget for more accurate development of the architecture, the entire system monitoring structure or configuration of orchestration;
• long development time of an application is expected;
• many developers are involved in a project - of course, microservice architecture can be used even in one-person projects but will perform even better when developed by a group of programmers;
• good preparation for long-term development is more important than short time for releasing an MVP version;
• the project involves many fields of IT which increases the chances of using different technologies and programming languages ​​(eg Gateway API in node.js due to asynchrony, graphs generating in Python due to the large library, recommendation system in C ++ due to the very short reaction time required);
• It’s not a typical CRUD application - if in the process of defining commands and actions plenty of them are UPDATE_ENTITY,  DELETE_ENTITY, CREATE_ENTITY, GET_ENTITY, in most cases it makes no sense using microservice architecture -  some popular MVC framework will perform better, eg Django.

That’s a lot of things to consider, but the truth is that in particular cases it’s really worth it. Microservices make adding new features much easier and faster than in monolithic architecture and enable using different languages for each service to better suit its needs. And if you are not able to meet most of the mentioned requirements, the good news is that microservices projects are pretty much made for outsourcing so you can fill the expertise gaps with a help of external specialists.  

Microservices and communication

Communication is the inherent characteristic of microservices. If you think that the way of communication between services is easy (such as in a monolith between modules or other forms of sharing application logic), you’re wrong. The world of timeouts, lost packages, event queues is waiting here for you. You should prepare yourself for tedious broker configuration and struggle for milliseconds in network communication (if we have many microservices depending on each other, they can generate really high network traffic, which as you know has a significant impact on the system's response time).

Ok, you'll probably think why he writes about it at all if there are always some problems with microservices - but it's not like that. Microservices, like everything, have their advantages and disadvantages, I could as well write a quite large list of monolith defects. It is important to know these defects, be prepared and take them into account when choosing the architecture for our project. And now it's time for our second guest.

Is Node.js good a choice for microservices?

We're in 2019, if you're interested in programming and you have not spent the last 10 years in a coma, you've probably heard something about Node.js. Let’s have a short reminder and consolidation because I often meet with a bad understanding of the topic. Node.js is not a programming language but a runtime environment that interprets and executes the code and has much more features but it's a topic for a separate article.  

So, back to the main topic, is Node.js a good choice for microservices? The answer is obvious - it depends :). As I mentioned before, in the world of microservices we have an unlimited choice of different technologies and languages ​​so we should choose them according to the needs depending on the preferences and skills of the team and the problem that the service should solve. So what do you need to know about Node.js and what are the pros of it? Here's a short list:

• asynchrony, which will be useful for API Gateway and other microservices strongly based on I/O (input/output) operations,
• relatively fast startup of applications running on it - it is useful when a quick restart of the service or dynamic scaling of the number of instances of a given service is required, depending on the demand;
• low demand for the operative memory - maybe it’s some exaggeration, because increasing memory is not a big deal but when our application consists of dozens of microservices, and each of them has several instances, the difference in memory can be significant, especially in times of cloud computing, where every gigabyte has to be paid extra;
• TypeScript - who doesn’t like this language? Native Node.js uses javascript, which is... everyone knows how it is to work with it. Fortunately, we can use TypeScript which will then be transpiled to javascript. 
• fast development, which is both an advantage and a disadvantage (lack of backward compatibility); however, there are more advantages than disadvantages. 

Ok, so we’ve just covered the theory. Time for some coding!

What is Moleculer?

Moleculer is a framework for writing microservices in node.js. Thanks to it, we do not have to worry (too much) about operational matters and we can focus on what generates money, that is business logic. It is relatively easy to use, efficient as demonstrated by the local and remote request benchmarks, and actively developed, which proves almost daily commits visible on the official repository. The main features of this framework are e.g. support of event-driven architecture with balancing or built-in caching solution. If you are interested in more details about those features, check the Moleculer official docs. 

And now I will show you how we can make microservices communicating with each other using Moleculer avoiding dealing with the entire transport stuff. The whole project will be based on the docker. The application will be very simple and its goal is only to present the basic possibilities, its business logic has been reduced to a minimum. It consists of 3 microservices:

• gateway - as the Gateway API;
• movies - responsible for the logic associated with films;
• email - used to send emails.

1. At the beginning we generate a new node.js package:

npm init

2. Install the required packages:

npm install --save moleculer amqplib express body-parser

It is a good practice to block all the versions of installed packages in package.json, thus avoiding problems with different versions of libraries.

3. We create the 'services' directory and in it a separate file for each microservice:

mkdir services && touch email.service.js && touch gateway.service.js && touch movies.service.js

4. Create Dockerfile in the main directory: 

FROM node:10.15-alpine
RUN mkdir /app
WORKDIR /app
ADD package*.json /app/
RUN npm install
ADD . /app/
CMD [ "npm", "run", "start" ]

5. It's a good idea to add .dockerignore with the node_modules/ entered to avoid copying it to the image.

6. We add a .env file that will be used to load environment variables in each of our microservices:

LOGLEVEL=info
TRANSPORTER=amqp://rabbitmq:5672
SERVICEDIR=services

TRANSPORTER contains information about the selected protocol and the tool used for communication, in our case it is RabbitMQ.

7. We create docker-compose.yml and define our services in it: 

version: '3.7'
services:
rabbitmq:
image: rabbitmq:3.7-alpine
gateway:
build:
context: .
image: service-gateway
env_file: .env
environment:
NODEID: "node-gateway"
SERVICES: gateway
PORT: 3000
ports:
- "3000:3000"
depends_on:
- rabbitmq
email:
build:
context: .
env_file: .env
environment:
NODEID: "node-email"
SERVICES: email
depends_on:
- rabbitmq
movies:
build:
context: .
env_file: .env
environment:
NODEID: "node-movies"
SERVICES: movies
depends_on:
- rabbitmq

It is important to define the environment variable SERVICES having the name of the corresponding microservice in each of the services (with docker compose). Thanks to this moleculer runner will know which microservice to run.

8. In the main directory, create the file moleculer.config.js containing the project configuration:

"use strict";
const os = require("os");
module.exports = {
nodeID: (process.env.NODEID ? process.env.NODEID + "-" : "") + os.hostname().toLowerCase(),
// metrics: true,
// cacher: true
};


In this file, we can set up an automatic cache or, for example, collect metrics and much more.

9. For the project to work we need to add a command to our `package.jsona '. In 'scripts' we add “start”: “moleculer-runner”

Now, with the docker-compose up command, we can launch our project. Of course, none of the services will work because we have not implemented any logic yet.

So let's start with the 'gateway' microservice. In the gateway.service.js file, we define our service by exporting an object according to the scheme (schema) containing the following fields:

• name - the name of our website;
• version - version of our service (frameworks supports versioning);
• settings - an object containing global variables for the service;
• actions - here we define our public methods that can be called by external services;
• methods - allows defining internal methods that are not available outside;
• events - here we define handlers that are executed when an event occurs.

It is worth mentioning the service lifecycle methods such as:

• created - called when the site instance is created;
• started - called when the broker starts all local services;
• stopped - called when the broker stops all local services.

With this knowledge, we can implement our microservice ‘gateway’. We add `name`, in`methods` we define routing and controllers known from express.js and in the created method we initialize our server. The whole code looks like this:

"use strict";
const express = require("express");
const bodyParser = require('body-parser')
module.exports = {
name: "gateway",
settings: {
port: process.env.PORT || 3000,
},
methods: {
initRoutes(app) {
app.get("/movies", this.getMovies);
app.get("/movies/:id", this.getMovie);
app.post("/movies", this.createMovie);
},
getMovies(req, res) {
return Promise.resolve()
.then(() => {
return this.broker.call("movies.listAll").then(movies => {
res.send(movies);
});
})
.catch(this.handleErr(res));
},
getMovie(req, res) {
const id = req.params.id;
return Promise.resolve()
.then(() => {
return this.broker.call("movies.getById", {id: id}).then(movie => {
res.send(movie);
});
})
.catch(this.handleErr(res));
},
createMovie(req, res) {
const payload = req.body;
return Promise.resolve()
.then(() => {
return this.broker.call("movies.create", { payload }).then(movie =>
res.send(movie)
);
})
.catch(this.handleErr(res));
},
handleErr(res) {
return err => {
res.status(err.code || 500).send(err.message);
};
}
},
created() {
const app = express();
app.use(bodyParser());
this.initRoutes(app);
this.app = app;
}
};

Everything looks like a standard application in express.js, but the difference is to call methods from another service, in this case 'movies'. this.broker.call("movies.listAll") returns the Promise object containing the result of the `listAll` action from the microservice 'movies'. Everything happens automatically, the developer doesn’t have to worry about how these data are sent. If we want to pass some parameters to the external method, as in the "createMovie" method, we add an additional parameter to the broker.call method containing the object with these parameters.

Microservice ‘movies’ contains the following implementation: 

"use strict";

const movies = [
{id: 1, title: 'Sharknado'},
{id: 2, title: 'Roma'},
];

module.exports = {
name: "movies",

actions: {
listAll(ctx) {
return Promise.resolve({ movies: movies });
},
getById(ctx) {
const id = Number(ctx.params.id);
return Promise.resolve(movies.find(movie => movie.id === id ));
},
create(ctx) {
const lastId = Math.max(...movies.map(movie => movie.id));
const movie = {
id: lastId + 1,
...ctx.params.payload,
};
movies.push(movie);
this.broker.emit("movie.created", movie);
return Promise.resolve(movie);
}
},
};

Of course, in a real app, our movies would be stored in an external database. This line is a novelty here: 

this.broker.emit("movie.created", movie);

In this way, we send to our system an event "movie.created" with payload containing information about the newly created movie. Our third 'email' service listens to this event and can react accordingly. The big advantage of this approach is that the operation performed after receiving the event will be done outside of the request-response cycle. The implementation of the 'email' service is as follows: 

"use strict";

module.exports = {
name: "email",
events: {
"movie.created": {
group: "other",
handler(payload) {
console.log('Recieved "movie.created" event in email service with payload: ', payload);
}
}
},
};

In this case, the only thing that this microservice does is write to the standard output information about the received event, while in a real application, it could, for example, send an email to people who could be interested in the new movie.

Is it worth to build microservices with Node.js?

As you can read here, building microservices with Node.js and Moleculer is worth your attention. In this short tutorial, we’ve created a basic app, but the main goal was to acquaint you with the microservices features and the way they communicate with each other. And remember - it is crucial to always be aware of their pros and cons and take them into account before starting a project. Take some time at the beginning to choose the right architecture and avoid all the problems that were presented in the introduction.