MongoDB collection format mass modifications

MongoDB is a NoSQL database and one of the main difference compared to relational DB is you can put in your collection documents with different structure. In fact you don’t really care about structure of documents and type of data you are storing.

This is particularly true when you start using your Mongo instance with langage like PHP or JS where the langage itself does not manage data type explicitly.

As a consequence your document format may not be optimized (like storing numbers in String and making search on them later) and once you decide to move your backend to a more structured language like Java you start to have to make this more robust and organized. At this point you may want to reformat your documents to match the right type, eventually update your previous documents to a new version.

This kind of operation can be done “easily” with Mongo scripting language. This post will details 2 of the current structure to manipulate and transform Documents.

Continue reading

Centos 7 – install a mongo cluster with 3 replica set

This post is describing how to setup a Mongo Db database in a clustering mode. I choose to deploy 3 replica set on 3 different bare metal server running centos 7 and to add 1 arbiter to reduce vote problem during a server crash or reboot.

You will find the different steps to make this configuration running, the way to secure it in a vlan and to activate the authentication.

I also added some elements on the way to backup it. Feel free to propose enhancement and links in the comments.

Continue reading

SpringBoot – Mongo / Create a Mutex to synchronize multiple frontends

When deploying a API in a scalable environment we usually try tom implement a stateless solution to avoid parallelism conflicts between the different nodes.

This works well for event management synchronously executed. But in some case these event need to have a part of the work managed asynchronously. It was my was my case when I had to process alarm report potentially generating multiple communications with execution time not predictable. To manage this I decided to process theses operation asynchronously. There are multiple architecture for managing this like Queuing (my preferred) of dedicate asynchronous engine. I’ve in fact preferred another way to do it for keeping my infrastructure more simple : adding in all of my nodes a batch function processing the alarms asynchronously.

The alarm events are written into the database and processed by the first of the batch taking it. This allow to preserve a fully redundant and simple architecture. That said I need to ensure only one of the process will execute this process at a time and potentially split the work to do between all the nodes. For making this I had to create a mutex solution.

This post describes the way to create a Mutex solution based on MongoDB with Spring Boot.

Continue reading

SpringBoot webservices full example

SpringBoot is a really efficient framework for creating webservices and much more. For this reason I’m using it to design my backend applications.

This framework is also well referenced on Internet and many people are downloading it and using it. All of this sounds really good and you will see you can make a webservice in less than 5 minutes following the many examples existing on Internet. As usually in this kind of technology once you have made the classical HelloWorld and university classical practices you have a lot of difficulties to make your first real program coupling different simple use-cases. So As I spent a couple of hours searching solution on Internet, this post will give you a full example of a project getting data from a MongoDB instance to provide a simple webservice.

Continue reading

Access to a remote mongodb with mac OsX client

I needed to access a mongodb instance to manage my collections. I was looking for something like phpmyadmin for mongo but the first tool I found was mongoclient.

This tool is an application for desktop provided for many OS including mac OsX. The quality of the tool is really good.

The mongodb was installed on my server remotely and protected by a firewall so the easiest way to access it was to open a ssh tunnel. Mongoclient is supposed to manage the ssh tunnel in the application but it was not working for me so I had to manually connect the tunnel :

ssh -L 27017:localhost:27017 -p 22 user@mongodb.server.com

Once connected it is possible to create a localhost connection with mongoclient.