Keepin' up with Nun-DB
It has been a year since I open-sourced Nun-DB. That happened when https://www.teknisa.com invited me to give a talk about real-time processing. At that time, I worked on Nun-DB for over six months, and I thought it would be an excellent opportunity to announce the project. It was called “Freira-DB” back then, and the goal of this post is to update the Nun-DB and what has happened during this one year.
What was the goal for the year?
The goal for the year in my yearly presentation to myself was “Release the Nun-DB 1.0 and use it in e-belle/gerir-me” I still if you do not know E-belle/Gerir-me are the products of my company (http://www.vilarika.com.br) they are SaaS where we provide a management system for small business in Brazil (That is why the side is all in PT-BR). For a long time, we had the necessity of using a real-time database. Still, I never wanted to use Firebase or other alternatives, so my goal was to introduce my new database to solve that problem. I needed it to be stable before doing that (cause you to know the product is used, and I did not want to cause downtime with the new changes). Before the end of this post, I will share how this plan went. If you cannot wait to see it, go to the “Going live with Nun-DB” subtitle.
The first big thing that has happened was the renaming from “Freira-DB” to “Nun-DB” I loved the Freira-DB name, but talking to people out of brazil and it was hard for them to get the name and even was hard pronounce it. So I decided to take a advise from a friend @Iucasazevedo and rename it to Nun-DB, more straightforward, cleaner, and much easier for the people to get the same name and made simple for the people to get it right. After renaming, it was time to start trying it out in some places other than in my examples, so I decide to make an NPM package out of the JS code I had made already.
The npm package and the JS version
The fist public lib I publish was to npm(https://www.npmjs.com/package/nun-db) that for me was clear the first step, once most of the use cases I had was in the front end, and an npm package was naturally the decision to start, I also needed a scala lib so I could integrate my backend (pure scala) with Nun-DB but it still not ready to publish, so I keep it only internal ( Let me know if you need I can work and get it out there), as I think the scala community may not be so interested in this kind of integration at the moment.
After getting it to NPM, I wanted to make it “deployable,” so I decided to build the Docker image and get it out there.
Release docker image
One of my big goals was to create an open-source alternative to Firebase (the database). I want to be just as simple as it, but 1: Free, 2: Open Source, 3: Small and direct. A docker container is a way lots of databases providers make their versions available to the community, which seemed the correct direction. By the time I did the talk back in 2019, I had a docker image already running, but today’s one is 30x smaller and uses 80% less memory for the same data, so I didn’t even consider that a valid docker version at that time.
So there was I with JS version running and an easy runner docker, so what is next? Get it to my own company’s product needs usage. The first real-world requirement came to life when I would not be able to use the WebSocket or TCP Socket communication seems less because of my product’s legacy code and the absence of great support for such protocols in the pre-existent code, so I came up with new http drive.
Implement http drive
Before going live with Nun-DB, I needed to add the http drive, as of the integration (with my legacy scala system), and it did not have native support to WebSocket, the http connection interface was needed to do it.
We needed to add passwords for different databases and make it possible to hide the admin password only in the backend. Those changes were introduced after I did the initial model to integrate with my SaaS solution. At first, we required all commands to auth with the admin password. That was not a good idea, mode commands can require only a DB auth to execute, and we can save the admin auth from being used only from the backend to the Nun-DB engine. Therefore one user cannot disturb others (by calling the Nun-DB API manually). This helped a lot to make Nun-DB easy to make safe.
Initially, Nun-DB was all on memory, but soon after starting the POC, I saw I would need to have some more durable data because I would restart the process often (while doing some changes on the fly). My idea for that was to add a command to snapshot the database to the disk with a command. Therefore, I would not need to worry about going to the disk for each command but instead only when the DB user wants the data to be stored in some hard disk to be restored if needed.
Going live with Nun-DB
The big thing to the year was going live with Nun-DB in gerir-me. All happened when I release a new mobile app that the users use to control their commission and calendar individually. Of course, the calendar needs to be real-time. That was the perfect integration point. I put out a blog post how it was implemented on here
Add command lines utilities
It turns out that to work in the day to day with Nun-DB, I needed a command-line tool I could use to save time, I started to create
sh files to do stuff, and so I decided to make at last a small subset of command built-in, so any new user can use it for the same things I do.
For that, I have come up with the following commands.
There were only a few bugs I needed to deal with until now. I credit this to Rust been such a good language and environment to create stable software, the one I had was fun to fix.
1) Database stop responding to the http request. That was the only time I had an unexpected behavior, and it turned out to be a bad error handling. (At some point, I even want to go back end get rid of all the unwrapping of my code) here
2) Using too much memory here
For a js developer with more than ten years of experience, writing this thing for the first time and going live with it in my real business and only having a few problems were quite surprised, to say the least, after one week of it running, I was looking at the logs and guessing I had forgotten to release the new version with Nun-DB included or that no users were using the latest version, I was quite a positive feeling when I noticed that it was running all the time. There was no fatal, no error, and no user has reported performance problems so far (and at that time, there had been +800 different users that had used that piece of code).
It sounded clear to me that Nub-DB needed to be horizontal scalable at the heart if I want to get anyone to use it, that is not only me. Horizontal replication can help me get high availability of what is vital to anyone operating in a real-time database.
I decided to implement this feature as soon as I start using it in my personal SAS product, it is not a must-have to my product itself, but it is a nice to have and would be again from the ground up the other services I use.
It took me more than three months to make it “talk,” there are several problems when you decide to implement a horizontally scalable database. How to create a cluster? Where do you write goals to? Where reads goals to? And many others. There are great books that help with the decisions, and there is no perfect approach to any implementation. Leader election is a big enough problem that will take too much time if you don’t take a pragmatic approach.
I have recently merged to branch with horizontal replication (even tho it is not fully implemented yet). I think I have enough that I can start using it in my own product and start fixing the problems I find when using it.
I need to improve the fail tolerance and implement some operation log (current operation only replicate data to the live nodes when the change occurred). This implementation must not be that hard once we don’t need to store the operation itself, only the keys that changed and use the primary as the source of truth and send the data to the new node once it comes online (soon, I will be working on this :) ).
Today, we can replicate data horizontally, which is already colled, and help with the initial proposal of making it possible to update without downtime.
What is next?
It is always hard to say exactly what is next when we build something like Nun-DB (an experimental project), but I can try to predict what is next to come. I intend to make Nun-DB became more popular and get others to use it, and help me to make it better. Therefore a few things are needed. Next, I am presenting some of my take on the next steps to Nun-DB.
Helm chart for kubernetes?
Deploying any distributed database is complicated, an helm chart would be a great way to simplify it and not scary people that wanna try it in many different ways.
Improve documentation (a lot)
Documentation may be boring to do, but it is such an important thing we all need to get used to doing it. We build things for others to use and to make it successful, and we need to make it so they can understand how to use it, and how to set it up and how to try it, and how not to try it. Today Nun-DB, documentation is a basically nil. I think next year will be basically about working on examples, documents, tries, helping people to try it out in many ways possible.
Examples examples examples
Working on examples, I think, will be the next big thing. I want to show how Nun-DB can be used in many different problems and solutions. Therefore the best way to do it is by doing it. I have already done it to react, vue, and node.js, but the example is never enough. I love learning from examples because it quickly allows me to see what I want to experiment in action with some good portion of code.
Observability is such a big thing for any infrastructure solution, and I bet it will be for Nun-DB. I care a lot about metrics and performance improvements; therefore, I want to instrument Nun-DB much more to become observable in many different ways, maybe implement a Prometheus collector to store some metrics. I would call it a success if so.
Let’s see, and maybe I can bring someone building something new to try it out. Perhaps I can give some consulting and deliver it as part of the infrastructure. I am not sure how it is going to happen yet. I know I want to get there. Soon I will figure out how.
Today I have an cluster running on linode where I can easily add instances and start bringing some users to see how it scales, but this is subject for a future post, how do we make this simple DB a platform others can simple use trust and scale.
- Leader election in rust the journey towards implementing nun-db leader election
- How to make redux TodoMVC example a real-time multiuser app with nun-db in 10 steps
- A fast-to-sync/search and space-optimized replication algorithm written in rust, The Nun-db data replication model
- NunDb How to backup one or all databases
- How to create your simple version of google analytics real-time using Nun-db
- Migrating a chat bot feature from Firebase to Nun-db
- Keepin' up with Nun-DB
- Going live with Nun-DB