This was an excellent year for Nun-db; at first, I thought it would take me a lot of time to get some adoption and bring external users to use it, but it was super quick.

The first external user

At the beginning of the year, the perfect opportunity to get a first external user for Nun-db showed up. When a friend of mine that runs an investing group on Telegram was delivering a new feature to store wallets to check the price at any time, he was using Firebase. So I decided to team up with him to migrate the chatbot to Nun-db I tell that story in the post Migrating a chatbot feature from Firebase to Nun-db. The code with Nun-db turned out to be much simpler than with Firebase because the use case was favorable to Nun-db data models.

My analytics

I started the year adding it to my blog as a secondary analytics tool, made this blog post about it here titled: How to create your simple version of google analytics real-time using Nun-db. It started as a simple online count visitors with a simple connection count. And in the end the year with a fully working page with page views by date, language and users, as you can see in the next print.

I should write about it again, now that it works and it is quite useful to me as today I use it more than google analytics indeed.

Mentoring to drive adoption

One of my ongoing strategies to drive new adopters is to do some mentoring; that is what I did with Pedro Henrique Köhler, former Electrical engineering looking for mentorship to grow as a Software engineer. My mentorship with him in react/js resulted in a blog post called Lessons learned creating a real-time multiplayer Tic-Tac-Toe game with Nun-DB published on Linkedin and it was, in fact, a great success both for the project and for him.

We spent like 4 hours parring on refactoring his code, making it more testable, and later integrating it with Nun-DB. You can check the code here on GitHub and the live version here. This was a great strategy, and I hope to repeat it with more young engineers the following year.

That made the project my favorite project on GitHub with +12 stars (Now it has more than +44 stars).

Going scalable

After these initial activities, I knew it was time to start the structural changes I would like to do. The bigger one of them makes it horizontal scalable. I decided to start with a single primary strategy to make it easier. It led to my most successful post so far A fast-to-sync/search and space-optimized replication algorithm written in Rust, The Nun-db data replication model, where I show my ideas and progress to make it a reality, it also made me show up in the newsletter This Week in Rust that brought so much traffic I started to get worried I should publish more stuff soon. It also showed up in the Rusttimes.

The same feature led to a second blog post with great success, too titled Leader election in Rust the journey towards implementing nun-db leader election that also showed up in the This week in rust 400.

This significant change took a couple of months to get in place and ready to be deployed to the running cluster where all the applications are running. However, it was super impressive to see at the deployment that it only took 2s to sync 778 databases to the replica of two new sets \o/.

Adding replication to the cluster sync in 2s 778 databases

2021-06-14T20:19:50.306804356Z replicating to tcp client in the addr: nun-db-0.nun-db.nun-db.svc.cluster.local:3013
2021-06-14T20:19:52.652748453Z Will snapshot the database org-610

After all these changes, I needed to improve the code and my test tools to make sure we were ready to move faster from now on. Additionally, I am convinced that Nun-db needs to be a leaderless database, making it much more scalable horizontally and easier to run on the edge.

Better testing tools

Improving the tests also led to a new blog post called https://mateusfreira.github.io/@mateusfreira-integration-tests-for-rust-apps-testing-command-line-tools-in-rust/ where I migrated most of the tests that I was using sh to native rust code, and I loved the results you can find this code change in the MR 51.

First issue from an external user

Another significant milestone was the first issue created by a user that I have node idea who they are Issue 54. Even though it was a simple question, and I am not even sure they are using Nun-db, I was super happy to see an external issue with my repository.

Bugs in replication

Not everything is flowers. There were some buts in the replication and distribution. The first one was falling to replicate election fixed here with a simple code change.

The second was a bit more complicated. I called it Mark op log as invalid when keys not safe yet. This one required a much more complex fix. The change was the +311 −44 lines change. And to be honest, I did not love the solution to invalidate the oplog file and require a full sync, but given the cost to keep it fast, it was the best for the moment. I may be returned to the same problem in the future when I am more used to this scale of problems.

Going the leaderless

As soon as I hit publish in the blog post Leader election in Rust the journey towards implementing nun-db leader election, I started to think it was not a good idea to have Nun-db as a single primary database, and in the company I am consulting for we are having big issues to scale the application because Mongo does not have a great way to horizontal scale with more than one node handing writes and read simultaneously. So I immediately start to work to make Nun-db a leaderless database natively, which will lead to a much more scalable solution. I can also run small instances of Nun-db in many places instead of significant instances in a few nodes. This will help us scale and get adoption without needing to inject too much money too earlier. The first step was to plan what I would like to do, and I added an MD file to be my source of truth here. And in fact, that required me to make the replication module much more stable. For example, in the MR 49 where I implemented acknowledge messages from the replication, we now have a much more safe replication algorithm, we may need to pay some CPU and one additional network hop for each message to make sure it is secure. Now next step is to make sure we can detect collisions on changes (One of the expected problems with leaderless databases), for that I am already implementing the versioning in the MRm 50, initially in memory and the next step it to make the same in the disk so we will be able to handle restarts in the nodes and restore in the same versioning as before.

This was one of the most exciting changes I have made this year, and it seems to be the correct way to go. Of course, I have no need for it in the applications using Nun-db in production nowadays, but I think it is essential to have them at least in good shape before driving too much adoption when I will probably spend most of the time I have to dedicate to Nun-db to help other succeed with it.

Observability

Improving observability is an exciting topic because you can quickly see the results in real life. I love to see charts and numbers and see things changing over time.

Initially, when I was writing Nun-db, I used to run unit tests for fun because no one was using the system. Then when I got a few people and apps using it, I used to tails logs and watched to see what was happening to have fun.

Now I can open a Grafana chart and see query time and how many operations it has been processing.

An essential step in that direction was the Prometheus exporter I wrote about in this blog and required two changes in the nun-db codebase in the PR 52 adding some oplog stats and finally in the PR 56 adding the rest of the metrics I wanted and also changing the name of the command to an adequate name the exporter is also open-sourced GitHub.

Once again, it got to This week in Rust, and it is always a great source of traffic, and every time my posts hit it, I few proud.

Current users

I have only a few users so, still possible for me to analyze all users. I have the case lets see how its goals.

E-belle

The first and more prominent user is my own ERP called “E-belle”(In PT-BR), and it uses Nun-db to notify in real-time the users when new appointments are registered for their organization in the application. Every web and mobile user will have a direct connection to Nun-db. So far, it has been a great success for that proposal. I still need to add more features to it tho I onboarded one single use case and moved on once it was working. The integration is still working gracefully too. A new database is created for each company registered in the application on Nun-db to support it. I think this may be a good use case other solutions can use.

It is easy for me to know all the databases E-belle is using with the next command:

nun-db --user $NUN_USER  -p $NUN_PWD --host "https://http.nundb.org" exec "use-db \$admin $NUN_PWD; keys" | tr "," "\n" | grep "org-" | wc -l
799

There are 799 databases in Nun-db only to support E-belle’s outstanding traffic.

Other applications

Stock-bot

The Stock-bot application is the application I spoke about in the blog post (“Migrating a chatbot feature from Firebase to Nun-db”)[https://mateusfreira.github.io/@mateusfreira-migrating-a-chat-bot-feature-from-firebase-to-nun-db/]. It is a closed source application, and I helped code it was the first application, not mine to use Nun-db.

House-scrapper

This is an application that scraps Brasilian sites to find the best prices for houses to buy, a friend of mine is building a business with that and is using Nun-db as its primary database. I never seeing saw this application running since it is a closed app for a few users, but it is by far the most complicated data model we hold, he stores an enormous amount of JSON data, and it has been working pretty well.

Tic-tac-toe

This is the application built by pedrokohler, and you can see its code in GitHub. It is a react application, and its integration with Nun-db is all written as a redux Middleware in this file, I am also showing in the next block of code.

I love this example because it proves that once the application is coded with react and redux, the integration with Nun-db is trivial, the mentorship with Pedrokohlera was to prove that in his small game.

Demo apps

Data-Analysis-demo

This was an experiment I still not ready to make is public, but the idea is to have a real-time collaborative data analysis tool. You put the dataset, share it with the others, debate the data, and apply filters, groups, and aggregations. Part of its code is available (here)[https://github.com/mateusfreira/nun-db-js/tree/master/examples/data-analysis]

Analytics-blog

This is my blog analytics app. I have been improving it slowly for some time. It counts visits and stores data. I use it as second analytics to backup google analytics and see how much traffic I am missing that is blocking analytics, usually. At least 50% of the traffic has been blocked what means I missed great information for a long time. It is a pure javascript written in a day, so I am is not super well thought. The code is also available on GitHub here.

React

I build this sample app using the react+redux todo sample app to show how seamlessly integrate your app with Nun-db. The code is available here, and the blog post about it is (“How to make redux TodoMVC example a real-time multi-user app with nun-db in 10 steps”)[https://mateusfreira.github.io/@mateusfreira-2021-06-30-how-to-making-redux-todomvc-example-a-real-time-multiuser-with-nun-db/]. You can also see it running here.

Vue

The sample of Nun-db integration with Vue js was the first one I wrote is available in the (Nun-db javascript lib example on GitHub)[https://github.com/mateusfreira/nun-db-js/tree/master/examples/vue] probably should write about it soon too.

What was the goal for the year?

In the last year, I guessed my next steps on Nun-db. Here is what happened.

• Helm chart for Kubernetes? I did not implement it. The adoption won’t start with people installing it directly but rather from using an already running instance I am running. With Firebase giving that option, I don’t think I will be able not to do and succeed.
• Improve documentation (a lot) I did improve a bit. I wrote about lots of stuff but probably should write more about it to make it more stable.
• Examples examples examples I implemented several good examples with Nun-db, as I will show in the rest of this post, but I am still not making them available to a broader audience.
• Monitoring In the last month of the year, I did make significant progress. I shared that in my previous blog post about ()[]
• Early adopters? Yep did manage to get some earlier adopters, and it was great. However, it is time to clean the house and onboard new users to make Nun-db more useful for others too.

How will I bring new users?

Still now sure how I will bring new users yet. I have two main ways to go:

1. Make onboarding new users super easy: Create a website to Nun-db where people can register and get a Database and a key and start using it by themselves. 1.1 Pros: that would probably bring much more users soon. 1.2 Cons: Less engaging and less committed to using the Nun-db for the long run.
2. Start picking consulting gigs and delivering Nun-db as solutions. As an experienced Software engineer, I refuse to consult gigs every month. However, some of them are interesting sometimes, and I can see Nun-db being a core component for many problems that companies look after me to solve.
3. Pro: In this case, the users would be much more engaged and committed to staying using Nun-db. I would also have a close relationship with them to help with issues and understand whether Nun-db is helping in their case. I can also add them to my already running environment and make some money out of it.
4. Cons: Much more work to acquire new users, which may lead to only getting a handful of new users for the following year.

The decision comes down to more users, less engaged users, less context, but the small cost of acquisition and no financial reward in the short term. Or fewer users, but more involved users more committed but with a much higher acquisition cost but may also lead to monetary compensation.

There is not a perfect solution. I don’t expect to have much more users now. I want to make Nun-db more scalable first but don’t want to let it get in the unused land. It has been three years since I started Nun-db, and I am really happy where it got now, but I need to make sure it will go in the right direction so it will still be relevant in the following years. I see many opportunities for adoption, and I want to get some of them and prove Nun-db can be a good bet.

Conclusion

Here I am at Fri Dec-31 14:29, finishing the writing of this article while my wife sleeps (she is going to work overnight), and thinking back this year was terrific. It is great to see how much time I spent working on Nun-db this year. Doing open-source is a long-time desire of mine. I am happy to see this project getting this far in time and feature, I still have a lot to learn and deliver to see Nun-db a successful database, yet it is already by far the most successful open source project I have ever started. I have taken part in other more successful projects like Dockbase, for example, but they were other ideas that I believed in at the time. I want to keep pushing Nun-db forward to see where it will get. Next year, I aim to get more adopters with more extensive applications. Nun-db is the core of more applications and brings more technical maturity to the project with my new learnings and experiment.

What to expect for the next year?

• Leaderless, this is already in progress, so it will keep Nun-db infra cheap while we grow and help me learn more complicated algorithms in distributed systems. The big challenge here is to go leaderless safe, and fast. I am sure I will have to sacrifice one aspect, yet leaderless can also bring us close to the infinity scale, which is excellent for my plan.

• Examples, examples, and examples I think good and well-explained examples are what will bring external eyes to Nun-db, so I hope to spend a good time working on working code examples to put out there in this very blog.

• Mentoring I also plan to do at least one more engineering that may help me bring new eyes and blood to the project.

• Use it in my product. I started Nun-db to be a component of a time tracker I was building years ago. I may put it to run again next year, it is, in fact, fully functional now, but I wanted to repay the APIs in elixir before getting it public. I may decide to make it public next year. I am also evaluating releasing a new payment system for my customer in Brazil based on a new technology released last year by the central bank that will have Nun-db as its heart and real-time infra.

There are multiple ways to move forward with Nun-db. Which one will be the best, faster and better time will tell. But, for now, I am happy to close the year as is and prepare to restart tomorrow, Jan 1st, with new energy.

Happy coding 2022, and let’s make the web more open and fast o/.