Integration tests in rust a multi-process test example
Testing is essential for developing good software. It is hard to keep speed and quality on software development without having a great test suite. I particularly like all levels of tests, and I think they are all vital, from the unit test, integration, to end-to-end tests. Each will run faster or slower than the other and cover a more significant part of the stack.
In this post, I am covering the integration tests in rust, using Nun-db as example. I am not covering the basic of tests in rust in this post so I advice you read Testing in rust before going to far if you are not familiar.
My goal with the integration tests is to cover parts of Nun-db where I need to run the database as a separate process and use it as an external user would, I used nun-db command-line tools to test and in the end, I assert the result of the command line to make sure it reply what I expected.
Why isn’t unit tests enough ?
Unit tests help to test one specific case. So, for example, when I call
set_key_value, the value should change.
The code above runs super fast in less than
100ms on my local machine. But it covers a small case. It makes sure my function
set_key_value is working as I expect.
But what if I have one deployment with three nodes running Nun-db, one being the primary and two others as secondary, and I want to make sure set the value in the primary node and reading from any of the secondary nodes I would get the correct value. This is a much more complicated state that would be pretty hard to cover with unit tests. Here is where integration tests shine, and rust provides some good alternatives for integration tests for command-line tools that help us write this kind of test quite easily.
The test case
In a deployment with three nodes running Nun-db, one being the primary and two others as secondaries. Set the value in the primary node and reading from some secondary and assert for the correct value.
To make that one test, we need:
- Start three nun-db processes in different ports and directories
- Connect them as a replica set
- Execute the command
set name mateusin the primary.
- Execute the command
get namefrom both secondaries.
- Kill all processes.
Starting 3 Nun-db processes
To run Nun-db, we need to pass several parameters, e.g., admin user name and password, as well as all the addresses for http, TCP-socket, web-socket, and the replicas-set addresses (The addresses to connect to the other nodes) and pass the
NUN_DBS_DIR env var to tell nun-db where to store the data files.
To support that, I used 2 libraries
predicates the following code to my
[dev-dependencies] //... assert_cmd = "0.10" predicates = "1"
The code we are going to build in rust would works like the following in bash:
NUN_DBS_DIR=/tmp/dbs nun-db --user $user -p $user start --http-address "$primaryHttpAddress" --tcp-address "$primaryTcpAddress" --ws-address "$wsPrimaryAddress" --replicate-address "$replicaSetAddrs"
Now lets see the same in rust:
The critical point here is the return of the
Child so I can control from the callee the process (To kill it, for example).
We need to start the other two additional processes in one single method and return the 3
Child processes to be administrated by the callee. Here I use a rust feature called Tuples. For simplicity I am omitting the code for the methods
start_secoundary_2 but you can read them from GitHub here.
At the end of the test, I wanted to kill all the processes. For that, I created a method to kill the replica sets.
Now I need to execute commands from the Nun-db command line to set and get the values I want.
I use the command exec of Nun-db
exec as I present next in code-block where I execute
cluster-state in bash.
nun-db -p $password -u $user --host "http://$host" exec "cluster-state";
I also build a helper function to perform the exec operations.
Here I returned an
Assert since the process will die as soon as the exec is done, and I will, in all cases, want to validate if the returned output has what I expected.
Finally, we can write the test we want. Watch the comments as I will use them to reference each step of our planned test.
The test above will take at least 5 seconds to run (remember the unit test takes less than 100ms, this one is 50x slower at least), but it covers a much bigger surface, since to this test to pass, the request parser and processors will also have to work as also expected the election and replication engines.
The rust API
Command helps interact with command lines tools straightforwardly, and when combined with
assert_cmdmakes it easy to create complicated test cases simply.
It has helped Nun-db build some challenging tests, and I bet it may also help you create tests for your project.
- Stop procrastinating and just fix your flaky tests, it may be catching nasty bugs
- An approach to hunt and fix non-reproducible bugs - Case study - Fixing a race conditions in Nun-db replication algorithm in rust
- Nun-db the debug command
- Keeping up with Nun-db 2021
- Writing a prometheus exporter in rust from idea to grafana chart
- Integration tests in rust a multi-process test example
- Leader election in rust the journey towards implementing nun-db leader election
- How to make redux TodoMVC example a real-time multiuser app with nun-db in 10 steps
- A fast-to-sync/search and space-optimized replication algorithm written in rust, The Nun-db data replication model
- NunDb How to backup one or all databases
- How to create your simple version of google analytics real-time using Nun-db
- Migrating a chat bot feature from Firebase to Nun-db
- Keepin' up with Nun-DB
- Going live with Nun-DB