Testing is essential for developing good software. It is hard to keep speed and quality on software development without having a great test suite. I particularly like all levels of tests, and I think they are all vital, from the unit test, integration, to end-to-end tests. Each will run faster or slower than the other and cover a more significant part of the stack.

In this post, I am covering the integration tests in rust, using Nun-db as example. I am not covering the basic of tests in rust in this post so I advice you read Testing in rust before going to far if you are not familiar.

My goal with the integration tests is to cover parts of Nun-db where I need to run the database as a separate process and use it as an external user would, I used nun-db command-line tools to test and in the end, I assert the result of the command line to make sure it reply what I expected.

Why isn’t unit tests enough ?

Unit tests help to test one specific case. So, for example, when I call set_key_value, the value should change.

The code above runs super fast in less than 100ms on my local machine. But it covers a small case. It makes sure my function set_key_value is working as I expect.

But what if I have one deployment with three nodes running Nun-db, one being the primary and two others as secondary, and I want to make sure set the value in the primary node and reading from any of the secondary nodes I would get the correct value. This is a much more complicated state that would be pretty hard to cover with unit tests. Here is where integration tests shine, and rust provides some good alternatives for integration tests for command-line tools that help us write this kind of test quite easily.

The test case

In a deployment with three nodes running Nun-db, one being the primary and two others as secondaries. Set the value in the primary node and reading from some secondary and assert for the correct value.

To make that one test, we need:

• Start three nun-db processes in different ports and directories
• Connect them as a replica set
• Execute the command set name mateus in the primary.
• Execute the command get name from both secondaries.
• Kill all processes.

Starting 3 Nun-db processes

To run Nun-db, we need to pass several parameters, e.g., admin user name and password, as well as all the addresses for http, TCP-socket, web-socket, and the replicas-set addresses (The addresses to connect to the other nodes) and pass the NUN_DBS_DIR env var to tell nun-db where to store the data files.

To support that, I used 2 libraries assert_cmd and predicates the following code to my Cargo.toml.

[dev-dependencies]
//...
assert_cmd = "0.10"
predicates = "1"

The code we are going to build in rust would works like the following in bash:

NUN_DBS_DIR=/tmp/dbs nun-db --user $user -p $user start --http-address "$primaryHttpAddress" --tcp-address "$primaryTcpAddress" --ws-address "$wsPrimaryAddress" --replicate-address "$replicaSetAddrs"

Now lets see the same in rust:

The critical point here is the return of the Child so I can control from the callee the process (To kill it, for example).

We need to start the other two additional processes in one single method and return the 3 Child processes to be administrated by the callee. Here I use a rust feature called Tuples. For simplicity I am omitting the code for the methods start_secoundary and start_secoundary_2 but you can read them from GitHub here.

At the end of the test, I wanted to kill all the processes. For that, I created a method to kill the replica sets.

Now I need to execute commands from the Nun-db command line to set and get the values I want.

I use the command exec of Nun-db exec as I present next in code-block where I execute cluster-state in bash.

nun-db -p $password -u $user --host "http://$host" exec "cluster-state";

I also build a helper function to perform the exec operations.

Here I returned an Assert since the process will die as soon as the exec is done, and I will, in all cases, want to validate if the returned output has what I expected.

Finally, we can write the test we want. Watch the comments as I will use them to reference each step of our planned test.

The test above will take at least 5 seconds to run (remember the unit test takes less than 100ms, this one is 50x slower at least), but it covers a much bigger surface, since to this test to pass, the request parser and processors will also have to work as also expected the election and replication engines.

Conclusion

The rust API Command helps interact with command lines tools straightforwardly, and when combined with assert_cmdmakes it easy to create complicated test cases simply.

It has helped Nun-db build some challenging tests, and I bet it may also help you create tests for your project.

Check out the test used as an example to this blog post in action on the Nun-db repository on GitHub here in a complete version, and here are the helpers.