End-to-End command line tool testing with BATS and (auto)expect

Having a good test suite is important for any software artifact. The same is true for cli tools, especially if these are meant to be used by devops/sre teams in the context of handling critical infrastructure.

In Workable, we have recently started to consolidate all of our previously dispersed tools under an umbrella utility written in go. The purpose of this article is not to talk about how to write such tools in go(there are plenty of excellent tutorials out there for this use case) nor to advocate for the specific language, but rather to discuss about our approach for end-to-end testing that is language-agnostic and can be used by any cli tool.

The main problem that we are trying to tackle is the creation of a consistent/reproducible e2e testing flow for cli tools that

a) tests the final deliverable exactly as is going to be used (i.e. through its invocation from the shell) and not by direct invocation of any of its functions or methods

b) handles declarative test scenarios where user interaction is needed

c) make the process repeatable and language agnostic

The usual stuff

When designing the testing strategy for new software the common approach is to follow the testing-pyramid, that describes what type of tests are needed and how the tests should be proportioned (see image below).

For our tool each new functionality always comes along with the first two types of tests:

- unit tests: these are dedicated tests that validate proper functionality of specific units. In the context of go the definition of unit is a bit debatable, in the sense of in most languages a unit usually refers to a class or a function. In go however the scope may be a bit more loosely defined, and so a unit may be a function, struct, method (see more on this recommended reading) or even as Bill Kennedy states, an entire package.

- integration tests: these are the suites that test more than one unit at the same time (that can extend the scope of testing beyond a single package); we also run integration tests by validating the proper functionality of our cli against external integrations, i.e. systems that the program uses to run properly (in our case some examples is out VCS provider — Github and our k8s clusters)

What we were missing and were dedicated on finding a way around, was a way to be able to test our program from the end user’s perspective, i.e. by simulating as closely as possible the way the cli will be invoked when it will actually be in use. This helps spot issues and bugs that otherwise are not easy to identify, given that neither unit nor integration tests perform on this level.

In our scenario(s), and given that we are working on a compiled language, we need a reliable e2e testing flow that:

1. compiles the code

2. executes the commands by either:

• passing several flags (non-interactive execution)

OR

• interacting with the binary (interactive execution)

3. validates that the program

• ran with success

AND

• its output were the expected one

BATS and (auto)expect to the rescue

Our testing method leverages 2 tools:

• BATS (Bash Automated Testing System): BATS is a testing framework based on (what else?) bash; quoting the source:

It provides a simple way to verify that the UNIX programs you write behave as expected. Bats test file is a Bash script with special syntax for defining test cases. Under the hood, each test case is just a function with a description.

• expect: expect. is a tcl-based utility that supports the automated execution of programs that … expect input from the user through the terminal; this is done through the composition and execution of a script that outlines the sequence of prompts and the inputs automatically provided to them.

A BATS test

This is by no means a BATS tutorial, so here is a Hello World type of BATS testing that assumes you already have it installed on your system.

To start walking our way through our final testing infrastructure, we will use BATS to test the simplest possible go binary so for anyone wishing to follow along go should also be installed.

Initialize a new go module in (an ideally) clean directory you are working on:

go mod init github.com/mygithubusername/justarandomprojectname

Create the typical Hello World go program we want to test in the binary level:

A proper end-to-end testing process should arguably:

1. compile the program above

2. invoke the compiled binary

3. validate its successful execution

The last item can be done with a combination (`AND`) of:

• checking the exit code was 0
• the output of the program was the expected one

Here is how all of the above are addressed with the BATS test below (comprised of just one test block — declared by the @test annotation)

note: do not forget to set the appropriate interpreter for the above script, as in

#!/usr/bin/env bats

• setup makes the necessary preparations before our test are executed; in this case, it just compiles our go program to a binary named mycli; note that by convention any cleanup is performed by the teardown function which we could easily have called to perform an rm mycli if we wanted to do so

important note: setup and teardown counterpart are special functions in the BATScontext; if defined, they run before and after (correspondingly) each test block and not the entire test suite defined within the BATS file

• the block annotated by @test is the actual BATS test which is named 1 — Evaluating our cli (using a number before the test name is of course at the discretion of the developer)
• invoking our cli to be tested via eval “run ./mycli” allows a) the exit code b) the output to be captured by the variables status and output respectively (note however that non-zero exit codes will make the test fail either way when run is not used)
• the test ends with the evaluation of the conditions we wanted to check, namely that

a) the non-zero exit code: [[ “$status” -eq 0 ]]

b) the comparison of the output with the expected value: [[ “$output” == “Hello World” ]]

Here is the actual execution:

▶ ./bats_testing.sh
✓ 1 - Evaluating our cli
1 test, 0 failures

In our context, we have adopted BATS more or less as an orchestrator that undertakes the tasks of invoking through its @test annotated functions the scripts that execute the actual binary and which is actually invoked through …

Expect

The following steps require for you to have the expect utility installed onto your system (brew install expect / apt update && apt install expect / yum install expect will most likely do the job for Mac / debian based / red-hat based linux respectively)

Let’s see how expect works, using another sample go program that prompts the user for input which just prints afterwards: (you can just change the contents of the file you used in the previous example)

Executing the above manually would go something like:

▶ go run main.go
Enter text: Testing
Testing

We will now create the go binary that will be invoked by our expect script (we will also automate this step in the next section)

go build -o mycli .

and here is the corresponding expect script

note: as was the case with the BATS script, you should set the interpreter directive accordingly in this case as well: #!/usr/bin/expect

Invoking the script above will automate its execution, since the program being executed/spawned will not be blocked expecting input as this will be provided by the expect script, something that is done via the following directive:

expect "Enter text: " { send "Testing\r" }

Here is the expect script’s execution output:

▶ ./myexpect.exp
spawn ./mycli
Enter text: Testing
Testing

Making our lifes easier with autoexpect

Writing long expect scripts for programs with multiple prompts for user input can become really tedious; this can be made easier to accomplish though through usage of autoexpect.

What this utility does, is that it keeps track of your interaction with a program requiring user input, and produces automagically the corresponding expect program, with all the steps you carried out scripted as they are being executed. We expand the above go interactive program a bit to showcase this a bit more effectively:

Instead of compiling the corresponding expect script ourselves, we can leverage autoexpect for this purpose as follows: (for MacOS, autoexpect should be installed alongside expect, at least when the later is installed via brew)

First we need to have the binary a priori ready:

go build -o mycli .

and then let autoexpect do the work for us:

▶ autoexpect ./mycli
autoexpect started, file is script.exp
Enter first name: Pantelis
Enter last name: Karamolegkos
Enter job title: SRE
First Name: Pantelis, Last Name: Karamolegkos, Job Title: SRE
autoexpect done, file is script.exp

The above process will create an expect file (by default named script.exp) which will incorporate our previously executed interaction and can of course be executed

▶ ./script.exp
spawn ./mycli
Enter first name: Pantelis
Enter last name: Karamolegkos
Enter job title: SRE
First Name: Pantelis, Last Name: Karamolegkos, Job Title: SRE

Putting all the pieces together

We saw some simple examples about usage of our main utilities comprising our proposed testing stack. However, real life scenarios are far more contrived. In this section, we will see an integrated example about how we can put all these building blocks together to drive this home.

Our go cli

We will use a go program that is a bit more complex than the samples cited above. This is a basic cli-tool that although simple will help us showcase some basic tests that rely on user interaction. We will create a simple `go` utility that:

1. accepts user provided flags (we will use go’s flagpackage for that purpose)
2. the program will be able to run in both:

• non — interactive mode, where all the required flags will be passed by the user upon invocation
• interactive mode, in which the program will ask interactively the user to provide the values to be assigned in the variables (pointers actually) returned by the corresponding flag declarations

Here is the code for that basic cli

As it becomes apparent from the usage of interactive flag, this cli has 2 modes of execution: interactive and non-interactive. What is more, to make our testing/execution validation process a little bit more interesting, in the case of interactive execution, the program’s output is not sent to stdout but rather on a file that is created on the fly.

After compiling our program:

▶ go build -o mycli .

we run our two modes of execution:

Interactive:

▶ go run main.go -interactive
Enter first name: Pantelis
Enter last name: Karamolegkos
Enter job title: SRE

Non-interactive:

▶ go run main.go -firstname Pantelis -lastname Karamolegkos -jobtitle SRE
First Name: Pantelis, Last Name: Karamolegkos, Job Title: SRE

Our expect script

Invoking the non-interactive form of execution within a test suite, is easy, since its a oneliner. However the interactive case, needs to have its invocation automated and this will be done through the help of autoexpect.

(you shoud remove testFile.txt in case it exists, before the following execution)

▶ autoexpect ./mycli -interactive
autoexpect started, file is script.exp
Enter first name: Pantelis
Enter last name: Karamolegkos
Enter job title: SRE
autoexpect done, file is script.exp

The only difference here compared to our initial autoexpect example usage is that this time, we need to pass the -interactive flag as the program requires it to run in interactive mode.

Final step: Orhcestrating with BATS

To put our test suite together, there is now only one thing missing: an orchestrator test script that should:

1. compile our gocode on the fly (so that we are certain we are testing the binary that was compiled from the exact same code we want to test)

2. invoke whatever commands can be invoked non-interactively

3. invoke any expectscripts that in their turn will undertake interactive execution of our binary

4. test the outputs (or any artifacts that should have been created whatsoever) against some expected outcome.

Validation of non interactive execution

Following our initial BATS example, we will address requirement (1) in the setup function and b in the same pattern as before, by validating that both the cli ends with 0exit code and that our output is the expected one.

…and by running it:

▶ ./bats_testing.sh
✓ 1 - Testing non interactive mode of execution
1 test, 0 failures

Validation of interactive execution

This case is a bit more contrived, since we now need (apart from compiling our program as before) to make sure:

• the execution once again exits with 0
• the file is created
• its contents are the expected ones

Here is one way to go about all of the above objectives, by leveraging (among others) the expect script.exp we created a few steps ago with autoexpect

And executing it:

▶ ./bats_testing.sh
✓ 1 - Testing interactive mode of execution
✓ 2 - Testing creation of output file
✓ 3 - Testing contents of the output file

Some notes on the last BATS testing file

• we achieve our objective by using 3 BATS test blocks
• we refrain from using a dedicated setup function since (as stated in the beginning of this article) it would be invoked before each test block execution which in our case is not necessary
• we declare the expected ouput (i.e. test file contents in this case) in a variable that is declared as a heredoc; this is convenient for the case of comparing actual vs expected output when we are dealing with multiline strings/vars; we also use a variable for the filename that is expected to be created as a good practice
• we validate that the expected test file has actually been created by checking the exit code of the ls $TEST_FILENAME command
• we capture the actual test file contents in the result variable, which is subsequently compared against the expected file contents (and whose comparison result determines the outcome of the 3rd test block)

Now let’s combine all the test cases (related to both interactive and non-interactive executions) in one BATS file (also re-numbering the test )

…and running it all together…

▶ ./bats_testing.sh
✓ 1 - Testing non interactive mode of execution
✓ 2 - Testing interactive mode of execution
✓ 3 - Testing creation of output file
✓ 4 - Testing contents output file

Aftermath

We went through a proposed approach about a way to make use of a tool that has been around for a long time (expect) and a more recent one, a bash-based testing framework (BATS) as a solution of testing command line tools on the binary level and validating its proper functionality in scenarios that resemble as closely as possible the actual invocation of the tool (including user interactions as well). It goes without saying that it’s up to the tester’s imagination / skills / experience to create far more convoluted testing scenarios than the ones we have showcased here.

Some further reading may incorporate things like:

• extra BATS — based libraries that support higher level functionality as filesystem-related assertions
• go’s expect library by Google that on one hand is native to go, on the other hand it makes easier things as variable manipulation, string/file contents handling/parsing etc;

PS. Thanks to Niky Riga (Systems Director in Workable) for proofreading this article.