Any sufficiently complex software system has bugs, and those of us who aspire to produce high quality work also seek to not only minimize these, but guarantee that our code does what we say it ought to.

One proven way to eliminate bugs, and ensure that code behaves as documented is to test the program. Easy enough to do by hand, when there isn’t much functionality. However, when the system grows more complex, and there are many possible environmental factors with various permutations, it quickly becomes obvious that we need to automate our testing.

This article aims to provide the reader with a few notions about software testing in general, and then concentrates on a specific example, using the test suite written in Tcl for Apache Rivet, in order to demonstrate a real-world approach to testing a particular program.

Software testing

Testing is often an afterthought, and even for large, complex systems, or expensive, proprietary software, testing is never going to directly generate revenue, or add new features to the program, and so it has to compete for scarce developer time. Even in free software work, more often than not, it’s more fun to spend time hacking on a cool new feature rather than writing test cases to make sure everything works exactly as advertised.

Testing is often an afterthought, and even for large, complex systems, or expensive, proprietary software, testing is never going to directly generate revenue, or add new features to the program, and so it has to compete for scarce developer time

This means that it’s important to try to get the most from the time you dedicate to testing—to get better value for money. In technical terms, it is desirable to maximize is the “code coverage” for the time invested. Coverage refers to how much of the code is exercised by the test suite. Think of running a simple program with options -x, -y, and -z. If you run it with the -x option, the “paths” through the other options will not be taken, and you won’t know if they really work or not. So you have to expand your code coverage to all three code paths. Generalizing, the two main approaches are “white box” and “black box” testing.

White box testing is testing a program based on knowledge of its internal workings. For example, writing a series of tests that give different input to each C function in a program, and the checking to ensure that it behaves correctly. Obviously, you need the source code, and the ability to rebuild the program in order to do this type of testing. One of the most important reasons to use this approach is that it is theoretically possible to test most or all of the code paths in the program. In practice though, the effort required to do this may be significant—imagine that you have a C function that takes a struct as input, and that struct is in turn generated by other functions, and so on. It’s clear that things can quickly get more complicated.

Black box (or “functional”) testing involves running a program against a specification to make sure it produces the correct output. For instance, testing to ensure the ls program works correctly. A very simple test would be to create a directory with files of known sizes and creation times, run the ls program, and compare its output with the known contents of the directory.

Apache Rivet

Apache Rivet is a server-side Tcl system for the creation of dynamic web pages. Think JSP or PHP, but using Tcl, a free, multi platform, general-purpose scripting language. For example:

<b><? puts "The date is: [clock format [clock seconds]]" ?></b>

It is best to test the software with as little modification to the environment as possible, meaning that the test suite will run using the copy of Apache already installed on the computer—having to create a special copy of Apache would defeat the purpose.

The goal of the test suite is to be able to start the web server with configuration options of our choosing, send HTTP requests, receive answers, stop the server, and then create a report. Because it’s so tightly integrated with Apache and Tcl, white box or unit test would be difficult. It would be very laborious to create and fill in all of the arguments that are passed to each function in the C code, because they usually reference complex C structures such as the Tcl interp struct, or the Apache request struct, which rely, in turn, on lots of configuration and set up. The effort required to make most of this work would probably be more than that involved in creating Rivet itself! So, a “black box” testing style would provide more coverage for the time dedicated to it. From the test suite, there much required, as the real work is in devising clever ways to test as much of Rivet’s functionality as possible. An application is required that allows program tests to be performed quickly and flexibly, and provides a lot of tools for interacting with Rivet and Apache:

• Reading and writing files in order to manipulate Rivet’s configuration files.
• Process control, to control the Apache process itself.
• Sockets and an implementation of the HTTP protocol in order to send requests to the Rivet-enabled web server.
• Good string matching and regular expression support.

Being a fan of Tcl, I choose.

The Tcl Test Suite ships as part of the core Tcl distribution, and is an excellent base upon which to build a series of tests for all kinds of applications

The Tcl test suite

The Tcl Test Suite ships as part of the core Tcl distribution, and is an excellent base upon which to build a series of tests for all kinds of applications. As noted Tcl expert Cameron Laird says:

Tcl tests automobile engines, emergency telephone circuits, chemical sensors, microprocessors, rocket components, industrial ovens, and much, much more. In the absence of any other knowledge, ANY software project should think of Tcl as its first choice for testing.

What does the suite itself provide? It defines several Tcl commands that are used in order to create and run a series of tests written in the Tcl language, and generate reports based on the results. It gives control over exactly which tests are wanted, and how the output is to be organized, as well as a number of commands to control the testing environment (files, directories, output, errors, and so on).

Returning to the example of testing ls, the following code contains some simplistic tests for the ls command.