Testing

Prerequisites

In this lesson we use a python library called nose. Basic understanding of python variables and functions are a necessary prerequisite. Some previous experience with the shell is expected, but isn’t mandatory.

Getting ready

Nothing to do: you’re ready to go!

Topics

Before relying on a new experimental device, an experimental scientist always establishes its accuracy. A new detector is calibrated when the scientist observes its responses to to known input signals. The results of this calibration are compared against the expected response. An experimental scientist would never conduct an experiment with uncalibrated detectors - the that would be unscientific. So too, simulations and analysis with untested software do not constitute science.

The collection of all of the tests for a given code is known as the test suite. You can think of the test suite as a bunch of pre-canned experiments that anyone can run. If all of the test pass, then the code is at least partially trustworthy. If any of the tests fail then the code is known to be incorrect with respect to whichever case failed. After this lesson, you will know to not trust software when its tests do not cover its claimed capabilities and when its tests do not pass.

In most other programming endeavors, if code is fundamentally wrong - even for years at a time - the impact of this error can be relatively small. Perhaps a website goes down, or a game crashes, or a days worth of writing is lost to a bug in your word processor. Scientific code, on the other hand, controls planes, weapons systems, satellites, agriculture, and most importantly scientific simulations and experiments. If the software that governs the computational or physical experiment is wrong, then disasters (such as false claims in a publication) will result.

This is not to say that scientists have a monopoly on software testing, simply that software cannot be called scientific unless it has been validated.

Testing is the calibration step of the computational simulation and analysis world: it lets the scientist trust their own work on a fundamental level and helps others to understand and trust their work as well. Furthermore, tests help you to never fix a bug a second time. Once a bug has been caught and a test has been written, that particular bug can never again re-enter the codebase unnoticed. So, whether motivated by principles or a desire to work more efficiently, all scientists can benefit from testing.