Dual-license your content for inclusion in The Perl 5 Wiki using this HOWTO, or join us for a chat on irc.freenode.net#PerlNet.

Perl testing tools

From PerlNet

Revision as of 02:27, 13 July 2007; view current revision
←Older revision | Newer revision→
Jump to: navigation, search


Perl Testing Tools

CPAN has a good selection of modules to help with automated testing. In addition to testing applications written in Perl, it's also common to find Perl used as an automated testing tool for projects developed in other languages like C or C++.

If you're new to Perl testing, start by reading Test::Tutorial. After that, playing around with Test::More should give you a practical feel for how easy it is to write tests in Perl.

Testing Strategies

For an introduction to Software Testing principles please visit wikipedia:Software testing.

Typical testing

In the vast majority of projects, formal testing is left to the end, if it is performed at all. Developers will probably test each component as they write it, but these tests are usually haphazard and thrown away.

Test Driven Development

In test driven development (TDD), writing test cases is as important (if not more important!) than writing code. These test cases are usually written prior to the code they test. Thus you write cases to cover your code's interface and the correct usage of each subroutine before you even start to think about the algorithms you may need to use. TDD code may take longer to write initially, as the author usually writes more lines of tests than lines of code, but it has a much shorter formal testing period as well as an excellent test suite which prevents code regression.

When new bugs are found after the development process, tests are added or updated to ensure the bugs remain fixed.

Unit testing

In unit testing, code is tested in small parts (units) to ensure that each of these work correctly.

Interface testing

Interface testing ensures that the code interfaces work as stated in the specification. These are useful in discovering cases where code authors have changed the argument order without consultation with other project members.

Regression testing

Fixing bugs in one part of code often causes them to reappear in other parts of code - sometimes with the same symptoms. To prevent code from "regressing" (suffering from bugs that have already been fixed previously), most people recommend regression testing.

This is just common sense. Instead of performing throw-away tests, regression testing suggests adding these tests to your test suite. Thus once you find a bug, you write a test which fails due to the presence of that bug. Once that test no longer fails, the bug has gone. These tests are collected into a test suite and run after every code change. Then if bugs re-emerge it's much easier to locate and remove them.

Black box and White box Testing

Simple black and white box testing can be carried out with any of Perl's testing tools. Although there is a conceptual distinction, this is rarely evident in the resulting test suites. Coverage testing can be carried out with Devel::Cover.

Black box testing

Black box testing treats the code as if it were unreadable. Instead, much like interface testing, the tester writes tests based on the specification. For example if the specification states that a given field will take a date, then the tests will include valid dates, invalid dates, things that look like dates, things which don't.

Black box tests are easier than white box tests, as they should not be affected by changes to the code internals.

White box testing

White box testing relies upon intimate knowledge of the code internals. This knowledge is then used to test all the code assumptions and boundaries. For example, if the code assumes that a particular field will only be 20 characters long, white box testing includes tests for 19, 20 and 21 characters. Prudent testing will also include tests for 0 and 1 characters for the same field.

White box testings aims to ensure that the code behaves correctly when its assumptions are incorrect.

Coverage testing

Coverage testing is a special case of white box testing and measures how well the test suite exercises the code. There are a range of metrics for coverage testing:

  • statement coverage - is every line of code tested at least once?
  • branch coverage - is every conditional branch tested at least once?
  • path coverage - is every possible path through the code tested at least once?
  • condition coverage - is every term in each condition tested at least once?

Both 100% conditional coverage and path coverage imply 100% branch coverage. Path coverage is usually the hardest to achieve, and as such path coverage is usually limited to paths within subroutines or other smaller sections of the code.

Achieving decent coverage testing is a worthy goal, as it often highlights areas of your code which are unreachable and tests areas of your code which are not usually exercised.

Coverage Example

An example of using Devel::Cover on Class::DBI can be found at http://perl.net.au/Class-DBI-cover/ .

TAP: Test Anything Protocol

The Test Anything Protocol (TAP) is a standard format for displaying results of tests. An example TAP session may look like:

ok 1 - Beverage::Coffee loaded.
ok 2 - Cup creation.
ok 3 - Kettle location.
ok 4 - Boiling water.
ok 5 - Black coffee.
not ok 6 - Milk location.    
# Milk not found in fridge.
ok 7 - White coffee.         # Skipped, no milk.
ok 8 - Espresso              # Skipped, espresso machine not available.
not ok 9 - Non-dairy creamer # TODO, unimplemented.

The TAP output can be easily parsed by Perl's testing frameworks, and numerous modules exist to generate TAP.

The plan

TAP output usually starts with a plan line:


which specifies how many tests are to follow. The above example specifies 9 tests.

This line is optional, but recommended. Should a test suite die part-way through, the plan allows the testing framework to recognise this situation, rather than assuming that all tests were completed.

In the case of rapid test-suite development, it can be irritating to have to remember to update the number of expected tests. As such it is possible to specify that there is no plan. Typically this results in the test harness counting the number of tests run and creating a plan line as the last line of TAP output.

The test line

After the plan come the test lines. Each of these can be broken down into several parts:

ok 1 Description # Directive
  • ok/not ok This part is required, and specifies whether the given test succeeded or failed
  • The test point number. Not strictly required, but helpful for identifying missing tests.
  • Description. A human readable description of what each test is.
  • Directive. Any string after a # is a directive. These allow developers to skip tests that depend on success of previous failed tests, or to create tests which are expected to fail at the moment, but which will hopefully succeed once functionality is added.


Any other lines are assumed to be diagnostics. In the case above, our diagnostic line:

# Milk not found in fridge.

tells us why the milk location test failed. We can also use diagnostic lines to segment test output, for example breaking up test results for database connection, template construction and data parsing.

More information

For more information on the TAP visit the Test Anything Protocol Wiki or read the TAP documentation. Petdance's journal covers where the TAP name came from.

Test::TAP::HTMLMatrix can be used to create a colourful html summary of TAP results.

TAP-compatible tools for other programming languages

Testing example

Perl tests are often written using Test::More. For those who haven't written many tests before, Test::Simple may be an easier place to start.

 # Specify our plan, how many tests we're writing
 use Test::More tests => 8;
 # or alternately, if we don't know how many:
 # use Test::More qw(no_plan);
 # Check that our module compiles and can be "use"d.
 BEGIN { use_ok( 'PerlNet::TestMe' ); }
 # Check our module can be required.  Very similar test to that above.
 require_ok( 'PerlNet::TestMe' );
 # There are a number of ways to generate the "ok" tests.  These are:
 # ok:   first argument is true, second argument is name of test.
 # is:   first argument equals (eq) second argument, third argument is name of test.
 # isnt: first argument does not equal (ne) the second, third is name of test
 # like: first argument matches regexp in second, third is name of test
 # unlike: first argument does not match regexp, third is name of test
 # cmp_ok: compares first and third argument with comparison in second.  Forth is test name.
 # Here are some examples
 ok( (1+1) == 2,       "Basic addition is working");
 is  ( 2 - 1, 1,       "Basic subtraction is working");
 isnt( 2 * 2, 5,       "Basic multiplication doesn't fail");
 like  ("PerlNet is great", qr/PerlNet/i,   "Finding PerlNet in a string");
 unlike("PerlNet is great", qr/PythonNet/i, "Not finding PythonNet in a string");
 cmp_ok($this, '==', $that, "Comparing $this and $that with integer ==");

Test::More provides a few other functions for testing including is_deeply which compares complex data structures for equality and can_ok, isa_ok which test object functionality. For more information, read the documentation.

Mock objects

Sometimes just testing your code can be really hard, because you need to setup so much before you have a system your code can use.

Say you have written code to find and delete duplicate customers records. For historical reasons, some customers are in an LDAP directory, others are in a Oracle database, and still others are in an XML file in a server in another office. To test your code before you go deleting records off the live systems, you might try to setup duplicates of all of these, a complex, costly and difficult process.

Or you could "just fake it". What if, in your code, where it tries to use these various systems, you could substitute in something that behaves just like an LDAP server, or Oracle DB, or remote server, but is in fact just a "mock-up" of a real system. Like those mock streets in Hollywood - from the front, they look just like a shop or garage, but behind their just framing and sandbags.

Mock objects are just like these mock streets - they have the same API as the real object their mocking, but inside they do almost nothing - maybe they just return an "OK" status, something like that.

In complex projects, many objects rely on other objects or layers of infrastructure. This can make testing these objects difficult when these collaborators do not exist, or it is impractical to instantiate them. Further, using mock objects allows you to isolate test failures from external influences.

Take the example of an LDAP directory. There is a Perl module for connecting to LDAP servers, Net::LDAP. If you write code that connects to an LDAP server, your probably going to write lines of code like

package My::Deduplicator
# make use of LDAP server
use Net::LDAP;
sub connectLDAP {
 my ($server) = @_;
 return Net::LDAP->new($server) || "server $server is down";

Now what if you wanted to test how your code behaves if the LDAP server is down ? In our example, it should return a message. You might be working in an environment where your not allowed to turn the LDAP server off, or you dont want to change your code to connect to a non-existent server. What you want is to mock the Net::LDAP module, so that the mock version behaves just like the real one does if the requested LDAP server is down.

Looking at the Net::LDAP POD, we can see that if the connection cannot be established, Net::LDAP->new() returns a false value. It might be undef, 0, or '0'. It doesn't matter, as long as our mock version also returns false as well.

If you carefully construct your test file to create a mock LDAP server first, then you can control exactly what happens when the call to Net::LDAP->new() is made.

# in t/ldap.t
use Test::MockObject;

use Test::More qw(no_plan); # tools for testing

# pretend the LDAP server is down...
my $mock = Test::MockObject->new();
# LDAP server is DOWN 
                    new => sub { return; },

use_ok("My::Deduplicator"); # ends up using mocked version we just created

# test response when server is down
is(My::Deduplicator::connectLDAP("MockedServer"), "server MockedServer is down", 'expected failure message');

Test::MockObject allows you to set all kinds of actions for your mock objects - return the same value all the time, cycle though values, log the call etc. You could mock an LDAP server that failed every third connection request, or at a random interval, or dropped connections after the bind stage? What about testing how your code behaves when you connect and bind OK, but the LDAP server falls over on the second search query - how would you set that up in real life ?

The common coding style for testing with mock objects is to:

  • Create instances of mock objects
  • Set state and expectations in the mock objects
  • Run tests for object that use the mocked object

Mock objects are very useful for situations where there is a lot of external complexity, where time is involved, where things happen in lots of different orders or there are complex chains of interaction e.g. a GUI spawns a process that connects to a server to FTP a file.

Specialised mock objects

Some mock object forms are very common, such as database replacements. Some of these are available on CPAN:

Further information

Mock objects are also covered in Perl Code Kata: Mocking Objects.

CPAN Modules

Perl Testing Resources

Recent perl.com Articles on Testing

Other Testing Resources