Upgrade Notice

The CPAN Testers Wiki site has been upgraded since you last accessed the site. Please press the F5 key or CTRL-R to refresh your browser cache to use the latest javascript and CSS files.

What is CPAN Testers?

[25/11/2005 - The following section was originally taken from the original article by Autrijus Tang]

When most people run make test, it's immediately followed by a make install; that is, people only test modules that they are going to use. But they may not have the time or inclination to report the outcome, for various reasons listed above.

CPAN Testers (http://www.cpantesters.org/) is an effort to set up a Quality Assurance (QA) team for CPAN modules. As its homepage states:

The objective of the group is to test as many of the
distributions on CPAN as possible, on as many platforms
as possible.

The ultimate goal is to improve the portability of the distributions on CPAN, and provide good feedback to the authors.

Central to its operation is a mailing list, cpan-testers@perl.org.

All recent CPAN uploads, as well as all test results, are forwarded to the cpan-testers mailing list's subscribers -- CPAN testers. After a distribution's release on CPAN, one or more testers will pick it up, test it, then feed the result back to the author and to the list. The testers don't have to know what the module is about; all they do is to ensure it's working as advertised.

You can query the reports for a CPAN distribution on the Reports website to see if it works on your platform, checking for any associated bug reports. The same information is also present at each module's page in http://search.cpan.org/.

Alas, while this system certainly works, it is far from perfect. Here are some of its shortcomings:

  • The integration between website and mailing list hadn't extended to the CPAN.pm shell, so checking for test results requires an additional step to navigate the web browser.
  • People who aren't subscribed to the cpan-testers list rarely submit test reports, so coverage of platforms is limited to a few popular ones; a heavily-used module may still be tested in only 2 or 3 platforms, hardly representing its entire user base.
  • There are no smoking (automatic cross-platform testing) mechanisms; all testers must manually fetch, extract, test and submit their test reports, including the same copy-and-paste toil we described earlier. This entry barrier has seriously limited the number of active volunteer testers.

Fortunately, CPANPLUS and CPAN-YACSmoke addressed most of these issues, and have made it significantly easier to participate in testing-related activities.