CPAN Testers is only made possible with the support of our sponsors.
For more information on sponsoring, please visit the I CPAN Testers website.

Upgrade Notice

The CPAN Testers Wiki site has been upgraded since you last accessed the site. Please press the F5 key or CTRL-R to refresh your browser cache to use the latest javascript and CSS files.

CPAN Test Reports

There are 4 grades of report which CPAN Testers can submit. These are:

PASS    - distribution built and tested correctly
FAIL    - distribution failed to test correctly
UNKNOWN - distribution failed to build, had no test suite or outcome was inconclusive
NA      - distribution is not applicable to this platform and/or version of Perl

Grade definitions last changed September 2008. Prior to that time, build failures were classified FAIL rather than UNKNOWN.

Report Grades

PASS

A distribution that passes all the tests in its test suite, is classed as a PASS.

It should be noted that this doesn't necessarily mean the distribution is perfect, as in several cases the author many only include a bare minimum of tests in their test suite.

Also PASS reports don't take into consideration any warnings that are reported during the testing. While not necessarily failures, warnings may indicate some level of latent issue that the author may be interested to know about. The ability to report these warnings is currently part of a TODO list.

FAIL

A distribution that build successfully, but fails one or more tests in its test suite is classed as a FAIL.

UNKNOWN

The UNKNOWN grade indicates that no tests could be run for this distribution. This could happen if a distribution fails during its build cycle (e.g. perl Makefile.PL, make) or if it has no test suite at all. Also, if the result of the distribution tests could not be determined by the test harness or smoker, the result is classed as UNKNOWN.

NA

A distribution that fails to build or fails its test suite because the platform or Perl version is not supported, is classed as NA (Not Applicable).

Examples of these kinds of distributions are Win32-OLE or Mac-Glue, which are not expected to run on other operating systems, or distributions which explicitly state they require a specific version of Perl or better, due to the features they require (e.g. Unicode) not being available in earlier versions.

Dependency Issues

Where a dependency fails, any dependent, including the distribution that instigated the test run, is considered to be 'on hold' and no test report should be sent. When the dependency is fixed, the dependents can then be tested and reports sent as appropriate.

Where a dependency reports NA, all dependents should be reported as NA and the appropriate report should be sent to the cpan-testers mailing list.

Report Formats

Currently the format of reports differs slightly between the reporting mechanism, although there are plans to merged the ideas into a single format.

Subject Line

The subject line of the reports needs to include the report class, the distribution that was tested and the platform it was tested on. There are several testing tools used to parse these reports, and any deviation to this format may mean that the reports are ignored.

The format is:

<class> <distribution name>-<distribution-version> <platform>

where class is one of:

PASS
FAIL
UNKNOWN
NA

Examples of this format are:

PASS Acme-Example-3.1.7 openbsd 4.0

Please Note that the distribution must be named and versioned, and not include its package extension, otherwise the parsing tools will ignore the report.

Content

The format of the content is currently under review, as different reporting mechanisms produced different reports. However, they all provide the following basic content:

  • An introduction to the author, noting this is an automated report
  • If a FAIL, UNKNOWN or NA report it should include the testing output
  • If a PASS report, including the testing output is optional, but prefered if any warnings were emitted
  • Optional tester comments
  • the output of Perl -V

Helpful hints, which are not present in all reports, but are prefered are:

  • A list of the prerequisites and the currently installed versions
  • ?