Upgrade Notice

The CPAN Testers Wiki site has been upgraded since you last accessed the site. Please press the F5 key or CTRL-R to refresh your browser cache to use the latest javascript and CSS files.

Roadmap

CPAN Testers is a complex ecosystem of tools and the complexity keeps growing as one tool or another adds features. This page is intended for discussion and brainstorming about current issues and challenges and how to manage complexity over time. Please add issues or comment on existing ones.

The roadmap is currently divided into "version 1" and "version 2" plans.

Version 1.0

The current CPAN Testers infrastructure:

  • CPANPLUS or CPAN::Reporter support manual or automated testing
  • Test::Reporter used for low-level report creation and transport
  • Reports received at cpan-testers mailing list
  • Reports archived on NNTP
  • Websites generated from statistics mined from NNTP

With authenticated SMTP transport now provided in Test::Reporter, there are few plans for the 1.0 roadmap beyond critical bug fixes.

Open issues:

Version 2.0

Version 2.0 is a proposed future architecture to address current limitations. Some desired features for 2.0:

  • Consistent grading logic across all tools
  • Reports sent via HTTP instead of email
  • Reports stored in a central database instead of NNTP
  • Author notification via a centralized service instead of by each individual tester
  • Reports with structured data instead of plain text (e.g. prereqs found)

The CPAN-Testers namespace was reserved for this next generation and some initial work on a database component was done as part of the Oslo QA Hackathon 2008.

Design notes

Testing components

These will be used by CPAN, CPANPLUS and/or smoke programs to create reports and submit them to a central database.

  • Harness to run tests and capture output
  • CPAN Testers report object
  • Grader to generate a report
  • Editor to let user edit or comment (text or GUI)
  • Transporter to submit report

These should live under the CPAN::Testers::* namespace. Most functions can be taken from either CPAN::Reporter or Test::Reporter with some refactoring.

See the CPAN Testers Client page for more.

Database components

This needs to support both current reports (just text) and future reports (structured data). In Oslo, the consensus was to focus on an object database, with the side benefit of supporting other projects that create or provide information about CPAN Distributions (e.g. CPANTS). Components include:

  • Generic data object classes
  • Database for central storage of data objects
  • Client library to query, submit to or retrieve from the database
  • Web server to receive CPAN Testers reports via HTTP
  • Web server to query central database of reports (for analysis or reporting)

See the CPAN Metabase page for more.

Statistics/reporting components

Downstream web servers or daemon services to analyze and/or act on reports submitted.

These should live under the CPAN::Testers::WWW::* namespace. Although there are currently distributions under the CPAN::WWW::Testers namespace, the plan is to move these to the newer namespace in future releases.

Author Notification System

  • Centralized notifications service for authors (e.g. email, RSS)
    • Should have a way to honor author preferences about notification, either through a web interface or through an AUTHOR.yml file on CPAN
      • A distribution YAML file would be too much of a load on the central server, as it would require continual CGI requests.
    • Some possible preferences to offer authors are: to opt out entirely, to only get reports about the most current version of their module, to only get one failure report per module version, perl version, or OS.
    • potentially allow people to sign up for reports about modules they don't own
      • This might be better to do via the RSS feeds.

The notifications system prototype is being based on a simple database, and looks to be the right way to go. It enables faster lookups and when there are around 10,000 reports to be reviewed in a single run, a quick lookup is important to allow the server to concentrate on other parts of the system.

TODO List:

  • set default to only notify on the latest version of a distribution
  • Include distribution name/version in subject line (suggested by Mark Stosberg)
    • Only works if one instance included in message
    • Could reference how many distributions listed in notifications if multiple distributions
  • include preferences for osname, perl version and distribution version
    • distribution version could be tricky, unless latest version is automatically included

Author Preferences Website

  • Website to display report summaries (and drill down to individual reports) by author or distribution
    • this should be possible with the dynamic version of the Reports website.

Other tools

  • Legacy gateway to capture reports submitted to perl.cpan.testers into new database

Implementation notes

  • Need to find hosting for database and notification service
    • currently being hosted by Birmingham.pm, plans are afoot to move to a new RAID enabled server.
  • Old reports need to be ported into the new database
    • extra field in the database to indicate the submission method, SMTP or HTTP, which together with the report id forms the primary key.
  • CPAN, CPANPLUS and smoking tools need to be rewritten to use new client tools
  • Existing stats/analytics sites will need to be ported to use new database
    • a minor change, as it should still be able to use the same SQL, assuming there will still be parsing of the reports into a metadata table.

Open issues

  • What mechanism should identify/authenticate testers?
    • Could leverage an existing mechanism (e.g. PAUSE) or could be a new service to manage API keys
      • PAUSE is only for CPAN Authors, there are many hundreds of testers who are not authors. We want to encourage almost anyone to be able to submit reports, however we do need to ensure that testers acknowledge their potential involvement. Some use unreachable email addresses, some are just regular users who are installing modules occasionally. Need to differentiate these testers from the high volume and/or responsive testers.
  • Disputing and/or withdrawing reports is a common request by some CPAN authors and testers
    • dynamic reports site should have a button that authors/testers can raise as an issue. This can then be reviewed by an admin to mark the report as incorrect. The report is not deleted, to avoid reparsing by accident or being deleted by accident, but marked in someway that the Reports, Statistics and any other site knows not to include that report.
    • Authors should not be able to delete reports, there needs to be a review process.
    • Testers should be asked to confirm a marked report.