CPAN Testers Statistics

Latest Summary - 3rd January 2009

CPAN Testers Stats - December Summary - Seasons End

December was quite a hive of activity, what with the new Preferences Administration site launch and the reworked Reports site also going live. There is still another member of the website family to be unveiled, but that might be a little longer arriving. In the meantime work towards CT2.0 is being seen as a higher priority for the moment. The data generation tools now encompass a MySQL database update, which is used to feed the Reports site. The first benefits from the change to the MySQL database have been to enable the new Reports site to go live, however in the longer term the AJAX API to data snippets are expected to be used on search.cpan.org, and pretty much anywhere that people can make use of it.

A big data cleanup was also undertaken last month with several entries getting sanitised. As some of the reports get truncated or parsing fails partially, some of the fields, such as osname and osvers can often be left blank or contain erroneous data. As such cleaning this up helps to provide more accurate numbers for the different platforms. Until we move to the complete HTTP submission system, it is likely that some descrepencies will still slip through, so if you spot any please let me know and I'll investigate where I can.

As mentioned last month, the 2009 QA Hackathon event in Birmingham will be featuring work on the CPAN Testers infrastructure. With several key collaborators planning to be there, we're hoping that a lot of the plans to get CT2.0 up and working consistently for all smokers will come to fruition.

We topped 135 testers submitting reports last month, so thank you very much to everyone involved. The mappings this month included 68 total addresses mapped, of which 23 were for newly identified testers.

One side effect of being able to map so many addresses to actual people over the last year, has meant the totals per month keep getting adjusted. With the most recent mappings several months that previous listed over 130 testers, have seen their counts notably reduced, to the point that now December 2008 is the only month to have listed more than 130 testers in a month. I expect that number to reduce too once there are a few addresses mapped in the future. As a result of all the mappings recently, I've held off my unmapped address mailout for the last few months, but I plan to sort that out this month. In the complete database we have 564 unmapped addresses, although only 68 were used during 2008. With a total of 2749 addresses being used, 564 is a significantly high number. My hope is that moving to the new HTTP submission system and releasing the new member of the website family will also help to reduce that number, especially those who use 'example.com' or 'ESMTP' as their domains. CT2.0 is coming ... soon :)

Announcement - 28th December 2008

Dynamic CPAN Testers Reports - Phase One

After the last major update to CPAN Testers Reports website, the plan had always been to move to a more dynamic site. While a fully fledged dynamic site is still planned, the move is being made gradually with significant updates. As such the last updates have now been made live.

The latest set of changes now mean that the site is now heavily reliant on Javascript running in the browser. With the previous updates and the large amount of data on some pages, the Javascript code previously used was taking a considerable amount of time to correctly render the page according to the user preferences. As such a major rethink was instigated and I'm very grateful to JJ, for providing a different perspective that has now been employed. All the data displayed is now driven by data files rather than HTML, thus allowing the pages to render much more quickly. Also to alieviate the loading on distribution pages, the display now only shows one version of a distribution at a time. In most cases this will be the latest version, but depending on your preferences this may differ slightly.

The preferences themselves have been updated and you are now able to filter on Perl verison and operating system, so you can see the reports that are specific for your installation requirements. The preferences are also now tied to the stats and summary graphics, so updating your preferences will also update these tables too.

The latter dynamic update, the summary graphics, now use AJAX to update the quick view of the PASS/FAIL/NA/UNKNOWN reports for the distribution versions or author distributions, depending upon which type of page you are looking at.

This does mean that if you don't have Javascript enabled in your browser then you are unlikely to experience the site as it was meant to be viewed. The next phased released however, will included a graceful downgrade to allow each page to be rendered as a flat HTML page with the ability to submit your choices of preferences via a traditional form. With the Googlebot and MSNbot hitting the site on a regular basis, it's likely that they will be harnessed to rebuild the site in future too. More updates coming soon.

Announcement - 8th December 2008

NEW CPAN Testers site - Preferences Administration

With no little amount of fanfare the CPAN Testers would like to unveil a new site to accompany the rest of the family.

The CPAN Testers Preferences Administration website is now available for authors to specify the preferences they would like to set for the receipt of reports and summaries. The site now drives the Daily Summary Reports that are produced by the Author Notification System. Currently only Daily Summaries are available, but as the system settles in, the ability to provide individual reports and Weekly Summaries will also be available.

The preferences are separated into two sections, the default preferences and the distribution preferences. The default preferences are applied when you have no specific distribution preferences defined. It is here you can also switch off the ability to receive reports at all, if you wish not to be bothered with CPAN Testers. The distribution preferences can be defined for each distribution which you have released to CPAN. If you choose to only define specific preferences for some of your distributions, the remainder will use the default preferences.

The preferences themselves allow you to specify which grades you are interested in, whether you wish to just be alerted to the first or all tuples, where a tuple is an instance of Distribution version, Perl version and Operating System. You can elect to just be notified about the latest release of a distribution or all versions, as well as the Perl versions and operating systems which you wish to be alerted to, or conversely not be alerted to.

As the system is only meant for CPAN authors, the backend is driven by your PAUSE credentials. However, please note that your password is never stored by the system, and is only used to verify your login with PAUSE itself (all over SSL).

One important note I would like to make authors aware of, is that if you are not receiving summaries, then login to the site and find all notifications have been disabled without you previously asking me to, you may want to check your PAUSE account directly. The system uses your 'PAUSEID at cpan.org' mailing address to send reports, and as such if you haven't correctly set up a forwarding mail address within PAUSE (it doesn't appear to be a default setting), then persistant bounces will disable all notifications within your account. Once you correct the PAUSE forwarding, you can then enable notifications and should then begin receiving summaries, as per your preferences.

If you find any problems in using the system. please me know.

Latest Summary - 6th December 2008

CPAN Testers Stats - November Summary - London Calling

November was very much a development period for CPAN Testers. There has been a lot of work going on in the backend, with Ask working on the new backend database and HTTP submission process, and myself working on several sites, including 2 new ones. There has also been some work by David Golden and Chris Williams on the smoker tools, so although we may have been publicly quiet, there has been a lot of effort being put in to push CPAN Testers forward.

One of the new sites, The CPAN Testers Preferences site is now ready to go. We finally got the SSL Certifcate approved, signed and installed, so look for a bigger announcement next week about that. The site will allow authors to set preference filters for the reporting they currently receive via the Author Notification Service. Currently only Daily Summary Reports are available, but this will be extended to the individual reports as well as Weekly Summary Reports, as soon as I have some time to add those reports in.

The more interesting site to be worked on is the main CPAN Testers Reports site. Following on from my use.perl post regarding Sorting CPAN, the backend code now uses the new database to correctly sort distribution versions. However, the biggest change to be made will be to make the site very dependant on Javascript. There is already Javascript on the site at the moment, but it wasn't written well enough to cope with some of the HUGE pages, and as such some author and distribution pages suffer from popups asking the user whether they wish to continue a long running script. This has now been resolved, and the new pages load a lot faster. In addition the filter mechanism has now been extended to include the ability to filter on the perl version and/or OS name. There is still a bit of work still be done, but again expect an announcement regarding the site in the next couple of weeks.

One thing to note about the Reports site, is that I am not currently coding it as I would a static site. If you don't have javascript enabled, then you will not be able to see the data. Once the dynamic site is available then a flat HTML version will be possible, together with the provision for AJAX requests to view specific results on your site.

At the London Perl Workshop, both Chris Williams and myself gave CPAN Testers related talks. Chris gave a more up to date version of the talk he present at YAPC::Europe in Copenhagen during the summer, while I presented a behind the scenes look at the files used to present CPAN, BACKPAN and parts of the CPAN Testers site, finishing up with the new Uploads Database. The event went very well and by all accounts all the talks were very well received. No doubt a few CPAN Testers talks will feature in some of the workshops and conferences during 2009 too.

One event that will be featuring the CPAN Testers project, will be the 2009 QA Hackathon. This is being held in Birmingham, and although this is a very dedicated event to bring together the developers involved in QA projects, if you have suggestions for attendees, would like to nominate a person we should contact, or even better still have a sponsor contact, then please visit the wiki and add a post to the site.

We topped 133 testers submitting reports last month, our highest ever, so thank you very much to everyone involved. This included 17 new addresses mapped, of which 6 new testers were identified.

Expect more news and announcements in the coming weeks as we move ever closer to CT2.0 :)

Announcement - 19th November 2008

Sorting CPAN

One of the problems with the CPAN Testers website resources, is that where an author listing of distributions, or the list of versions for a distribution, is required, a lot of backend trawling is done. This is due to the current backends having to refer to 3 sources to get those lists. Even then the resulting lists aren't quite correct, as the version sorting can be slightly weird when you have to take into account every author has slightly different perception of versioning. Sort::Versions goes a long way, but it isn't 100% accurate. The only really accurate way of sorting is on the release date of a distribution, which until now hasn't existed in a single form.

For a couple of months now, CPAN Testers has had it's own BACKPAN and CPAN mirrors. Of the 3 sources these are represented by Parse::BACKPAN::Packages and Parse::CPAN::Distributions. and the 2 index files they use. These can take a long time to parse, and as they don't parse and return any release date for distributions and their version, using Sort::Versions is a reasonable alternative. However, there is a third source and that is the CPAN Uploads that are announced by PAUSE. Due to the time lag of the mirrors, very often a release can be made and not be available to CPAN for several hours, so while no CPAN Testers reports might exist, it's still important to know the latest version.

Previously the last source is the only one that contained any release date information, which prompted me to think about doing it for the other sources. Surprisingly quickly, using the local CPAN Testers copies of BACKPAN and CPAN, I was able to build a basic database of upload data, and tag each with 'backpan', 'cpan' or 'upload', to indicate in what state the release was currently at. Queries now take fractions of seconds instead of several seconds. But, and perhaps more importantly, the sorting of distributions actually makes more sense!

The new database is being integrated into the backend code at the moment, but for those that might wish to have this information available for their own uses, the complete database is publicly available at the following locations:

These will now be updated daily, and once everything else is in place will eventually updated hourly.

Latest Summary - 3nd November 2008

CPAN Testers Stats - October Summary - Rust Never Sleeps

At the beginning of the last month, my automated mailer for authors and testers went live. For the most part many will never see the mails it produces, which is good, as it will mean that authors are uploading packages nicely formatted, and testers aren't sending out mangled reports. There have been an average of 3 or 4 authors a month who upload distributions without an archive extension accepted by CPAN/CPANPLUS or missing a version number. With testers it varies depending whether a bogus upload has filtered through the system, but again there are at least a few a month. This check now runs daily, as opposed to when I ran it manually roughly once a month. As a consequence it will hopefully help authors more quickly to know when they have uploaded a package that wasn't quite packaged correctly, and help testers spot when their test queue has got a bad package. If you do receive a mail from the system and don't believe your package/report should have been included, please let me know. It should be noted that authors can upload any archive format they choose, or any file format for that matter. However, to be most useful to end users, particularly those using the automated tools such as CPAN.pm and CPANPLUS, it is recommended that the following archive formats are used: .tar.gz, tgz, .tar.bz2 and .zip. In addition the format for a reliable upload is <distribution name>-<distribution version>.<archive extension>.

Also at the beginning of the month, we had a slight hiccup with the update to the Reports site. Unfortunately Léon was in the process of moving his server and the file used by Parse-BACKPAN-Packages became unavailable. Leon quickly sent me a copy of the file and I pointed the code at a local copy of the file instead of the remote one. However, it did prompt me to think about possibly running a mirror BACKPAN server, and thus creating the file myself. At least that way if anything happens to Leon's server, there is an alternative location from which the file can be grabbed from. Once I've got it working to my satisfaction, I'll shall implement a patch to Parse-BACKPAN-Packages so that it knows about alternative locations. As a consequence, this new BACKPAN mirror is now available publicly via http://backpan.cpantesters.org.

The CPAN Testers Wiki got an update this month too. Following some XHTML fixes and a little bit of a code clean up, I also added syndication feeds for the Recent Changes, as requested by David Golden. The fixes to the XHTML templates, CSS and RSS have hopefully helped to provide a cleaner interface.

Between 13th and 19th October there were some signification changes to the infrastructure of CPAN Testers. The fact that nobody appears to have noticed was a good thing. Expect an official announce soon :)

Over the course of the month, JJ came up with an alternative method of displaying the reports information on the Reports website. Many authors have complained (loudly), that due to the large number of reports their distributions have amassed, it gets difficult to see the important data from the irrelavent (to them). As a first attempt to minimise this the default selections within the Javascript ensured that only CPAN available distributions, tested against unpatched versions of Perl, were seen on each page. However, this also has the problem that for large listings the Javascript can still take a large amount of time to render the page correctly. For some visitors it can also be a bit disconcerting when the page initially locks while it runs the javascript, then half of the page suddenly disappears. This was always a short term measure, and following discussions with JJ, he has come up with a solution that uses a JSON data file, which loads in the browser with the predefined defaults. The new default will be to ONLY render the current distribution, with the ability to "Expand All" should you wish to see the other distribution versions. This will then mean no disappearing report data and no long running javascript. My thanks to JJ for persisting with this and figuring out the best way to render everything. The changes aren't live yet, but expect an update at some point this month.

Other website developments have continued at quite a pace and while some work has mostly been to improve XHTML compatibility and fixes cosmetic bugs, the new sites for Report Administration and Author Reporting Preferences are coming along very well. More news on those in the coming weeks. The CPAN Testers git repository has now been live for several weeks and will be expanding once more of the backend tools and code are cleaned up and packaged.

Again we had 126 testers submitting reports last month, our second highest by month, with 16 new addresses mapped, of which we had 6 new testers identified.

Work continues on for the websites and mailing systems along with numerous other changes within the CPAN Testers ecosystem. More news soon.

Latest Summary - 2nd October 2008

CPAN Testers Stats - September Summary - Rainbow Rising

Well a lot has been happening this month! After a quiet revamp of the Statistics website, the work to complete the revamped Reports site continued on. At least for a few days. Following some rather large threads on the perl-qa mailing list, it was decided that part of the change for CPAN Testers, to bring about a centralised author notification system, should be brought forward. With the increased volume of reports and the fact that some authors are uninterested in the reports as a whole, a daily summary report was generated, as part of the Authors Notification System, and a new version of Test-Reporter was released, which would stop smoke testers sending reports directly to the authors. However, this turn of events also received an equal if not bigger resistance, as a number of authors requested to have their direct mails turned back on! Unfortunately, we won't be turning them back on (yet!), but we do have plans to enable the direct mails to those authors who want them, by way of the Preferences system, which will be launched as soon as we have the pieces in place. This was to be part of a major CPAN Testers 2.0 launch, but it looks like we may end up releasing parts in stages now. However, if an author wishes to still receive these emails in the short term, they can download David Golden's App-CPAN-Testers-Remailer, and configure it to read the NNTP server as and when it updates.

The new Reports site has now finally been launched, and initial feedback was positive. There are still some improvements to be made, but after 3 months of solid work, please don't expect anything soon. The biggest drawback with the current static site, is that it creates 3.7GB worth of files, and takes several hours just to generate those files that need updating each day. The dynamic site will improve on this considerably, by only caching files that are actually requested. In addition the preferences system can then use AJAX calls to update the page rather than trying to load everything up front. This will then mean that the update system can be made much more responsive, and once the full HTTP submission system is working, the websites can be updated every few hours. Though perhaps more importantly, those authors who wish to receive reports the moment they hit the server, can then open the floodgates to their own mailservers as much as they want ;)

Chris and David have been hard at work releasing updates to CPANPLUS-YACSmoke and CPAN-Reporter respectively, with David also patching CPAN.pm and CPANPLUS to implement some of the fixes to the toolchain for testers. The testing tools have moved on leaps and bounds within the last year, and setting up an automated smoke tester is now much easier than it once was. With David and Chris being much more responsive to the updates of the smoke test tools at the moment, it should be noted that CPAN-YACSmoke hasn't had much attention from Rob or myself. There is a version 0.04 in existence privately, which may see a final release, but largely the plan is to move any useful functionality into CPANPLUS-YACSmoke, where Chris can look after it more attentively.

One of the changes implemented by David, was following a clarification by Graham Barr over the use of the UNKNOWN grade. For the last few years it has been taken to mean that the test suite could not be found for the distribution, and therefore it is unknown whether the distribution passes or not. However, Graham has since stated that the grade was to also include those instances in the build stage where something goes wrong that is unclear whether the fault is with the distribution, environment, a prerequisite or something else. Previously the build stage errors were reported as FAIL, and have been a great cause of debate for authors when FAILs appear that were out of their control. The reassignment to UNKNOWN will hopefully allow authors to ignore these reports, and for testers to investigate further whether they need to fix their own environment or send a specific bug report. Following on from this, David Golden has written a "nagbot" that will be emailing testers to alert them when their testing environment need upgrading.

Another quiet launch was the CPAN Testers Development Site. This is a central resource that is updated daily with the data files and source code snapshots for the collection of CPAN Testers websites, so you have a central source to refer to. However, for the source code repositories for the whole CPAN Testers ecosystem, you should refer to the Contributing page of the CPAN Testers Wiki. Here you will find all the known repositories for the smoke testing tools and website generation tools. Patches, as always, are welcome, so if you feel a burning desire to fix some part of the system, now you know where to find the latest code versions.

Again we had over 100 testers submitting reports last month, with 23 new addresses mapped, of which we had 15 new testers identified. Rick Myers, a recent addition to the CPAN Testers community, caught out Chris with an extra 481 reports to become top tester of the month, and we've seen a slight rise in testers last month too, with 120 posting reports.

I expect a quiet a month for October, as we let the dust settle and allow David to follow Chris (nothing new there then ;)) and become a new Dad. That's not to say we will be idle, as there is still plenty to be getting on with, as we move closer to CT2.0.

Announcement - 20th September 2008

NEW CPAN Testers Reports

As soon as DNS catches up for you, the all new CPAN Testers Reports site is now live. There are plenty of changes and fixes that have gone into this release, and my thanks specifically to Gabor Szabo and David E. Wheeler for contributing patches.

So what do you get with the new site? Well thanks to Gabor, a pass matrix has been added to each distribution page. On each author page, the distributions now link to their own respective pages. David E Wheeler's patch has been incorporated to provide an RSS feed that excludes the PASS reports. In addition the RSS feeds now limit to at most the last 100 reports. For the author page, the very latest release is listed, and not just the latest with reports, so authors can see more immediately whether their latest version is known about. On the distribution page, all known releases are listed, with those having no reports having text saying that. This latter change has been a cause for concern with some authors, as the latest versions were often getting missed. Unfortunately this was due to everything being referenced as per the BACKPAN files, and did not include everything that was still on CPAN. There have been many minor fixes, which all help to contribute to better usability.

However, the biggest change is the ability to select what you wish to see on the page. The options currently available are only the beginning, as these were all the easy ones to sort out. More complex changes, such as being able to filter on OS and/or Perl version, and preferences cookie recording, are planned for a later release along with many other improvements. Although the most notable change will be to turn the site into a dynamic one, which will be able to respond to new reports much more immediately.

If you do have further suggestions for improvement, please post them to RT, so I don't forget them.

For those that are wondering what has happened to the testers.db.bz2 file, that file is no longer generated for the new site, as it doesn't contain enough data for the current cpantesters.org sites. The new file is at http://devel.cpantesters.org/cpanstats.db.gz, and contains several changes from the previous schema. To see the current schema, see CPAN::WWW::Testers::Generator. In the short term, if you still need the old schema formatted database file you can grab it from http://cpantesters.perl.org/testers.db.bz2. The old site is currently maintained by Robert and Ask at perl.org and will continue to be updated as per the old schema until at least the new site has had a chance to settle down, and probably not until there is a more interactive API available to the data. However, if you are using the old database, please think about moving over to the new format, as the contents of the new database are now more actively monitored and maintained.

And latestly a request. Assuming nothing breaks and the new changes work out, the definitive domain for all the CPAN Testers related websites should be cpantesters.org. As such, if you have documentation in your modules or on your websites that references any other domain (whether for the Reports site or the Statistics site for example), could you update them to use only the cpantesters.org domain, with the correct sub-domain (Reports is www.cpantesters.org, Statistics is stats.cpantesters.org and the Wiki is wiki.cpantesters.org, etc.), for your next release. The reason being that other domain names may or may not work in the future, and the use of the new domain name will make it easier for everyone to remember the sites they wish to go to for anything related to CPAN Testers. It also doesn't hurt having all these references adding to search engine rankings either.

Latest Update - 12th September 2008

The CPAN Testers Websites - What's Next?

Yesterday I announced the launch of the notification system (with very mixed reactions), part of a whole swathe of improvements planned for CPAN Testers. The next phase for the notification system is to implement a preferences website, on which authors can fine tune the alerting they wish to receive, even down to particular os, perl version, distribution and distribution version. So for example if an author wants to receive mails for their distributions, except their Acme ones, or only for those reports run on VMS, then they can. I'm not going to commit to when the preferences site will be live, as there is much work behind the scenes still yet to do, but all being well I plan to have something live in the coming months.

Since taking on the code base for CPAN-WWW-Testers-Generator and CPAN-WWW-Testers, I have been hard at work implementing many improvements for these, as well as packaging CPAN-Testers-WWW-Statistics. You may not have noticed the improvements in the database generation tools and process, but you may have noticed some of the clean ups on the Statistics site. One of these clean-ups has been to finally add an RSS feed and formalise the blog.

Another update has been to move the non-CPAN code and data to a new CPAN Testers Development site. My primary intention with this is to provide links to the data that is used or generated behind the scenes, that isn't appropriate for CPAN. However, a further intention is to provide a suitable one-hit place to find all the development links you will need if you wish to get involved in working with the CPAN Testers code base. At the moment only links to the CPAN versions are included, but eventually git/svn/cvs/whatever repositories will also be listed. This will hopefully encourage patches and the like for adding more functionality to the websites and bring us closer to CPAN Testers 2.0.

However, there is still one part of the puzzle to be unveiled, and that is the new CPAN Testers Reports website. On Monday I gave a quick preview to MiltonKeynes.pm, which gained some positive comments, and will hopefully meet with approval with everyone else (though I doubt it after the flak I've been getting over the past week) once it receives its finishing touches. The site currently builds to over 3.7GB as a static site, so the move to a dynamic site will reduce that somewhat. Lots of fixes for the static site have been implemented, and many more are still planned. Expect its public launch soon.

There is lots more planned for CPAN Testers 2.0, but much of that will be behind the scenes in the way that reports get sent to the central server. If you want to get involved and help improve CPAN Testers, please consider joining the cpan-testers-discuss mailing list. David Golden has started a Roadmap page on the wiki, so please put forward your suggestions and ideas.

Announcement - 11th September 2008

All Your Report Are Belong To Us

Today I'm pleased to announce the launch of the CPAN Testers' "Collated Email Notification of Tester Reports for Authors Service" [1], a centralised notification system to send authors email updates for the reports, that were previously sent by testers to the authors directly. David has now updated Test-Reporter to suppress the sending of mails to authors, and the CPAN Tester community are currently upgrading their environments with the latest version. As such the current direct mails should now tail off, with the new notification system taking over.

Following some heated discussions last week, part of the plans for CPAN Testers 2.0 have been brought forward. As much of the data recording and presentation is already fairly centralised, it has long been planned to make the author notificatons more centralised to provide more consistent reporting. Due to some very heated threads recently, David Golden and myself decided we should look at putting a centralised email alerting system into place sooner rather than later. David has been looking at alerting testers to broken environments, old CPAN Testers tools and badly formed reports, which will hopefully be live soon, while I have taken on the first part of the whole author notification and preferences system.

The prototype for alerting authors went live this lunchtime (UK time), though if you haven't received an email yet, don't fret it is just likely that none of your distributions have been tested since Saturday in order to receive the mails. As this is just the first part of the full preferences system, I've had to set some defaults. These have been set to notify authors of the FAIL reports, which are the FIRST instance of a distribution/platform/perl combination. Currently all the known PAUSE ids have been entered into the database with these defaults, apart from a few authors who have expressed other default preferences. If you do not wish to receive these alerts, or would like to alter the preferences, please send me an email (barbie[at]cpan.org) and I'll make the updates as requested.

The notification system will run once a day and collate a list of the links to the reports via the web interface on the NNTP server. As such, you will no longer receive huge reports in your inbox, but at most just one email a day containing a list of links to your reports. You can then view them at your leisure. If you have any problems with the notifications themselves, please let me know, and I would welcome any suggestions for improvements. I may not be able to implement them straight away (and may already have some on my TODO list), but please feel free to email me privately, comment here, or join the cpan-testers-discuss mailing list and join the dicussions.

[1] If you can think of a suitable word beginning with 'L' to replace 'Service' let me know ;)

Latest Summary - 3rd September 2008

CPAN Testers Stats - August Summary - Summer Fun

Assembled CPAN Testers at The CPAN Testers BOF in Copenhagen (click to see large image)

This month we hit the 2 millionth post to cpan-testers, thanks to Chris Williams. I congratulated him after his talk in Copenhagen, but he was more relieved to find out that the report itself was a PASS :) I still need to check which was the 2 millionth verified report, but I suspect Chris will claim that too. With such a high volume of reports, it's getting more urgent to move to something other than SMTP. There have been further discussions about updating the transport method and hopefully there should be some movement on that during the latter part of the year. In the meantime, if you're interested in getting involved in the way CPAN Testers gets reported in the future, feel free to join cpan-testers-discuss@perl.org, and add your thoughts.

Having taken over the code that runs the CPAN Testers Reports website, I have been hard at work behind the scenes updating all the code that generates the database and websites. As the CPAN Testers Statistics site worked from an enhanced version of the database, the first hurdle has been to merge the two and ensure both websites can be rebuilt from the same database. Despite a few hiccups with the NNTP server, that part of the process has gone very well. As a consequence several code releases of CPAN-WWW-Testers-Generator have already hit CPAN. The code for the Reports and Statistics site is currently being packaged and will be released soon too. The two sites meanwhile are being heavily tested with the new database, and again all seems to be running smoothly. For some thoughts into my plans for the future of the Reports site, see my use.perl posting.

Last month featured an excursion by many testers to Copenhagen, for the annual YAPC::Europe Perl Conference. Chris Williams presented 'Rough Guide to CPAN Testing', and we held a CPAN Testers BOF at the end of the first day. We had several people along to discuss several aspects of CPAN testing, and Adam Kennedy's PITA idea also got a lot of interest. After the BOF we managed to get a group photo of some of the testers. Next year I hope it will be an even bigger collection of testers :)

Both at the end of Chris' talk and during the first set of Lightning Talks, JJ and I announced that with the support of Birmingham.pm, we would be hosting the second QA Hackathon in Birmingham. We have already spoken to several people, who we hope will be able to attend, and we're hoping a few others will also be able to get involved too. More details will be announced in due course, but expect more CPAN Testers work to be involved in the hackathon.

Again we had over 100 testers submitting reports last month, with 49 new addresses mapped, of which we had 16 new testers identified. Once I've got the current work out of the way, my plan is to graph the number of new and core testers we see each month. I'm interested to see what effect the promotion of CPAN Testers at major events, or significant online promotions, have on the uptake of new testers. After that I hope to look at the platforms tested in a little more depth.

While trawling the back catalogue of articles on the NNTP server, in order to verify the database was building correctly, I uncovered a couple of posts from Brian King; FAQ First Draft Thread and FAQ Second Draft Thread. Interesting to see we still have the same basic principals of CPAN Testing now :)

Latest Summary - 4th August 2008

CPAN Testers Stats - July Summary - The Times They Are A-Changin'

First off an apology. It seems that I used a legitimate email address in a rant back last year, about the use of tester email addresses. At the time, the address didn't appear to be real and had given the impression that the address was being used as a spamtrap. The owner of the address has since contacted me to inform me that it is real, and that it doesn't get used as a spamtrap. So my sincere apologies to Rob for casting doubt on his email address. However, the address in question had caused confusion, so Rob has kindly altered the address he now sends from, so that authors can feel more comfortable about sending any queries to him. Many thanks to Rob for this.

At the beginning of the month, I launched a new domain for CPAN Testers, cpantesters.org. The new domain will hopefully prove more helpful in remembering all the sites used in the CPAN Testers family of sites, and maybe even start including a whole bunch of new ones. In order to make it even more easier to find other sites, linking between the sites with a standard dropdown is now planned, thanks to the suggestions of a few people. The reports site is currently receiving an update and will be rsync'ing to the public site soon.

August sees many of us here in Europe heading to Copenhagen, for the annual YAPC::Europe Perl Conference. On the first day of the event, on Wednesday 13th August, Chris Williams will be presenting 'Rough Guide to CPAN Testing', with a CPAN Testers BOF planned at the end of the day, so be sure to come along to find out more. I'm also hoping to add to my collection of photos of CPAN Testers, so please make sure you come and say hello to me during the conference if you are a CPAN Tester.

As mentioned last month, Leon has given me the go ahead to update the code behind the scenes for the data collection and creation of the reports site. The first part of the switch over was to merge the code that does the data collection, with the previously forked code that is currently used by the Statistics site and CPAN Dependencies site. When I forked, there were a few fields that weren't previously being recorded that were needed (e.g. the tester email address) and a few that weren't used by the stats site that were dropped. Rather than having two separate systems it makes sense to merge these changes. Plus in the last month or so, I've picked up on reports that were sent in Base64 or Quoted-Printable that were previously not being record properly. There are some further changes, such as the fact that the stats database also parses the UPLOAD articles, and a whole bunch of new tests, which have resulted in quite a nicely packaged little distribution. Before launching it on the world, I'm running it in by generating the database from scratch for all articles. Originally I was going to drop the creation of the articles database, which is held locally, but decided to keep it, as it makes look ups a lot quicker. However, it's currently over 7GB and is still not half of the way through the rebuild! The lastest complete stats database is just over 400MB, and is much more manageable. Thankfully disk space isn't an issue yet, but as a consequence I've made the creation of the articles database optional. Assuming anyone else is mad enough to try and generate this stuff ;)

This time next month, I hope to having all the CPAN Testers sites running off the same database, and will hopefully have the updated reports site online for all. Thanks again to everyone for the patches, ideas and support. It's much appreciated.

Again we had over 100 testers submitting reports last month, with 43 new addresses mapped, of which we had 7 new testers identified. After David Golden's magnificent effort last month, his bots have settled down now to a more sedate pace. As a consequence Chris has once again taken the top spot this month, and has even posted his highest number of reports in a single month. No doubt he's been getting his daughter to help out while he's been at work ;)

Latest Summary - 2nd July 2008

CPAN Testers Stats - June Summary - Bound For Glory

Largely thanks to David Golden, we seem to have broken all records again. David already warned me at the end of May that he was planning to top 100,000 reports in a month, and 3 weeks into June he did just that, after which he started aiming for 150,000! With 138,311 reports finally submitted, David is now our top monthly tester, with Chris and Andreas vying for second place. Although it has to said that Chris does have rather a good excuse for not meeting David's challenge last month, as his latest project, Niamh Olivia Williams, was born last Wednesday. Apparently Chris has machines prepped and ready for her to help him pass David for July ;)

Right across the board, the reports have all hit their highs, with a total of 262,817 reports being submitted during the month. The stats website suffered a little with the daily updates, although this was due to two different parsing errors that surfaced in a few reports, which have now been fixed. The daily updates now seem to be running nice and smoothly, and I'm now looking to finish off some further automation work.

Another of our top testers, Jon Allen, has also helped to identify Mac OS X versions that don't see much testing. While Leopard (the latest release) has some testing on 5.10.0 and 5.8.8, it would be beneficial to see a lot more, for those releases of the perl interpreter. Tiger is seeing the majority of testing, with 5.10.0 and 5.8.8 making up the bulk of reports, so a few more on 5.8.7 and 5.8.6 would be good to see. Lastly, Panther testing seems to concentrate on 5.8.1, 5.8.2 and 5.8.3, and would definitely benefit from more testing on later releases of Perl. Jon has written a script that I hope to adapt to provide a better breakdown for other OSs in future updates.

June also featured YAPC::NA in Chicago, where I presented "How To Be A CPAN Tester". The talk was mostly a brief introduction to CPAN Testing, with several references to the installation and configuration of the tools on the wiki, plus a list of the community websites used. The talk appeared to go down well, and at the BOF later the same day, I got a few interested parties that were looking to setup a CPAN testing box of some sort. Hopefully this also means we'll be getting another Sun machine to test with a variety of perls too. Next month Chris Williams will be giving his version of the talk at YAPC::Europe in Copenhagen.

During my talk in Chicago, I also announced that Léon Brocard, who has been looking after the CPAN Testers Reports site for several years, has handed the reins over to me. After which it seems there was a lot of interest in improving the report capabilities of the site. Gabor Szabo and Adam Kennedy had ideas for updating the site, and יובל קוג'מן (Yuval Kogman, aka nothingmuch) has also since hinted at some ideas too, so over the coming months expect some additional functionality to appear, plus further intergration with some of the other CPAN Tester sites. My thanks to Léon for keeping the site going for all these years and for the major improvements he made, particularly with the use of the YUI colour charts. You can also blame him for getting me involved with CPAN Testers in the first place, after his BOF presentation in 2003 at YAPC::Europe in Paris :)

Again we had over 100 testers submitting reports last month, with 20 new addresses mapped, of which we had 12 new testers identified. In addition, with so many addresses unmatched, I reran a check through the complete set of reports to see if I could find some matches for addresses from old reports. The result was a further mapping of 48 addresses. I suspect some of the still unmatched addresses from the early days of reporting have long since fallen into disuse, so it's unlikely that I'll be able to resolve them. However, if you were a CPAN Tester, especially from 1999 to about 2003, although later is good too, please check the leader board and see if any of your old addresses appear in the list. Let me know, and I'll update the lists.

Latest Summary - 3rd June 2008

CPAN Testers Stats - May Summary - The Great Wide Open

Once again Andreas took a stab at gaining on Chris, and for the second time in recent times, managed to submit the most reports for this month. It's impressive to see that all the major testers are running multiple test environments, to help cover multiple platform/perl combinations. There are roughly 1000 basic setups possible, with many more variations with libraries and compilers, etc, so although we're certainly making quite an impact, we're still a long way from covering the whole matrix. So if you do have a spare box, please think whether it could help with CPAN Testers coverage.

Once again the testers have broken several previous records. New reports topped 176793 for last month, with all the variations, PASS, FAIL, NA and UNKNOWN all seeing their highest counts since CPAN Testers records began. It would have been nice to see the FAIL and UNKNOWN counts not increase, but they're still fairly consistent at 13% of the total. Andreas raised the barrier a little higher, having submitted 54764 reports last month. So far he's the only tester to have broken the 50,000 barrier ... twice. However, I suspect that some testers will be pushing 100,000 reports by the end of the year.

This month we've had 14 further addresses added to the list, which includes 7 new testers, giving us the second highest number of testers, 117, in a month. One of the new testers, Jon Allen, you might also know as the guy behind the facelift of perldoc.perl.org, as well as being another Birmingham.pm'er. Jon happened to notice that there weren't a lot of Mac reports, so offered to give the CPAN Testers ago. He wrote up his experience of installing and configuring CPAN::Reporter::Smoker, which you can read on the CPAN Testers Wiki. If you're interested in setting up a similar automated testing bot, it's a good howto. In the coming months I shall endeavour to write a similar one for CPAN::YACSmoke.

Notice any significant difference in the stats site these days? For those that haven't, the site is now updated some time after 2am (Central European Time) every day. After regularly tinkering with the scripts behind the scenes to make the database more reliable on a daily basis, has now meant I can make the switch to running a site refresh on a daily basis too. With that done, I'm now looking at automating the Bad Upload and Bad Report emails that I send out every month. This will hopefully help authors to quickly spot when they've uploaded a badly formatted distribution, rather having to wait for weeks to be alerted, and for testers to quickly block testing certain distributions by updating the configuration. In the longer term I'd also like to have a report parser that can quickly spot where a testbot has gone rogue, thus alerting the tester to investigate quickly. This will then hopefully give authors a bit more confidence in the system, so they don't ignore reports out of hand.

A further addition to the site came last week, with the "Find A Tester" feature. As several authors are now reading the web interface for the NNTP server, they are finding it difficult to figure out who the tester is for their distributions, particularly when the reports don't get mailed to them. As such I've set up a simple script behind the scenes that does the correct lookup. Hopefully by requesting the NNTP ID of the report, this will avoid me having to put restrictive spam measures on the script. If you intend to use the form, please use it wisely.

Latest Summary - 6th May 2008

CPAN Testers Stats - April Summary - Rise Of The Machines

Not to be out done after last month, Chris has once again stepped up his testing, with Andreas closely following his heels. Slaven has also dramatically ramped up his testing, and the top 3 testers are now producing over 20,000 reports more than their nearest rivals. The totals didn't quite beat last month's records, but it was still a phenomenal month nonetheless.

After suggestions from several people, I revised the graphs of the Graphs page. Using the Google Chart API (as suggested by simonm), I managed to generate some better graphs, however, with the extreme values in the data and the large amount of data, the first attempts resulted in errors. As such the main graph is now only for the uploads, reports and passes, while the last graph now lists all the non-pass results. The presentation is much better than the previous attempts with GD and I've been quite impressed with the Chart API. I might start looking at some other forms of the charts in the coming months.

Earle also suggested I sort out the numbering in the matrices, which seemed a reasonable requested, so I've also fixed that too :)

Following my half yearly request to unknown testers, I received 8 responses, which have been added to the addresses list, together with 20 further address additions, 3 of which are from new testers. There are still a large number of testers who are unknown, so hopefully they'll get around to responding in the next month.

In other news my talk submission for YAPC::NA has been accepted. Once again I'll be talking about How To Be A CPAN Tester, which will be based on last year's talk, but with all the updates happening over the last few months, should include quite a lot of news for future changes.

Latest Summary - 7th Apr 2008

CPAN Testers Stats - March Summary - New Beginnings

Well someone has finally blown Chris Williams out of the water. Chris was steadfastly producing his 30,000+ reports as he's been doing for the past 5 months, but after tipping him off a while back about Andreas catching him, Andreas decided to out do himself this month. Thankfully the cpan-testers mailing list has been turned off for outbound, otherwise Andreas' 50683 reports would have probably wiped out the servers at perl.org. In fact I suspect he may have only ramped up because the outbound mail was turned off. As a consequence the total number of reports submitted last month, far exceeded anything we've seen before, topping 150858 reports! With Slaven Rezic, Imacat and Daniel Bosold also stepping up their testing, I'm expected to see this level of report submissions to continue.

Despite all the testing currently being performed, it is perhaps interesting to note that Perl 5.8.8, 5.10.0 and 5.11.0 are the 3 most tested versions of the perl interpreter. While these are likely to be the 3 versions of most interest to many developers, it would be good to keep the presence of earlier perls, especially 5.005, 5.6.1, 5.6.2 and a selection of 5.8.*. I know some of the testers are certainly trying to cover some of the traditional platforms, but if you have any machines that don't get regularly tested, and can spare the clock cycles, please see if getting involved in CPAN Testers is something that you could do. At the moment NetBSD seems to have the best coverage of testings, so if you have a Darwin, Solaris, Win32 or other unusual platform, all test reports will be gratefully received.

16 more addresses mappings, including 9 new testers. As of this morning, my 6 monthly tester clarification mail has been sent out to 60 addresses used in the last 6 months that haven't been correctly mapped. If you receive one of these mails, please take the time to complete it and send it back to me.

The Oslo QA Workshop seems to have been a worthwhile experience for CPAN Testers, particularly David Golden, Ricardo J Signes and Jonas B Nielson, who between them have helped to generate the HTTP transporter layer for updating a central CPAN Testers database into shape. Ask Bjørn Hansen is already working on the central store, so this is a very welcome piece of the solution towards moving towards removing the email transport layer altogether. The final solution is still a while off yet, but things are moving, which is great. So thanks guys for taking the time out to work on the code.

I recently attended the UKUUG 2008 Spring Conference, during which a discussion arose around CPAN Testers, and how other dynamic language repositories handle their testing. It would seem that Perl is head and shoulders above every other language repository, never mind dynamic languages. There were several developers of other languages that were truly amazed at the voluntary effort being put into ensuring CPAN is as well tested as possible. So a hearty pat on the back to all the CPAN Testers, you're setting a standard for others to aspire to. And to quote Mike Whitaker ... Perl is SO not dead :)

Latest Summary - 3rd Mar 2008

CPAN Testers Stats - February Summary - End of an Era

The cpan-testers mailing list finally got turned off for outbound email. The outbound part of the mailing list has been a growing cause for concern for Ask and Robert, as with the increasing volume of reports, the mail server at perl.org has been suffering, and consequently all the other mailing lists, particularly all the Perl Monger group and Perl project mailing lists have been seeing severe delays. Reports should still be sent to the list, but all access and reference to them should be done through the NNTP server or the web based list. PAUSE Uploads can still be received via email thanks to the new cpan-uploads mailing list. This is read-only mailing list, and any mail sent to it will be ignored. Authors and Testers wishing to continue discussions for any particular reports should direct them to the cpan-testers-discuss mailing list. See CPAN Testers Wiki, particularly the Mailing Lists page, for further details.

Despite the slight drop in testers and thus reports last month, it is worth noting that both the number of platforms (70) and the number of perls (21) are the highest we've ever had. Of course we would be delighted to see more of both ;) Over the past few months it's been interesting to see that all the top 5 testers are European based, and last month Serguei Trouchelle, who has been a tester for just under a year, has leapt into 5th place. Well done Serguei, and to all the testers for contributing their efforts.

6 more addresses mappings, including 3 new testers. Even though there have been several new addresses spotted, and even some that have been around for a while, I think it's getting time again for my mail requesting clarification of who people are, so expect an email sometime this month if your email address isn't too obvious. Some are kind enough to include hints for me, but still the majority like to disguise their involvement.

The Oslo Perl Mongers are organising the Oslo QA Hackathon in April, followed by the Go Open Conference, where Adam Kennedy and Michael Schwern will be presenting. For various reasons it looks like a number of those involved with CPAN Testers won't be able to make the event, but there are plenty of TAP and standard Test tool authors going that it should be a good event to attend. I'm planning to host a CPAN Testers BOF at both YAPC::NA and YAPC::Europe, with more of a mind to figure out some specifics for the future of CPAN Testers. In the meantime, the TAP and Test toolchain are likely to get a good round of tuits during the Hackathon, so hopefully we'll all see the benefits of the Hackathon.

Latest Summary - 5th Feb 2008

CPAN Testers Stats - January Summary - One Million Reports

Last month marked quite a momentous occasion, as David Cantrell submitted the one millionth report. I think Chris was after that accolade, but he'll just have to make do with being the highest test report submitter :) Thanks to the increased interest in bulk testing from our current top testers, we reach this point much sooner than I would have expected a few years ago. Thanks to these guys a large portion of CPAN has already been tested on 5.10, and we now have many reports across nearly all versions of Perl 5, certainly those known to be available in production environments.

Slaven Rezić's CPAN Testers Matrix is now accessible from each CPAN distribution page, as is David Cantrell's CPAN Dependencies site, although with the expected increase in traffic, Dave is currently moving the site to a new box. Using the database generated for the stats site, Andy Armstrong has created a slightly different presentation of the success/failure graph that's on the stats site. I currently create all the graphs statically, which is fine when I'm updating once a month. However, if I get the site updated more frequently, then it would be useful to enable you to click on the report you would like to see and create more professional looking graph/chart. I may well investigate more of manyeyes later.

Interest in CPAN Testing is increasing, even if some authors still appear to be completely unaware of who and what CPAN Testers are. One thing for anyone confused by these reports, they are generated by computer, and are often unattended. As such instructions in the README file don't get read. If you have a prerequisite that is outside of the MakeMaker or Module::Build prerequisite framework, try and detect it using Devel-CheckLib. The Notes For CPAN Authors wiki page was specifically written for authors who need pointers to enable them to achieve a clean install. Although it should be noted, that while CPAN Testers aren't necessarily average users, they are trying to recreate environments that a brand new user would likely experience. The reports aren't there to harangue or berate you, they are trying to highlight problems that occurred during the build and testing processes. In some cases this may highlight problems in the automated test environment, which testers try very hard to resolve as soon as possible, but in the bigger picture we are making CPAN a more reliable and enviable code repository.

17 more addresses mappings, including 8 new testers. Welcome aboard folks.

Latest Summary - 2nd Jan 2008

CPAN Testers Stats - December Summary - Strength to Strength

More mappings with 13 new address mappings, including 6 new testers. Welcome to the new testers and hopefully we'll see even more in 2008. As we close out 2007, it's good to see that there has been a healthy interest in CPAN Testers. Even if it has been considered as an annoyance for some. There are now 61 different platforms being tested and 20 different versions of Perl. Regards the latter value the patch levels have been ignored in the stats, although you can still get at the raw data in the SQLite DB if you wish. The several of the major league testers have upgraded their environments to additional include Perl 5.10.0, which is now leading the test reports for the month. Not too surprising seeing the responsiveness of CPAN Testers these days.

Taking a look at the tester submissions over the last month, it's impressive to see so many now regularly submitting several thousand a month. And we should not undervalued the efforts of those only submitting a few hundred a month, as only a few years ago they would have been topping the monthly stats. Although ActiveState have effectively stopped promoting Perl 5.6.1 on their page (it is still available though), the testing of the two latest releases on Windows has probably been the best it has been for some considerable time. The major operating systems on the whole seem well represented, although it would be nice to see some VMS, AIX, HP-UX or other less used, but still popular, operating systems in there.

The number of badly formatted uploads and reports seem to be diminishing these days too, which has been good. We still get the odd one creeping through, but hopefully sometime during 2008 my alert script won't be compiling any emails for me :) One thing I do want to reiterate though, is that if you're going to get involved with CPAN Testing, please use an email address that people can contact you at. There are still a few who are using unreachable addresses and it makes it very difficult for authors to understand why something went wrong if you don't respond.

The stats database has now been successfully ported to the website box, with nightly remote backups just in case anything goes wrong. The code changes seem to have improved matters considerable and the database updates now take just under an hour to complete. Considering the rise in reports, I'm quite pleased with the improvements. Now to start looking at updating the website nightly :)

Latest Summary - 12th Dec 2007

CPAN Testers Stats - November Summary

Sorry for the delay in getting the stats site update this month, for various reasons I've been busy with other things and unfortunately the stats update has been a lower priority. I'm looking at updating the code at the moment to speed the process up and hopefully automate the site updates on a more regular basis.

19 new mappings, including 7 new testers this month, so welcome to you guys and thanks for getting involved. After our dramatic rise in testers over the last couple of months, unfortunately that number has dropped back again slightly this month. I would also like to thank, once again the testers who include their name or tag, that helps me to identify them. Many are new testers who aren't CPAN authors, so it makes my life a lot easier. Chris once again breaks his monthly record for most reports in a month with over 32,000 reports. After an early lead at the beginning of the month, I was hoping Andreas might finally take top spot for the month ... just to give Chris a bit of competition :) All the top ten testers have stepped up their testing this month, with all of them submitting over 1,000 reports, so a big thanks to them and all the CPAN Testers for taking the time to test and submitting their reports. 55 different perls got tested last month, which is the highest we've ever had. Although it should be noted that the rise was mostly due to the different patch versions. I'm considering folding those into the main release, as the CPAN Testers results site does this.

Latest Summary - 2nd Nov 2007

CPAN Testers Stats - October Summary - Testers New & Old

After successfully mapping 54 addresses from September up to mid-October, I sent out the six-month request to all outstanding addresses used so far this year. After only a short time I'd already received several emails from testers updating their details for me, mapping a further 14 addresses. It is notable that several new and current testers are now putting their names, or a least some form of reference in their addresses, that makes it very easy for my scripts to reference them, so thanks very much for that.

One thing that does still irritate me though is testers who use an illegal, malformed or unreachable address. Aside from the stats, if as an author I receive a rather unusual test result and want to follow it up, who do I contact? 'root@dmz1' or 'root@bender.domain' is of no used whatsoever. If you're going to get involved with CPAN Testers, ** PLEASE ** use a legitimate and accessible email address so that authors can contact you if they need to.

If you are a current tester and use email addresses like this, please change them so you can be contact by authors and not have communication end in a black hole. It has already been proposed that testers should register, somewhat more officially, that they wish to become testers, which the more I think about it, is probably the right way to go. We currently have a large number of testers regularly posting reports, even if some are only sending 2 or 3 a month. Those that send with badly configured test environments, we should be catching and encouraging to improve their setup if we can. Registering your interest might be a better way to enable that.

Thanks to the efforts of our top 4 testers, my nightly scripts now take nearly 10 hours to run at the end of the month! The basic update takes roughly 2-3 hours and then the backend analysis and verification takes anything up to 8 hours to run at the moment. I'm looking to improve these as I'd like to be able to automate the publishing of the stats on a daily basis. The database is already published daily, so it shouldn't be too difficult. However, the verification process enables me to check that any uploads and reports haven't been missed or badly formatted, so that I can send the authors or testers emails to alert them to any problems. By improving the automation, I'm hoping this can be done rather more quickly to raise the alerts earlier.

In mid-October Andreas was looking to take over Chris' monthly top tester spot, having submitted twice as many as Chris at the time. Since I happened to mention it to Chris when I saw him the other week, Chris has taken back his regular lead. Andreas has now taken up 3rd position on the leader board, and making a good attempt to take 2nd place. Slaven Rezic is also making notable headway with his testing setup, to the point where he is now the 14th tester to have submitted over 10,000 reports, jumping from 24th to 10th place. Well done Slaven. The other notable mover is Nigel Horne who has moved from 11th to 9th. Several new testers have joined the crew, with both Matthew Musgrove and Phil Monsen making a a very healthy debut, to the point we had 128 testers during October. Overall we have 600 known testers and 546 unmatched email addresses, making up 1146 listed testers at the moment, with only 10 unmatched email addresses for last month.

One of the reasons for increased reports this month was due to the highest number of distribution uploads to CPAN in a single month, with over 1,500 packages being released. I have analysed how many of those are different versions of the same distribution, but maybe that's something I can add when I have a little more time.

If you want to get involved in CPAN Testing, please checkout the CPAN Testers Wiki for details of how to set things up. You can also join the CPAN Testers Discussion mailing list if you need any further advice or if you have any suggestions for improving the tools or resources.

Latest Summary - 2nd Oct 2007

CPAN Testers Stats - September Summary - Breaking Records!

There was a dramatic change in reports last month, with a significant increase in the number of FAIL and UNKNOWN reports. This is in part to a change in the way that some of the reports were being generated. It has been discussed on the CPAN Testers Discussion mailing list and on the Perl QA mailing list, as well as between the main developers of the tools, and several have been generated due to libraries being unavailable on the testing platform and a FAIL or UNKNOWN report being generated. These reports should have been treated as a missing prerequisite and not have been generated for the tested distribution. However, as the installer tool chain treats libraries and other Perl distributions differently, this makes it a little tricky to notify the automated testing tools that the reporting should be abandoned. Thankfully Andreas and David have quickly patched and released upgraded versions of CPAN and CPAN-Reporter, so if you're using that side of the tool chain please upgrade now. CPANPLUS already has a mechanism to detect library failures, so YACSmoke users should not being the same issue, however if you do, please let me know so I can investigate further. For further discussion and David Golden's clarification of the above, please see my use.perl journal entry.

We broke several records for last month in the history of CPAN Testers. Aside from the highest FAIL and UNKNOWN reports being submitted, we also had the most PASS and NA reports in a single month, the most testers in a single month, the most platforms tested in a single month, the most perls tested in a single month, the highest number of reports submitted from a single tester in a single month. But then we did also have the most UPLOADS in a single month too, so it's perhaps not too surprising we've seen some high numbers during September.

This month also the first reports being submitted for 5.10, all 15 patch variants! It has made me wonder whether merging the patched versions into their main release version would be a good thing, to simplify the stats. I'll keep them as they are for now, but anyone has a strong opinion one way or the other, let me know.

Currently in the CPAN Testers Statistics database there are 556 unaccounted tester addresses, with 1318 addresses mapped to 558 real testers. Only 1 new tester spotted this month, although I've added 31 new address mappings. This is great in advance of my sixth month request for confirmation, as there are several addresses that just use 'root' or similar, and it makes it difficult to know they are owned by. So hopefully when I get the chance to send the mails out, at least some of them will reply with more details. If you get a mail, please take the time to update me.

Even though only 1 new tester was spotted, it is wonderful to see several previous testers, who have dropped off for one reason or another, submitting reports again last month. So much so that we had 120 testers submitting reports in September, the highest we've ever had and the first time we've had over 100 testers in a single month. However, it should be noted that ten only submitted 1 report ... but they all count :)

Virtually all the top testers upped their game last month, with all but David Cantrell in the top 6 beating their previous month's number of submitted reports. Chris had another phenomenal month, submitting over 25,000 reports, such that his total in the leader board is a few reports under the combined submissions of his next 4 rivals. David made the leap from 4th to 2nd, while Andreas leapt from 7th to 5th, although I expect at the current rate Andreas will be in the 3rd place by the end of the year. Gabor and Slaven have also made notable jumps further up the standings and following their increased submissions after this month, I expect we'll be seeing them both in the top 20 shortly. The biggest jump this month is from Johannes Plunien, from 33rd to 22nd, who as mentioned last month only started testing in August. It just goes to show that you don't need to be a tester for long to make a worthwhile impact in CPAN Testing.

If you want to get involved in CPAN Testing there is now a couple of established places to get help and advice. The definite resource for setting things up is the CPAN Testers Wiki, with the CPAN Testers Discussion mailing list your first point of call for advice or suggestions for improving the tools or resources. There are some changes planned for the future, so if you're willing to muck in and help out, your help would be greatly appreciated. See the discussion mailing list for more details.

Latest Summary - 2nd Sept 2007

CPAN Testers Stats - August Summary - The Running Man

Andreas, Chris & Dave (click to see large image)

The heat is on for Chris Williams, he only thought Dave Cantrell was chasing his heels, but Andreas Künig has also stepped up his game too. It has been a staggering month for the sheer volume of reports. In fact halfway through the month I did a quick check to see the current state of testing, and discovered that after only 15 days of testing in August, there nearly as many reports as there were for the whole of July! Andreas was less than 1000 reports behind Chris and Dave was less than 1500 behind Chris. However, Chris certainly rose to the challenge.

The use of virtual machines and space on spare boxes by some of the testers has helped to give coverage to a lot of platforms that weren't seeing much exposure. In part this is one of the reasons I created the CPAN Testers Statistics site in the first place, so I'm glad that worked. The Win32 platform has also been making a very significant return to form over the last 6 months as a regular testing platform, which extremely welcome. All the major platforms are well represented these days, even Irix. But more are always welcomed.

Again over the last 6 months, there has been a dramatic increase in the range of perls getting tested, once again largely thanks largely to Chris, David and Andreas. while some testers use virtual environments, which is something that PITA will eventually be in a good position to take even further, it's also interesting to see failures where the default install of Perl is not the one being used for testing. It's highlighted some bad assumptions by authors and has helped to improve some of code on CPAN.

Although there are a significant number of address still unmapped, I have managed to resolve 16 new tester address mappings and 7 new address mappings for known testers this month. However, there is a new challenger who has Chris, Dave and Andreas in his sights, and that's Johannes Plunien. Johannes only started testing last month and has already settled into 4th place for the monthly listings. It is great to see new testers getting involved.

At this year's YAPC::Europe in Vienna, the CPAN Testers BOF proved very fruitful, if only for a photo opportunity for Andreas, Chris and Dave. There were a few additional thoughts discussed, including the change to the subject line of emails. This is something that David Golden recently suggested to Adam Foxson and myself, and although we all agree on the change, the details still need to be clarified. Also discussed was the move towards a non-email based method of fuelling CPAN Testers. Currently the email/nntp method is the primary announcement and report notification repository. However, Andreas has already been looking at an alternative method of announcing the recent uploads from PAUSE and providing a last week, last month, last quarter and last year list should not be too difficult to achieve.

For sometime now there has been discussions about the method of reporting results. There has also been the concern from some authors that they may want to receive reports for particular distributions, for only FAIL reports, or not at all. The initial discussion centred around the use of the META.yml, which although might work, requires parsing being added to the all the tools. However, using a HTTP method of recording the results and then using the central server to send authors the email, it may be more appropriate to allow authors to register to the service (via PAUSE IDs) and alter their settings. At the moment most of this is still in the discussion stage, but it will be a major change to the way CPAN Testers currently works. We still expect the email mailing list, and thus the nntp group, to still exist, as they can be useful for general discussions regarding specific failure reports, but the aim will be to reduce the amount of mail that authors receive, which are not applicable or not wanted.

After photographing Andreas, Chris, Dave and Lars Dieckow before the BOF, I've also thought about expanding the Stats Leader Board. If you meet me at future events, introduce yourself and I'll add you to the role call :)