2.50 status update and statistics

It has been 2 months since we launched 2.50 alpha and I would like to give you an update on our progress in the stabilization process.

In short, it is taking longer than we expected and we still are finding many issues. As you might know from our wiki, our criteria for completing the alpha testing and starting beta is to reach zero critical and major defects, and less than 100 minor and trivial defects combined.

During 2.40, this process took us 8 weeks so we were expecting that it would have taken roughly the same amount of time for 2.50, but things are not going in the same way.

We started the 2.40 alpha cycle on May 23, 2008 with an overall backlog of 278 open defects. In the 8 weeks till July 18th, when we published beta, 357 new defects were logged and we closed a total of 534, leaving the backlog level at 95 open defects, in line with our exit criteria. The historical trend is summarized in the image below.

When we started 2.50 alpha on the November 21st, we had a similar backlog of 271 product defects. However, since then and up to the end of January 2009, 646 defects have been reported and we have closed 684 defects, leaving the current backlog at 233.
The 2.50 historical trend is in the image below and as you can see, 10 weeks into the alpha cycle, the progress is very uncertain and we have been blocked at this level for a while.

How do we justify that? Have we been sleeping on the job? Actually, we made a lot of progress:

• We fixed almost precisely the same number of defects per week in 2.50 as we did in 2.40 (68.4 defects per week in 2.50 vs. 66.7 defects per week in 2.40);
• We published 9 alpha versions, both in source code format and in the handy appliance format to ease testing
• We manned the Modularity Program that supports about 20 early adopters that are actively developing extensions on top of 2.50.
  

The thing is... the defect inflow has been 40% higher in 2.50 than in 2.40 (65 defects per week vs 45 defects per week) and this is what is slowing us down.

But then... how do we explain this increase? I really do not have a very strong argument to explain it in definitive terms, but I can put forward some theories:

• We do a much better job at testing compared to previous releases. In particular, compared to 2.40, our dedicated QA team has grown more than 300%
• Our community usage of previous releases has grown dramatically in recent months and many users are stressing the systems in ways it was not stressed before. Of the 646 reported defects, 155 (or 24%) have been reported against a production release.
  
• Our community is testing this early release very actively: 2.50 alpha gets downloaded around 800 times a week (without even counting people who install from sources using our SVN repository) and around 10% of the defects are reported by community testers.
  

The complete explanation is probably a mix of all the above points. Overall, this means that 2.50 is going to be the best release ever in terms of quality. Our exit criteria, in fact, is not time driven but quality driven and we will not release the beta version until the intended level of quality has been reached.

Because we do not have a time boundary, it is difficult for us to predict the availability date of 2.50 beta. We hope that it is going to be within a few weeks but you can monitor the progress of our stabilization effort live: if you are an iGoogle user, click here to add a Google Gadget with daily update of our progress graph as in the image below.

You can help us in this process by contributing to the testing effort. Please do download our latest alpha release and test it out. You can participate in the Community testing program organized by our QA team provided or just verify whatever feature is most important to you. In any case, please do not forget to report your defects!