12.04 Quality Engineering Retrospective

Ubuntu 12.04 LTS Development Release (Precise Pangolin) Beta 2 is (most likely) going to be released today. This means we are getting quite close to final release! I have been running Precise as my only OS on both of my computers for months now, and it is far and away my favorite desktop I've ever used. It is beautiful, fast, and robust. This post is about the robust part.

After we release Beta 2, we should continue see the Ubuntu and Ubuntu Server improving day by day and to quickly achieve release quality. I have asked Kate Stewart, our release manager, to do everything in her power to ensure that starting with Final Freeze on April 12th each daily image is high enough quality that it could be our final release.

Why am I so confident that Ubuntu will only get better and better? Because Ubuntu stayed of usable quality throughout the development cycle. This created a virtuous cycle, where it was easier to develop and test with, so was then easier to maintain the quality.

After the last UDS, I described how we planned to maintain quality throughout the release. We followed those plans, and got the expected results. I am very very proud of

But rather than repeating the activities that we did, I thought I would look back and see what values arose from those practices. I think it was the following values that really had the impact, and that we should build on for 12.10 and beyond.

1. Verify and fix before landing major changes in Ubuntu
2. Not waiting when something breaks to take action
3. Test for testability, then test rigorously

Let me provide some specific example for each.

Verify And Fix Before Landing

Previously, teams would rush to meet certain development milestones, with the goal of meeting the letter of the law. A package had to be uploaded before Feature Freeze, for example, so a team would just push what they had, even if it was not proven to work, or even know not to work at all!

In 12.04 we took a different approach with packages that tended to have significant impact on the usability of Ubuntu, or that were otherwise important. In fact, the xorg team has been following this approach for many releases, using their "edgers" PPA and calls for testing. For many releases new versions fof X were vetted by a community of dedicated community testers before being uploaded to the development version of Ubuntu. In 12.04, they took this even a step further. In previous releases, while different parts of the X stack were building, user of the development release might upgrade while the archive was in an inconsistent state, because different parts of the new X stack were built while others were still builiding or waiting to build. This could result in situations where X was uninstalled altoghter! In 12.04, the X team actually built the X stack separetly, and then copied the binaries into the archive. Totally verified and fixed before landing!

Many folks have noted the dramatically increased robustness of Unity during the 12.04 development cycle. The Unity team did a lot of work to tune and improve their development processes. This included using a PPA for each new release of Unity, and then having that release rigorously tested (with test cases, a testing tool, etc...) by community members with different kinds of graphics hardware and other setups. Then regressions and problems were fixed in the PPA, testing repeated, and only then being uploaded to the development release.

Ultimately, though, I think Ubuntu Cloud must take the prize for rigor in this area, with their OpenStack testing and verification process. On each and every commit to OpenStack uptream, OpenStack gets deployed to a Canonical Cloud test bed (deployed with Juju, of course), then a full suite of tests run. If the tests pass, it gets automatically built into a PPA. When the team is ready to do a release into Ubuntu, they can make the many necessary tweaks in the PPA before uploading it to Ubuntu. This level of Precision allowed the Server team to stay with cutting edge OpenStack, while maintaning a system that was always working, and therefore testable.

Don't Wait when Something Breaks

This value has really taken hold in the Ubuntu community, and it has really helped. There are 2 areas that I monitor each morning. First, I check how the arcvhices look. I can do this because the Plus One Maintenance team, led by Colin Watson and Martin Pitt in turns, have written a tool that finds problems in he archives. Furthermore, each morning they strive to fix those problems. In this way, uninstallable packages and other problems are fixed before we try to spin that day's daily ISO.

After spinning the daily ISO the QA team runs a set of smoke tests on them. If the tests can run, or fail, the right engineering teams are notified, and either they try to fix the tests, or fix the test failure so we can try spinning the CD again. The daily response meant that it was pretty certain that issues were introduced in the last 24 hours, which in turn made them easier and faster to resolve.

Still, Ubuntu development is incredibly rapid. We didn't want to set up a situation where people were afraid to make changes because they might break something. Therefore, from the beginning of the cycle, we accepted that our testing would not catch everything, and that some things would break. So, we set the goal of quickly reverting changes that caused the development release to be hard to test or use. We only had to resort to this a few times. For example, at one point, LightDM was not able to load any but the default desktop. As a result, it was not possible to use desktops like Kubuntu, Xubuntu, etc... The change was reverted the same day so that testing could continue.

Test for Testability, then Test Rigorously

So, we now have automated testing of Canonical upstream code, as well as daily images and daily upgrade testing. However, we don't consider this the end of the testing process, but the beginning. In other words, we use the automated tests to tell us if the code trunks and images are worth testing harder.

In 12.04 development, we evolved our community testing practices to meet this needs. In the past we would do a "call for testing" which mean "please update and try out Ubuntu, let me notice if anything broke". In 12.04 a "call for testing" changed to include test cases so that we could know what worked, not just what broke, coverage of hardware and configurations by recruiting community members who had the right setups, and organized results.

This thought process was not limited to our only Canonical produced code, however. Before or soon after introducing potentially distruptive changes Nicholas Skaggs, our new Community Team member, collects test cases from the relevant developers, and than organizes community members to execute those test cases. He is also organizing these tests at important milestones, such as Beta 1 and now Beta 2.