Friday, March 30, 2012

Thanks mterry! (Quickly Tutorial Updated) :)

So I decided I had to bit the bullet this morning and update the ubuntu-application tutorial for Quickly, since desktopcouch is no longer supporter and I therefore removed CouchGrid from quickly.widgets. So I start looking through the tutorial to make notes about what I need to change, and I find everything already fixed by Michael Terry! Amazing. I love working (or in this case not working) on open source projects ;)

Thursday, March 29, 2012

12.04 Quality Engineering Retrospective

Ubuntu 12.04 LTS Development Release (Precise Pangolin) Beta 2 is (most likely) going to be released today. This means we are getting quite close to final release! I have been running Precise as my only OS on both of my computers for months now, and it is far and away my favorite desktop I've ever used. It is beautiful, fast, and robust. This post is about the robust part.

After we release Beta 2, we should continue see the Ubuntu and Ubuntu Server improving day by day and to quickly achieve release quality. I have asked Kate Stewart, our release manager, to do everything in her power to ensure that starting with Final Freeze on April 12th each daily image is high enough quality that it could be our final release.

Why am I so confident that Ubuntu will only get better and better? Because Ubuntu stayed of usable quality throughout the development cycle. This created a virtuous cycle, where it was easier to develop and test with, so was then easier to maintain the quality.

After the last UDS, I described how we planned to maintain quality throughout the release. We followed those plans, and got the expected results. I am very very proud of

But rather than repeating the activities that we did, I thought I would look back and see what values arose from those practices. I think it was the following values that really had the impact, and that we should build on for 12.10 and beyond.
  1. Verify and fix before landing major changes in Ubuntu
  2. Not waiting when something breaks to take action
  3. Test for testability, then test rigorously
Let me provide some specific example for each.

Verify And Fix Before Landing

Previously, teams would rush to meet certain development milestones, with the goal of meeting the letter of the law. A package had to be uploaded before Feature Freeze, for example, so a team would just push what they had, even if it was not proven to work, or even know not to work at all!

In 12.04 we took a different approach with packages that tended to have significant impact on the usability of Ubuntu, or that were otherwise important. In fact, the xorg team has been following this approach for many releases, using their "edgers" PPA and calls for testing. For many releases new versions fof X were vetted by a community of dedicated community testers before being uploaded to the development version of Ubuntu. In 12.04, they took this even a step further. In previous releases, while different parts of the X stack were building, user of the development release might upgrade while the archive was in an inconsistent state, because different parts of the new X stack were built while others were still builiding or waiting to build. This could result in situations where X was uninstalled altoghter! In 12.04, the X team actually built the X stack separetly, and then copied the binaries into the archive. Totally verified and fixed before landing!

Many folks have noted the dramatically increased robustness of Unity during the 12.04 development cycle. The Unity team did a lot of work to tune and improve their development processes. This included using a PPA for each new release of Unity, and then having that release rigorously tested (with test cases, a testing tool, etc...) by community members with different kinds of graphics hardware and other setups. Then regressions and problems were fixed in the PPA, testing repeated, and only then being uploaded to the development release.

Ultimately, though, I think Ubuntu Cloud must take the prize for rigor in this area, with their OpenStack testing and verification process. On each and every commit to OpenStack uptream, OpenStack gets deployed to a Canonical Cloud test bed (deployed with Juju, of course), then a full suite of tests run. If the tests pass, it gets automatically built into a PPA. When the team is ready to do a release into Ubuntu, they can make the many necessary tweaks in the PPA before uploading it to Ubuntu. This level of Precision allowed the Server team to stay with cutting edge OpenStack, while maintaning a system that was always working, and therefore testable.

Don't Wait when Something Breaks

This value has really taken hold in the Ubuntu community, and it has really helped. There are 2 areas that I monitor each morning. First, I check how the arcvhices look. I can do this because the Plus One Maintenance team, led by Colin Watson and Martin Pitt in turns, have written a tool that finds problems in he archives. Furthermore, each morning they strive to fix those problems. In this way, uninstallable packages and other problems are fixed before we try to spin that day's daily ISO.

After spinning the daily ISO the QA team runs a set of smoke tests on them. If the tests can run, or fail, the right engineering teams are notified, and either they try to fix the tests, or fix the test failure so we can try spinning the CD again. The daily response meant that it was pretty certain that issues were introduced in the last 24 hours, which in turn made them easier and faster to resolve.

Still, Ubuntu development is incredibly rapid. We didn't want to set up a situation where people were afraid to make changes because they might break something. Therefore, from the beginning of the cycle, we accepted that our testing would not catch everything, and that some things would break. So, we set the goal of quickly reverting changes that caused the development release to be hard to test or use. We only had to resort to this a few times. For example, at one point, LightDM was not able to load any but the default desktop. As a result, it was not possible to use desktops like Kubuntu, Xubuntu, etc... The change was reverted the same day so that testing could continue.

Test for Testability, then Test Rigorously

So, we now have automated testing of Canonical upstream code, as well as daily images and daily upgrade testing. However, we don't consider this the end of the testing process, but the beginning. In other words, we use the automated tests to tell us if the code trunks and images are worth testing harder.

In 12.04 development, we evolved our community testing practices to meet this needs. In the past we would do a "call for testing" which mean "please update and try out Ubuntu, let me notice if anything broke". In 12.04 a "call for testing" changed to include test cases so that we could know what worked, not just what broke, coverage of hardware and configurations by recruiting community members who had the right setups, and organized results.

This thought process was not limited to our only Canonical produced code, however. Before or soon after introducing potentially distruptive changes Nicholas Skaggs, our new Community Team member, collects test cases from the relevant developers, and than organizes community members to execute those test cases. He is also organizing these tests at important milestones, such as Beta 1 and now Beta 2.

Wednesday, March 21, 2012

I less than 3 Pychart (or how I turned tracking 10 bugs into a programming task)

We are getting closer to release! Beta2 freeze is tomorrow. Quality in 12.04 is looking very good today. However, we still see hundreds of bugs get fixed across desktop and server between now and April 26th. In the past, I've found that in the flury of activity it's easy to lose track of the most important bugs in all that noise, and then some scrambling ensues.

To counteract this, at least for myself, I had a couple of calls, with Jason Warner (Desktop Engineering Manager), Robbie Williamson (Server Engineering Manager), and Steve Langasek (Foundations Engineering Manager). We talked about what bugs we had (that we know about now) that would actually keep us from releasing as scheduled. We have a term called "release blocking bug", but in point of fact, almost none of them would actually keep us from releasing. The kinds of bugs that would truly make us slip a ubuntu release are ones that cause problems with existing OSs in multi-boot situations, serious bugs in the installer, serious bugs in update manager, bugs that result in a loss of networking, etc... Bugs that can reasonably fixed in an update do not block the release.

We decided that the best way to keep track of the very few bugs like this is to continue to track them as normal, but to set their importance as critical.

There is another set of bugs that I also ask the team to focus on. This set is more aspirational. I want us to fix all of the upgrade bugs that we find from automated testing, or at least all of the High and Critical importance ones. I would sincerely love to see every upgrade go smoothly for all of the millions of people who will be upgrading to Precise.

So, when I am going to start talking about pychart? Right now, in fact! Keeping tabs of bugs is boring, so must be automated, and I love automating things with Python. So, I wrote a program that scrapes the data from those 2 pages, store the info in a sqlite database, and generate a line graph each time I run it.

You can see all the code here if you want, but I doubt you do, it's pretty hacky. But, it was fun to bring together the ecellent json, HTMLParser, sqlite3, and pychart libraries.

Here's the pychart money shot:
        xaxis= axis.X(label=_("Date"), tic_interval=1,format = format_date)
yaxis = axis.Y(tic_interval = 2, label="Open Bugs")
ar = area.T(x_axis=xaxis, y_axis=yaxis, y_range=(0,None))
plot = line_plot.T(label="upgrade", data=graph_list, ycol=1,
plot2 = line_plot.T(label="blockers", data=graph_list, ycol=2, tick_mark=tick_mark.square)
ar.add_plot(plot, plot2)

can = canvas.init("/home/rick/Documents/bugs.png","png")
print ar.draw(can)
def format_date(ordinal):
d =
return "/a60{}" + d.strftime("%b %d, %y")