Monday, November 19, 2012

The Road to 14.04




Copenhagen was quite a UDS. I found it to be very "tight" and well organized. The sessions seemed to be extra productive  I think compressing UDS down to four days helped us be more focused, and also less tired by the end. Maybe next one should be three days? Copenhagen was also great for the content. Working on getting the desktop running on the Nexus 7 was very interesting and fun. Also, we made a lot of progress in terms of how we make Ubuntu. I think this will be an unusually fun cycle thanks to some of the changes to our development process.
Sleestacks are monsters bent on keeping you in the past

One of the great things about my job is that I get to talk to so many people about their vision for Ubuntu. As you can imagine, I run into a lot of variety there. However, there is a certain shared vision that has come together. I think we all look forward to 14.04, we can mostly agree about what we want that release to look like.

Seeing the future through conversations with community members, developers, users, etc....
The 14.04 Vision
Imagine that you are running Ubuntu 14.04. What will your experience be like?

  • Robust
  • Fresh
  • Easy to Make
  • Ubiquitous

First, the quality will impeccable  You will not even think about up-time  The system will work close to perfectly. You will eagerly take updates and upgrades knowing that they will only make your system better. Applications will run smoothly and won't cause any system-wide problems.
Secondly, you will be able to get the freshest apps that you want on you client machines, and the freshest workloads on you servers. As a developer, you'll be able to deliver your applications directly to end users, and be able to easily update those applications.
Thirdly, will have an extremely efficient release process, one that also inspires confidence in developers. Good changes will reach end users quickly, while mistakes are easily caught and corrected well before users are exposed to them.
Finally, by 14.04 Ubuntu will run everywhere. The same Ubuntu will be on you phone, your tablet, your netbook, your laptop, your workstation, you cloud server hosts, and the instances powering workloads in your public and private clouds. The same product with the same engineering running everywhere. A simpler world with Free code everywhere.

How Do We Get There?

"It's more important to know where you are going than to get their quickly"
We have laid out the steps necessary to achieve this vision. We intend to make this real! In fact, we've already achieved some of the things necessary to get to where we want to where we want to be for 14.04. Overall, by 14.04 we need to:

  • Assure quality at every step of the development process
  • Improve application sand-boxing
  • Simplify the release schedule
  • Implement continuous integration in the Ubuntu development process
  • By 14.04 expand Ubuntu to include mobile form factors, such as phones and tablets

Assured Quality at Every Step

Leaping through time ensuring everything stays on track
In 13.04 we will change our full testing cadence from testing at each Alpha or Beta milestone, to doing full test runs every 2 weeks. This is about a 3 times increase in the rate of manual community testing. Furthermore, we will test more broadly, more deeply, and more rigorously, so that we will have a more complete view of the quality of Ubuntu during the development release.
We will also be leveraging some previous work to create a GUI testing tool. We call this tool "Autopilot" and it is designed to drive all the components of Unity in a testing environment. In 13.04 we will see expanded usage of this tool, and critically, dramatically more tests written. We will then be able to catch regressions in the Ubuntu user experience earlier, and ensure that fewer regressions make their way into the development release.
The one and only Martin Pitt has implemented a new test harness along with tests for GNOME. In this way, the Canonical QA labs will be able to identify regressions in GNOME as soon as they are introduced. This should allow GNOME developers to more quickly spot and fix problems, raising the overall quality of the GNOME code and improving the velocity of the GNOME developers.
Finally, after 13.04 ships, we will start doing updates in a new way. After 13.04 is a stable release, updates to that release will not be delivered to all users when available. Rather, updates will go out to a small number of users, and the system will automatically monitory whoopsie-daisy to ensure that users aren't experiencing issues due to the update before releasing the update to yet more users. We call this "phased updates".

Ubuntu Continuous Integration

Every day is like the day before, Daily Quality
There are 2 new things that we are doing in 13.04 to get us to the world of 14.04 where releases are easy and confidence inducing. First, Colin Watson has set up the Ubuntu build system so that all packages are built in a staging area (by reusing the proposed pocket for the development release). Only when a package is built succesffull along with all of it's dependencies are the packages copied into the release pocket and go out to the wider development release. This means that there will be no more breakages due to out of sync packages when you update. Compiz and Nux will always be built together before they are copied over. The whole xorg stack too.
Building things in proposed provides an opportunity to assure the quality of the packages before they go into the release pocket. This will be accomplished with auto-package testing. Essentially, tests that come with a package will be run when the package is built. Additionally any package that depends on the new package will also run it's auto-package tests. The package will only be copied into the release pocket when all of the tests pass!

Start Application Insulation

Protecting the world from pure evil

By 14.04 we expect most applications to be run in a secure manner, so that poorly written or even malicious applications will have limited opportunity to do damage to a users system. In 13.04 the Security Team is moving ahead with lots of work to enable App Armour throughout Ubuntu, in addition to isolating some common infrastructure in use today, such as online accounts, gnome-keyring, and even dbus.
In this way, applications will be able to run and access only the small subset of system that is relevant to them. When a user installs an application it will come with an AppArmour profile that will ensure that the kernel can insulate the system from the application appropriately. The fruits of this labor should be widely visible by 13.10.

Simplified Releases

A time turner literally creates more time
Ubuntu has traditionally held a serious of Alphas and Betas. These had the purpose of ensuring that we had an installable image at least a few times during the release, and to provide an opportunity to do some wide testing of the system. This meant that several times throughout the release cycle we would stop development on Ubuntu, freeze the archive and roll a release.
Since the advent of daily quality, Ubuntu can install pretty much every day. Furthermore, we are opting for much more frequent testing than the milestones allowed. Therefore, the Alphas and Betas have limited utility, but would have continued to sap our development velocity. So, in 13.04, Ubuntu is making the bold move of skipping all Alphas, and having just a single Beta! This also allowed us to extend certain freezes, especially Feature Freeze. The new schedule has a much more time for finishing features and fixing bugs, and much less time in freezes.






7 comments:

  1. "will be able to identify regressions in GNOME as soon as they are introduced. This should allow GNOME developers to more quickly spot and fix problems, raising the overall quality of the GNOME code and improving the velocity of the GNOME developers."

    All sounds nice but this does not really hold as you are tracking GNOME one release late, no?

    ReplyDelete
    Replies
    1. In Ubuntu, yes, for 13.04 the desktop team is sticking with stable. However, in our QA labs, we can and will be (if we aren't already) running tests for each commit to tip of GNOME so developers can catch regressions asap after they are introduced.

      Delete
  2. I'm supper excited about all the new changes/features you mentioned.
    But honestly one part I didn't understand very well was App security. How exactly are you going to ensure that all the packages in Ubuntu repositories are safe for users to install and use. Are you using an automatic security system that monitors applications' behavior (like anti-viruses) or what?

    ReplyDelete
    Replies
    1. The plan is to use AppArmour and automatically apply an AppArmour profile to applications at build time. You can look through the blueprints in the links above to get more details. There is work to do to allow AppArmour to cover everything necessary. So in 13.04, expect to see changes that allow AppArmour to work, and then expect in 13.10 a lot discussion about how to apply those capabilities.

      Delete
  3. I would just like to ask a simple question. Are you going to allow updates of non-essential packages separate of system packages, much like what we can see today in App stores of mobile systems or the Mac App Store?

    It would also be a great idea if you could implement delta updates in Ubuntu.

    ReplyDelete
    Replies
    1. Delta updates. That's something I'm waiting for and would affect Canonical's bandwidth bill a lot.

      I remember when Chrome changed its compression algorithm, even a small percentage reduction in their (already) delta updates represented a huge win for google in the end of the day ($$).

      Downloading time would be a great side effect too.

      For reference, for a usual 10MB update, google managed to reduce it to 78kb using delta updates[1].

      [1] http://blog.chromium.org/2009/07/smaller-is-faster-and-safer-too.html

      Delete
  4. Please provide delta debs. in india broadband is slow and costly. and offline updates are not easily ppssible in linux. this alone feature will bring you millions of users from developing countries.

    ReplyDelete