Automated Software Update tests with Mozmill

Release testing which has to be done by QA right before a new release of Firefox will be offered to our users is still an area where lot of manual work is involved. That means we run Smoketests and the Basic Functional Tests (BFT’s) against the build candidate. As I have already written there is ongoing work with Mozmill to get those work fully automated in the future. But that are not the only tests we have to run…

Since ever Firefox is supporting automatic updates we also have to check that each and every user will get the right update package for the installed version of Firefox. Most of our users should run the latest version of Firefox but there are also cases where people don’t update immediately or even don’t want to upgrade to the next major version of Firefox. Given that updates have to be delivered to each of the supported branches (e.g. Firefox 3.0.0.x and Firefox 3.5.x) and also as major update for upgrading to the next major version. We also have a community beta program running where users can help testing beta versions of the next Firefox version. Those users will get a separate update offer on another update channel.

Finally there are 4 different channels we have to test for en-US and some of our P1 localized builds. In detail those are in the right order:

• betatest: This channel makes sure that updates which will be delivered to beta users will pass.
• beta: Beta testers will get their updates on that channel.
• releasetest: This channel tests the update snippets which have been pushed to our official download mirrors.
• release: Default channel for all Firefox installations to get the next version.

For each of those mentioned channels we offer partial and complete updates. The former one will be used if the latest minor version of Firefox is in use, e.g. a user wants to update from 3.5.4 to 3.5.5, while the latter one is for all other versions of the same branch. If an update fails to apply which could happen due to different reasons like a download problem, users will not get stuck on their installed version. In such a case a fallback update will be downloaded which is identical to the complete update. If that fails too the same process will be started again after a given time.

Until now these tests had to be done manually by us. An example can be seen in the update section of the Firefox 3.5.5 test plan. So we normally tests updates on all supported platforms, for each update type (minor, major), and make sure that fallback updates will pass.

With the new software update tests for Mozmill which I have finished two days ago, we can easily automate this process now. The only manual steps which have to be done is to prepare the tests by downloading the necessary builds for all the platforms and place them in their own folders. Once that is done the automated test can be started. It will use all builds within a given folder and runs tests updates for the specified channel. The results are printed in wiki format to the console and only have to be copied to the appropriate Wiki page.

If you are interested in running those update tests you have to install Mozmill on your machine and clone our Mozmill test repository. Detailed steps can be found in the Mozmill test creation tutorial on QMO.

This is a big step forward in a direction where we can run update checks against each localized build of Firefox and can make sure that updates are successfully applied and don’t fail or cause any sort of failure. And it will give QA more time to focus on other topics.

If you are interested and want to know more about Mozmill then join us in #QA on IRC or subscribe to the Mozmill developer list.