Bug 570790: Due to a broken pyPI package of jsbridge the extension wasn’t working properly on all platforms. At least on Windows the files which have been accidentally added to the tar.gz file, caused an error when loading files from the components and chrome folder of the extension. We have removed all those instances to make sure the extension will work again.

Bug 568839: Installing binary extensions with Mozmill was broken in version 2.4.2. All files which have been extracted from the XPI were handled as ASCII files. With this fix Mozmill is now able to install any version of extensions successfully.

If you still notice any other problems please get in contact with us.

If you are an extension author and interested to see your extension tested in a daily fashion by our automated test-runs in the QA-lab, you should definitely join the testday on Friday, May 28th. We really want to encourage you to think about the implementation of automated functional tests for your own extension. And it’s not only helpful for the extension itself, but can also help us to identify regression in the Firefox development cycle as early as possible.

To have an introduction available, we were working closely together with Google to have example tests in our mozmill-tests repository. Learn how Mozmill tests will be written and how they can be run in Firefox. The Mozmill team will be around the whole day to assist you wherever possible.

If you are interested in the testday, you should read through the following documentation about the creation of testscripts for extensions.

You can also attend when you have general questions about Mozmill or when you want to help in fixing or creating Mozmill tests for Firefox. Get ready and join us in #testday next Friday.

That means we have the following correlations now:

default => Firefox 3.7
mozilla1.9.2 => Firefox 3.6.x
mozilla1.9.1 => Firefox 3.5.x

More details about branch handling for our Mozmill tests can be found on MDC.

The Mozmill test repository is available under the following two locations:

http://hg.mozilla.org/qa/mozmill-tests/
http://github.com/whimboo/mozmill-tests

A list of fixes, which have been made it into this release, can be found on Bugzilla. Lets give an explanation for some of those:

• Bug 542000: With former Mozmill releases there were needs to install the Python for Windows extensions (pywin32). If you have used the MozillaBuild environment, changes to the registry were needed. All that work hasn’t to be done from now on. Installing MozillaBuild is enough to prepare the system for Mozmill. A big thanks goes to Atul Varma who removed that dependency and made our life more compelling.
• Bug 543501: One of our upcoming projects is the execution of Mozmill tests against add-ons. Therefore we have to make sure that the usage of Mozmill will be as easy as possible. That’s why the misleading –plugin command line option has been changed to –addons. Make sure to use this option from now on.
• Bug 548446: Mozmill had problems with installing some of the existing extensions because it was not able to find the extension id. Thanks goes to Jonathan!
• Bug 544896: If you have long test-runs it was possible that the software update dialog came up and removed the focus from the window currently under test. That’s why the automatic update for the application has been disabled. If you want to run tests against that feature make sure to enable the preference app.update.enabled before.
• Bug 558404 and Bug 559152: Some controller functions have been updated which missed some details about failures. Now we always have a stacktrace with full information available.

The new version of the add-on can be found on addons.mozilla.org. It’s under review right now but can already be installed. For all the others who are using the command line client of Mozmill, you can simply run the “easy_install -U mozmill” command to update to the new 1.4.1 release.

If you have questions don’t hesitate to send your feedback to the mozmill-dev mailing list or directly contact us on IRC in the #qa channel.

But lets talk in-detail about the progress…

During the last All Hands week in December 2009 I had to hold a presentation in one of our QA group meetings about the current state of the Mozmill tests and about the software update tests. Everyone was excited to see the improvements especially for the update tests which do not need any cumbersome file edits anymore. Finally we had an interesting discussion and collected some ideas about how to run those Mozmill tests in the future. As result we decided to setup a machine in the QA lab which will run Mozmill tests once a day against nightly builds of Namoroka and Shiretoko on OS X, Linux, and Windows. Further we wanted to use it as our machine to run Mozmill tests against release candidates and to verify the update channels. No sooner said than done…

Directly after the Christmas holidays my colleague Al ordered a Mac Mini and did the initial setup routine. He handed me over a system with 3 partitions. One for OS X 10.5, another one for OS X 10.6, and a third partition which is used for all the virtual machines and other necessary data. While talking about virtual machines the following will be in use: Ubuntu 9.10, Windows 2000, Windows XP, Windows Vista, and Windows 7.

My first task was to setup the initial Mozmill environment for all those operating systems. There is a nice documentation up on MDC which helps to make the task as easy as possible. To be able to share builds, scripts, and other stuff between the vms each of those has the shared folders feature enabled and a virtual drive connected to the data partition.

For the daily test-runs against en-US nightly builds of Firefox 3.6.x (Namoroka) and Firefox 3.5.x (Shiretoko), we decided to have OS X 10.5, Ubuntu 9.10, and Windows XP always open. With the help of the crontab on Linux and OS X, and the scheduled tasks on Windows we are able to start the tests at the same time each day. Given by the availability of the update snippets we have chosen 8am PDT for now. I’m not sure if that will work every day due to some slightly time shifts but it’s still something we have to improve. To run all the available tests I have written a wrapper script in Python which runs each of the different tests sequentially. This script will be used from any of the above mentioned vms. The order how the tests are run is given by:

• Software Update tests: With the software update tests we check that nightly updates can be downloaded and applied via the software update dialog. Also we make sure to run the successive Mozmill tests against the most recent nightly build.
• Normal tests: Those tests cover most of our implemented tests for the Smoketests, BFT, and FFT test group which do not require a restart of the browser.
• Restart tests: The restart tests cover the remaining tests and are able to even test complex paths which require a restart of the browser.

Now someone may ask how do we track the results. There is no-one sitting on that machine. That’s correct. Therefor we use the rich feature set of Mozmill 1.4 which is able to send the results as a JSON object to any CouchDB server. For our usage inside QA such a server has been setup a while ago and is also used by our Mozmill tests now. It’s publicly accessible which means that everyone can check and analyze our daily results. If you are interested check-out the summary page on brasstacks. Click on any of the entries to see all the details of each test-run. For the moment the web interface is really simple and hackish but it gives us the chance to easily track failed tests.

Within the next weeks or months a couple of improvements have to be made on the client side but also for the web interface. In general it covers:

• Finalize the decision which application information has to be acquired by Mozmill and has to be send to brasstacks.
• Make Mozmill more robust against failures. The biggest problem we have are modal dialogs which can cause a hang of the complete test-run.
• The software update tests have to be fixed to send results for each test module. Actually only for the first module results are available.
• Complete new design views are needed to give an easy and structured access to all of the existent results. That is probably the biggest chunk. A separate blog post will follow once we start to work on that item.

Even with those tasks in the queue we are ready to run daily Mozmill tests continuously each day and use the same machine for our release testing work.

As usual we do not rest, so expect more to hear about Mozmill testing in the near future.

Since ever Firefox is supporting automatic updates we also have to check that each and every user will get the right update package for the installed version of Firefox. Most of our users should run the latest version of Firefox but there are also cases where people don’t update immediately or even don’t want to upgrade to the next major version of Firefox. Given that updates have to be delivered to each of the supported branches (e.g. Firefox 3.0.0.x and Firefox 3.5.x) and also as major update for upgrading to the next major version. We also have a community beta program running where users can help testing beta versions of the next Firefox version. Those users will get a separate update offer on another update channel.

Finally there are 4 different channels we have to test for en-US and some of our P1 localized builds. In detail those are in the right order:

• betatest: This channel makes sure that updates which will be delivered to beta users will pass.
• beta: Beta testers will get their updates on that channel.
• releasetest: This channel tests the update snippets which have been pushed to our official download mirrors.
• release: Default channel for all Firefox installations to get the next version.

For each of those mentioned channels we offer partial and complete updates. The former one will be used if the latest minor version of Firefox is in use, e.g. a user wants to update from 3.5.4 to 3.5.5, while the latter one is for all other versions of the same branch. If an update fails to apply which could happen due to different reasons like a download problem, users will not get stuck on their installed version. In such a case a fallback update will be downloaded which is identical to the complete update. If that fails too the same process will be started again after a given time.

Until now these tests had to be done manually by us. An example can be seen in the update section of the Firefox 3.5.5 test plan. So we normally tests updates on all supported platforms, for each update type (minor, major), and make sure that fallback updates will pass.

With the new software update tests for Mozmill which I have finished two days ago, we can easily automate this process now. The only manual steps which have to be done is to prepare the tests by downloading the necessary builds for all the platforms and place them in their own folders. Once that is done the automated test can be started. It will use all builds within a given folder and runs tests updates for the specified channel. The results are printed in wiki format to the console and only have to be copied to the appropriate Wiki page.

If you are interested in running those update tests you have to install Mozmill on your machine and clone our Mozmill test repository. Detailed steps can be found in the Mozmill test creation tutorial on QMO.

This is a big step forward in a direction where we can run update checks against each localized build of Firefox and can make sure that updates are successfully applied and don’t fail or cause any sort of failure. And it will give QA more time to focus on other topics.

If you are interested and want to know more about Mozmill then join us in #QA on IRC or subscribe to the Mozmill developer list.