Daily automated Mozmill test-runs in the QA Lab

Starting today the Mozilla QA lab has it’s own automated Mozmill test-run in place which gets executed once a day. That means I do not have to run those tests manually anymore and we also have public available results which will give us more exposure regarding failed tests. Really, it’s a great step forward for us.

But lets talk in-detail about the progress…

During the last All Hands week in December 2009 I had to hold a presentation in one of our QA group meetings about the current state of the Mozmill tests and about the software update tests. Everyone was excited to see the improvements especially for the update tests which do not need any cumbersome file edits anymore. Finally we had an interesting discussion and collected some ideas about how to run those Mozmill tests in the future. As result we decided to setup a machine in the QA lab which will run Mozmill tests once a day against nightly builds of Namoroka and Shiretoko on OS X, Linux, and Windows. Further we wanted to use it as our machine to run Mozmill tests against release candidates and to verify the update channels. No sooner said than done…

Directly after the Christmas holidays my colleague Al ordered a Mac Mini and did the initial setup routine. He handed me over a system with 3 partitions. One for OS X 10.5, another one for OS X 10.6, and a third partition which is used for all the virtual machines and other necessary data. While talking about virtual machines the following will be in use: Ubuntu 9.10, Windows 2000, Windows XP, Windows Vista, and Windows 7.

My first task was to setup the initial Mozmill environment for all those operating systems. There is a nice documentation up on MDC which helps to make the task as easy as possible. To be able to share builds, scripts, and other stuff between the vms each of those has the shared folders feature enabled and a virtual drive connected to the data partition.

For the daily test-runs against en-US nightly builds of Firefox 3.6.x (Namoroka) and Firefox 3.5.x (Shiretoko), we decided to have OS X 10.5, Ubuntu 9.10, and Windows XP always open. With the help of the crontab on Linux and OS X, and the scheduled tasks on Windows we are able to start the tests at the same time each day. Given by the availability of the update snippets we have chosen 8am PDT for now. I’m not sure if that will work every day due to some slightly time shifts but it’s still something we have to improve. To run all the available tests I have written a wrapper script in Python which runs each of the different tests sequentially. This script will be used from any of the above mentioned vms. The order how the tests are run is given by:

  • Software Update tests: With the software update tests we check that nightly updates can be downloaded and applied via the software update dialog. Also we make sure to run the successive Mozmill tests against the most recent nightly build.
  • Normal tests: Those tests cover most of our implemented tests for the Smoketests, BFT, and FFT test group which do not require a restart of the browser.
  • Restart tests: The restart tests cover the remaining tests and are able to even test complex paths which require a restart of the browser.

Now someone may ask how do we track the results. There is no-one sitting on that machine. That’s correct. Therefor we use the rich feature set of Mozmill 1.4 which is able to send the results as a JSON object to any CouchDB server. For our usage inside QA such a server has been setup a while ago and is also used by our Mozmill tests now. It’s publicly accessible which means that everyone can check and analyze our daily results. If you are interested check-out the summary page on brasstacks. Click on any of the entries to see all the details of each test-run. For the moment the web interface is really simple and hackish but it gives us the chance to easily track failed tests.

Within the next weeks or months a couple of improvements have to be made on the client side but also for the web interface. In general it covers:

  • Finalize the decision which application information has to be acquired by Mozmill and has to be send to brasstacks.
  • Make Mozmill more robust against failures. The biggest problem we have are modal dialogs which can cause a hang of the complete test-run.
  • The software update tests have to be fixed to send results for each test module. Actually only for the first module results are available.
  • Complete new design views are needed to give an easy and structured access to all of the existent results. That is probably the biggest chunk. A separate blog post will follow once we start to work on that item.

Even with those tasks in the queue we are ready to run daily Mozmill tests continuously each day and use the same machine for our release testing work.

As usual we do not rest, so expect more to hear about Mozmill testing in the near future.

Mozmill 1.4 released

The new version of Mozmill brings some under the hood changes to the Python back-end.

The patch on Bug 533227 adds report support for restart tests which can push results to a couchdb instance now. With Mozmill 1.3 only normal Mozmill tests had that capability.

Further we have improved the handling of Python callbacks from within Mozmill tests. While the former version of Mozmill has only supported those callbacks for restart tests, you can use them in normal tests (Bug 538058) now. The only restriction is that the test-run has to be started from the command line client. Users with only the extension installed will not have access to this feature. In the future we will probably use it to implement handling of system level elements and events. Ongoing work can be checked on Bug 525060.

Another great enhancement we are proud to announce are the simplified steps to install Mozmill as command line client on Windows. The only requirement we have is the Mozilla Build environment. There is no need to install Python or the pywin32 extension in parallel anymore. Complete instructions can be found on MDC. A big thanks goes to Atul Varma for his contribution. You helped us a lot!

For more information check-out the complete list of fixes.

Mozmill 1.4 is still in the review queue on addons.mozilla.org. It will take a bit until it is officially available there. So for now grab a copy from Mikeal’s repository and give it a try.

Mozmill 1.3 released

Given the quick review on AMO (many thanks to you guys that this happened under a week!) the Mozmill team can call out that Mozmill 1.3 has been released. It’s available for download on addons.mozilla.com.

This release is a big step forward by adding a couple of new features and fixing some important bugs which have been found by users and have been introduced by the last release. A complete list can be found on Bugzilla.

Let’s give a short overview and mention some of the fixes/features:

  1. Bug 509912: We have updated the maxVersion for all applications. So Mozmill is not compatible up to Firefox 3.7a1pre, Thunderbird 3.1a1pre, and SeaMonkey 2.1a1
  2. Bug 508643: From now on new profiles are created by Firefox itself. Before that fix we have used the files from within the default profile folder of the default application. That caused failures for localized builds because the profile has been initialized with wrong profile data.
  3. Bug 516729: Tests failed by clicking on elements inside the content area if the window was too small. Now with this fix elements will be scrolled into view before clicking on them.
  4. Bug 522990: Nested elements in the content or chrome document weren’t correctly reported by the inspector which didn’t let you get an element string.
  5. Bug 512789: Both controller.check and controller.radio functions have been updated to work now.
  6. Bug 515072: A second parameter has been added for controller.assertJS which let you specify an object which can be accessed via “subject” from within that function. It allows to show a more detailed information for a failing test.
  7. Bug 500987: Restart tests can pass variables between test modules. There is a persisted property available by default which can be used to set/get user-defined values.
  8. Bug 515209: Restart tests can have a callback handler written in Python which can be called asynchronously.

If something has been regressed since the last version please file a bug under Testing/Mozmill on Bugzilla.

If you are interested and want to know more about Mozmill then join us in #QA on IRC or subscribe to the Mozmill developer list.

Automated Software Update tests with Mozmill

Release testing which has to be done by QA right before a new release of Firefox will be offered to our users is still an area where lot of manual work is involved. That means we run Smoketests and the Basic Functional Tests (BFT’s) against the build candidate. As I have already written there is ongoing work with Mozmill to get those work fully automated in the future. But that are not the only tests we have to run…

Since ever Firefox is supporting automatic updates we also have to check that each and every user will get the right update package for the installed version of Firefox. Most of our users should run the latest version of Firefox but there are also cases where people don’t update immediately or even don’t want to upgrade to the next major version of Firefox. Given that updates have to be delivered to each of the supported branches (e.g. Firefox 3.0.0.x and Firefox 3.5.x) and also as major update for upgrading to the next major version. We also have a community beta program running where users can help testing beta versions of the next Firefox version. Those users will get a separate update offer on another update channel.

Finally there are 4 different channels we have to test for en-US and some of our P1 localized builds. In detail those are in the right order:

  • betatest: This channel makes sure that updates which will be delivered to beta users will pass.
  • beta: Beta testers will get their updates on that channel.
  • releasetest: This channel tests the update snippets which have been pushed to our official download mirrors.
  • release: Default channel for all Firefox installations to get the next version.

For each of those mentioned channels we offer partial and complete updates. The former one will be used if the latest minor version of Firefox is in use, e.g. a user wants to update from 3.5.4 to 3.5.5, while the latter one is for all other versions of the same branch. If an update fails to apply which could happen due to different reasons like a download problem, users will not get stuck on their installed version. In such a case a fallback update will be downloaded which is identical to the complete update. If that fails too the same process will be started again after a given time.

Until now these tests had to be done manually by us. An example can be seen in the update section of the Firefox 3.5.5 test plan. So we normally tests updates on all supported platforms, for each update type (minor, major), and make sure that fallback updates will pass.

With the new software update tests for Mozmill which I have finished two days ago, we can easily automate this process now. The only manual steps which have to be done is to prepare the tests by downloading the necessary builds for all the platforms and place them in their own folders. Once that is done the automated test can be started. It will use all builds within a given folder and runs tests updates for the specified channel. The results are printed in wiki format to the console and only have to be copied to the appropriate Wiki page.

If you are interested in running those update tests you have to install Mozmill on your machine and clone our Mozmill test repository. Detailed steps can be found in the Mozmill test creation tutorial on QMO.

This is a big step forward in a direction where we can run update checks against each localized build of Firefox and can make sure that updates are successfully applied and don’t fail or cause any sort of failure. And it will give QA more time to focus on other topics.

If you are interested and want to know more about Mozmill then join us in #QA on IRC or subscribe to the Mozmill developer list.

MozMill 1.3 beta 1 available for testing

Nearly 3 month after we have released Mozmill 1.2 we are close to our next release of Mozmill. Lots of bugs have been fixed and even a couple of new features were implemented. A nearly complete list you can find on Bugzilla.

Everyone who is using Mozmill regularly is welcome to help us in testing the beta version. As long as no big issues will come up the release of Mozmill 1.3 will happen next week.

If you want to test the extension please download it from Github.

Users of the pyPI packages only have to run “easy_install -U mozmill” to get the latest packages for Mozmill, JSBridge, and Mozrunner.

Thanks in advance.

Mozmill status and goals for 2009

As a couple of you already know Mozilla QA runs its own set of automated functional tests which are separated from the tests in the automated test suite. The main goal for us is to shorten the test duration for manual functional tests so those tests will be run more often. There are Smoketests, BFT’s (basic functional tests), and FFT (full functional tests) available on Litmus which get partly run by QA during release testing or at any time by contributors. Given the fact that all those tests need a lot of time to execute manually, we are working on getting most of those tests automated.

There is a question which I get asked very often from developers: “Why do we need Mozmill tests when we already have a suite of automated tests available?” The clear answer is that those tests are used to simulate user actions on UI elements the same way as when a user would sit in-front of the computer. That means that for example clicks on hidden or disabled elements shouldn’t trigger the execution of the underlying command. That’s the difference to Mochitests which always trigger the command when the synthesizeMouse function is used to click on an element. Another really helpful feature is the capability to run restart tests of any sort. That’s not possible with the existing test harnesses which makes Mozmill test unique.

Seeing the importance of those tests we want to have a full suite of BFT tests for Mozmill by end of Q2 in 2010. The total number of 196 doable tests, except the ones which require OS level interaction, would allow us to run 82% of the tests automated. At the moment 65 of the tests have been already finished and can be run with Mozmill against builds from the 1.9.1 and 1.9.2 branch. For detailed information about the current state and actual work please check the Google spreadsheet.

To get more tests automated the following goals have been set by the QA execution team for Q4 in 2009:

  • Firefox 3.6 will be released this quarter. To enhance our testing we want to automate all of the tests in 4 BFT subgroups prioritized as P1. In general these subgroups are: Awesomebar, Add-ons Manager, Download-Manager, and Tabbed Browsing. This will incite us to write the next 40 tests. A list of all available subgroups and their prioritization can be found in the Feature Ownership document.
  • For us who are working on release testing, software update tests have to be performed for the betatest, beta, releasetest, and release channels. Given the manual work which have to be performed here automation will help a lot. I will finalize my software update tests so they can be run by everyone.
  • Running Mozmill tests you will get results reported in the terminal. Even with the integrated capability to send those reports to a server we don’t have a web frontend to display those results. We want to use Brastacks to visualize Mozmill results similar to the Fennec test results. This work will be a joined effort with Testdev.

If anyone is interested in helping us to write or maintain Mozmill tests you can read more about it in the test creation tutorial or simply join us on IRC and get in contact with whimboo or aakashd. But you can also send me a mail or comment on this blog post. Thanks!

“Mozmill meets L10n” slides available

Over the last weekend Mozilla Camp Europe 2009 has been taken place in Prague. About 150 people from l10n, qa, dev, and advocacy were invited to join this conference which Mozilla Europe is organizing each year.

Given my project to get manual Litmus tests automated with Mozmill I have prepared some slides with a special focus on l10n. But sadly I wasn’t able to join the conference because of sickness. I have to say a big thanks to my colleague Marcia Knous and also to one of our main contributors for Sunbird tests Merike Sell who both hold the session. As informed at the end of the session via IRC the talk was a great success and a lot of questions were ask.

Due to the amount of sessions not everyone was able to join the Mozmill session. Also given all the people who weren’t be able to come I have uploaded my slides for all of you now. Please check the embedded Slideshare content below:

Because I haven’t got any feedback from localizers so far I’m anxious to hear what you think about the usefulness of Mozmill and testing with localized builds. Given by the current number we have over 70 official locales available which are not tested by automated tests and require manual testing from localizers and contributors on a regular basis. With all the 250 BFT and another 750 FFT tests enabled in Litmus manual testing is a time taking action. Running all the tests with Mozmill will take much lesser time, could be run more often, and could cover all platforms which will result in a higher quality of Firefox and helps us to minimize any new regressions for our huge user base.

Please check the following questions I’m interested in getting an answer:

  1. How often does your l1on team run Litmus tests against your locale whether those are BFT/FFT or the localizer test-run?
  2. Would you like to see much of those tests automated and are you interested in running those tests on your local machine for each major and stability release?
  3. Are you interested to help QA in writing Mozmill tests so we have most of them available as soon as possible?
  4. Do you have further ideas how Mozmill can be used in the l10n area additionally to the points I have pointed out in my slides?

Thanks in advance for your feedback!

Mozmill-test repository branched for Firefox 3.5

In the last weeks we discovered a couple of changes in the upcoming 1.9.2 code base which forced us to branch our mozmill-test repository. At first we weren’t sure if we should create separate repositories for the 1.9.1 and 1.9.2 branches or handle everything through multiple heads in the same repository. We made the decision to use the first way (see bug 509290) which finally ended up as the wrong way due to some limitations of hgweb. Means we had to revert all the changes and are now using named branches in the same repository.

If you want to use Mozmill to run tests against Shiretoko / Firefox 3.5.x or Namoroka then you can switch between the different branches with the commands seen below.

$ hg clone http://hg.mozilla.org/qa/mozmill-tests
$ cd mozmill-tests
$ hg branches
default 108:d8529dd87a77
mozilla1.9.1 106:221ecb1a8ea9
$ hg branch
$ hg up -C mozilla1.9.1
$ hg branch
$ hg up -C default
$ hg branch

Given the output default is our current trunk version of the Mozmill tests while the mozilla1.9.1 branch is for Firefox 3.5.x.