Daily automated Mozmill test-runs in the QA Lab

Starting today the Mozilla QA lab has it’s own automated Mozmill test-run in place which gets executed once a day. That means I do not have to run those tests manually anymore and we also have public available results which will give us more exposure regarding failed tests. Really, it’s a great step forward for us.

But lets talk in-detail about the progress…

During the last All Hands week in December 2009 I had to hold a presentation in one of our QA group meetings about the current state of the Mozmill tests and about the software update tests. Everyone was excited to see the improvements especially for the update tests which do not need any cumbersome file edits anymore. Finally we had an interesting discussion and collected some ideas about how to run those Mozmill tests in the future. As result we decided to setup a machine in the QA lab which will run Mozmill tests once a day against nightly builds of Namoroka and Shiretoko on OS X, Linux, and Windows. Further we wanted to use it as our machine to run Mozmill tests against release candidates and to verify the update channels. No sooner said than done…

Directly after the Christmas holidays my colleague Al ordered a Mac Mini and did the initial setup routine. He handed me over a system with 3 partitions. One for OS X 10.5, another one for OS X 10.6, and a third partition which is used for all the virtual machines and other necessary data. While talking about virtual machines the following will be in use: Ubuntu 9.10, Windows 2000, Windows XP, Windows Vista, and Windows 7.

My first task was to setup the initial Mozmill environment for all those operating systems. There is a nice documentation up on MDC which helps to make the task as easy as possible. To be able to share builds, scripts, and other stuff between the vms each of those has the shared folders feature enabled and a virtual drive connected to the data partition.

For the daily test-runs against en-US nightly builds of Firefox 3.6.x (Namoroka) and Firefox 3.5.x (Shiretoko), we decided to have OS X 10.5, Ubuntu 9.10, and Windows XP always open. With the help of the crontab on Linux and OS X, and the scheduled tasks on Windows we are able to start the tests at the same time each day. Given by the availability of the update snippets we have chosen 8am PDT for now. I’m not sure if that will work every day due to some slightly time shifts but it’s still something we have to improve. To run all the available tests I have written a wrapper script in Python which runs each of the different tests sequentially. This script will be used from any of the above mentioned vms. The order how the tests are run is given by:

  • Software Update tests: With the software update tests we check that nightly updates can be downloaded and applied via the software update dialog. Also we make sure to run the successive Mozmill tests against the most recent nightly build.
  • Normal tests: Those tests cover most of our implemented tests for the Smoketests, BFT, and FFT test group which do not require a restart of the browser.
  • Restart tests: The restart tests cover the remaining tests and are able to even test complex paths which require a restart of the browser.

Now someone may ask how do we track the results. There is no-one sitting on that machine. That’s correct. Therefor we use the rich feature set of Mozmill 1.4 which is able to send the results as a JSON object to any CouchDB server. For our usage inside QA such a server has been setup a while ago and is also used by our Mozmill tests now. It’s publicly accessible which means that everyone can check and analyze our daily results. If you are interested check-out the summary page on brasstacks. Click on any of the entries to see all the details of each test-run. For the moment the web interface is really simple and hackish but it gives us the chance to easily track failed tests.

Within the next weeks or months a couple of improvements have to be made on the client side but also for the web interface. In general it covers:

  • Finalize the decision which application information has to be acquired by Mozmill and has to be send to brasstacks.
  • Make Mozmill more robust against failures. The biggest problem we have are modal dialogs which can cause a hang of the complete test-run.
  • The software update tests have to be fixed to send results for each test module. Actually only for the first module results are available.
  • Complete new design views are needed to give an easy and structured access to all of the existent results. That is probably the biggest chunk. A separate blog post will follow once we start to work on that item.

Even with those tasks in the queue we are ready to run daily Mozmill tests continuously each day and use the same machine for our release testing work.

As usual we do not rest, so expect more to hear about Mozmill testing in the near future.