Tag Archives: jenkins

Firefox Automation report – week 1/2 2014

I promised to keep up with our updates over the last week but given a major breakage in the freshly released version of Mozmill 2.0.4, I had a full week of work to get the fix out. I promise that during this week I will write reports for the weeks in January.


With the new year our team has been reorganized and we are part of the Mozilla QA team again. That means we will have a way closer relationship to any feature owner, and also working towards in bringing more automation knowledge to everyone. The goals for our team are getting worked out and I will present those in one of my following blog posts. As of now you can find our team page on the Mozilla wiki under Firefox Automation.

Since the landing of all the new features for Mozmill-CI on our staging machine before Christmas, we have no longer experienced any crash of the Jenkins master. Given that Henrik pushed all the changes to our production system. We are totally happy that the incremental updates made our system that stable, and that Mozilla QA doesn’t have cope with outages.

Henrik and Jarek were both working on the mozfile 1.1 release to make it more stable in terms of removing files when those are still in use or don’t have the right permissions set.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 1 and week 2.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the first Firefox Automation meeting of week 2.

Automation Development report – week 51/52 2013

Wow, somehow I totally missed to send out reports for our automation work. Most likely that happened because of the amount of work I had in the past couple of weeks. So for now lets do a final update before the title will be updated to ‘Firefox Automation report’ by the year 2014.


We have released Mozmill 2.0.3 to fix a couple of issues (see dependencies on bug 950831 seen with Firefox Metro builds and our Firefox shutdown code. We pushed those changes together with the releases of mozmill-automation 2.0.3 and the new mozmill-environment files to our mozmill-ci staging instance for baking.

Henrik was able to finish the work in setting up our new mozmill-ci staging instance in the SCL3 datacenter. Please see bug 947108 for details. With it we have the identical environment as the production instance and can see regressions immediately and not when we merge to production, which was pretty bad in the past couple of week. So RIP old staging server!

One of our goals for quarter 3 in 2013 was to setup a web based configutation tool for ondemand testruns in mozmill-ci, which can be used by QA people to trigger testruns for beta and release builds. Cosmin jumped on it and got the first version implemented. You can find a running instance on Github for now. Later we want to make the tool available via http://www.mozqa.com.

To make our mozmill-ci system more stable, Henrik pushed a large set of new features and fixes to the staging instance. Our plan was to let it bake over the Christmas holidays with the hope that Jenkins will run way more stable now.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 51 and week 52.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the last Automation Development meeting of week 51.

Automation Development report – week 47/48 2013

This is again an overview about the highlights of our work in week 47 and 48 this year. Sorry for the delay of a couple of days but some critical work was holding me off from writing this post.


Henrik and Dave were working on a couple of Mozmill-CI updates, which have been pushed to production. The first batch of features and bug fixes landed in week 47. It included the monitoring plugin for Jenkins, which hopefully will help us to figure out the reasons of Java crashes. Also we can finally run project branches via our CI now, even it can only be done manually as of now. It is important for our upcoming tests for the Holly branch (Firefox Nightly without the Australis feature). The second batch landed by last week was intended to upgrade our system to Mozmill 2.0.1. Sadly we failed in it due to a couple of other failures, which we haven’t seen before on our staging server. So we partly reverted back the latest commits for production and we all are working hard on getting those issues fixed.

With the detected failures by upgrading to Mozmill 2.0.1, Henrik has noticed that one of those existed because of an incompatibility of mozbase with Python 2.6. See bug 944361 for details. To solve this we upgraded our OS X 10.6 boxes to Python 2.7.3, so all machines are running the same version of Python now. As a very nice side-effect we noticed a speed improvement by running our Mozmill tests of about 25%!

Henrik pushed a couple of releases to pypi, which include mozprofile 0.17 with a lot of add-on manager related bug fixes, mozdownload 1.10 (see details), and mozmill 2.0.1 (see details)

To be prepared for executing the first Metro tests with Mozmill we had to prepare the mozmill-tests repository for handling multiple applications. Therefore Andreea refactored nearly the whole repository. You can find details on bug 927397.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 47 and week 48.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the last two Automation Development meetings of week 47 and week 48.

Automation Development report – week 36 2013

It looks like since everyone is back from the summer vacations we keep our productivity kinda high over the last couple of weeks. During the last week we finished again a couple of important tasks.


As announced in last weeks report we were going to upgrade our Mozmill continuous integration system to the latest LTS release of Jenkins. That task has been done now, after a full week of observation on our staging instance. Everything looked fine, so Henrik pushed the upgrade to production on Monday. Since then the system hasn’t shown any instability, but gave us a fantastic improvement in terms of memory usage. The mean heap usage dropped from about 1.6 GB to only 800 MB, which is only 50% compared to the formerly usage! Also with other updates for used plugins we were able to close some long outstanding issues.

As requested by Mozilla QA a while ago we were finally able to get our Windows 8.1 enabled and activated for testing. The crashes we have seen here frequently when logging into those machines were really related to the remote desktop service. With a hint from Clint Talbert to use VNC instead (what we actually do already on all the buildbot machines), we were able to stop the crashes by simply using UltraVNC now. Since the tests are getting run on that platform no specific regressions have been observed. So as of now all looks good for a release on Windows 8.1.

Given that the release of Mozmill 2.0 is really close, we started to work on a transition plan, which would allow us to easily flip between Mozmill 1.5 and 2.0 in our mozmill-ci system. To accomplish that we have decided to release mozmill-automation as a package on PyPI from now on. That means the appropriate version will be pre-installed in the mozmill-environment, which the Jenkins jobs are using. A bit of work has to be done on the CI side too, but once that happened we will have a second CI instance, which will run Mozmill 2.0 in parallel to our production system. If all is green we will release 2.0 next week.

Henrik is also working to get the mozmill-l10n tests moved to our production mozmill-ci. As of now those were running on an old Mac Mini located outside the datacenter. All the code it was using was outdated too, so after lots of updates we pushed the changes to staging last week. By next week I will give some more details about it, probably by a separate blog post.

But also on the Gaia front we were able to make some good progress. Dave found the time to finally create a command line interface tool to directly communicate with Gaia via simple commands. You should really check this out!

A big step for us was also the fact that Dave was able to get the Gaia-UI endurance tests landed into the main Mozilla Gaia repository for the v1-train! With that landing the Jenkins instance has been updated to run the tests with nightly builds from the new location now. If you are interested in running those tests on your own, you shouldn’t miss to read the blog post from Rob in how to run the Gaia-UI endurance tests.

In terms of Eideticker Dave also was able to make progress and setup the first working version of the Eideticker CI. Over the curse of the next couple of week it will get further improvements, so we will have a trusted source of test results.

Last but not least Dave also pushed a new version of Marionette Client 0.5.37, which is able to forward command line arguments to the application under test by simply specifying them via –app-arg. That new feature was necessary for us to get failures in B2G desktop better investigated.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad.

Automation Development report – week 34 2013

Last week was a really productive week, when we look over the accomplished tasks. Across different test frameworks and tools we got a lot done. Especially the mozdownload project raises interests by even more people, who want to help with their own contribution. Thank you all who were involved!


To appreciate all the hard work done by Johannes Vogel on the mozdownload project we promoted him to the reviewer status. With this recognition he is now able to help us driving this project forward with a faster pace by doing code reviews. A delicacy was also his patch for getting the long outstanding test infrastructure working for mozdownload. It landed last week, and gave Dave Hunt the opportunity to get Travis CI support added.

Over the past couple of Firefox branch merges we experienced troubles with our Mozmill tests execution within mozmill-ci. This was caused by a bug in one of the plugins in use, which only checks the default branch of a Mercury repository for changes, but not any other named branch. Henrik Skupin landed a workaround as an interim solution, which replaced the SCM trigger with a timed trigger in Jenkins. Now the CI is checking for updates each 5 minutes. For details please also see his post on the mailing list.

Further Henrik was able to finish off his patch for pulsetranslator on bug 857980. With it in place it let us send out individual normalized Pulse notifications for beta and release candidate builds produced by repack jobs. As result individual locales beside en-US can be tested with mozmill-ci now. Step by step our team will constantly increase the number of locales to test over the curse of the next few weeks. For details and a follow-up discussion please check the thread on the mailing list.

To stay on top of running our Mozmill tests on the latest OS versions, Henrik added 6 new Ubuntu 13.04 nodes to our CI cluster. Three of them are running a 32bit version while the other three have 64bit installed. Most likely as of today we will remove the Ubuntu 12.10 machines from the cluster.

Sadly we are still scratching our head around the current problems with Windows 8.1 preview release. Due to issues with the LoginManager we most likely see our nodes getting disconnected when an user is logging into such a machine via VNC. The failure thrown doesn’t seem to be that known yet, so fixing the problem and bringing those nodes online will not happen soon. If anyone has ideas what it could be, please let us know!

Thanks to our automation members from the Softvision team (Andreea Matei, Andrei Eftimie, Mario Garbi, and Cosmin Malutan) we were able to close out 4 of our existing mozmill-test failures and also got 2 enhancements landed for Mozmill 2.0.

Last but not least we want to thank Johannes for his contribution on the mozmill-dashboard project. With one of his latest patches it’s now possible to correctly bookmark our individual dashboard instances with unique bookmark names.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad.