Category Archives: Mozilla

Survey about sharing information inside the Firefox Automation team

Within the Firefox Automation team we were suffering a bit in sharing information about our work over the last couple of months. That mainly happened because I was alone and not able to blog more often than once in a quarter. The same applies to our dev-automation mailing list which mostly only received emails from Travis CI with testing results.

Given that the team has been increased to 4 people now (beside me this is Maja Frydrychowicz, Syd Polk, and David Burns, we want to be more open again and also trying to get more people involved into our projects. To ensure that we do not make use of the wrong communication channels – depending where most of our readers are – I have setup a little survey. It will only take you a minute to go through but it will help us a lot to know more about the preferences of our automation geeks. So please take that little time and help us.

The survey can be found here and is open until end of November 2015:

Thank you a lot!

Firefox Automation report – Q3 2015

It’s time for another Firefox Automation report! It’s incredible how fast a quarter passes by without that I have time to write reports more often. Hopefully it will change soon – news will be posted in a follow-up blog post.

Ok, so what happened last quarter for our projects.


One of my deliverables in Q3 was to create mozharness scripts for our various tests in the firefox-ui-tests repository, so that our custom runner scripts can be replaced. This gives us a way more stable system and additional features like crash report handling, which are necessary to reach the tier 2 level in Treeherder.

After some refactoring of the firefox ui tests, scripts for the functional and update tests were needed. But before those could be implemented I had to spent some time in refactoring some modules of mozharness to make them better configurable for non-buildbot jobs. All that worked pretty fine and finally the entry scripts have been written. Something special for them is that they even have different customers, so extra configuration files had to be placed. In detail it’s us who run the tests in Jenkins for nightly builds, and partly for release builds. On the other side Release Engineering want to run our update tests on their own hardware when releases have to be tested.

By the end of September all work has been finished. If you are interested in more details feel free to check the tracking bug 1192369.


Our Jenkins instance got lots of updates for various new features and necessary changes. All in all I pushed 27 commits which affected 53 files.

Here a list of the major changes:

  • Refactoring of the test jobs has been started so that those can be used for mozharness driven firefox-ui-tests later in Q4. The work has not been finished and will be continued in Q4. Especially the refactoring for report submission to Treeherder even for aborted builds will be a large change.

  • A lot of time had to be spent in fixing the update tests for all the changes which were coming in with the Funsize project of Release Engineering. Due to missing properties in the Mozilla Pulse messages update tests could no longer be triggered for nightly builds. Therefore the handling of Pulse messages has been completely rewritten to allow the handling of similar Pulse messages as sent out from TaskCluster. That work was actually not planned and has been stolen me quite some time from other projects.

  • A separation of functional and remote tests didn’t make that much sense. Especially because both types are actually functional tests. As result they have been merged together into the functional tests. You can still run remote tests only by using --tag remote; similar for tests with local testcases by using `–tag local.

  • We stopped running tests for mozilla-esr31 builds due to Firefox ESR31 is no longer supported.

  • To lower the amount of machines we have to maintain and to getting closer what’s being run on Buildbot, we stopped running tests on Ubuntu 14.10. Means we only run on Ubuntu LTS releases from now on. Also we stopped tests for OS X 10.8. The nodes will be re-used for OS X 10.11 once released.

  • We experienced Java crashes due to low memory conditions of our Jenkins production master again. This was kinda critical because the server is not getting restarted automatically. After some investigation I assumed that the problem is due to the 32bit architecture of the VM. Given that it has 8GB of memory a 64bit version of Ubuntu should have been better used. So we replaced the machine and so far everything looks fine.

  • Totally surprising we had to release once more a bugfix release of Mozmill. This time the framework didn’t work at all due to the enforcement of add-on signing. So Mozmill has been released.

mozdownload 1.18 released

Today we have released mozdownload 1.18 to PyPI. The reason why I think it’s worth a blog post is that with this version we finally added support for a sane API. With it available using the mozdownload code in your own script is getting much easier. So there is no need to instantiate a specific scraper anymore but a factory scraper is doing all the work depending on the options it gets.

Here some examples:

from mozdownload import FactoryScraper
scraper = FactoryScraper('release', version='40.0.3', locale='de')
from mozdownload import FactoryScraper
scraper = FactoryScraper('candidate', version='41.0b9', platform='win32')
from mozdownload import FactoryScraper
scraper = FactoryScraper('daily', branch='mozilla-aurora')

If you are using mozdownload via its API you can also easily get the remote URL and the local filename:

from mozdownload import FactoryScraper
scraper = FactoryScraper('daily', branch='mozilla-aurora')
print scraper.url
print scraper.filename

Hereby the factory class is smart enough to only select those passed-in options which are appropriate for the type of scraper. If you have to download different types of builds you will enjoy that feature given that only the scraper type has to be changed and all other options could be still passed-in.

We hope that this new feature will help you by integrating mozdownload into your own project. There is no need anymore by using its command line interface through a subprocess call.

The complete changelog can be found here.

Firefox Automation report – Q2 2015

It’s been a while since I wrote my last Firefox automation report, so lets do another one to wrap up everything happened in Q2 this year. As you may remember from my last report the team has been cut down to only myself, and beside that I was away the whole April. Means only 2 months worth of work will be covered this time.

In general it was a tough quarter for me. Working alone and having to maintain all of our infrastructure and keeping both of our tests (Mozmill and Marionette) green was more work as expected. But it still gave me enough time to finish my two deliverables:

  • Ensure better visibility of Firefox UI test results for sheriffs and developers by uploading them to treeherder
  • Finalize features for Firefox UI update tests, and help to get them running on RelEng hardware

Firefox UI Test Results on Treeherder

With the transition of Mozmill tests to Marionette we no longer needed the mozmill-dashboard. Especially not since nearly no job in our Mozmill CI is running Mozmill anymore. Given that we also weren’t happy with the dashboard solution and the usage of couchdb under the hood, I was happy that a great replacement exist and our Marionette tests can make use of.

Treeherder is the new reporting system for tests being run in buildbot continuous integration. Because of that it covers all products and their appropriate supported branches. That’s why it it’s kinda perfect for us.

Before I was able to get started a bit of investigation had to be done, and I also checked some other projects which make use of treeherder reporting. That information was kinda helpful even with lots of undocumented code. Given that our tests are located outside of the development tree in its own Github repository some integration can be handled more loosely compared to in-tree tests. With that respect we do not appear as Tier-1 or Tier-2 but results are listed under the Tier-3 level, which is hidden by default. But that’s fine given that our goal was to bring up reports to treeherder first. Going into details will be a follow-up task.

For reporting results to Treeherder the Python package treeherder-client can be used. It’s a collection of different classes which help with authentication, collecting job details, and finally uploading the data to the server. It’s documentation can be found on readthedocs and meanwhile contains a good number of example code which makes it easier to get your code implemented.

Before you can actually upload reports the names and symbols of the groups and jobs have to be defined. For our tests I have chosen the three group symbols “Fu”, “Ff”, and “Fr”. Each of those stays for “(F)irefox UI Tests” and the first letter of the testrun name. We currently have “updates”, “functional”, and “remote”. As job name the locale of Firefox will be used. That results in an output like the following:

Treeherder Results

Whether jobs are passing or failing it is recommended to always add the log files as generated by running the tests to the report. It will not happen via data URLs but as artifacts with a link to an upload location. In our case we make use of Amazon S3 as storage service.

With all the pieces implemented we can now report to treeherder and covering Nightly, Aurora (Developer Edition), Beta, and Release builds. As of now reporting only happens against the staging instance of Treeherder, but soon we will be able to report to production. If you want to have a sneak peak how it works, just follow this link for Nightly builds.

More details about Treeherder reporting I will do later this quarter when the next pieces have been implemented.

Firefox UI Update Tests under RelEng

My second deliverable was to assist Armen Zambrano in getting the Firefox UI Update tests run for beta and release builds on Release Engineering infrastructure. This was a kinda important goal for us given that until now it was a manually triggered process with lots of human errors on our own infrastructure. That means lots of failures if you do not correctly setup the configuration for the tests, and a slower processing of builds due to our limited available infrastructure. So moving this out of our area made total sense.

Given that Armen had already done a fair amount of work when I came back from my PTO, I majorly fixed issues for the tests and the libraries as pointed out by him. All that allowed us to finally run our tests on Release Engineering infrastructure even with a couple of failures at the beginning for the first beta. But those were smaller issues and got fixed quickly. Since then we seem to have good results. If you want to have a look in how that works, you should check the Marionette update tests wiki page.

Sadly some of the requirements haven’t been completely finished yet. So the Quality Engineering team cannot stop running the tests themselves. But that will happen once bug 1182796 has been fixed and deployed to production.

Oh, and if you wonder where the results are located… Those are not getting sent to Treeherder but to an internal mailing list as used for every other automation results.

Other Work

Beside the deliverables I got some more work done. Mainly for the firefox-ui-tests and mozmill-ci.

While the test coverage has not really been increased, I had a couple of regressions to fix as caused by changes in Firefox. But we also landed some new features thankfully as contributed by community members. Once all that was done and we agreed to have kinda stable tests, new branches have been created in the repository. That was necessary to add support for each supported version of Firefox down to ESR 38.0, and to be able to run the tests in our Mozmill CI via Jenkins. More about that you will find below. The only task I still haven’t had time for yet was the creation of proper documentation about our tests. I hope that I will find the time in Q3.

Mozmill CI got the most changes in Q2 compared to all the former quarters. This is related to the transition from Mozmill tests to Marionette tests. More details why we got rid of Mozmill tests can be found in this post. With that we decided to get rid of most of the tests and mainly start from scratch by only porting the security and update tests over to Marionette. The complete replacement in Mozmill and all its jobs can be seen on issue 576 on Github. In detail we have the following major changes:

  • Run all jobs with Marionette beside Firefox ESR 31.0 which is not supported by Marionette, and ondemand_update jobs because they still have to be run by Quality Engineering.
  • Reduced number of platforms. We got rid of Windows Vista, Ubuntu 14.10, and OS X 10.7 whereby the latter machines have been re-used for OS X 10.10.
  • No usage of a pre-configured environments anymore, but creating it from fresh for each test-run by installing Python packages from the internal PyPI mirror.
  • Sending test results to treeherder and giving public access for everyone.
  • Stopped sending emails for failures to our mozmill-ci mailing list in favor of having treeherder results.

All changes in Mozmill CI can be seen on Github.

Last but not least we also had two releases of mozdownload in Q2. Both had a good amount of features included. For details you can check the changelog.

I hope that gave you a good quick read on the stuff I was working on last quarter. Maybe in Q3 I will find the time to blog more often and in more detail. Lets see.

Firefox Automation report – Q1 2015

As you may have noticed I was not able to come up with status reports of the Firefox Automation team during the whole last quarter. I feel sad about it, but there was simply no time to keep up with those blog posts. Even now I’m not sure how often I will be able to blog. So maybe I will aim to do it at least once a quarter or if possible once a month.

You may ask how it comes? The answer is simple. Our team faced some changes and finally a massive loss of core members. Which means from the former 6 people only myself are remaining. Since end of February all 5 former team members from Softvision are no longer participating in any of the maintained projects. Thanks to all of them for the great help over all the last months and years! But every project we own is now on my own shoulders. And this is kinda hell of work with downsides like not being able to do as many reviews as I want for side projects. One positive thing at least was that I got pulled back into the A-Team at the same time. With that move I’m once more closer again to all the people who care about the basics of all test infrastructure at Mozilla. I feel back home.

So what have I done the whole last quarter… First, it was always the ongoing daily work for maintaining our Mozmill CI system. This was usually a job for a dedicated person all the last months. The amount of work can sometimes eat up a whole day. Especially if several regressions have been found or incompatible changes in Firefox have been landed. Seeing my deliverables for Q1 it was clear that we have to cut down the time to spent on those failures. As result we started to partially skip tests which were failing. There was no time to get any of those fixed. Happily the latest version of Mozmill is still working kinda nicely so no other work had to be dedicated for this project.

Most of my time during the last quarter I actually had to spent on Marionette, especially building up wrapper scripts for being able to use Marionette as test framework for Firefox Desktop. This was a kinda large change for us but totally important in terms of maintenance burden and sustainability. The code base of Mozmill is kinda outdated and features like Electrolysis (e10s) will totally break it. Given that a rewrite of the test framework is too cost-intensive the decision has been made to transition our Mozmill tests over to Marionette. Side-effect was that a lot of missing features had to be implemented in Marionette to bring it at a level as what Mozmill offers. Thanks for the amazing work goes to Andrew Halberstadt, David Burns, Jonathan Griffin, and especially Chris Manchester.

For the new UI driven tests for Firefox Desktop we created the firefox-ui-tests repository at Github. We decided on that name to make it clear to which product the tests belong to, and also to get rid of any relationship to the underling test framework name. This repository contains the harness extensions around Marionette, a separate puppeteer library for back-end and UI modules, and last but not least the tests themselves. As goal for Q1 we had to get the basics working including the full set of remote security tests, and most important the update tests. A lot of help on the security tests we got from Barbara Miller our intern from December to March. She did great amount of work here, and also assisted other community members in getting their code done. Finally we got all the security tests converted.

My own focus beside the harness pieces were the update tests. Given the complete refactoring of those Mozmill tests we were able to easily port them over to Marionette. We tried to keep the class structure as is, and only did enhancements where necessary. Here Bob Silverberg helped with two big chunks of work which I’m gladly thankful about! Thanks a lot! With all modules in-place I finally converted the update tests and got them running for each version of Firefox down to 38.0, which will be the next ESR release and kinda important to be tested with Marionette. For stability and ease of contribution we added support for Travis CI to our new repository. It helps us a lot with reviews of patches from community members, and they also can see immediately if changes they have done are working as expected.

The next big chunk of work will be to get those tests running in Mozmill CI (to be renamed) and the test reporting to use Treeherder. Also we want to get our update tests for Firefox releases executed by the RelEng system, to further reduce the amount of time for signoffs from QE. About this work I will talk more in my next blog post. So please stay tuned.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agendas, the video recordings, and notes from the Firefox Automation meetings. Please note that since end of February we no longer host a meeting due to the low attendance and other meetings like the A-team ones, where I have to report my status.

Firefox Automation report – week 51/52 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 51 and 52 of 2014. I’m sorry for this very late post but changes to our team, which I will get to in my next upcoming post, caught me up with lots of more work and didn’t give me the time for writing status reports.


Henrik started work towards a Mozmill 2.1 release. Therefore he had to upgrade a couple of mozbase packages first to get latest Mozmill code on master working again. Once done the patch for handling parent sections in manifest files finally landed, which was originally written by Andrei Eftimie and was sitting around for a while. That addition allows us to use mozhttpd for serving test data via a local HTTP server. Last but not least another important feature went in, which let us better handle application disconnects. There are still some more bugs to fix before we can actually release version 2.1 of Mozmill.

Given that we only have the capacity to fix the most important issues for the Mozmill test framework, Henrik started to mass close existing bugs for Mozmill. So only a hand-full of bugs will remain open. If there is something important you want to see fixed, we would encourage you to start working on the appropriate bug.

For Mozmill CI we got the new Ubuntu 14.10 boxes up and running in our staging environment. Once we can be sure they are stable enough, they will also be enabled in production.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 51 and week 52.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meeting of week 51 and week 52.

Firefox Automation report – week 49/50 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 49 and 50.


During the first week of December the all-hands work week happened in Portland. Those were some great and inspiring days, full of talks, discussions, and conversations about various things. Given that I do not see my colleagues that often in real life, I have taken this opportunity to talk to everyone who is partly or fully involved in projects of our automation team. There are various big goals in front of us, so clearing questions and finding the next steps to tackle ongoing problems was really important. Finally we came out with a long list of todo items and more clarity about so far unclear tasks.

In week 50 we got some updates landed for Mozmill CI. Given a regression from the blacklist landing, our l10n tests haven’t been executed for any locale of the Firefox Developer Edition. Since the fix landed, we have seen problems with access keys in nearly each locale for a new test, which covers the context menu of web content.

Also we would like to welcome Barbara Miller in our team. She joined us as an intern via the FOSS outreach program as driven by Gnome. She will be with us until March and will mainly work on testdaybot and the conversion of Mozmill tests to Marionette. The latter project is called m21s and details can be found on its project page. Soon I will post more details about it.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 49 and week 50.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meeting of week 48. Due to the Mozilla all-hands workweek there was no meeting in week 49.

Firefox Automation report – week 47/48 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 47 and 48.


Most of the work during those two weeks made by myself were related to get [Jenkins]( upgraded on our Mozmill CI systems to the most recent LTS version 1.580.1. This was a somewhat critical task given the huge number of issue as mentioned in my last Firefox Automation report. On November 17th we were finally able to get all the code changes landed on our production machine after testing it for a couple of days on staging.

The upgrade was not that easy given that lots of code had to be touched, and the new LTS release still showed some weird behavior when connecting slave nodes via JLNP. As result we had to stop using this connection method in favor of the plain java command. This change was actually not that bad because it’s better to automate and doesn’t bring up the connection warning dialog.

Surprisingly the huge HTTP session usage as reported by the Monitoring plugin was a problem introduced by this plugin itself. So a simple upgrade to the latest plugin version solved this problem, and we will no longer get an additional HTTP connection whenever a slave node connects and which never was released. Once we had a total freeze of the machine because of that.

Another helpful improvement in Jenkins was the fix for a JUnit plugin bug, which caused concurrent builds to hang, until the former build in the queue has been finished. This added a large pile of waiting time to our Mozmill test jobs, which was very annoying for QA’s release testing work – especially for the update tests. Since this upgrade the problem is gone and we can process builds a lot faster.

Beside the upgrade work, I also noticed that one of the Jenkins plugins in use, it’s actually the XShell plugin, failed to correctly kill the running application on the slave machine in case of an job is getting aborted. The result of that is that following tests will fail on that machine until the not killed job has been finished. I filed a Jenkins bug and did a temporary backout of the offending change in that plugin.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 47 and week 48.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 47 and week 48.

Firefox Automation report – week 45/46 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 45 and 46.


In our Mozmill-CI environment we had a couple of frozen Windows machines, which were running with 100% CPU load and 0MB of memory used. Those values came from the vSphere client, and didn’t give us that much information. Henrik checked the affected machines after a reboot, and none of them had any suspicious entries in the event viewer either. But he noticed that most of our VMs were running a very outdated version of the VMware tools. So he upgraded all of them, and activated the automatic install during a reboot. Since then the problem is gone. If you see something similar for your virtual machines, make sure to check that used version!

Further work has been done for Mozmill CI. So were finally able to get rid of all the traces for Firefox 24.0ESR since it is no longer supported. Further we also setup our new Ubuntu 14.04 (LTS) machines in staging and production, which will soon replace the old Ubuntu 12.04 (LTS) machines. A list of the changes can be found here.

Beside all that Henrik has started to work on the next Jenkins v1.580.1 (LTS) version bump for the new and more stable release of Jenkins. Lots of work might be necessary here.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 45 and week 46.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 45 and week 46.

Firefox Automation report – week 43/44 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 43 and 44.


In preparation for the QA-wide demonstration of Mozmill-CI, Henrik reorganized our documentation to allow everyone a simple local setup of the tool. Along that we did the remaining deployment of latest code to our production instance.

Henrik also worked on the upgrade of Jenkins to latest LTS version 1.565.3, and we were able to push this upgrade to our staging instance for observation. Further he got the Pulse Guardian support implemented.

Mozmill 2.0.9 and Mozmill-Automation 2.0.9 have been released, and if you are curious what is included you want to check this post.

One of our major goals over the next 2 quarters is to replace Mozmill as test framework for our functional tests for Firefox with Marionette. Together with the A-Team Henrik got started on the initial work, which is currently covered in the firefox-greenlight-tests repository. More to come later…

Beside all that work we have to say good bye to one of our SoftVision team members.October the 29th was the last day for Daniel on the project. So thank’s for all your work!

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 43 and week 44.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 43 and week 44.