Firefox Automation report – Q3 2015

It’s time for another Firefox Automation report! It’s incredible how fast a quarter passes by without that I have time to write reports more often. Hopefully it will change soon – news will be posted in a follow-up blog post.

Ok, so what happened last quarter for our projects.

Mozharness

One of my deliverables in Q3 was to create mozharness scripts for our various tests in the firefox-ui-tests repository, so that our custom runner scripts can be replaced. This gives us a way more stable system and additional features like crash report handling, which are necessary to reach the tier 2 level in Treeherder.

After some refactoring of the firefox ui tests, scripts for the functional and update tests were needed. But before those could be implemented I had to spent some time in refactoring some modules of mozharness to make them better configurable for non-buildbot jobs. All that worked pretty fine and finally the entry scripts have been written. Something special for them is that they even have different customers, so extra configuration files had to be placed. In detail it’s us who run the tests in Jenkins for nightly builds, and partly for release builds. On the other side Release Engineering want to run our update tests on their own hardware when releases have to be tested.

By the end of September all work has been finished. If you are interested in more details feel free to check the tracking bug 1192369.

Mozmill-CI

Our Jenkins instance got lots of updates for various new features and necessary changes. All in all I pushed 27 commits which affected 53 files.

Here a list of the major changes:

  • Refactoring of the test jobs has been started so that those can be used for mozharness driven firefox-ui-tests later in Q4. The work has not been finished and will be continued in Q4. Especially the refactoring for report submission to Treeherder even for aborted builds will be a large change.

  • A lot of time had to be spent in fixing the update tests for all the changes which were coming in with the Funsize project of Release Engineering. Due to missing properties in the Mozilla Pulse messages update tests could no longer be triggered for nightly builds. Therefore the handling of Pulse messages has been completely rewritten to allow the handling of similar Pulse messages as sent out from TaskCluster. That work was actually not planned and has been stolen me quite some time from other projects.

  • A separation of functional and remote tests didn’t make that much sense. Especially because both types are actually functional tests. As result they have been merged together into the functional tests. You can still run remote tests only by using --tag remote; similar for tests with local testcases by using `–tag local.

  • We stopped running tests for mozilla-esr31 builds due to Firefox ESR31 is no longer supported.

  • To lower the amount of machines we have to maintain and to getting closer what’s being run on Buildbot, we stopped running tests on Ubuntu 14.10. Means we only run on Ubuntu LTS releases from now on. Also we stopped tests for OS X 10.8. The nodes will be re-used for OS X 10.11 once released.

  • We experienced Java crashes due to low memory conditions of our Jenkins production master again. This was kinda critical because the server is not getting restarted automatically. After some investigation I assumed that the problem is due to the 32bit architecture of the VM. Given that it has 8GB of memory a 64bit version of Ubuntu should have been better used. So we replaced the machine and so far everything looks fine.

  • Totally surprising we had to release once more a bugfix release of Mozmill. This time the framework didn’t work at all due to the enforcement of add-on signing. So Mozmill 2.0.10.2 has been released.