Firefox Automation report – Q2 2015

It’s been a while since I wrote my last Firefox automation report, so lets do another one to wrap up everything happened in Q2 this year. As you may remember from my last report the team has been cut down to only myself, and beside that I was away the whole April. Means only 2 months worth of work will be covered this time.

In general it was a tough quarter for me. Working alone and having to maintain all of our infrastructure and keeping both of our tests (Mozmill and Marionette) green was more work as expected. But it still gave me enough time to finish my two deliverables:

• Ensure better visibility of Firefox UI test results for sheriffs and developers by uploading them to treeherder
• Finalize features for Firefox UI update tests, and help to get them running on RelEng hardware

Firefox UI Test Results on Treeherder

With the transition of Mozmill tests to Marionette we no longer needed the mozmill-dashboard. Especially not since nearly no job in our Mozmill CI is running Mozmill anymore. Given that we also weren’t happy with the dashboard solution and the usage of couchdb under the hood, I was happy that a great replacement exist and our Marionette tests can make use of.

Treeherder is the new reporting system for tests being run in buildbot continuous integration. Because of that it covers all products and their appropriate supported branches. That’s why it it’s kinda perfect for us.

Before I was able to get started a bit of investigation had to be done, and I also checked some other projects which make use of treeherder reporting. That information was kinda helpful even with lots of undocumented code. Given that our tests are located outside of the development tree in its own Github repository some integration can be handled more loosely compared to in-tree tests. With that respect we do not appear as Tier-1 or Tier-2 but results are listed under the Tier-3 level, which is hidden by default. But that’s fine given that our goal was to bring up reports to treeherder first. Going into details will be a follow-up task.

For reporting results to Treeherder the Python package treeherder-client can be used. It’s a collection of different classes which help with authentication, collecting job details, and finally uploading the data to the server. It’s documentation can be found on readthedocs and meanwhile contains a good number of example code which makes it easier to get your code implemented.

Before you can actually upload reports the names and symbols of the groups and jobs have to be defined. For our tests I have chosen the three group symbols “Fu”, “Ff”, and “Fr”. Each of those stays for “(F)irefox UI Tests” and the first letter of the testrun name. We currently have “updates”, “functional”, and “remote”. As job name the locale of Firefox will be used. That results in an output like the following:

Whether jobs are passing or failing it is recommended to always add the log files as generated by running the tests to the report. It will not happen via data URLs but as artifacts with a link to an upload location. In our case we make use of Amazon S3 as storage service.

With all the pieces implemented we can now report to treeherder and covering Nightly, Aurora (Developer Edition), Beta, and Release builds. As of now reporting only happens against the staging instance of Treeherder, but soon we will be able to report to production. If you want to have a sneak peak how it works, just follow this link for Nightly builds.

More details about Treeherder reporting I will do later this quarter when the next pieces have been implemented.

Firefox UI Update Tests under RelEng

My second deliverable was to assist Armen Zambrano in getting the Firefox UI Update tests run for beta and release builds on Release Engineering infrastructure. This was a kinda important goal for us given that until now it was a manually triggered process with lots of human errors on our own infrastructure. That means lots of failures if you do not correctly setup the configuration for the tests, and a slower processing of builds due to our limited available infrastructure. So moving this out of our area made total sense.

Given that Armen had already done a fair amount of work when I came back from my PTO, I majorly fixed issues for the tests and the libraries as pointed out by him. All that allowed us to finally run our tests on Release Engineering infrastructure even with a couple of failures at the beginning for the first beta. But those were smaller issues and got fixed quickly. Since then we seem to have good results. If you want to have a look in how that works, you should check the Marionette update tests wiki page.

Sadly some of the requirements haven’t been completely finished yet. So the Quality Engineering team cannot stop running the tests themselves. But that will happen once bug 1182796 has been fixed and deployed to production.

Oh, and if you wonder where the results are located… Those are not getting sent to Treeherder but to an internal mailing list as used for every other automation results.

Other Work

Beside the deliverables I got some more work done. Mainly for the firefox-ui-tests and mozmill-ci.

While the test coverage has not really been increased, I had a couple of regressions to fix as caused by changes in Firefox. But we also landed some new features thankfully as contributed by community members. Once all that was done and we agreed to have kinda stable tests, new branches have been created in the repository. That was necessary to add support for each supported version of Firefox down to ESR 38.0, and to be able to run the tests in our Mozmill CI via Jenkins. More about that you will find below. The only task I still haven’t had time for yet was the creation of proper documentation about our tests. I hope that I will find the time in Q3.

Mozmill CI got the most changes in Q2 compared to all the former quarters. This is related to the transition from Mozmill tests to Marionette tests. More details why we got rid of Mozmill tests can be found in this post. With that we decided to get rid of most of the tests and mainly start from scratch by only porting the security and update tests over to Marionette. The complete replacement in Mozmill and all its jobs can be seen on issue 576 on Github. In detail we have the following major changes:

• Run all jobs with Marionette beside Firefox ESR 31.0 which is not supported by Marionette, and ondemand_update jobs because they still have to be run by Quality Engineering.
• Reduced number of platforms. We got rid of Windows Vista, Ubuntu 14.10, and OS X 10.7 whereby the latter machines have been re-used for OS X 10.10.
• No usage of a pre-configured environments anymore, but creating it from fresh for each test-run by installing Python packages from the internal PyPI mirror.
• Sending test results to treeherder and giving public access for everyone.
• Stopped sending emails for failures to our mozmill-ci mailing list in favor of having treeherder results.

All changes in Mozmill CI can be seen on Github.

Last but not least we also had two releases of mozdownload in Q2. Both had a good amount of features included. For details you can check the changelog.

I hope that gave you a good quick read on the stuff I was working on last quarter. Maybe in Q3 I will find the time to blog more often and in more detail. Lets see.