Tag Archives: firefox

Firefox Automation report – week 9/10 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 9 and 10. I for myself was a week on vacation. A bit of relaxing before the work on the TPS test framework should get started.

Highlights

In preparation to run Mozmill tests for Firefox Metro in our Mozmill-CI system, Andreea has started to get support for Metro builds and appropriate tests included.

With the help from Henrik we got Mozmill 2.0.6 released. It contains a helpful fix for waitForPageLoad(), which let you know about the page being loaded and its status in case of a failure. This might help us to nail down the intermittent failures when loading remote and even local pages. But the most important part of this release is indeed the support of mozcrash. Even that we cannot have a full support yet due to missing symbol files for daily builds on ftp.mozilla.org, we can at least show that a crash was happening during a testrun, and let the user know about the local minidump files.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 9 and week 10.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 9 and week 10.

Firefox Automation report – week 5/6 2014

A lot of things were happening in weeks 5 and 6, and we made some good progress regards the stability of Mozmill.

Highlights

The unexpected and intermittent Jenkins crashes our Mozmill CI system was affected with are totally gone now. Most likely the delayed creation of jobs made that possible, which also gave Jenkins a bit more breath and not bomb it with hundreds of API calls.

For the upcoming release of Mozmill 2.0.4 a version bump for package dependencies was necessary for mozdownload. So we released mozdownload 1.11. Sadly a newly introduced regression in packaging caused us to release mozdownload 1.11.1 a day later.

After a lot of work for stability improvements we were able to release Mozmill 2.0.4. This version is one with the largest amount of changes in the last couple of months. Restarts and shutdowns of the application is way better handled by Mozmill now. Sadly we noticed another problem during restarts of the application on OS X (bug 966234) which forced us to fix mozrunner.

Henrik released mozrunner 5.34 which includes a fix how mozrunner retrieves the state of the application during a restart. It was failing here by telling us that the application has quit while it was still running. As result Mozmill started a new Firefox process, which was not able to access the still used profile. A follow-up Mozmill release was necessary, so we went for testing it.

As another great highlight for community members who are usually not able to attend our Firefox Automation meetings, we have started to record our meetings now. So if you want to replay the meetings please check our archive.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 5 and week 6.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the Firefox Automation meetings of week 5 and week 6.

Firefox Automation report – week 3/4 2014

Due to the high work load and a week of vacation I was not able to give some updates for work done by the Firefox Automation team. In the next days I really want to catch up with the reports, and bring you all on the latest state.

Highlights

After the staging system for Mozmill CI has been setup by IT and Henrik got all the VMs connected, also the remaining Mac Minis for OS X 10.6 to 10.9 have been delivered. That means our staging system is complete and can be used to test upcoming updates, and to investigate failures.

For the production system the Ubuntu 13.04 machines have been replaced by 13.10. It’s again a bit late but other work was stopping us from updating earlier. The next update to 14.04 should become live faster.

Beside the above news we also had 2 major blockers. First one was a top crasher of Firefox caused by the cycle collector. Henrik filed it as bug 956284 and together with Tim Taubert we got it fixed kinda quick. The second one was actually a critical problem with Mozmill, which didn’t let us successfully run restart tests anymore. As it has been turned out the zombie processes, which were affecting us for a while, kept the socks server port open, and the new Firefox process couldn’t start its own server. As result JSBridge failed to establish a connection. Henrik got this fixed on bug 956315

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 3 and week 4.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the Firefox Automation meetings of week 3 and week 4.

Firefox Automation report – week 1/2 2014

I promised to keep up with our updates over the last week but given a major breakage in the freshly released version of Mozmill 2.0.4, I had a full week of work to get the fix out. I promise that during this week I will write reports for the weeks in January.

Highlights

With the new year our team has been reorganized and we are part of the Mozilla QA team again. That means we will have a way closer relationship to any feature owner, and also working towards in bringing more automation knowledge to everyone. The goals for our team are getting worked out and I will present those in one of my following blog posts. As of now you can find our team page on the Mozilla wiki under Firefox Automation.

Since the landing of all the new features for Mozmill-CI on our staging machine before Christmas, we have no longer experienced any crash of the Jenkins master. Given that Henrik pushed all the changes to our production system. We are totally happy that the incremental updates made our system that stable, and that Mozilla QA doesn’t have cope with outages.

Henrik and Jarek were both working on the mozfile 1.1 release to make it more stable in terms of removing files when those are still in use or don’t have the right permissions set.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 1 and week 2.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the first Firefox Automation meeting of week 2.

Automation Development report – week 49/50 2013

It’s getting closer to Christmas. So here the 2nd last automation development report for 2013. Also please note that all team members including myself, who were dedicated to Firefox Desktop automation, have been transitioned back from the A-Team into the Mozilla QA team. This will enable us to have a better relationship with QA feature owners, and get them trained for writing automated tests for Firefox. Therefor my posts will be named “Firefox Automation” in the future.

Highlights

With the latest release of Firefox 26.0 a couple of merges in our mozmill-tests repository had to be done. Work involved here was all done by Andreea and Andrei on bug 945708

To get rid of failed attempts to remove files after a testrun with Mozmill, Henrik was working on a new version of mozfile, which includes a method called remove() now. It should be used by any code given that it tries multiple times to remove files or directories if access is getting denied on Windows systems.

We released Mozmill 2.0.2 with a couple of minor fixes and the above mentioned file removal fixes. Beside that mozmill-automation 2.0.2 and new mozmill-environments have been released.

We are still working on the remaining issues with Mozmill 2.0 and are hoping to get them fixed as soon as possible. So that an upgrade to the 2.0.x branch can happen.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 49 and week 50.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the last two Automation Development meetings of week 49 and week 50.

Automation Development report – week 47/48 2013

This is again an overview about the highlights of our work in week 47 and 48 this year. Sorry for the delay of a couple of days but some critical work was holding me off from writing this post.

Highlights

Henrik and Dave were working on a couple of Mozmill-CI updates, which have been pushed to production. The first batch of features and bug fixes landed in week 47. It included the monitoring plugin for Jenkins, which hopefully will help us to figure out the reasons of Java crashes. Also we can finally run project branches via our CI now, even it can only be done manually as of now. It is important for our upcoming tests for the Holly branch (Firefox Nightly without the Australis feature). The second batch landed by last week was intended to upgrade our system to Mozmill 2.0.1. Sadly we failed in it due to a couple of other failures, which we haven’t seen before on our staging server. So we partly reverted back the latest commits for production and we all are working hard on getting those issues fixed.

With the detected failures by upgrading to Mozmill 2.0.1, Henrik has noticed that one of those existed because of an incompatibility of mozbase with Python 2.6. See bug 944361 for details. To solve this we upgraded our OS X 10.6 boxes to Python 2.7.3, so all machines are running the same version of Python now. As a very nice side-effect we noticed a speed improvement by running our Mozmill tests of about 25%!

Henrik pushed a couple of releases to pypi, which include mozprofile 0.17 with a lot of add-on manager related bug fixes, mozdownload 1.10 (see details), and mozmill 2.0.1 (see details)

To be prepared for executing the first Metro tests with Mozmill we had to prepare the mozmill-tests repository for handling multiple applications. Therefore Andreea refactored nearly the whole repository. You can find details on bug 927397.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 47 and week 48.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the last two Automation Development meetings of week 47 and week 48.

Automation Development report – week 45/46 2013

After the last report over a two week cycle, I have to follow-up with another one for the weeks 45 and 46. Due to my move I had limited availability and to fix some important other stuff. So hopefully this is the last report over a two weeks period in the near future.

Highlights

To be able to release Mozmill 2.0.1 as soon as possible Henrik had to fix a lot of existent bugs for mozprofile in conjunction with its add-on manager class. Those fixes were necessary because our restart tests were broken due to an inappropriate clean-up of add-ons after closing Firefox. At the end 10 bugs have been fixed.

We are close to get our first Metro tests running with Mozmill on Windows 8 and 8.1. But before we can really do that, the whole mozmill-tests repository has to be refactored to support multiple applications. Here both Andreea and Andrei are doing most of the work.

Dave and Henrik were both working on a couple of Mozmill-CI issues, which will help us to better diagnose the memory issues and random crashes of the Jenkins Java process. Everything has been merged to our staging server and has to bake a bit before a push to production will happen.

Henrik added a new feature to Mozmill-CI which allows us to run tests for builds off project branches. This will come into play really soon when we have to execute daily tests for the upcoming holly branch, which is mozilla-central without the Australis UX.

Dave released gaiatest 0.19, b2gpopulate 0.11, and b2gperf 0.12 to resolve leaks in performance tests.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 45 and week 46.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the last two Automation Development meetings of week 45 and week 46.

Mozdownload 1.10 released

As of now we have released a new version of mozdownload. The 1.10 release contains a couple of new features, bug fixes, and absolutely to mention a suite of tests. A big thank you goes here to Johannes and Jarek. Both spend a lot of time again, to make it a successful and lesser bug prone release.

Here some major items to highlight:

  • Addition of tests for all types of scrapers and a lot for command line options
  • Output of all candidate builds founds if build number has been specified
  • Add support for tinderbox stub installer
  • Allow to download files with different extensions than exe

A full list of features can be found in the changelog

Automation Development report – week 43/44 2013

Given that I was partly away in the last two weeks I haven’t had the time to write another summary report of our work yet. So lets combine the last two weeks in a single blog post this time.

Highlights

To be able to continue our work to speed-up testruns in Mozmill CI Henrik upgraded our Mozmill-CI system to Jenkins 1.509.4. This step was necessary so that we can investigate the slowness in sending out emails for finished testruns, which could take up to 25 minutes! Beside that Henrik also disabled the sending of emails for successful testruns given that those were mostly blocking us, and no-one really uses them. That was already a real boost and our system was able to execute all of the 245 update tests for the 25.0 release in 25 minutes – across all platforms!

If you are interested in our goals you might want to have a look at Henrik’s post about the Automation Development team goals for Q4.

As noted in the last couple of Automation Development reports, our Windows 8.1 64bit nodes were really unstable and restarted at random times. With a lot of research Henrik was able to identify the problem. Finally it turned out a bug in VMWare vSphere you could circumvent by tweaking the CPUID mask of the machine. All the details about investigation and problem solving can be found on bug 916746.

Together with Adrian from the IT team Henrik setup another slave node for each supported platform of the Mozmill-CI production instance. That means for the 16 different platforms (including 32bit vs. 64bit) we are running 64 slaves now. That is a lot, and the need for Puppet support is more important than ever before. Manually keeping those machines up-2-date is a real pain for me.

More and more we are facing a really nasty behavior of Jenkins when it cannot delete the workspace for all kinds of jobs on Windows. It forces us to log into the machines, delete the whole Jenkins folder, and re-connect the slave. As of now we have no idea what causes this problem, but it might be related to a restart of the slave and some permission problems. Has anyone else seen the same and shed some light on us?

Speaking about Mozmill we are still blocked on some addon-on related issues in mozprofile before we can release version 2.0.1. Henrik is heavily working on those bugs and hopefully we will have a new mozprofile release next week.

Dave participated in the Mozilla Festival this year, and while he was really busy over that weekend, he found the time to get started on the Mozmill CI configuration generator for ondemand tests. We will collect the requirements for the first version soon, and get started to further hack on it.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 43 and week 44.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the last two Automation Development meetings of week 43 and week 44.

Automation Development team goals for Q4 2013

I know that I’m a bit late in publishing our Automation Development team goals for the last quarter this year. But better late than never. I think there are some very interesting tasks we want to tackle on over the course of the next three months. And we would invite everyone who has interest to join us, learn new stuff, and simply have fun with coding.

Given that we are a part of the https://wiki.mozilla.org/Auto-tools team aka the A-Team, we inherit some of the overall team goals for Q4 2013. As of now all of our own work is concentrated around Desktop QA Automation and B2G (Firefox OS). Collecting those various pieces we came up with the following list of prioritized projects.

Cross-Team Goals

Given that one of our core principles is to take care of test automation across different teams, and to teach everyone with interests under this topic, we have some tasks which have to get finished off by working with people from a couple of other teams. Those are:

Switch Mozmill CI from Mozmill 1.5.x to Mozmill 2.0

After we have released Mozmill 2.0 by end of September we now have to work on getting our CI system upgraded too. The necessary work is listed on issue 329 of our issue tracker on Github.

There is not that much to do here beside fixing two bugs, which block us from an upgrade. That’s a recent change which comes with Firefox 26.0 and confused our restart test handling. For more details see bug 922995. And on the other side our update testrun leaves mounted DMG images behind on OS X 10.6. Therefore see issue 90.

Get FirefoxMetro tests executed in Mozmill CI production

Given that the QA team gave us a couple of Metro tests to automate, we have to add support to run all of them in our CI system. There is no support for that type of application yet. So a good amount of work lays in-front of us. All tasks are covered by issue 215.

Mozmill 2.0 already comes with MetroFirefox support but still misses some critical pieces. As of now it’s not able to set the installed application as the default browser on the system. Starting with Windows 8 this system level change is way harder to do. If anyone of you has ideas in how to do that please let us know.

AWSY (Are we slim yet) Firefox OS Memory Tests

We have to develop test automation to add Firefox OS to http://www.areweslimyet.com. The test, which is an orangutan-based script driving the gaia-ui, will be run on the B2G emulator, using TBPL builds from mozilla-central. Memory dumps will be grabbed and results posted to AWSY, with the ultimate goal in identifying and preventing Firefox OS memory leak regressions. For more information please see the tracking bug.

Get SoftVision to lead the automation efforts for QA with our oversight

Our team members from the test automation team at Softvision are helping us for a long time now in maintaining the Mozmill tests repository by fixing broken tests and creating new ones. Also they have assisted in the final work for the Mozmill 2.0 release, and other tools we own. To give them more responsibility we want to bring all of them into the position to handle the Mozmill-CI system on their own. This is necessary given the much broader scope of tasks everyone in our team is assigned to. Therefore we will do trainings and demos in our Ask an expert meetings.

Team Goals

There are also some additional goals, which do not scale across teams, but are also important to note.

Deploy Eideticker CI system for Firefox OS

We have to implement a continuous integration system for the Firefox OS start-up and scrolling tests using an external camera. More details about this goal can be found on bug 904837.

Extend Mozmill CI to cover most of our localized builds

To lower the amount of possibly broken beta and release builds we have to extend our tests to cover nearly all supported locales. Therefore we have to bring more nodes online in our CI system, and further enhance the tests to run more efficient. The ultimate goals would be to test all of our >90 locales. If you are interested to help us please let us know or check issue 270.

Create web application for on-demand update jobs

We want to reduce the failure rate when setting up the configuration for on-demand update tests in our Mozmill-CI system. Therefore a simple web application has to be created which allows us to pick the configuration like what can be done with the trychooser. If you want to help please check bug 915164

Stretched Goals

Last but not least we also have goals which are stretched and we might work on if there is time left.

Setup try-server like Mozmill CI system

To help everyone who is working on creating new or fixing broken Mozmill tests, we want to have a try server like system. That means you could push your patch and it will be tested across platforms and Firefox versions.

Investigate requirements for community driving tool

One of the goals from Henrik is to lower the amount of volunteers who are leaving our team or even Mozilla because of lack of activity from our side. Given that there is no good way in keeping in touch with everyone yet, a tool has to be created or extended. So maybe we can chime into the Black Hole.

As you can see there is a lot of work, which is awaiting to be finished by this quarter. We always appreciate your willingness and interest in helping us. So if one or some of those goals have drawn your interest let us know.