Automation Development team goals for Q4 2013

I know that I’m a bit late in publishing our Automation Development team goals for the last quarter this year. But better late than never. I think there are some very interesting tasks we want to tackle on over the course of the next three months. And we would invite everyone who has interest to join us, learn new stuff, and simply have fun with coding.

Given that we are a part of the https://wiki.mozilla.org/Auto-tools team aka the A-Team, we inherit some of the overall team goals for Q4 2013. As of now all of our own work is concentrated around Desktop QA Automation and B2G (Firefox OS). Collecting those various pieces we came up with the following list of prioritized projects.

Cross-Team Goals

Given that one of our core principles is to take care of test automation across different teams, and to teach everyone with interests under this topic, we have some tasks which have to get finished off by working with people from a couple of other teams. Those are:

Switch Mozmill CI from Mozmill 1.5.x to Mozmill 2.0

After we have released Mozmill 2.0 by end of September we now have to work on getting our CI system upgraded too. The necessary work is listed on issue 329 of our issue tracker on Github.

There is not that much to do here beside fixing two bugs, which block us from an upgrade. That’s a recent change which comes with Firefox 26.0 and confused our restart test handling. For more details see bug 922995. And on the other side our update testrun leaves mounted DMG images behind on OS X 10.6. Therefore see issue 90.

Get FirefoxMetro tests executed in Mozmill CI production

Given that the QA team gave us a couple of Metro tests to automate, we have to add support to run all of them in our CI system. There is no support for that type of application yet. So a good amount of work lays in-front of us. All tasks are covered by issue 215.

Mozmill 2.0 already comes with MetroFirefox support but still misses some critical pieces. As of now it’s not able to set the installed application as the default browser on the system. Starting with Windows 8 this system level change is way harder to do. If anyone of you has ideas in how to do that please let us know.

AWSY (Are we slim yet) Firefox OS Memory Tests

We have to develop test automation to add Firefox OS to http://www.areweslimyet.com. The test, which is an orangutan-based script driving the gaia-ui, will be run on the B2G emulator, using TBPL builds from mozilla-central. Memory dumps will be grabbed and results posted to AWSY, with the ultimate goal in identifying and preventing Firefox OS memory leak regressions. For more information please see the tracking bug.

Get SoftVision to lead the automation efforts for QA with our oversight

Our team members from the test automation team at Softvision are helping us for a long time now in maintaining the Mozmill tests repository by fixing broken tests and creating new ones. Also they have assisted in the final work for the Mozmill 2.0 release, and other tools we own. To give them more responsibility we want to bring all of them into the position to handle the Mozmill-CI system on their own. This is necessary given the much broader scope of tasks everyone in our team is assigned to. Therefore we will do trainings and demos in our Ask an expert meetings.

Team Goals

There are also some additional goals, which do not scale across teams, but are also important to note.

Deploy Eideticker CI system for Firefox OS

We have to implement a continuous integration system for the Firefox OS start-up and scrolling tests using an external camera. More details about this goal can be found on bug 904837.

Extend Mozmill CI to cover most of our localized builds

To lower the amount of possibly broken beta and release builds we have to extend our tests to cover nearly all supported locales. Therefore we have to bring more nodes online in our CI system, and further enhance the tests to run more efficient. The ultimate goals would be to test all of our >90 locales. If you are interested to help us please let us know or check issue 270.

Create web application for on-demand update jobs

We want to reduce the failure rate when setting up the configuration for on-demand update tests in our Mozmill-CI system. Therefore a simple web application has to be created which allows us to pick the configuration like what can be done with the trychooser. If you want to help please check bug 915164

Stretched Goals

Last but not least we also have goals which are stretched and we might work on if there is time left.

Setup try-server like Mozmill CI system

To help everyone who is working on creating new or fixing broken Mozmill tests, we want to have a try server like system. That means you could push your patch and it will be tested across platforms and Firefox versions.

Investigate requirements for community driving tool

One of the goals from Henrik is to lower the amount of volunteers who are leaving our team or even Mozilla because of lack of activity from our side. Given that there is no good way in keeping in touch with everyone yet, a tool has to be created or extended. So maybe we can chime into the Black Hole.

As you can see there is a lot of work, which is awaiting to be finished by this quarter. We always appreciate your willingness and interest in helping us. So if one or some of those goals have drawn your interest let us know.