Review of Automation Services goals in Q1 2012

Now, three weeks after Q1 of 2012 has been passed by I finally have the time to give some details about our work happened during the first three months.

As usual we set a couple of goals we wanted to see by the end of the quarter. All of them were kinda important so we did our best to get all of them done – and we succeeded even if it was in the last minute! So everyone of us was happy about the goal achievements.

Below you will find an overview of each of those projects. For more detailed information I will follow-up with additional blog posts by next week.

Mozmill CI system for release, daily, and l10n builds

It has been taken a while for us to get to a state when we absolutely had to work on a continuous integration solution. Seeing that more and more tests are getting automated with Mozmill and that we still require manual interaction, or crontab jobs and tasks on Windows to get tests executed, we had to find a better way in getting tests run. After a bit of investigation we started by using Jenkins (formerly known as Hudson). And since beginning of this month we now have a system which runs our Mozmill tests for all the supported daily and beta candidate builds, and for each tinderbox build based on l10n check-ins on whichever branch. As trigger we make use of Mozilla Pulse which informs us directly when builds have been made available. Further we also had to ensure that tests can be triggered by on-demand, because update tests cannot be covered by Pulse yet and sometimes we have to re-run tests against a specific build. All in all the system works great but still needs a lot of improvements on which we will work on in the next couple of months.

Automation system for the Add-ons compatibile by default feature

With the change in Firefox 11 to make all add-ons compatible by default, we have been asked to create a system to automatically test add-on compatibility for the top 100 add-ons. As result a new testrun for Mozmill has been created which runs our endurance, update, and endurance tests again. That way we want to make sure that there are no performance regressions involved and updates of Firefox will be applied without any failures.

MemChaser – An extension to visualize memory and GC/CC related activities

Given the amount of manual testing for GC/CC related issues in Firefox by end of 2011, I had the idea to create an extension which makes it easy for users to track memory related activities. That means that all the important information is displayed directly in the add-on bar and don’t have to be retrieved from ‘about:memory‘ or the Error Console (which constantly wipes out old entries). The idea got approved as goal so Dave Hunt, David Guo, and myself were heavily working on getting this extension released on addons.mozilla.org as soon as possible. Meanwhile even more features have been implemented and more are about to come in the next couple of months. If you want to follow the development of the extension check the Github repository.

Mozmill Dashboard – Validation method for uploaded test reports

During a security review of our dashboard by end of 2011 it has been turned out that arbitrary data can be uploaded to our dashboard. The reason was that we haven’t checked the data which gets uploaded before storing it in our database. So we have had to implement a validation method for the CouchDB database. Now the report data gets checked and if the method fails the document will not be stored.

As mentioned before I will follow-up with detailed information later. So please watch out for upcoming blog posts in the next days.

Mozmill 1.5.11 released

Pushing out releases are fun! Especially if new regressions for patches landed a month ago will be identified 15 minutes after the release actually happened. Exactly that fooled us this time, and we even had to pull down Mozmill 1.5.10 from PyPI. As a Mozmill user you will probably have noticed that we skipped the announcement for Mozmill 1.5.10.

As by late yesterday Mozmill 1.5.11 is now available for everyone on PyPI. This version will be the last one we expect to see for the 1.5 branch. Current work will only happen on Mozmill 2.0, so it can be released by end of this quarter. If you are interested in helping us, please let us know and we definitely will find something for you.

As usual I would like to point out new features and bug fixes which are part of version 1.5.11:

  • Bug 686320: JSBridge is not limited to a single port (24242) anymore but probes for a free one now. This allows you to run multiple concurrent testruns with Mozmill at the same time. Keep in mind that the focus manager has to be put into testing mode for this.
  • Bug 696834: Securable modules have not been imported as singletons so that sharing data between modules didn’t work.
  • Bug 709773: The Mozmill API has been updated internally to let customers override the manifest (-m) and tests (-t) settings provided by the command line options.
  • Bug 727660: HTTP proxies are now supported for sending reports to a remote database.
  • Bug 740773: With the update of our Mozmill Dashboard invalid reports will be declined. Mozmill hasn’t shown any failure message yet which was kinda unhelpful in tracking down broken reports.

With the changes as mentioned above we can now go ahead and work on the transition of our Firefox tests to Mozmill 2.0.

MemChaser 0.3 is out

Late on Thursday we got our MemChaser 0.3 release out the door. Compared to the last releases this version comes with a ton of new features and bug fixes. Most of them have been requested by users and from the JS team so we put some extra focus on those.

As you can see in the image new ui elements have been added. The memory and garbage collector related items are equivalent to the buttons in ‘about:memory‘ and help those users who want to trigger a memory clean-up more often but don’t want to keep the before mentioned page open. The reaction of Firefox and the updated memory and GC/CC values can be seen in the MemChaser widget right after the triggered action has been finished.

Also we have updated our code which retrieves GC/CC related information to use the new Garbage Collector API. This allows us to get all the information and not have to scrape the console messages anymore for particular elements. That makes our life a lot easier and as benefit we can even store the complete set of data to the log file for further investigation. This log file is now a valid JSON file and will make it easier for you to analyze the data. If you wondered (with the former versions of MemChaser) where the log files have been stored, you can now open the containing folder via the new panel element too. If you are not satisfied with the location go ahead and change the target folder in the add-on preferences to any folder you like.

To help new users of the extension to understand the entries in the widget, tooltips have been added. Those will inform you about the background of the individual values. This was a request by a couple of users so we wanted to have this included too.

If you are interested in all the changes happened in the 0.3 release you can find the complete list as usual on our MemChaser issue tracker.

This version of MemChaser has still been built on top of the Add-on SDK 1.5, because we wanted to get out all of those new features also to our Firefox 10.0 user base which is about 28% of all our users. This was more important to us compared to fixes we would have benefit from the SDK 1.6. But at this time we want to announce that version 0.3 will be the last release which officially supports Firefox 10. With version 0.3.1 which will be released in the next one or two weeks a version bump to the SDK 1.6 will be done to fix the before mentioned issues like memory leaks.

If you are interested in helping us in the implementation of new features or fixing bugs, please check our issue tracker for open issues and contact us on IRC in the #automation channel.