Memchaser 0.6 has been released

The Firefox Automation team would like to announce the release of memchaser 0.6. After nearly a year of no real feature updates, but also some weeks of not being able to run Memchaser in Firefox Aurora (34.0a2) at all (due to a regression), we decided to release the current state of development as a public release. We are aware that we still do not fully support the default Australis theme since Firefox 29.0, but that’s an issue, which takes some more time to finish up.

Changes in 0.6

  • Upgrade to Add-on SDK 1.17 (#201)
  • Fix test_start_stop_logging for ‘File not found’ error (#199)
  • contentURL expects a string in Panel() and Widget() constructors (#198)
  • Bring the minimizeMemory() implementation up to date (#193)
  • Use ‘residentFast’ instead of ‘resident’ for memory reporter (#192)
  • Use postMessage for widget and panel communication instead port object (#185)
  • Support incremental cycle collection statistics (#187)
  • Added a usage section to the README (#178)
  • Change require statements and update the SDK to 1.14 (#174)
  • Simplify .travis.yml (#172)
  • Use . (dot) to include files while invoking /bin/sh (#169)
  • Use failonerror attribute in exec tasks (#168)

For all the details about the 0.6 release please check our issue tracker on Github.

Nightly Tester Tools 3.3 released

Having a good experience when testing Firefox Nightly builds should be self-evident. Therefor the QA Automation Services team is maintaining the Nightly Tester Tools extension, which brings a couple of useful features to ease the testing efforts.

While the extension works fine most of the time, we unfortunately fail at times when a version bump for Firefox happens. Exactly this action repeats in a 6 week interval and is the release cycle of Firefox.

So what happens here? The extension registers binary components for the crashme feature in the manifest file and that’s why it is not handled as a ‘compatible by default’ extension by the add-ons manager. So right after a version bump the extension gets marked as incompatible until the compatibility information on addons.mozilla.org has been updated. This results in an unavailability of the extension in Nightly builds for a couple of hours or even a day, depending on when one of us has time to actually do the version bump. That had to be changed, and thankfully Szabolcs Hubai has spent some time on it to fix the problem and to make our extension ‘compatible by default’. From now on you will never see the Nightly Tester Tools disabled after a version bump of Firefox, which is great. I hope you all will enjoy it!

With the 3.3 release we also ship some other fixes which you can read about in the changelog.

You want to help in the development of the extension? Feel free to contact us as usual via MozIRC or our mailing list.

Review of Automation Services goals in Q1 2012

Now, three weeks after Q1 of 2012 has been passed by I finally have the time to give some details about our work happened during the first three months.

As usual we set a couple of goals we wanted to see by the end of the quarter. All of them were kinda important so we did our best to get all of them done – and we succeeded even if it was in the last minute! So everyone of us was happy about the goal achievements.

Below you will find an overview of each of those projects. For more detailed information I will follow-up with additional blog posts by next week.

Mozmill CI system for release, daily, and l10n builds

It has been taken a while for us to get to a state when we absolutely had to work on a continuous integration solution. Seeing that more and more tests are getting automated with Mozmill and that we still require manual interaction, or crontab jobs and tasks on Windows to get tests executed, we had to find a better way in getting tests run. After a bit of investigation we started by using Jenkins (formerly known as Hudson). And since beginning of this month we now have a system which runs our Mozmill tests for all the supported daily and beta candidate builds, and for each tinderbox build based on l10n check-ins on whichever branch. As trigger we make use of Mozilla Pulse which informs us directly when builds have been made available. Further we also had to ensure that tests can be triggered by on-demand, because update tests cannot be covered by Pulse yet and sometimes we have to re-run tests against a specific build. All in all the system works great but still needs a lot of improvements on which we will work on in the next couple of months.

Automation system for the Add-ons compatibile by default feature

With the change in Firefox 11 to make all add-ons compatible by default, we have been asked to create a system to automatically test add-on compatibility for the top 100 add-ons. As result a new testrun for Mozmill has been created which runs our endurance, update, and endurance tests again. That way we want to make sure that there are no performance regressions involved and updates of Firefox will be applied without any failures.

MemChaser – An extension to visualize memory and GC/CC related activities

Given the amount of manual testing for GC/CC related issues in Firefox by end of 2011, I had the idea to create an extension which makes it easy for users to track memory related activities. That means that all the important information is displayed directly in the add-on bar and don’t have to be retrieved from ‘about:memory‘ or the Error Console (which constantly wipes out old entries). The idea got approved as goal so Dave Hunt, David Guo, and myself were heavily working on getting this extension released on addons.mozilla.org as soon as possible. Meanwhile even more features have been implemented and more are about to come in the next couple of months. If you want to follow the development of the extension check the Github repository.

Mozmill Dashboard – Validation method for uploaded test reports

During a security review of our dashboard by end of 2011 it has been turned out that arbitrary data can be uploaded to our dashboard. The reason was that we haven’t checked the data which gets uploaded before storing it in our database. So we have had to implement a validation method for the CouchDB database. Now the report data gets checked and if the method fails the document will not be stored.

As mentioned before I will follow-up with detailed information later. So please watch out for upcoming blog posts in the next days.

MemChaser 0.3 is out

Late on Thursday we got our MemChaser 0.3 release out the door. Compared to the last releases this version comes with a ton of new features and bug fixes. Most of them have been requested by users and from the JS team so we put some extra focus on those.

As you can see in the image new ui elements have been added. The memory and garbage collector related items are equivalent to the buttons in ‘about:memory‘ and help those users who want to trigger a memory clean-up more often but don’t want to keep the before mentioned page open. The reaction of Firefox and the updated memory and GC/CC values can be seen in the MemChaser widget right after the triggered action has been finished.

Also we have updated our code which retrieves GC/CC related information to use the new Garbage Collector API. This allows us to get all the information and not have to scrape the console messages anymore for particular elements. That makes our life a lot easier and as benefit we can even store the complete set of data to the log file for further investigation. This log file is now a valid JSON file and will make it easier for you to analyze the data. If you wondered (with the former versions of MemChaser) where the log files have been stored, you can now open the containing folder via the new panel element too. If you are not satisfied with the location go ahead and change the target folder in the add-on preferences to any folder you like.

To help new users of the extension to understand the entries in the widget, tooltips have been added. Those will inform you about the background of the individual values. This was a request by a couple of users so we wanted to have this included too.

If you are interested in all the changes happened in the 0.3 release you can find the complete list as usual on our MemChaser issue tracker.

This version of MemChaser has still been built on top of the Add-on SDK 1.5, because we wanted to get out all of those new features also to our Firefox 10.0 user base which is about 28% of all our users. This was more important to us compared to fixes we would have benefit from the SDK 1.6. But at this time we want to announce that version 0.3 will be the last release which officially supports Firefox 10. With version 0.3.1 which will be released in the next one or two weeks a version bump to the SDK 1.6 will be done to fix the before mentioned issues like memory leaks.

If you are interested in helping us in the implementation of new features or fixing bugs, please check our issue tracker for open issues and contact us on IRC in the #automation channel.

MemChaser 0.2 released

Exactly one month after we have released our initial version of MemChaser, version 0.2 has been made publicly available. You can install the add-on as usual from addons.mozilla.org or if it is already installed, simply check for updates within the Add-ons Manager.

A couple of subtle changes have been made which will give a better experience for users. So we have combined the formerly two widgets in the Add-on bar into a single one to prevent other extensions from inserting their widgets in-between. At the same time the width has been reduced to not waste too much space (keep in mind that we are still blocked on a SDK bug which doesn’t let the widget size adapt to the content length dynamically).

Given the recent landing of the incremental garbage collector in Nightly builds of Firefox we had to do some extra unplanned updates to support this new feature. When you are running a Nightly build and an incremental GC happens, the ‘GC‘ label will be replaced with ‘iGC‘.

Further we have improved the parsing of the GC/CC console messages to allow us to add more interested entries dynamically. So this time we have added the reason and type (global or compartmental) for a GC, and the number of collected entries from the cycle collector. All three will be written together with all the other data to the log file and are not visible in the add-on bar.

As reported by an user MemChaser v0.1.1 was affected by some memory leaks which were caused by the Add-on SDK 1.4. Given the tireless efforts of the SDK devs nearly all of those leaks (except for widgets) have been fixed in version 1.5, which has been recently released and is now used by MemChaser 0.2.

Beside all those visible changes we also made some improvements to our development process. So we have added the Add-on SDK as a submodule to our own repository on Github. Also to get more traction we made the decision to move the MemChaser repository to the Mozilla account. And last but not least we implemented the Travis CI integration to get our code automatically tested with each check-in.

If you are interested in further information about the changes feel free to check our list of issues for 0.2.

We would also like to welcome everyone who has interests to help us with coding, ui work, or documentation. There are a lot of features we want to implement and need talented people for. Stop by on one of our channels and get in contact with us.

Results of the second Add-ons Manager testday

Last Friday, June the 11th, we had our second testday for exploratory testing the new Add-ons Manager. It was again well attended and we had a couple of fantastic discussions across its whole duration. If you weren’t able to attend but interested in details about the discussions, you can read through the chat transcription which lists any specific detail.

Given our last testday end of April, I have continued my idea to see the testday covering more than only the PDT timezone. That means this time it has also been started 8am UTC and ended 5pm PDT. At the beginning we had lesser action but after lunch time in Europe more and more people joined and participated in discussions. The most active people in the channel were aaronmt, aleksej, dark_skeleton, gabe2300, kbrosnan, kinger, mossop, smaug, tchung, tobbi, tonymec, unfocused, wx24, and myself.

After all we were able to identify 11 new bugs which is much lesser than the last time. But I think it speaks for the work which happened in that area the last one and a half month.

We thank everyone who has participated in that testday and made it a success again. Now with the new themes approaching in the near future, another testday has to be targeted. Stay tuned and check for updates regularly.

Testday for creating Mozmill Tests for Add-ons

Mozmill is not only able to execute functional and unit tests against applications, which are build on top of the Gecko platform, but can also be used to run those type of tests against extensions.

If you are an extension author and interested to see your extension tested in a daily fashion by our automated test-runs in the QA-lab, you should definitely join the testday on Friday, May 28th. We really want to encourage you to think about the implementation of automated functional tests for your own extension. And it’s not only helpful for the extension itself, but can also help us to identify regression in the Firefox development cycle as early as possible.

To have an introduction available, we were working closely together with Google to have example tests in our mozmill-tests repository. Learn how Mozmill tests will be written and how they can be run in Firefox. The Mozmill team will be around the whole day to assist you wherever possible.

If you are interested in the testday, you should read through the following documentation about the creation of testscripts for extensions.

You can also attend when you have general questions about Mozmill or when you want to help in fixing or creating Mozmill tests for Firefox. Get ready and join us in #testday next Friday.

Testday for Testscripting your Add-on with MozMill

Mozmill, which is a framework for running functional tests, can be used for any application which is built on top of the Mozilla platform. This includes Firefox, Thunderbird, SeaMonkey, and many others. But it’s not only possible to test the application itself. Instead it can also be used to run any type of functional tests for installed add-ons.

To stimulate add-on authors to create their own Mozmill tests, Mozilla QA is holding a testday on Friday, March 5th, which is fully devoted to Mozmill testscripting for your add-on. Learn how Mozmill tests will be written and how they can be run in Firefox. The Mozmill team will be around the whole day to assist you wherever possible.

If you are interested in the testday, you should read through the following documentation about the creation of testscripts for extensions.

You can also attend, when you have general questions about Mozmill or when you want to help in creating Mozmill tests for Firefox. Get ready and join us in #testday next Friday.