Tag Archives: QA

Firefox Automation report – week 41/42 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 41 and 42.

With the beginning of October we also have some minor changes in responsibilities of tasks. While our team members from SoftVision mainly care about any kind of Mozmill tests related requests and related CI failures, Henrik is doing all the rest including the framework and the maintenance of Mozmill CI.

Highlights

With the support for all locales testing in Mozmill-CI for any Firefox beta and final release, Andreea finished her blacklist patch. With that we can easily mark locales not to be tested, and get rid of the long white-list entries.

We spun up our first OS X 10.10 machine in our staging environment of Mozmill CI for testing the new OS version. We hit a couple of issues, especially some incompatibilities with mozrunner, which need to be fixed first before we can get started in running our tests on 10.10.

In the second week of October Teodor Druta joined the Softvision team, and he will assist all the others with working on Mozmill tests.

But we also had to fight a lot with Flash crashes on our testing machines. So we have seen about 23 crashes on Windows machines per day. And that all with the regular release version of Flash, which we re-installed because of a crash we have seen before was fixed. But the healthy period did resist long, and we had to revert back to the debug version without the protect mode. Lets see for how long we have to keep the debug version active.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 41 and week 42.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 41 and week 42.

Firefox Automation report – week 9/10 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 9 and 10. I for myself was a week on vacation. A bit of relaxing before the work on the TPS test framework should get started.

Highlights

In preparation to run Mozmill tests for Firefox Metro in our Mozmill-CI system, Andreea has started to get support for Metro builds and appropriate tests included.

With the help from Henrik we got Mozmill 2.0.6 released. It contains a helpful fix for waitForPageLoad(), which let you know about the page being loaded and its status in case of a failure. This might help us to nail down the intermittent failures when loading remote and even local pages. But the most important part of this release is indeed the support of mozcrash. Even that we cannot have a full support yet due to missing symbol files for daily builds on ftp.mozilla.org, we can at least show that a crash was happening during a testrun, and let the user know about the local minidump files.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 9 and week 10.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 9 and week 10.

Double-check to disable Firefox Sync when doing a regression test to not loose important profile data

Yesterday I was doing a regression test for Firefox to determine the changeset which brought in a very annoying behavior into Aurora builds. Therefore I copied my profile to not destroy my original profile data, and was working on that copy. All was working fine and I was able to reduce the whole profile to only the sessionstore.js and prefs.js files.

The evil awakening came today morning when I tried to log into a website via my personal profile. By surprise the user name and password hasn’t been filled in automatically, and a check for the saved passwords have revealed that NONE are stored anymore! All were gone forever.

After some thoughts I figured out that sync was the problem here, because I missed to disconnect it in the copied profile. So after I have deleted the signons.sqlite and other files it also removed all the information from my sync account, which got distributed to all my other machines too. So on each of them the passwords are gone.

And if that isn’t bad enough, the backup tool on my Ubuntu machine had a hick-up lately and deleted all the old backup files on the backup drive. Not sure how that have come, but with that I completely lost all my stored passwords, and other synced data. :(

So a warning for everyone who is doing regression tests for Firefox and has Sync enabled: Please do NOT forget to disconnect Sync for the testing profile and double-check that it has been turned off.

New ‘in-qa-testsuite’ flag available for Mozilla QA driven test frameworks

Given the importance of automation in QA the Automation Development team had a discussion lately how to improve tracking of QA covered automated testcases in Bugzilla. In the past we haven’t had a way to mark code in bugs for new features or regression fixes as being covered by Mozmill automation.

As we have agreed on it is important to scale our visibility and make it easier for both developers and QA to determine if a feature has automated tests, whether in the tree, in the Mozmill tests repository, or any other framework we would have to create in the future. A whiteboard entry didn’t make much sense for us, and to better align with the existent ‘in-testsuite’ flag, we decided on the ‘in-qa-testsuite‘ flag.

So if you think that a feature needs coverage by Mozmill, make sure to set this flag and assign myself (use “:whimboo” as name). I will then ensure the request will be handled appropriately and the test gets implemented.

Further documentation on automation frameworks and this flag will be available soon. I will blog about once it is ready.

MemChaser 0.2 released

Exactly one month after we have released our initial version of MemChaser, version 0.2 has been made publicly available. You can install the add-on as usual from addons.mozilla.org or if it is already installed, simply check for updates within the Add-ons Manager.

A couple of subtle changes have been made which will give a better experience for users. So we have combined the formerly two widgets in the Add-on bar into a single one to prevent other extensions from inserting their widgets in-between. At the same time the width has been reduced to not waste too much space (keep in mind that we are still blocked on a SDK bug which doesn’t let the widget size adapt to the content length dynamically).

Given the recent landing of the incremental garbage collector in Nightly builds of Firefox we had to do some extra unplanned updates to support this new feature. When you are running a Nightly build and an incremental GC happens, the ‘GC‘ label will be replaced with ‘iGC‘.

Further we have improved the parsing of the GC/CC console messages to allow us to add more interested entries dynamically. So this time we have added the reason and type (global or compartmental) for a GC, and the number of collected entries from the cycle collector. All three will be written together with all the other data to the log file and are not visible in the add-on bar.

As reported by an user MemChaser v0.1.1 was affected by some memory leaks which were caused by the Add-on SDK 1.4. Given the tireless efforts of the SDK devs nearly all of those leaks (except for widgets) have been fixed in version 1.5, which has been recently released and is now used by MemChaser 0.2.

Beside all those visible changes we also made some improvements to our development process. So we have added the Add-on SDK as a submodule to our own repository on Github. Also to get more traction we made the decision to move the MemChaser repository to the Mozilla account. And last but not least we implemented the Travis CI integration to get our code automatically tested with each check-in.

If you are interested in further information about the changes feel free to check our list of issues for 0.2.

We would also like to welcome everyone who has interests to help us with coding, ui work, or documentation. There are a lot of features we want to implement and need talented people for. Stop by on one of our channels and get in contact with us.

Mozmill 1.5.2 and changes for running Firefox tests

As Clint Talbert has already written on Friday last week, Mozmill 1.5.2 has finally been released. A lot of great new stuff is now part of Mozmill core and we are happy to use it in our tests and automation scripts in the next couple of weeks.

The following changes will have to be made:

I’m looking forward to see all the mentioned enhancements active in March. Those are a big step forward, also for our users of the Mozmill Crowd extension.

New Firefox 3.7 branch for the mozmill-test repository

A couple of minutes ago I have branched the mozmill-test repository for our upcoming Firefox 3.7 work. With that addition we can start to update our tests on the default
branch to make them compatible with current Developer Preview releases and Minefield builds.

That means we have the following correlations now:

default => Firefox 3.7
mozilla1.9.2 => Firefox 3.6.x
mozilla1.9.1 => Firefox 3.5.x

More details about branch handling for our Mozmill tests can be found on MDC.

The Mozmill test repository is available under the following two locations:

http://hg.mozilla.org/qa/mozmill-tests/
http://github.com/whimboo/mozmill-tests

Testday: Exploratory testing the new Add-ons Manager

As you have probably already read about in a couple of blog post from Jennifer Boriss, Dave Townsend, and Blair McBride, the next major version of Firefox will contain a shiny new Add-ons Manager. If you wanna know more details, you should check out the design documents.

Even with the development for this feature still in progress, Mozilla QA will hold a testday for exploratory testing the new user interface on Friday, April 30th. For this time we will not use any Litmus test but running tests based on the current test plan.

If you are interested in getting your fingers wet and to help us in making it a helpful and less buggy feature, read through the test day documentation and join us next Friday on IRC.

Testday for Testscripting your Add-on with MozMill

Mozmill, which is a framework for running functional tests, can be used for any application which is built on top of the Mozilla platform. This includes Firefox, Thunderbird, SeaMonkey, and many others. But it’s not only possible to test the application itself. Instead it can also be used to run any type of functional tests for installed add-ons.

To stimulate add-on authors to create their own Mozmill tests, Mozilla QA is holding a testday on Friday, March 5th, which is fully devoted to Mozmill testscripting for your add-on. Learn how Mozmill tests will be written and how they can be run in Firefox. The Mozmill team will be around the whole day to assist you wherever possible.

If you are interested in the testday, you should read through the following documentation about the creation of testscripts for extensions.

You can also attend, when you have general questions about Mozmill or when you want to help in creating Mozmill tests for Firefox. Get ready and join us in #testday next Friday.