Tag Archives: QA

Firefox Automation report – week 9/10 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 9 and 10. I for myself was a week on vacation. A bit of relaxing before the work on the TPS test framework should get started.

Highlights

In preparation to run Mozmill tests for Firefox Metro in our Mozmill-CI system, Andreea has started to get support for Metro builds and appropriate tests included.

With the help from Henrik we got Mozmill 2.0.6 released. It contains a helpful fix for waitForPageLoad(), which let you know about the page being loaded and its status in case of a failure. This might help us to nail down the intermittent failures when loading remote and even local pages. But the most important part of this release is indeed the support of mozcrash. Even that we cannot have a full support yet due to missing symbol files for daily builds on ftp.mozilla.org, we can at least show that a crash was happening during a testrun, and let the user know about the local minidump files.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 9 and week 10.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 9 and week 10.

Double-check to disable Firefox Sync when doing a regression test to not loose important profile data

Yesterday I was doing a regression test for Firefox to determine the changeset which brought in a very annoying behavior into Aurora builds. Therefore I copied my profile to not destroy my original profile data, and was working on that copy. All was working fine and I was able to reduce the whole profile to only the sessionstore.js and prefs.js files.

The evil awakening came today morning when I tried to log into a website via my personal profile. By surprise the user name and password hasn’t been filled in automatically, and a check for the saved passwords have revealed that NONE are stored anymore! All were gone forever.

After some thoughts I figured out that sync was the problem here, because I missed to disconnect it in the copied profile. So after I have deleted the signons.sqlite and other files it also removed all the information from my sync account, which got distributed to all my other machines too. So on each of them the passwords are gone.

And if that isn’t bad enough, the backup tool on my Ubuntu machine had a hick-up lately and deleted all the old backup files on the backup drive. Not sure how that have come, but with that I completely lost all my stored passwords, and other synced data. :(

So a warning for everyone who is doing regression tests for Firefox and has Sync enabled: Please do NOT forget to disconnect Sync for the testing profile and double-check that it has been turned off.

New ‘in-qa-testsuite’ flag available for Mozilla QA driven test frameworks

Given the importance of automation in QA the Automation Development team had a discussion lately how to improve tracking of QA covered automated testcases in Bugzilla. In the past we haven’t had a way to mark code in bugs for new features or regression fixes as being covered by Mozmill automation.

As we have agreed on it is important to scale our visibility and make it easier for both developers and QA to determine if a feature has automated tests, whether in the tree, in the Mozmill tests repository, or any other framework we would have to create in the future. A whiteboard entry didn’t make much sense for us, and to better align with the existent ‘in-testsuite’ flag, we decided on the ‘in-qa-testsuite‘ flag.

So if you think that a feature needs coverage by Mozmill, make sure to set this flag and assign myself (use “:whimboo” as name). I will then ensure the request will be handled appropriately and the test gets implemented.

Further documentation on automation frameworks and this flag will be available soon. I will blog about once it is ready.

MemChaser 0.2 released

Exactly one month after we have released our initial version of MemChaser, version 0.2 has been made publicly available. You can install the add-on as usual from addons.mozilla.org or if it is already installed, simply check for updates within the Add-ons Manager.

A couple of subtle changes have been made which will give a better experience for users. So we have combined the formerly two widgets in the Add-on bar into a single one to prevent other extensions from inserting their widgets in-between. At the same time the width has been reduced to not waste too much space (keep in mind that we are still blocked on a SDK bug which doesn’t let the widget size adapt to the content length dynamically).

Given the recent landing of the incremental garbage collector in Nightly builds of Firefox we had to do some extra unplanned updates to support this new feature. When you are running a Nightly build and an incremental GC happens, the ‘GC‘ label will be replaced with ‘iGC‘.

Further we have improved the parsing of the GC/CC console messages to allow us to add more interested entries dynamically. So this time we have added the reason and type (global or compartmental) for a GC, and the number of collected entries from the cycle collector. All three will be written together with all the other data to the log file and are not visible in the add-on bar.

As reported by an user MemChaser v0.1.1 was affected by some memory leaks which were caused by the Add-on SDK 1.4. Given the tireless efforts of the SDK devs nearly all of those leaks (except for widgets) have been fixed in version 1.5, which has been recently released and is now used by MemChaser 0.2.

Beside all those visible changes we also made some improvements to our development process. So we have added the Add-on SDK as a submodule to our own repository on Github. Also to get more traction we made the decision to move the MemChaser repository to the Mozilla account. And last but not least we implemented the Travis CI integration to get our code automatically tested with each check-in.

If you are interested in further information about the changes feel free to check our list of issues for 0.2.

We would also like to welcome everyone who has interests to help us with coding, ui work, or documentation. There are a lot of features we want to implement and need talented people for. Stop by on one of our channels and get in contact with us.

Mozmill 1.5.2 and changes for running Firefox tests

As Clint Talbert has already written on Friday last week, Mozmill 1.5.2 has finally been released. A lot of great new stuff is now part of Mozmill core and we are happy to use it in our tests and automation scripts in the next couple of weeks.

The following changes will have to be made:

I’m looking forward to see all the mentioned enhancements active in March. Those are a big step forward, also for our users of the Mozmill Crowd extension.

New Firefox 3.7 branch for the mozmill-test repository

A couple of minutes ago I have branched the mozmill-test repository for our upcoming Firefox 3.7 work. With that addition we can start to update our tests on the default
branch to make them compatible with current Developer Preview releases and Minefield builds.

That means we have the following correlations now:

default => Firefox 3.7
mozilla1.9.2 => Firefox 3.6.x
mozilla1.9.1 => Firefox 3.5.x

More details about branch handling for our Mozmill tests can be found on MDC.

The Mozmill test repository is available under the following two locations:

http://hg.mozilla.org/qa/mozmill-tests/
http://github.com/whimboo/mozmill-tests

Testday: Exploratory testing the new Add-ons Manager

As you have probably already read about in a couple of blog post from Jennifer Boriss, Dave Townsend, and Blair McBride, the next major version of Firefox will contain a shiny new Add-ons Manager. If you wanna know more details, you should check out the design documents.

Even with the development for this feature still in progress, Mozilla QA will hold a testday for exploratory testing the new user interface on Friday, April 30th. For this time we will not use any Litmus test but running tests based on the current test plan.

If you are interested in getting your fingers wet and to help us in making it a helpful and less buggy feature, read through the test day documentation and join us next Friday on IRC.

Testday for Testscripting your Add-on with MozMill

Mozmill, which is a framework for running functional tests, can be used for any application which is built on top of the Mozilla platform. This includes Firefox, Thunderbird, SeaMonkey, and many others. But it’s not only possible to test the application itself. Instead it can also be used to run any type of functional tests for installed add-ons.

To stimulate add-on authors to create their own Mozmill tests, Mozilla QA is holding a testday on Friday, March 5th, which is fully devoted to Mozmill testscripting for your add-on. Learn how Mozmill tests will be written and how they can be run in Firefox. The Mozmill team will be around the whole day to assist you wherever possible.

If you are interested in the testday, you should read through the following documentation about the creation of testscripts for extensions.

You can also attend, when you have general questions about Mozmill or when you want to help in creating Mozmill tests for Firefox. Get ready and join us in #testday next Friday.

Mozmill 1.3 released

Given the quick review on AMO (many thanks to you guys that this happened under a week!) the Mozmill team can call out that Mozmill 1.3 has been released. It’s available for download on addons.mozilla.com.

This release is a big step forward by adding a couple of new features and fixing some important bugs which have been found by users and have been introduced by the last release. A complete list can be found on Bugzilla.

Let’s give a short overview and mention some of the fixes/features:

  1. Bug 509912: We have updated the maxVersion for all applications. So Mozmill is not compatible up to Firefox 3.7a1pre, Thunderbird 3.1a1pre, and SeaMonkey 2.1a1
  2. Bug 508643: From now on new profiles are created by Firefox itself. Before that fix we have used the files from within the default profile folder of the default application. That caused failures for localized builds because the profile has been initialized with wrong profile data.
  3. Bug 516729: Tests failed by clicking on elements inside the content area if the window was too small. Now with this fix elements will be scrolled into view before clicking on them.
  4. Bug 522990: Nested elements in the content or chrome document weren’t correctly reported by the inspector which didn’t let you get an element string.
  5. Bug 512789: Both controller.check and controller.radio functions have been updated to work now.
  6. Bug 515072: A second parameter has been added for controller.assertJS which let you specify an object which can be accessed via “subject” from within that function. It allows to show a more detailed information for a failing test.
  7. Bug 500987: Restart tests can pass variables between test modules. There is a persisted property available by default which can be used to set/get user-defined values.
  8. Bug 515209: Restart tests can have a callback handler written in Python which can be called asynchronously.

If something has been regressed since the last version please file a bug under Testing/Mozmill on Bugzilla.

If you are interested and want to know more about Mozmill then join us in #QA on IRC or subscribe to the Mozmill developer list.