That means we have the following correlations now:

default => Firefox 3.7
mozilla1.9.2 => Firefox 3.6.x
mozilla1.9.1 => Firefox 3.5.x

More details about branch handling for our Mozmill tests can be found on MDC.

The Mozmill test repository is available under the following two locations:

http://hg.mozilla.org/qa/mozmill-tests/
http://github.com/whimboo/mozmill-tests

Even with the development for this feature still in progress, Mozilla QA will hold a testday for exploratory testing the new user interface on Friday, April 30th. For this time we will not use any Litmus test but running tests based on the current test plan.

If you are interested in getting your fingers wet and to help us in making it a helpful and less buggy feature, read through the test day documentation and join us next Friday on IRC.

To stimulate add-on authors to create their own Mozmill tests, Mozilla QA is holding a testday on Friday, March 5th, which is fully devoted to Mozmill testscripting for your add-on. Learn how Mozmill tests will be written and how they can be run in Firefox. The Mozmill team will be around the whole day to assist you wherever possible.

If you are interested in the testday, you should read through the following documentation about the creation of testscripts for extensions.

You can also attend, when you have general questions about Mozmill or when you want to help in creating Mozmill tests for Firefox. Get ready and join us in #testday next Friday.

This release is a big step forward by adding a couple of new features and fixing some important bugs which have been found by users and have been introduced by the last release. A complete list can be found on Bugzilla.

Let’s give a short overview and mention some of the fixes/features:

1. Bug 509912: We have updated the maxVersion for all applications. So Mozmill is not compatible up to Firefox 3.7a1pre, Thunderbird 3.1a1pre, and SeaMonkey 2.1a1
2. Bug 508643: From now on new profiles are created by Firefox itself. Before that fix we have used the files from within the default profile folder of the default application. That caused failures for localized builds because the profile has been initialized with wrong profile data.
3. Bug 516729: Tests failed by clicking on elements inside the content area if the window was too small. Now with this fix elements will be scrolled into view before clicking on them.
4. Bug 522990: Nested elements in the content or chrome document weren’t correctly reported by the inspector which didn’t let you get an element string.
5. Bug 512789: Both controller.check and controller.radio functions have been updated to work now.
6. Bug 515072: A second parameter has been added for controller.assertJS which let you specify an object which can be accessed via “subject” from within that function. It allows to show a more detailed information for a failing test.
7. Bug 500987: Restart tests can pass variables between test modules. There is a persisted property available by default which can be used to set/get user-defined values.
8. Bug 515209: Restart tests can have a callback handler written in Python which can be called asynchronously.

If something has been regressed since the last version please file a bug under Testing/Mozmill on Bugzilla.

If you are interested and want to know more about Mozmill then join us in #QA on IRC or subscribe to the Mozmill developer list.

Since ever Firefox is supporting automatic updates we also have to check that each and every user will get the right update package for the installed version of Firefox. Most of our users should run the latest version of Firefox but there are also cases where people don’t update immediately or even don’t want to upgrade to the next major version of Firefox. Given that updates have to be delivered to each of the supported branches (e.g. Firefox 3.0.0.x and Firefox 3.5.x) and also as major update for upgrading to the next major version. We also have a community beta program running where users can help testing beta versions of the next Firefox version. Those users will get a separate update offer on another update channel.

Finally there are 4 different channels we have to test for en-US and some of our P1 localized builds. In detail those are in the right order:

• betatest: This channel makes sure that updates which will be delivered to beta users will pass.
• beta: Beta testers will get their updates on that channel.
• releasetest: This channel tests the update snippets which have been pushed to our official download mirrors.
• release: Default channel for all Firefox installations to get the next version.

For each of those mentioned channels we offer partial and complete updates. The former one will be used if the latest minor version of Firefox is in use, e.g. a user wants to update from 3.5.4 to 3.5.5, while the latter one is for all other versions of the same branch. If an update fails to apply which could happen due to different reasons like a download problem, users will not get stuck on their installed version. In such a case a fallback update will be downloaded which is identical to the complete update. If that fails too the same process will be started again after a given time.

Until now these tests had to be done manually by us. An example can be seen in the update section of the Firefox 3.5.5 test plan. So we normally tests updates on all supported platforms, for each update type (minor, major), and make sure that fallback updates will pass.

With the new software update tests for Mozmill which I have finished two days ago, we can easily automate this process now. The only manual steps which have to be done is to prepare the tests by downloading the necessary builds for all the platforms and place them in their own folders. Once that is done the automated test can be started. It will use all builds within a given folder and runs tests updates for the specified channel. The results are printed in wiki format to the console and only have to be copied to the appropriate Wiki page.

If you are interested in running those update tests you have to install Mozmill on your machine and clone our Mozmill test repository. Detailed steps can be found in the Mozmill test creation tutorial on QMO.

This is a big step forward in a direction where we can run update checks against each localized build of Firefox and can make sure that updates are successfully applied and don’t fail or cause any sort of failure. And it will give QA more time to focus on other topics.

If you are interested and want to know more about Mozmill then join us in #QA on IRC or subscribe to the Mozmill developer list.

Everyone who is using Mozmill regularly is welcome to help us in testing the beta version. As long as no big issues will come up the release of Mozmill 1.3 will happen next week.

If you want to test the extension please download it from Github.

Users of the pyPI packages only have to run “easy_install -U mozmill” to get the latest packages for Mozmill, JSBridge, and Mozrunner.

Thanks in advance.

There is a question which I get asked very often from developers: “Why do we need Mozmill tests when we already have a suite of automated tests available?” The clear answer is that those tests are used to simulate user actions on UI elements the same way as when a user would sit in-front of the computer. That means that for example clicks on hidden or disabled elements shouldn’t trigger the execution of the underlying command. That’s the difference to Mochitests which always trigger the command when the synthesizeMouse function is used to click on an element. Another really helpful feature is the capability to run restart tests of any sort. That’s not possible with the existing test harnesses which makes Mozmill test unique.

Seeing the importance of those tests we want to have a full suite of BFT tests for Mozmill by end of Q2 in 2010. The total number of 196 doable tests, except the ones which require OS level interaction, would allow us to run 82% of the tests automated. At the moment 65 of the tests have been already finished and can be run with Mozmill against builds from the 1.9.1 and 1.9.2 branch. For detailed information about the current state and actual work please check the Google spreadsheet.

To get more tests automated the following goals have been set by the QA execution team for Q4 in 2009:

• Firefox 3.6 will be released this quarter. To enhance our testing we want to automate all of the tests in 4 BFT subgroups prioritized as P1. In general these subgroups are: Awesomebar, Add-ons Manager, Download-Manager, and Tabbed Browsing. This will incite us to write the next 40 tests. A list of all available subgroups and their prioritization can be found in the Feature Ownership document.
• For us who are working on release testing, software update tests have to be performed for the betatest, beta, releasetest, and release channels. Given the manual work which have to be performed here automation will help a lot. I will finalize my software update tests so they can be run by everyone.
• Running Mozmill tests you will get results reported in the terminal. Even with the integrated capability to send those reports to a server we don’t have a web frontend to display those results. We want to use Brastacks to visualize Mozmill results similar to the Fennec test results. This work will be a joined effort with Testdev.

If anyone is interested in helping us to write or maintain Mozmill tests you can read more about it in the test creation tutorial or simply join us on IRC and get in contact with whimboo or aakashd. But you can also send me a mail or comment on this blog post. Thanks!

Given my project to get manual Litmus tests automated with Mozmill I have prepared some slides with a special focus on l10n. But sadly I wasn’t able to join the conference because of sickness. I have to say a big thanks to my colleague Marcia Knous and also to one of our main contributors for Sunbird tests Merike Sell who both hold the session. As informed at the end of the session via IRC the talk was a great success and a lot of questions were ask.

Due to the amount of sessions not everyone was able to join the Mozmill session. Also given all the people who weren’t be able to come I have uploaded my slides for all of you now. Please check the embedded Slideshare content below:

Feedback
Because I haven’t got any feedback from localizers so far I’m anxious to hear what you think about the usefulness of Mozmill and testing with localized builds. Given by the current number we have over 70 official locales available which are not tested by automated tests and require manual testing from localizers and contributors on a regular basis. With all the 250 BFT and another 750 FFT tests enabled in Litmus manual testing is a time taking action. Running all the tests with Mozmill will take much lesser time, could be run more often, and could cover all platforms which will result in a higher quality of Firefox and helps us to minimize any new regressions for our huge user base.

Please check the following questions I’m interested in getting an answer:

1. How often does your l1on team run Litmus tests against your locale whether those are BFT/FFT or the localizer test-run?
2. Would you like to see much of those tests automated and are you interested in running those tests on your local machine for each major and stability release?
3. Are you interested to help QA in writing Mozmill tests so we have most of them available as soon as possible?
4. Do you have further ideas how Mozmill can be used in the l10n area additionally to the points I have pointed out in my slides?

Thanks in advance for your feedback!

It was an unbelievable Monday. And definitely the best one ever in my life. After 8 years of active contribution to the Mozilla project, I was getting hired by Mozilla. Means, from now on I work as a full-time QA engineer in the QA execution team. And like any other remotely connected office-less person I’m working from home. For now I’m based in Karlsruhe which is located at the west side of Germany. But after years away from my homeland I will move back and enjoy the living in the beautiful capital of Saxony, called Dresden.

If anyone is interested on which topics I’ll work, here some raw data for now:

1. MozMill test creation for functional ui testing
2. Litmus testcase creation
3. Release testing for Firefox 3.0.x/3.1 and Thunderbird 2.0.0.x
4. Content creation for QMO
5. Community work
6. The usual Bugzilla work
7. …

Thanks everybody for the warm welcome last week. I’m really anxious to see how everything is managed in the headquarters. And also to meet anyone of you over there next month.

But for now it should be enough. Lets dig into MozMill and create some helpful UI tests…

The next step will be to get the list of patches, which have been checked-in between these builds. Until now I did that by taking the build id from each build and entering it in the date fields on the pushlog page on hg.mozilla.org. The problem here is, that you have to remember to add the time difference from PST to UTC or even your local time. I forgot this a couple of times and wondered why none of the listed check-ins were related in any way to the regression I’ve found.

But today I got a nice hint from Ted Mielczarek how to get the correct list of check-ins more easily. With his comment on Bugzilla he pointed out that recent builds offer the changeset information on the “about:buildconfig ” page:

The only thing you have to do now is to copy the changeset id for both versions and to create an URL like this one which will show you the list of check-ins.

For the moment it seems like that there is no UI for such a query but it would be great to have one in the future. But even in the current stage it will boost your work flow to finish any regression test.

Update: I’ve filed bug 471321 which covers the missing ui issue.