As every year I have spent a bit of my time before the event to prepare a schedule including interesting talks and other organizational stuff. But as it has been turned out again, setting up a schedule for talks is useless for me, because I would sit most of the time in the Mozilla DevRoom anyway. There are way too many interesting presentations and discussions that it is hard for me to leave.

This year some bits were different. Given my cold I was not able to participate in the Beer Event in the Delirium Cafe. It’s usually the perfect initial come together of this conference to greet people and have the first conversations. I kinda missed it, but the remaining work for this week did a good job in not letting me think about if for so long. The other news we noticed with the arrival at the campus of the University Solbosch. Surprisingly the locations have been modified a bit to give everyone more space. The biggest change definitely was the move of the booths to another building, which wasn’t that close to the rest of the areas of interest. So I headed over only once to check our Mozilla booth. IMO we haven’t had such a great location this time but given by our staff a lot of people stopped by already. While checking the other booths I also found the one of the Jenkins project, where Kohsuke Kawaguchi – the creator of Jenkins (formerly Hudson) – was able to help me with all of my questions. I call this a success – thanks Kohsuke!

As my personal highlight I had an own talk about Continuous Integration with Mozmill scheduled on Sunday at noon. It went pretty well and I got some positive feedback. If you wasn’t able to make it, you can check my slides here:

The most impressive conversation and follow-up work happened definitely with the guys from the Mozilla Add-on SDK project. Starting from my quick question about a foreseeable support for Thunderbird, it ended-up in a lot of discussions and a final hacking session at Sunday afternoon, which I sadly wasn’t able to attend. So I’m absolutely interested in the results. Oh, and while traveling back home I have spent about an hour to contribute my first patch to the project, which will let the builder add contributor names to the extensions install.rdf file.

All in all it was a stunning weekend and I hope to be able to attend it next year again. So a big thanks to Benoit and the Belgian community for taking care of the organization!

We have to admit that not much work happened on the extension in the last couple of months. Reasons are mostly other important projects and mainly the development of the new pre-configured environment for Mozmill. There are still some outstanding improvements to implement before we can release a new version of the environment.

If you are interested to help out please get in contact with us via our team page or any other public channel.

We had to release this version to fix a recent regression which has been identified in Mozmill 1.5.7, and which caused a hard-stop when an user restart has been requested. The issue did only exist on OS X.

Beside this fix we now enabled the new boolean preference called ‘focusmanager.testmode’ by default. It will allow us to run multiple Mozmill tests concurrently on the same machine. Support for the focusmanager testing mode has already been made available on Nightly and Aurora builds. Also there is a good chance to get this patch landed in an upcoming Firefox 10.0.1 release. I will follow-up with another blog post with more details once the final landing has happened and we can actually make use of it across our supported branches.

Beside that we also pushed another fix which re-enables the installation of add-ons from the application scope. It has been accidentally disabled in a former release.

Happy testing!

A list of fixes can be found on Bugzilla.

We also have moved our Mozmill environments for all platforms from my own Mozilla people account to a new secure location on mozqa.com. In the future please always use this new location to grab the latest version of the environment.

For more information please see the tracking bug for Mozmill Crowd 0.1.4.

So thanks to everyone for their help in letting us get the following stuff out:

• Add support for Linux3 kernels (Bug 664564)
• Disable any preference for caching XUL elements (Bug 661588)
• The private property .documentLoaded has been renamed to .mozmillDocumentLoaded and a public API has been added to avoid namespace conflicts on window objects (Bug 661408)
• Don’t install distributed extensions from the application folder per default (Bug 661008)
• Add support for iFrames to the waitForPageLoad() method (Bug 659000)
• Expose full stack frames again after the Error properties are not enumerable anymore (Bug 650646)
• Add Mozmill version to the report document (Bug 636746)

If you haven’t upgraded yet, please do so by running: pip install --upgrade mozmill

This command will also upgrade all dependent tools necessary by Mozmill. For questions just drop by in our #mozmill channel on IRC.

Given those results we have decided to extend our daily test-runs and not only run the functional and update tests, but also cover endurance tests, add-on tests, and the new remote tests.

Especially with the endurance tests, Dave Hunt has shown that we can retrieve a lot of useful memory related information and find regressions or bad interactions with add-ons very easily. So it makes a lot of sense to additionally perform those tests on Nightly and Aurora builds of Firefox. The test results can be found on our dashboard.

But also for add-on authors who have written Mozmill tests for their add-ons, we now provide test results. Regressions in the add-on or in Firefox itself can now be identified as close as possible. Currently only 2 add-ons have Mozmill tests and are being tested. If you are an add-on author get in contact with us, and we can work together to get your add-on tested.

Remote tests, which we haven’t talked about yet, are tests which connect the testing of the Firefox UI and remote websites. Those are fairly new, means they have been added recently, and currently only cover tests for the ‘Get Add-ons’ pane of the new Add-ons Manager. Those were necessary to create because Selenium can not automate the installation of add-ons. For the latest test results see our dashboard.

We are now looking forward to any regression we can detect with the newly added test-runs. And those we can not only identify by our daily test-runs, but also from reports sent by users of the Mozmill Crowd add-on. So please help us testing Firefox by sending your test reports. Thanks!

With the hiding of the menu bar on Windows we also had to introduce some more launch points for the extension. Now you can not only reach it via the Tools menu, but also in the Firefox button or the Add-on bar.

Beside the changes for the extension we have also updated the Mozmill test environment. If you are already an user of Mozmill Crowd, please open the chosen storage folder and delete the mozmill-*.zip and mozmill-env folder. With the next test-run Mozmill Crowd will automatically download and install the new version. With the upcoming Mozmill Crowd 0.2 this process will be fully automatic then. But as long as it is not available please run those steps manually. Thanks.

1. We have two new named branches:
   • aurora for builds on the Aurora channel
   • beta for builds on the Beta channel
2. From now on new Mozmill tests will be first checked into the aurora branch. The reason is because we want to have our new tests stable, and there will be too many changes happening for Nightly builds which could cause our tests to fail. Whenever a merge for Firefox happens we will do the same for our new tests. That means we merge all changes from aurora to default. That way we can ensure to test all new features from the last Firefox release in Nightly builds on a daily basis. The same strategy will apply to the beta channel, which we will merge all the tests into once a merge for Firefox happened.
3. There will be situations which will cause us to check-in new tests on default first, but those should be really rarely and we will handle it on a case-by-case basis.

That’s why please take care that you always work on the aurora branch when you create new Mozmill tests.

Any questions or proposals? Feel free to ask.

With Mozmill Crowd 0.1.2 we now support the Endurance Tests, which have been used a lot in the last couple of weeks to find memory leaks in Firefox. Now as part of the crowd extension, those tests are as easy to execute as all the other supported test-runs. Only if you would like to run those tests with a given set of add-ons installed, you will have to run the tests outside of the extension. Dave Hunt wrote an excellent article about this topic recently.

Further we were also able to remove the dependency for the Python headers on Linux. That means there is no pre-requisite anymore which has to be fulfilled before Mozmill Crowd will work. Only on OS X you will have to install the Developer Tools because VirtualEnv requires it to work. We hope to fix that by not using VirtualEnv in one of the upcoming releases.

A complete list of fixed bugs you can find on the tracking bug for the Mozmill Crowd 0.1.2 release.

Important Note: The extension is still not able to detect if an updated test environment has been made available. If you already have used MozMill Crowd before make sure that you manually delete the ‘mozmill-env.zip’ file and ‘mozmill-env’ folder in the selected storage folder. Triggering a new test-run will automatically download and install the newest version of the environment.

Also please check out our tomorrows testday which is completely devoted to the MozMill Crowd extension and the test execution in Firefox 4.

As more and more different runs are getting added the structure of the whole repository, we have agreed on in 2009, doesn’t make sense anymore. Also looking ahead to our upcoming API refactoring which will land soon, we had to chose a new structure of the whole repository. A discussion was necessary to make sure the new hierarchy will fit our needs for the next couple of months or even years. And I can say for sure that we will add a lot more of test-runs and those need the right place to live.

To get this step accomplished, I worked on top of the specs and got the new structure implemented today. If you are interested in check bug 637306. So without reading through the bug you may ask, what has been changed? Lets compare in detail the structure before and after my work.

Before:

• addons (Tests for add-ons)
• firefox (Folder of mixed tests for Firefox)
  • crashTests (Crash tests)
  • enduranceTests (Endurance tests)
  • helperClasses (Tests for shared code)
  • l10nTests (l10n tests)
  • restartTests (Functional restart tests)
  • softwareUpdate (Update tests)
  • test-files (Test data served via httpd.js)
  • test* (Folders which contain functional tests)
• shared-modules (Shared code divided into components)
• templates (Simple templates for starters)

After:

• data (Test data served via httpd.js)
• lib (Shared code divided into components)
  • tests (Tests for shared code)
• tests (Folder for the supported test-runs)
  • addons (Tests for add-ons)
  • crash (Crash tests)
  • endurance (Endurance tests)
  • functional (Functional tests for Firefox)
    • restartTests (Functional restart tests)
  • l10n (l10n tests)
  • update (Update tests)
• templates (Simple templates for starters)

As you can see sections are clearly defined and it’s easy to find the code you are looking for. It’s the perfect base for our future work, you will hear about in the next couple of weeks.

The following changes will have to be made:

• With the nearly complete refactoring of the Mozmill Python modules our Mozmill wrapper used by the automation scripts is no longer necessary and needs to be removed. For me it means I have to complete rewrite the automation scripts. With those changes implemented we will no longer be backward compatible to former Mozmill releases.
• Mozmill can now run a list of specified tests from the command line. We want to use that feature to allow users of the Mozmill Crowd extension to run selected tests in a given order. That means that the automation scripts will have to support this feature, and I will take care about in the rewrite.
• We have disabled the compatibility check for add-ons in Firefox 4. That was necessary because lot of add-ons aren’t compatible yet, and weren’t correctly installed from the command line when specified via the –addons option. Now we also want to support it in our automation scripts.
• Failures for fatal and non-fatal assertions are now using the same structure, which is important for our Mozmill Dashboard and especially for the l10n testrun results which mostly contain results of non-fatal assertions.

I’m looking forward to see all the mentioned enhancements active in March. Those are a big step forward, also for our users of the Mozmill Crowd extension.

I have to thank everyone for the amazing feedback. A lot ideas have been collected during those 2 days, which now need to be injected into our roadmap. One of the most important items is definitely the way how we offer a testing solution for add-ons. Because of the Jetpack team already offers unit tests via the Addon SDK framework, we should work together to get more add-on authors to create tests for their add-ons, but also to being able to collect more test results from our testing community. It’s absolutely an item for me after Firefox 4 has been released and we have finished our refactoring of the shared modules used by our Mozmill tests. So assume updates by next quarter.

Beside my own activities I was able to visit some other booth and finally got in contact with Bernhard Wiedemann from the openSUSE project, who now also runs our Mozmill tests on a daily basis on their machines. I’m kinda inspired by some of the features they are using, especially the creation of a video for the complete test-run. Features like those would have been already helped us a lot. It’s nothing more frustrating to not being able to reproduce an issue because of lack of information.

A big thanks goes also to the planning team of Mozilla who took care of anything related over this weekend. Thanks for the accommodation, the dinner and bowling party on Saturday and all the other small but necessary stuff. As always it was great to get in contact with our and external communities.

Oh, and some pictures have been uploaded to Flick.