We have to admit that not much work happened on the extension in the last couple of months. Reasons are mostly other important projects and mainly the development of the new pre-configured environment for Mozmill. There are still some outstanding improvements to implement before we can release a new version of the environment.

If you are interested to help out please get in contact with us via our team page or any other public channel.

We also have moved our Mozmill environments for all platforms from my own Mozilla people account to a new secure location on mozqa.com. In the future please always use this new location to grab the latest version of the environment.

For more information please see the tracking bug for Mozmill Crowd 0.1.4.

As result we have raised bug 676224 two weeks ago with the request to create a public facing mailing list and newsgroup. The creation has been a bit delayed but finally happened last Friday. So if you want to participate in automation, feel free to join by subscribing yourself to one of the following communication channels:

• Mailing List: https://lists.mozilla.org/listinfo/dev-automation
• Newsgroup: news://news.mozilla.org/mozilla.dev.automation
• Google Groups: http://groups.google.com/group/mozilla.dev.automation

We are looking forward to being able to discuss automation topics with you in a more collaborated fashion as before.

Given those results we have decided to extend our daily test-runs and not only run the functional and update tests, but also cover endurance tests, add-on tests, and the new remote tests.

Especially with the endurance tests, Dave Hunt has shown that we can retrieve a lot of useful memory related information and find regressions or bad interactions with add-ons very easily. So it makes a lot of sense to additionally perform those tests on Nightly and Aurora builds of Firefox. The test results can be found on our dashboard.

But also for add-on authors who have written Mozmill tests for their add-ons, we now provide test results. Regressions in the add-on or in Firefox itself can now be identified as close as possible. Currently only 2 add-ons have Mozmill tests and are being tested. If you are an add-on author get in contact with us, and we can work together to get your add-on tested.

Remote tests, which we haven’t talked about yet, are tests which connect the testing of the Firefox UI and remote websites. Those are fairly new, means they have been added recently, and currently only cover tests for the ‘Get Add-ons’ pane of the new Add-ons Manager. Those were necessary to create because Selenium can not automate the installation of add-ons. For the latest test results see our dashboard.

We are now looking forward to any regression we can detect with the newly added test-runs. And those we can not only identify by our daily test-runs, but also from reports sent by users of the Mozmill Crowd add-on. So please help us testing Firefox by sending your test reports. Thanks!

With the hiding of the menu bar on Windows we also had to introduce some more launch points for the extension. Now you can not only reach it via the Tools menu, but also in the Firefox button or the Add-on bar.

Beside the changes for the extension we have also updated the Mozmill test environment. If you are already an user of Mozmill Crowd, please open the chosen storage folder and delete the mozmill-*.zip and mozmill-env folder. With the next test-run Mozmill Crowd will automatically download and install the new version. With the upcoming Mozmill Crowd 0.2 this process will be fully automatic then. But as long as it is not available please run those steps manually. Thanks.

1. We have two new named branches:
   • aurora for builds on the Aurora channel
   • beta for builds on the Beta channel
2. From now on new Mozmill tests will be first checked into the aurora branch. The reason is because we want to have our new tests stable, and there will be too many changes happening for Nightly builds which could cause our tests to fail. Whenever a merge for Firefox happens we will do the same for our new tests. That means we merge all changes from aurora to default. That way we can ensure to test all new features from the last Firefox release in Nightly builds on a daily basis. The same strategy will apply to the beta channel, which we will merge all the tests into once a merge for Firefox happened.
3. There will be situations which will cause us to check-in new tests on default first, but those should be really rarely and we will handle it on a case-by-case basis.

That’s why please take care that you always work on the aurora branch when you create new Mozmill tests.

Any questions or proposals? Feel free to ask.

With Mozmill Crowd 0.1.2 we now support the Endurance Tests, which have been used a lot in the last couple of weeks to find memory leaks in Firefox. Now as part of the crowd extension, those tests are as easy to execute as all the other supported test-runs. Only if you would like to run those tests with a given set of add-ons installed, you will have to run the tests outside of the extension. Dave Hunt wrote an excellent article about this topic recently.

Further we were also able to remove the dependency for the Python headers on Linux. That means there is no pre-requisite anymore which has to be fulfilled before Mozmill Crowd will work. Only on OS X you will have to install the Developer Tools because VirtualEnv requires it to work. We hope to fix that by not using VirtualEnv in one of the upcoming releases.

A complete list of fixed bugs you can find on the tracking bug for the Mozmill Crowd 0.1.2 release.

Important Note: The extension is still not able to detect if an updated test environment has been made available. If you already have used MozMill Crowd before make sure that you manually delete the ‘mozmill-env.zip’ file and ‘mozmill-env’ folder in the selected storage folder. Triggering a new test-run will automatically download and install the newest version of the environment.

Also please check out our tomorrows testday which is completely devoted to the MozMill Crowd extension and the test execution in Firefox 4.

I have to thank everyone for the amazing feedback. A lot ideas have been collected during those 2 days, which now need to be injected into our roadmap. One of the most important items is definitely the way how we offer a testing solution for add-ons. Because of the Jetpack team already offers unit tests via the Addon SDK framework, we should work together to get more add-on authors to create tests for their add-ons, but also to being able to collect more test results from our testing community. It’s absolutely an item for me after Firefox 4 has been released and we have finished our refactoring of the shared modules used by our Mozmill tests. So assume updates by next quarter.

Beside my own activities I was able to visit some other booth and finally got in contact with Bernhard Wiedemann from the openSUSE project, who now also runs our Mozmill tests on a daily basis on their machines. I’m kinda inspired by some of the features they are using, especially the creation of a video for the complete test-run. Features like those would have been already helped us a lot. It’s nothing more frustrating to not being able to reproduce an issue because of lack of information.

A big thanks goes also to the planning team of Mozilla who took care of anything related over this weekend. Thanks for the accommodation, the dinner and bowling party on Saturday and all the other small but necessary stuff. As always it was great to get in contact with our and external communities.

Oh, and some pictures have been uploaded to Flick.

• Save the screenshots which are automatically taken by the l10n test-run into the screenshots folder of the extension storage. That allows you to access those a way easier, compared to before when you had to locate those first in the temporary folder.
• Disable the slow script warning dialog which was unnecessary before, because we already know that the test-run will take longer than the 20s which are set as default.
• After a successful test-run show the logging information in the output section of the extension main window.

All those changes were part of Bug 627108. If you wanna give the extension a try, you can find the Mozmill Crowd 0.1.1 release on AMO.

By today the new “Mozilla QA” product has been created on Bugzilla under the Others section. At the same time 5 components have been created, which mostly fit the active projects of the QA Test Automation team. If you want to see details, check Bug 612578. Once a new component has been created, lot of you already know what happens. Right, normally hundreds of bugs have to be moved to the new location. I did the same a couple of minutes ago for our Mozmill Tests. Finally an amount of 508 bugs have been landed in the components under the Mozmill QA product. That move also cleaned-up all Firefox, Toolkit, and Core components we have used before to track the creation of Mozmill tests.

The old “Mozmill Tests” component under Testing has already been deleted. That means for any new issues please file bugs under “Mozilla QA > Mozmill Tests”.

If you want to watch ongoing work, you can subscribe to the following QA aliases:

• automation-team-dashboard@mozilla-qa.bugs – Team dashboard (starts soon)
• mozmill-automation@mozilla-qa.bugs – Automation scripts (updates, add-ons)
• mozmill-crowd-extension@mozilla-qa.bugs – The Mozmill Crowd extension
• mozmill-result-dashboard@mozilla-qa.bugs – Our Mozmill Result dashboard

The overall topic for this evening was how Mozilla is using Selenium and other tools to ensure the quality of their products. Therefore I have been asked to do a presentation about our client-side ui testing framework Mozmill. I was pleased to do so, because I thought that not that many people from that group have already seen how client tests for Firefox look like. Especially not when those get run directly from an extension in the crowd. But another reason also was our ongoing work for Mozmill itself and that we are thinking about using parts of Webdriver, which is a component of the upcoming Selenium 2.0 release. We already have contacts to the developers from Google, and I wanted to finally meet them and have a discussion to get an impression about the status of Webdriver.

I already arrived on Nov 2nd and had nearly one more day to finish up my slides and the extension itself. Everything went very well, even with a change in Minefield which has been broken the IPC-PIPE component my extension is relying on. By using an older Minefield build everything was ready a couple of hours before the meetup. The remaining time I mostly spent together with Simon Stewart, the leading person for the Webdriver development. He gave me a great insight in how native events are implemented, but also pointed out some open questions and still missing features. Personally I really missed the native events on OS X feature. I hope that it will find its way into Webdriver. But overall it looks great and once the focus issues are solved, we can try to investigate which parts we can re-use in Mozmill. Thanks Simon and Eran! Oh, and you are still welcome to do the same with our code.

Finally at 7pm the first chunk of people appeared and we had a welcome snack to create the first contacts. Half an hour later we have started with the first presentation in a really nice and already crowded lecture room. I believe about 50 interested people were attending the meetup – even with the underground strike happening in London this day. So Stephen Donner was heating-up everyone with technical details about how Selenium is used at Mozilla and how tests are getting run. Right after Simon Stewart entered the stage and while getting applause he talked about the current status of Selenium 2. After a following short break with more pizza and coke we continued with the second round of presenters. That means David Burns started to talk about how to use Selenium for client-side performance monitoring. He showed some interesting examples with Firebug’s YSlow add-on. Once he had finished his Q&A part, the stage was ready for myself. In the following 20 minutes people were able to see which topics Mozilla QA is working on, how strong we rely on and are proud of our community, and finally how the current state of the Mozmill Crowd extension is working. People were amazed of the demo, even it didn’t worked out that well because of a broken screen assignment between my MBP and the two big screens behind me. But that didn’t matter. I got lot of great feedback and questions afterward. Thanks everyone!

If you are interested in details of my talk, check my slideshare presentation:

If you have further ideas or proposals you want to tell us about, please use the following EtherPad document. We would appreciate it.

Update:I should mention that the extension is still under development and we target a release for September. If you want to follow the process, check the Mozmill Crowd project page.

If you are an extension author and interested to see your extension tested in a daily fashion by our automated test-runs in the QA-lab, you should definitely join the testday on Friday, May 28th. We really want to encourage you to think about the implementation of automated functional tests for your own extension. And it’s not only helpful for the extension itself, but can also help us to identify regression in the Firefox development cycle as early as possible.

To have an introduction available, we were working closely together with Google to have example tests in our mozmill-tests repository. Learn how Mozmill tests will be written and how they can be run in Firefox. The Mozmill team will be around the whole day to assist you wherever possible.

If you are interested in the testday, you should read through the following documentation about the creation of testscripts for extensions.

You can also attend when you have general questions about Mozmill or when you want to help in fixing or creating Mozmill tests for Firefox. Get ready and join us in #testday next Friday.

Even with the development for this feature still in progress, Mozilla QA will hold a testday for exploratory testing the new user interface on Friday, April 30th. For this time we will not use any Litmus test but running tests based on the current test plan.

If you are interested in getting your fingers wet and to help us in making it a helpful and less buggy feature, read through the test day documentation and join us next Friday on IRC.