Some days ago we noticed that with the Mozmill 1.5.12 release we accidentally dropped the support for Firefox 3.6 or the Gecko 1.9.2 platform. It has been introduced with the fix for the global per compartment issue on bug 751424, especially with the addition of the ‘outer-window-destroyed’ observer notification listener.
Given the EOL of Firefox 3.6 and the Gecko 3.6 platform we do not intent to push another hotfix release for Mozmill 1.5 at this time. If you have to test applications which are based on Gecko 1.9.2 you should use Mozmill 1.5.11.
If you feel that this doesn’t solve your problem and we should re-add support for Firefox 3.6, please get in contact with us and together we can figure out what will be the best strategy for you.
As you have probably already read we had some changes to our team. With the transition over into the Automation and Tools Engineering team we can now focus even more on test harnesses, the APIs, and the test creation process while educating others how to write automated tests. Given that new responsibility I want to announce a change in how we will manage Mozmill tests from now on. All that is done with the idea of having a faster pace in getting tests automated and fixed.
- We have to do reviews faster. Right now I see a lot of time wasted because the reviewer of tests is in a different timezone and cannot react immediately to requests. That means that patches which have been put up for review will not get reviewed before the evening in Europe. Revising the patch has to happen the next day. Normally we have at least two iterations on a patch so it takes about 2 or 3 days until the patch is ready for check-in. I would like to reduce this time to 1 or at maximum 2 days by promoting at least us (Dave Hunt and me) as reviewers. This applies to new and broken tests. Please split up review requests between us.
- We have to ensure that the test covers the appropriate manual test on Litmus. Not sure how close we are at this right now because I never did those reviews since they moved over to Anthony Hughes. If we were good in that area lets keep up that work, otherwise we have to improve. That said we will split up the process in a review and feedback section. While the reviewer takes care about the actual code implementation, the person for feedback checks the test
for completeness. Only if both areas are fulfilled the test can be checked in. While Dave and myself will take care about reviews, Anthony is the one who knows the most of the current Firefox development and should act as the feedback person for now. Both can happen independently and should not be canceled if the other request gets denied.
- We have to fix broken tests and do not only skip them. In the past few months a couple of tests were failing and have been marked as skipped. It has happened a couple of times so that we now end-up with a lot of skipped tests. That doesn’t really help us and we have to get better. So before we create and work on new tests we have to make sure to really fix the broken tests. There should be used a round robin method between all of us (Automation Development and Desktop Automation) so that we will not have any skipped tests due to failures for each of the branches. I think that should be doable.
- We have to start the education. Learning from the experience of users while giving reviews will train Dave and me to see where the problems are located. That means that we can make sure to get more and better documentation up for those areas. If we wouldn’t work on reviews we would simply miss those parts. As result everyone who wants to contribute will benefit from and become a writer for automated tests in a shorter time.
I’m sure that with all those changes we can drastically improve the code quality and coverage of automated functional tests with Mozmill.
Some changes to our team happened lately I want to quickly talk about before starting to dive into our team goals for this quarter. So what’s going on?
Automation is a key part when it comes to qualifying new and existing code. Over the last years it’s getting more and more important at Mozilla for developers and QA to proof the quality of Firefox on a fast pace without the hassle of manual testing. With the change in the Firefox release process to a rapid train model a ton of different builds have to be tested by QA on a regular basis. Doing all that manually is impossible. A test framework which has a high impact for QA is Mozmill and the tests and automation tools written around by the Automation Services team over the last couple of months help a lot. But at the current state it’s not enough. We have to be better and cover way more of the manual testing with automated tests.
To make that possible the Automation Services team is now part of the Automation and Tools Engineering team (aka A-team) and has been renamed to Automation Development. We, which means Dave Hunt and myself, are part of the crew who will be responsible for any kind of automation question. For now we can only cover the Mozmill and Selenium frameworks but with additional team members added hopefully soon, more and more harnesses can be covered. Another key aspect for our crew will be mentorship and education, which means that we are eager to bring you into the automation land and to know all the stuff to write automated tests on your own. Sounds interesting? It definitely is, and I like it a lot!
So what will we work on during the current quarter?
There is a lot of work which has to be done for Mozmill to finally get out version 2.0 and to update our automation scripts and Mozmill tests, including a new API to write the tests. So those will be our prioritized projects.
- Automation Scripts: Those scripts drive our Mozmill test-runs for different versions of Firefox. They have been written in the early stages around Mozmill 1.0 and need a complete refactor to work with Mozmill 2.0. Also a couple of features will have to be moved to MozBase so that even other harnesses can benefit from. This is all Python related work.
Beside those projects we will also continue our work in the following areas:
- Mozmill CI: More and more tests are getting covered by our new Mozmill CI system, which I will talk about more in detail soon. So we have to move this instance to an ESX cluster to have a ton of machines for test execution, include handling of the DTC (default to compatible) test-run, and make sure that the upcoming Firefox Beta Nightly builds are handled too.
- Mozmill Tests: We take a stab in handling and coordinating the creation of new Mozmill tests and fixing the broken ones. The reason for this change I will also explain in a separate blog post shortly. But to say in general we need more test coverage and skipped/failing tests over a long time are bad practice and counter-productive. So lets get all of them re-enabled again and new tests written.
- Addons: There will not be that much time for us this quarter to continue developing the add-ons we own. But we will make sure to fix breakage for MemChaser and Nightly Tester Tools if those occur. If you want to help out, you are more then welcome!
- WebApps Testing: While this is not on our official goals for this quarter we have to assist the WebApps QA team to figure out the right test harnesses to create automated functional tests for open web apps. It will be a challenge and we are kinda interested to see that upcoming.
As you can see there are plenty of projects to work on. Any of those are public and are waiting for your contribution. If you are interested in working with us and to raise your personal skills, please contact us via our public mailing list or #automation IRC channel.
Something we have learned over the weekend was: Never say never. After we have released the 1.5.11 release of Mozmill on April 19th, we were sure to not have to ship any other release off this branch. We want to concentrate our work on Mozmill 2.0 and get it out as soon as possible.
But things can change, and sometimes faster as expected. So by last Friday (it was already late in the evening) Mike Conley contacted me on IRC and mentioned that Mozmill will be broken starting with tomorrows build. The reason for it was the ‘Global-Per-Compartment’ patch which has been landed in Nightly builds of Firefox and Thunderbird at that time. Great! Why do things like those happen
just before when I’m already in my weekend?
Anyway, given the importance of this bug (it broke the whole test infrastructure of Thunderbird which mostly relies on Mozmill) I was talking immediately to Clint Talbert. Given the good bug report from John Schoenick we were directly aware of what the problem was and agreed on a new API to get the issue fixed.
So I have spent a good portion of my Saturday and Sunday evening to put together a patch which removed the broken code in waitForPageLoad() and replaces it with the new API. So by now we do no longer add a custom ‘mozmillDocumentLoaded’ property to all windows, but have a nice windowMap object that handles all the loaded states of each window on its own. I think that’s great and modifying the DOM of each window was even a bad idea since the beginning of the project. Good to see that killed.
That means early today we got all patches landed and I was able to release Mozmill 1.5.12 to PyPI and even the uploaded extension on addons.mozilla.org has been approved in the meantime. If you haven’t updated yet and you make use of Nightly builds, you should do it now!
Lets cross the fingers that no other unexpected issues will arise and force another 1.5 branch release.
Having a good experience when testing Firefox Nightly builds should be self-evident. Therefor the QA Automation Services team is maintaining the Nightly Tester Tools extension, which brings a couple of useful features to ease the testing efforts.
While the extension works fine most of the time, we unfortunately fail at times when a version bump for Firefox happens. Exactly this action repeats in a 6 week interval and is the release cycle of Firefox.
So what happens here? The extension registers binary components for the crashme feature in the manifest file and that’s why it is not handled as a ‘compatible by default’ extension by the add-ons manager. So right after a version bump the extension gets marked as incompatible until the compatibility information on addons.mozilla.org has been updated. This results in an unavailability of the extension in Nightly builds for a couple of hours or even a day, depending on when one of us has time to actually do the version bump. That had to be changed, and thankfully Szabolcs Hubai has spent some time on it to fix the problem and to make our extension ‘compatible by default’. From now on you will never see the Nightly Tester Tools disabled after a version bump of Firefox, which is great. I hope you all will enjoy it!
With the 3.3 release we also ship some other fixes which you can read about in the changelog.
You want to help in the development of the extension? Feel free to contact us as usual via MozIRC or our mailing list.