On Friday Mozmill 1.5.4 has been finally released, including a lot of bug fixes and a couple of new features. This time it has been taken a bit longer as usual to get the release out, but the A-team is also working kinda hard on the Mozmill 2 code, where we expect a preview release kinda soon.
So thanks to everyone for their help in letting us get the following stuff out:
- Add support for Linux3 kernels (Bug 664564)
- Disable any preference for caching XUL elements (Bug 661588)
- The private property .documentLoaded has been renamed to .mozmillDocumentLoaded and a public API has been added to avoid namespace conflicts on window objects (Bug 661408)
- Don’t install distributed extensions from the application folder per default (Bug 661008)
- Add support for iFrames to the waitForPageLoad() method (Bug 659000)
- Expose full stack frames again after the Error properties are not enumerable anymore (Bug 650646)
- Add Mozmill version to the report document (Bug 636746)
If you haven’t upgraded yet, please do so by running: pip install --upgrade mozmill
This command will also upgrade all dependent tools necessary by Mozmill. For questions just drop by in our #mozmill channel on IRC.
Tags: automation, firefox, mozmill, mozqa, testing
As part of the discussions during our QA Automation Services work week end of July, we have decided to be more open to the public. We have seen that in the past mostly all conversations regarding tools, tasks, and status updates happened on our internal mailing list, which is not accessible by any of our contributors. Well, that’s counter-productive because we have dozens of smart and awesome community members out there who probably want to help and learn the techniques in automated testing. And we should give those the possibility to easily participate.
As result we have raised bug 676224 two weeks ago with the request to create a public facing mailing list and newsgroup. The creation has been a bit delayed but finally happened last Friday. So if you want to participate in automation, feel free to join by subscribing yourself to one of the following communication channels:
- Mailing List: https://lists.mozilla.org/listinfo/dev-automation
- Newsgroup: news://news.mozilla.org/mozilla.dev.automation
- Google Groups: http://groups.google.com/group/mozilla.dev.automation
We are looking forward to being able to discuss automation topics with you in a more collaborated fashion as before.
Tags: automation, mozilla, mozqa, testing
For a long time, exactly since January 29th in 2010, the Mozilla QA team is running daily Mozmill tests to prove the functionality of Firefox and the update system across all supported platforms and release branches. With those tests we were able to find a couple of regressions, which then were fixed before the next release of Firefox.
Given those results we have decided to extend our daily test-runs and not only run the functional and update tests, but also cover endurance tests, add-on tests, and the new remote tests.
Especially with the endurance tests, Dave Hunt has shown that we can retrieve a lot of useful memory related information and find regressions or bad interactions with add-ons very easily. So it makes a lot of sense to additionally perform those tests on Nightly and Aurora builds of Firefox. The test results can be found on our dashboard.
But also for add-on authors who have written Mozmill tests for their add-ons, we now provide test results. Regressions in the add-on or in Firefox itself can now be identified as close as possible. Currently only 2 add-ons have Mozmill tests and are being tested. If you are an add-on author get in contact with us, and we can work together to get your add-on tested.
Remote tests, which we haven’t talked about yet, are tests which connect the testing of the Firefox UI and remote websites. Those are fairly new, means they have been added recently, and currently only cover tests for the ‘Get Add-ons’ pane of the new Add-ons Manager. Those were necessary to create because Selenium can not automate the installation of add-ons. For the latest test results see our dashboard.
We are now looking forward to any regression we can detect with the newly added test-runs. And those we can not only identify by our daily test-runs, but also from reports sent by users of the Mozmill Crowd add-on. So please help us testing Firefox by sending your test reports. Thanks!
QA Automation Services is happy to announce that Mozmill Crowd 0.1.3 has been released. With this minor release Mozmill Crowd now got its own branding for the extension itself, and on AMO. Thanks goes to Tara Shahian and her team for their support in creating the icon. We think it looks brilliant. What do you think?
With the hiding of the menu bar on Windows we also had to introduce some more launch points for the extension. Now you can not only reach it via the Tools menu, but also in the Firefox button or the Add-on bar.
Beside the changes for the extension we have also updated the Mozmill test environment. If you are already an user of Mozmill Crowd, please open the chosen storage folder and delete the mozmill-*.zip and mozmill-env folder. With the next test-run Mozmill Crowd will automatically download and install the new version. With the upcoming Mozmill Crowd 0.2 this process will be fully automatic then. But as long as it is not available please run those steps manually. Thanks.
Tags: automation, crowd sourcing, firefox, mozilla, mozmill, mozqa, testing
As blogged by Release Engineering last week the Firefox 3.5 builds have been stopped. That means there are no more daily nightly builds available Mozilla QA will have to test with Mozmill. According to this action we also have turned off our daily test-runs for the latest Firefox 3.5.x builds on all platforms. At the same time the mozilla-1.9.1 branch of our mozmill-tests repository has been closed and as a good result, test failures which couldn’t be fixes due to limitations in Mozmill don’t exist anymore.
Today the builds for the new rapid release cycle have been pushed. With this change we have to update some of our automation tools and the way how we operate with the mozmill-tests repository. The following changes
coming into effect by now:
- We have two new named branches:
- aurora for builds on the Aurora channel
- beta for builds on the Beta channel
- From now on new Mozmill tests will be first checked into the aurora branch. The reason is because we want to have our new tests stable, and there will be too many changes happening for Nightly builds which could cause our tests to fail. Whenever a merge for Firefox happens we will do the same for our new tests. That means we merge all changes from aurora to default. That way we can ensure to test all new features from the last Firefox release in Nightly builds on a daily basis. The same strategy will apply to the beta channel, which we will merge all the tests into once a merge for Firefox happened.
- There will be situations which will cause us to check-in new tests on default first, but those should be really rarely and we will handle it on a case-by-case basis.
That’s why please take care that you always work on the aurora branch when you create new Mozmill tests.
Any questions or proposals? Feel free to ask.
Tags: automation, firefox, mozilla, mozmill, testing
The Mozilla QA Automation team proudly presents the next version of the MozMill Crowd extension which has been released today.
With Mozmill Crowd 0.1.2 we now support the Endurance Tests, which have been used a lot in the last couple of weeks to find memory leaks in Firefox. Now as part of the crowd extension, those tests are as easy to execute as all the other supported test-runs. Only if you would like to run those tests with a given set of add-ons installed, you will have to run the tests outside of the extension. Dave Hunt wrote an excellent article about this topic recently.
Further we were also able to remove the dependency for the Python headers on Linux. That means there is no pre-requisite anymore which has to be fulfilled before Mozmill Crowd will work. Only on OS X you will have to install the Developer Tools because VirtualEnv requires it to work. We hope to fix that by not using VirtualEnv in one of the upcoming releases.
A complete list of fixed bugs you can find on the tracking bug for the Mozmill Crowd 0.1.2 release.
Important Note: The extension is still not able to detect if an updated test environment has been made available. If you already have used MozMill Crowd before make sure that you manually delete the ‘mozmill-env.zip’ file and ‘mozmill-env’ folder in the selected storage folder. Triggering a new test-run will automatically download and install the newest version of the environment.
Also please check out our tomorrows testday which is completely devoted to the MozMill Crowd extension and the test execution in Firefox 4.
Tags: automation, crowd, firefox, mozilla, mozmill, mozqa, testing
Since I have been started to work on Mozmill tests for Firefox early 2009, our QA Automation Team has been grown and got a lot of work done. A fair amount of functional tests have been created, followed by a separate test-run of automated software update tests. Those two kinds of tests have been proven to reduce the time for release testing for QA drastically, and as result we have added even more test-runs like for add-ons, l10n, and lately for endurance tests.
As more and more different runs are getting added the structure of the whole repository, we have agreed on in 2009, doesn’t make sense anymore. Also looking ahead to our upcoming API refactoring which will land soon, we had to chose a new structure of the whole repository. A discussion was necessary to make sure the new hierarchy will fit our needs for the next couple of months or even years. And I can say for sure that we will add a lot more of test-runs and those need the right place to live.
To get this step accomplished, I worked on top of the specs and got the new structure implemented today. If you are interested in check bug 637306. So without reading through the bug you may ask, what has been changed? Lets compare in detail the structure before and after my work.
Before:
- addons (Tests for add-ons)
- firefox (Folder of mixed tests for Firefox)
- crashTests (Crash tests)
- enduranceTests (Endurance tests)
- helperClasses (Tests for shared code)
- l10nTests (l10n tests)
- restartTests (Functional restart tests)
- softwareUpdate (Update tests)
- test-files (Test data served via httpd.js)
- test* (Folders which contain functional tests)
- shared-modules (Shared code divided into components)
- templates (Simple templates for starters)
After:
- data (Test data served via httpd.js)
- lib (Shared code divided into components)
- tests (Tests for shared code)
- tests (Folder for the supported test-runs)
- addons (Tests for add-ons)
- crash (Crash tests)
- endurance (Endurance tests)
- functional (Functional tests for Firefox)
- restartTests (Functional restart tests)
- l10n (l10n tests)
- update (Update tests)
- templates (Simple templates for starters)
As you can see sections are clearly defined and it’s easy to find the code you are looking for. It’s the perfect base for our future work, you will hear about in the next couple of weeks.
Tags: automation, firefox, mozmill, mozqa, testing
As Clint Talbert has already written on Friday last week, Mozmill 1.5.2 has finally been released. A lot of great new stuff is now part of Mozmill core and we are happy to use it in our tests and automation scripts in the next couple of weeks.
The following changes will have to be made:
- With the nearly complete refactoring of the Mozmill Python modules our Mozmill wrapper used by the automation scripts is no longer necessary and needs to be removed. For me it means I have to complete rewrite the automation scripts. With those changes implemented we will no longer be backward compatible to former Mozmill releases.
- Mozmill can now run a list of specified tests from the command line. We want to use that feature to allow users of the Mozmill Crowd extension to run selected tests in a given order. That means that the automation scripts will have to support this feature, and I will take care about in the rewrite.
- We have disabled the compatibility check for add-ons in Firefox 4. That was necessary because lot of add-ons aren’t compatible yet, and weren’t correctly installed from the command line when specified via the –addons option. Now we also want to support it in our automation scripts.
- Failures for fatal and non-fatal assertions are now using the same structure, which is important for our Mozmill Dashboard and especially for the l10n testrun results which mostly contain results of non-fatal assertions.
I’m looking forward to see all the mentioned enhancements active in March. Those are a big step forward, also for our users of the Mozmill Crowd extension.
Tags: automation, firefox, mozmill, QA, testing
A week ago I had the pleasure to attend the FOSDEM (Free and Open source Software Developers’ European Meeting) in Brussels. As every year Mozilla has been setup its own booth and also got assigned an own developer room for talks. I have used this opportunity to spread the word about our crowd-testing initiative with Mozmill Crowd. Thanks to the committee my talk also got approved and I was able to demonstrate our latest activities in-front of a couple of people. If you weren’t able to attend, you can check my slides below or on Slideshare:
I have to thank everyone for the amazing feedback. A lot ideas have been collected during those 2 days, which now need to be injected into our roadmap. One of the most important items is definitely the way how we offer a testing solution for add-ons. Because of the Jetpack team already offers unit tests via the Addon SDK framework, we should work together to get more add-on authors to create tests for their add-ons, but also to being able to collect more test results from our testing community. It’s absolutely an item for me after Firefox 4 has been released and we have finished our refactoring of the shared modules used by our Mozmill tests. So assume updates by next quarter.
Beside my own activities I was able to visit some other booth and finally got in contact with Bernhard Wiedemann from the openSUSE project, who now also runs our Mozmill tests on a daily basis on their machines. I’m kinda inspired by some of the features they are using, especially the creation of a video for the complete test-run. Features like those would have been already helped us a lot. It’s nothing more frustrating to not being able to reproduce an issue because of lack of information.
A big thanks goes also to the planning team of Mozilla who took care of anything related over this weekend. Thanks for the accommodation, the dinner and bowling party on Saturday and all the other small but necessary stuff. As always it was great to get in contact with our and external communities.
Oh, and some pictures have been uploaded to Flick.
Tags: automation, crowd, event, firefox, fosdem, mozilla, mozmill, mozqa, testing
Recent Comments