The Firefox Automation team would like to announce the release of memchaser 0.6. After nearly a year of no real feature updates, but also some weeks of not being able to run Memchaser in Firefox Aurora (34.0a2) at all (due to a regression), we decided to release the current state of development as a public release. We are aware that we still do not fully support the default Australis theme since Firefox 29.0, but that’s an issue, which takes some more time to finish up.
Changes in 0.6
- Upgrade to Add-on SDK 1.17 (#201)
- Fix test_start_stop_logging for ‘File not found’ error (#199)
- contentURL expects a string in Panel() and Widget() constructors (#198)
- Bring the minimizeMemory() implementation up to date (#193)
- Use ‘residentFast’ instead of ‘resident’ for memory reporter (#192)
- Use postMessage for widget and panel communication instead port object (#185)
- Support incremental cycle collection statistics (#187)
- Added a usage section to the README (#178)
- Change require statements and update the SDK to 1.14 (#174)
- Simplify .travis.yml (#172)
- Use . (dot) to include files while invoking /bin/sh (#169)
- Use failonerror attribute in exec tasks (#168)
For all the details about the 0.6 release please check our issue tracker on Github.
The Firefox Automation team would like to announce the release of mozdownload 1.12. Without any other release of our universal download tool in the last 7 months, a couple of nice new features and bug fixes will make this release even more useful. You can upgrade your installation easily via pip, or by downloading it from PyPI.
Changes in 1.12
- Display selected build when downloading (#149)
- Add support for downloading B2G desktop builds (#104)
- Download candidate builds from candidates/ and not nightly/ (#218)
- Add Travis CI build status and PyPI version badges to README (#220)
- Add Python 2.6 to test matrix (#210)
- Fix broken download of mac64 tinderbox builds (#144)
- Allow download even if content-length header is missing (#194)
- Convert run_tests script to Python (#168)
- Ensure that –date option is a valid date (#196)
The Firefox Automation team would like to announce the release of Mozmill 2.0.7 and Mozmill 2.0.8. Both versions had to be released in such a short time frame to ensure continuing support for Firefox. Some latest changes done for Firefox Nightly broke Mozmill, or at least made it misbehaving. If you run tests with Mozmill ensure to upgrade to the latest version. You can do this via PyPI, or simply download the already pre-configured environment.
Changes in 2.0.7
- Bug 1066295 – testMultipleLoads.js is failing due to new HTTP Cache v2
- Bug 1000098 – Fix testPageLoad.js test for invalid cert page
- Bug 1065436 – Disable e10s until full support landed
- Bug 1062773 – Disconnect errors invalidate the report
- Bug 999393 – Expose assert and expect by default in sub modules
- Bug 970820 – Mozmill aborts with socket.timeout when trying to send the report
Changes in 2.0.8
- Bug 1067939 – JSBridge and Mozmill broken due to ‘let’ changes in bug 1001090
Please keep in mind that Mozmill 2.0 does not support electrolysis (e10s) builds of Firefox yet. We are working hard to get full support for e10s added, and hope it will be done until the next version bump mid of October.
Thanks everyone who was helping with those releases!
In this post you can find an overview about the work happened in the Firefox Automation team during week 29 and 30.
During week 29 it was time again to merge the mozmill-tests branches to support the upcoming release of Firefox 31.0. All necessary work has been handled on bug 1036881, which also included the creation of the new esr31 branch. Accordingly we also had to update our mozmill-ci system, and got the support landed on production.
The RelEng team asked us if we could help in setting up Mozmill update tests for testing the new update server aka Balrog. Henrik investigated the necessary tasks, and implemented the override-update-url feature in our tests and the mozmill-automation update script. Finally he was able to release mozmill-automation 220.127.116.11 two hours before heading out for 2 weeks of vacation. That means Mozmill CI could be used to test updates for the new update server.
For more granular updates of each individual team member please visit our weekly team etherpad for week 29 and week 30.
If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 29 and week 30.
In this post you can find an overview about the work happened in the Firefox Automation team during week 27 and 28.
Henrik continued his work on our QA specific PuppetAgain setup. One of the blockers for us was bug 997721, which is the full proxy support on OS X and Linux. By week 27 we were finally able to get this finished. Further Henrik also got the manifest for installing Java done.
On TPS we also made progress. So Cosmin got the Pulse listener script implemented for the Coversheet CI, which triggers TPS tests whenever new nightly builds of Firefox have been made available. Further a couple of fixes for Mozrunner were necessary given that the 6.0 release caused a couple of regressions for TPS. As result we agreed on to pin Python package dependencies to specific versions of mozrunner and related packages.
One big thing for our team is also to assist people in the decision, if automated tests are possible for certain Firefox features. The questions mainly come up for tests, which cannot be implemented for any of our developer driven test frameworks due to limitations on buildbot (no network access allowed, restart of the application, and others…). To be more successful in the future, Henrik started a discussion on the dev-quality mailing list. We hope to get the proposed process established for bug verification.
For more granular updates of each individual team member please visit our weekly team etherpad for week 27 and week 28.
If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 27 and week 28.
I promised to keep up with our updates over the last week but given a major breakage in the freshly released version of Mozmill 2.0.4, I had a full week of work to get the fix out. I promise that during this week I will write reports for the weeks in January.
With the new year our team has been reorganized and we are part of the Mozilla QA team again. That means we will have a way closer relationship to any feature owner, and also working towards in bringing more automation knowledge to everyone. The goals for our team are getting worked out and I will present those in one of my following blog posts. As of now you can find our team page on the Mozilla wiki under Firefox Automation.
Since the landing of all the new features for Mozmill-CI on our staging machine before Christmas, we have no longer experienced any crash of the Jenkins master. Given that Henrik pushed all the changes to our production system. We are totally happy that the incremental updates made our system that stable, and that Mozilla QA doesn’t have cope with outages.
Henrik and Jarek were both working on the mozfile 1.1 release to make it more stable in terms of removing files when those are still in use or don’t have the right permissions set.
For more granular updates of each individual team member please visit our weekly team etherpad for week 1 and week 2.
If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the first Firefox Automation meeting of week 2.
It’s was clearly a wow! The way how our first “Ask an Expert” session went was promising. I haven’t thought that we would be able to fill a complete hour with discussions right away – but finally we ended up with additional 15 minutes to get the next steps set for Mike Kaply’s last question how to run automated tests for autoconf.
So what is that amazing on such a session? Given that in the past we as team were mostly working on Bugzilla to get a plan how to solve a specific problem, a live discussion is indeed a lot more helpful. Every attendee can directly participate, ask if there are unclear steps or if there are better ways to accomplish the wanted goal. The discussion is ongoing until you really met your expectation, or the time is up. That’s the difference to threaded conversations via email, where you should expect delays in the turnaround. You will lose valuable time which is even harder when you already have a strong timeline to finish up a feature.
The Automation Development team strongly believes that we all can work together to improve the Test Automation area at Mozilla even more. To achieve that goal we will continue in having live discussions each week and to help others in increasing their skills in automation.
And as a hint for all MozCamp attendees: If you are around keep an eye out for our “Make Sure Your Code Works” hacking session on Saturday at 3:05pm. I will post details about it before MozCamp but just as a heads-up so you can mark it in your calendar.
Given the importance of automation in QA the Automation Development team had a discussion lately how to improve tracking of QA covered automated testcases in Bugzilla. In the past we haven’t had a way to mark code in bugs for new features or regression fixes as being covered by Mozmill automation.
As we have agreed on it is important to scale our visibility and make it easier for both developers and QA to determine if a feature has automated tests, whether in the tree, in the Mozmill tests repository, or any other framework we would have to create in the future. A whiteboard entry didn’t make much sense for us, and to better align with the existent ‘in-testsuite’ flag, we decided on the ‘in-qa-testsuite‘ flag.
So if you think that a feature needs coverage by Mozmill, make sure to set this flag and assign myself (use “:whimboo” as name). I will then ensure the request will be handled appropriately and the test gets implemented.
Further documentation on automation frameworks and this flag will be available soon. I will blog about once it is ready.
Some days ago we noticed that with the Mozmill 1.5.12 release we accidentally dropped the support for Firefox 3.6 or the Gecko 1.9.2 platform. It has been introduced with the fix for the global per compartment issue on bug 751424, especially with the addition of the ‘outer-window-destroyed’ observer notification listener.
Given the EOL of Firefox 3.6 and the Gecko 3.6 platform we do not intent to push another hotfix release for Mozmill 1.5 at this time. If you have to test applications which are based on Gecko 1.9.2 you should use Mozmill 1.5.11.
If you feel that this doesn’t solve your problem and we should re-add support for Firefox 3.6, please get in contact with us and together we can figure out what will be the best strategy for you.
Some changes to our team happened lately I want to quickly talk about before starting to dive into our team goals for this quarter. So what’s going on?
Automation is a key part when it comes to qualifying new and existing code. Over the last years it’s getting more and more important at Mozilla for developers and QA to proof the quality of Firefox on a fast pace without the hassle of manual testing. With the change in the Firefox release process to a rapid train model a ton of different builds have to be tested by QA on a regular basis. Doing all that manually is impossible. A test framework which has a high impact for QA is Mozmill and the tests and automation tools written around by the Automation Services team over the last couple of months help a lot. But at the current state it’s not enough. We have to be better and cover way more of the manual testing with automated tests.
To make that possible the Automation Services team is now part of the Automation and Tools Engineering team (aka A-team) and has been renamed to Automation Development. We, which means Dave Hunt and myself, are part of the crew who will be responsible for any kind of automation question. For now we can only cover the Mozmill and Selenium frameworks but with additional team members added hopefully soon, more and more harnesses can be covered. Another key aspect for our crew will be mentorship and education, which means that we are eager to bring you into the automation land and to know all the stuff to write automated tests on your own. Sounds interesting? It definitely is, and I like it a lot!
So what will we work on during the current quarter?
There is a lot of work which has to be done for Mozmill to finally get out version 2.0 and to update our automation scripts and Mozmill tests, including a new API to write the tests. So those will be our prioritized projects.
- Automation Scripts: Those scripts drive our Mozmill test-runs for different versions of Firefox. They have been written in the early stages around Mozmill 1.0 and need a complete refactor to work with Mozmill 2.0. Also a couple of features will have to be moved to MozBase so that even other harnesses can benefit from. This is all Python related work.
Beside those projects we will also continue our work in the following areas:
- Mozmill CI: More and more tests are getting covered by our new Mozmill CI system, which I will talk about more in detail soon. So we have to move this instance to an ESX cluster to have a ton of machines for test execution, include handling of the DTC (default to compatible) test-run, and make sure that the upcoming Firefox Beta Nightly builds are handled too.
- Mozmill Tests: We take a stab in handling and coordinating the creation of new Mozmill tests and fixing the broken ones. The reason for this change I will also explain in a separate blog post shortly. But to say in general we need more test coverage and skipped/failing tests over a long time are bad practice and counter-productive. So lets get all of them re-enabled again and new tests written.
- Addons: There will not be that much time for us this quarter to continue developing the add-ons we own. But we will make sure to fix breakage for MemChaser and Nightly Tester Tools if those occur. If you want to help out, you are more then welcome!
- WebApps Testing: While this is not on our official goals for this quarter we have to assist the WebApps QA team to figure out the right test harnesses to create automated functional tests for open web apps. It will be a challenge and we are kinda interested to see that upcoming.
As you can see there are plenty of projects to work on. Any of those are public and are waiting for your contribution. If you are interested in working with us and to raise your personal skills, please contact us via our public mailing list or #automation IRC channel.