In this post you can find an overview about the work happened in the Firefox Automation team during week 49 and 50.
During the first week of December the all-hands work week happened in Portland. Those were some great and inspiring days, full of talks, discussions, and conversations about various things. Given that I do not see my colleagues that often in real life, I have taken this opportunity to talk to everyone who is partly or fully involved in projects of our automation team. There are various big goals in front of us, so clearing questions and finding the next steps to tackle ongoing problems was really important. Finally we came out with a long list of todo items and more clarity about so far unclear tasks.
In week 50 we got some updates landed for Mozmill CI. Given a regression from the blacklist landing, our l10n tests haven’t been executed for any locale of the Firefox Developer Edition. Since the fix landed, we have seen problems with access keys in nearly each locale for a new test, which covers the context menu of web content.
Also we would like to welcome Barbara Miller in our team. She joined us as an intern via the FOSS outreach program as driven by Gnome. She will be with us until March and will mainly work on testdaybot and the conversion of Mozmill tests to Marionette. The latter project is called m21s and details can be found on its project page. Soon I will post more details about it.
For more granular updates of each individual team member please visit our weekly team etherpad for week 49 and week 50.
If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meeting of week 48. Due to the Mozilla all-hands workweek there was no meeting in week 49.
The Firefox Automation team would like to announce the release of memchaser 0.6. After nearly a year of no real feature updates, but also some weeks of not being able to run Memchaser in Firefox Aurora (34.0a2) at all (due to a regression), we decided to release the current state of development as a public release. We are aware that we still do not fully support the default Australis theme since Firefox 29.0, but that’s an issue, which takes some more time to finish up.
Changes in 0.6
- Upgrade to Add-on SDK 1.17 (#201)
- Fix test_start_stop_logging for ‘File not found’ error (#199)
- contentURL expects a string in Panel() and Widget() constructors (#198)
- Bring the minimizeMemory() implementation up to date (#193)
- Use ‘residentFast’ instead of ‘resident’ for memory reporter (#192)
- Use postMessage for widget and panel communication instead port object (#185)
- Support incremental cycle collection statistics (#187)
- Added a usage section to the README (#178)
- Change require statements and update the SDK to 1.14 (#174)
- Simplify .travis.yml (#172)
- Use . (dot) to include files while invoking /bin/sh (#169)
- Use failonerror attribute in exec tasks (#168)
For all the details about the 0.6 release please check our issue tracker on Github.
The Firefox Automation team would like to announce the release of mozdownload 1.12. Without any other release of our universal download tool in the last 7 months, a couple of nice new features and bug fixes will make this release even more useful. You can upgrade your installation easily via pip, or by downloading it from PyPI.
Changes in 1.12
- Display selected build when downloading (#149)
- Add support for downloading B2G desktop builds (#104)
- Download candidate builds from candidates/ and not nightly/ (#218)
- Add Travis CI build status and PyPI version badges to README (#220)
- Add Python 2.6 to test matrix (#210)
- Fix broken download of mac64 tinderbox builds (#144)
- Allow download even if content-length header is missing (#194)
- Convert run_tests script to Python (#168)
- Ensure that –date option is a valid date (#196)
The Firefox Automation team would like to announce the release of Mozmill 2.0.7 and Mozmill 2.0.8. Both versions had to be released in such a short time frame to ensure continuing support for Firefox. Some latest changes done for Firefox Nightly broke Mozmill, or at least made it misbehaving. If you run tests with Mozmill ensure to upgrade to the latest version. You can do this via PyPI, or simply download the already pre-configured environment.
Changes in 2.0.7
- Bug 1066295 – testMultipleLoads.js is failing due to new HTTP Cache v2
- Bug 1000098 – Fix testPageLoad.js test for invalid cert page
- Bug 1065436 – Disable e10s until full support landed
- Bug 1062773 – Disconnect errors invalidate the report
- Bug 999393 – Expose assert and expect by default in sub modules
- Bug 970820 – Mozmill aborts with socket.timeout when trying to send the report
Changes in 2.0.8
- Bug 1067939 – JSBridge and Mozmill broken due to ‘let’ changes in bug 1001090
Please keep in mind that Mozmill 2.0 does not support electrolysis (e10s) builds of Firefox yet. We are working hard to get full support for e10s added, and hope it will be done until the next version bump mid of October.
Thanks everyone who was helping with those releases!
In this post you can find an overview about the work happened in the Firefox Automation team during week 29 and 30.
During week 29 it was time again to merge the mozmill-tests branches to support the upcoming release of Firefox 31.0. All necessary work has been handled on bug 1036881, which also included the creation of the new esr31 branch. Accordingly we also had to update our mozmill-ci system, and got the support landed on production.
The RelEng team asked us if we could help in setting up Mozmill update tests for testing the new update server aka Balrog. Henrik investigated the necessary tasks, and implemented the override-update-url feature in our tests and the mozmill-automation update script. Finally he was able to release mozmill-automation 126.96.36.199 two hours before heading out for 2 weeks of vacation. That means Mozmill CI could be used to test updates for the new update server.
For more granular updates of each individual team member please visit our weekly team etherpad for week 29 and week 30.
If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 29 and week 30.
In this post you can find an overview about the work happened in the Firefox Automation team during week 27 and 28.
Henrik continued his work on our QA specific PuppetAgain setup. One of the blockers for us was bug 997721, which is the full proxy support on OS X and Linux. By week 27 we were finally able to get this finished. Further Henrik also got the manifest for installing Java done.
On TPS we also made progress. So Cosmin got the Pulse listener script implemented for the Coversheet CI, which triggers TPS tests whenever new nightly builds of Firefox have been made available. Further a couple of fixes for Mozrunner were necessary given that the 6.0 release caused a couple of regressions for TPS. As result we agreed on to pin Python package dependencies to specific versions of mozrunner and related packages.
One big thing for our team is also to assist people in the decision, if automated tests are possible for certain Firefox features. The questions mainly come up for tests, which cannot be implemented for any of our developer driven test frameworks due to limitations on buildbot (no network access allowed, restart of the application, and others…). To be more successful in the future, Henrik started a discussion on the dev-quality mailing list. We hope to get the proposed process established for bug verification.
For more granular updates of each individual team member please visit our weekly team etherpad for week 27 and week 28.
If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 27 and week 28.
I promised to keep up with our updates over the last week but given a major breakage in the freshly released version of Mozmill 2.0.4, I had a full week of work to get the fix out. I promise that during this week I will write reports for the weeks in January.
With the new year our team has been reorganized and we are part of the Mozilla QA team again. That means we will have a way closer relationship to any feature owner, and also working towards in bringing more automation knowledge to everyone. The goals for our team are getting worked out and I will present those in one of my following blog posts. As of now you can find our team page on the Mozilla wiki under Firefox Automation.
Since the landing of all the new features for Mozmill-CI on our staging machine before Christmas, we have no longer experienced any crash of the Jenkins master. Given that Henrik pushed all the changes to our production system. We are totally happy that the incremental updates made our system that stable, and that Mozilla QA doesn’t have cope with outages.
Henrik and Jarek were both working on the mozfile 1.1 release to make it more stable in terms of removing files when those are still in use or don’t have the right permissions set.
For more granular updates of each individual team member please visit our weekly team etherpad for week 1 and week 2.
If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the first Firefox Automation meeting of week 2.
It’s was clearly a wow! The way how our first “Ask an Expert” session went was promising. I haven’t thought that we would be able to fill a complete hour with discussions right away – but finally we ended up with additional 15 minutes to get the next steps set for Mike Kaply’s last question how to run automated tests for autoconf.
So what is that amazing on such a session? Given that in the past we as team were mostly working on Bugzilla to get a plan how to solve a specific problem, a live discussion is indeed a lot more helpful. Every attendee can directly participate, ask if there are unclear steps or if there are better ways to accomplish the wanted goal. The discussion is ongoing until you really met your expectation, or the time is up. That’s the difference to threaded conversations via email, where you should expect delays in the turnaround. You will lose valuable time which is even harder when you already have a strong timeline to finish up a feature.
The Automation Development team strongly believes that we all can work together to improve the Test Automation area at Mozilla even more. To achieve that goal we will continue in having live discussions each week and to help others in increasing their skills in automation.
And as a hint for all MozCamp attendees: If you are around keep an eye out for our “Make Sure Your Code Works” hacking session on Saturday at 3:05pm. I will post details about it before MozCamp but just as a heads-up so you can mark it in your calendar.
Given the importance of automation in QA the Automation Development team had a discussion lately how to improve tracking of QA covered automated testcases in Bugzilla. In the past we haven’t had a way to mark code in bugs for new features or regression fixes as being covered by Mozmill automation.
As we have agreed on it is important to scale our visibility and make it easier for both developers and QA to determine if a feature has automated tests, whether in the tree, in the Mozmill tests repository, or any other framework we would have to create in the future. A whiteboard entry didn’t make much sense for us, and to better align with the existent ‘in-testsuite’ flag, we decided on the ‘in-qa-testsuite‘ flag.
So if you think that a feature needs coverage by Mozmill, make sure to set this flag and assign myself (use “:whimboo” as name). I will then ensure the request will be handled appropriately and the test gets implemented.
Further documentation on automation frameworks and this flag will be available soon. I will blog about once it is ready.
Some days ago we noticed that with the Mozmill 1.5.12 release we accidentally dropped the support for Firefox 3.6 or the Gecko 1.9.2 platform. It has been introduced with the fix for the global per compartment issue on bug 751424, especially with the addition of the ‘outer-window-destroyed’ observer notification listener.
Given the EOL of Firefox 3.6 and the Gecko 3.6 platform we do not intent to push another hotfix release for Mozmill 1.5 at this time. If you have to test applications which are based on Gecko 1.9.2 you should use Mozmill 1.5.11.
If you feel that this doesn’t solve your problem and we should re-add support for Firefox 3.6, please get in contact with us and together we can figure out what will be the best strategy for you.