Tag Archives: mozqa

Firefox Automation report – week 51/52 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 51 and 52 of 2014. I’m sorry for this very late post but changes to our team, which I will get to in my next upcoming post, caught me up with lots of more work and didn’t give me the time for writing status reports.

Highlights

Henrik started work towards a Mozmill 2.1 release. Therefore he had to upgrade a couple of mozbase packages first to get latest Mozmill code on master working again. Once done the patch for handling parent sections in manifest files finally landed, which was originally written by Andrei Eftimie and was sitting around for a while. That addition allows us to use mozhttpd for serving test data via a local HTTP server. Last but not least another important feature went in, which let us better handle application disconnects. There are still some more bugs to fix before we can actually release version 2.1 of Mozmill.

Given that we only have the capacity to fix the most important issues for the Mozmill test framework, Henrik started to mass close existing bugs for Mozmill. So only a hand-full of bugs will remain open. If there is something important you want to see fixed, we would encourage you to start working on the appropriate bug.

For Mozmill CI we got the new Ubuntu 14.10 boxes up and running in our staging environment. Once we can be sure they are stable enough, they will also be enabled in production.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 51 and week 52.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meeting of week 51 and week 52.

Firefox Automation report – week 49/50 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 49 and 50.

Highlights

During the first week of December the all-hands work week happened in Portland. Those were some great and inspiring days, full of talks, discussions, and conversations about various things. Given that I do not see my colleagues that often in real life, I have taken this opportunity to talk to everyone who is partly or fully involved in projects of our automation team. There are various big goals in front of us, so clearing questions and finding the next steps to tackle ongoing problems was really important. Finally we came out with a long list of todo items and more clarity about so far unclear tasks.

In week 50 we got some updates landed for Mozmill CI. Given a regression from the blacklist landing, our l10n tests haven’t been executed for any locale of the Firefox Developer Edition. Since the fix landed, we have seen problems with access keys in nearly each locale for a new test, which covers the context menu of web content.

Also we would like to welcome Barbara Miller in our team. She joined us as an intern via the FOSS outreach program as driven by Gnome. She will be with us until March and will mainly work on testdaybot and the conversion of Mozmill tests to Marionette. The latter project is called m21s and details can be found on its project page. Soon I will post more details about it.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 49 and week 50.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meeting of week 48. Due to the Mozilla all-hands workweek there was no meeting in week 49.

Memchaser 0.6 has been released

The Firefox Automation team would like to announce the release of memchaser 0.6. After nearly a year of no real feature updates, but also some weeks of not being able to run Memchaser in Firefox Aurora (34.0a2) at all (due to a regression), we decided to release the current state of development as a public release. We are aware that we still do not fully support the default Australis theme since Firefox 29.0, but that’s an issue, which takes some more time to finish up.

Changes in 0.6

  • Upgrade to Add-on SDK 1.17 (#201)
  • Fix test_start_stop_logging for ‘File not found’ error (#199)
  • contentURL expects a string in Panel() and Widget() constructors (#198)
  • Bring the minimizeMemory() implementation up to date (#193)
  • Use ‘residentFast’ instead of ‘resident’ for memory reporter (#192)
  • Use postMessage for widget and panel communication instead port object (#185)
  • Support incremental cycle collection statistics (#187)
  • Added a usage section to the README (#178)
  • Change require statements and update the SDK to 1.14 (#174)
  • Simplify .travis.yml (#172)
  • Use . (dot) to include files while invoking /bin/sh (#169)
  • Use failonerror attribute in exec tasks (#168)

For all the details about the 0.6 release please check our issue tracker on Github.

mozdownload 1.12 has been released

The Firefox Automation team would like to announce the release of mozdownload 1.12. Without any other release of our universal download tool in the last 7 months, a couple of nice new features and bug fixes will make this release even more useful. You can upgrade your installation easily via pip, or by downloading it from PyPI.

Changes in 1.12

  • Display selected build when downloading (#149)
  • Add support for downloading B2G desktop builds (#104)
  • Download candidate builds from candidates/ and not nightly/ (#218)
  • Add Travis CI build status and PyPI version badges to README (#220)
  • Add Python 2.6 to test matrix (#210)
  • Fix broken download of mac64 tinderbox builds (#144)
  • Allow download even if content-length header is missing (#194)
  • Convert run_tests script to Python (#168)
  • Ensure that –date option is a valid date (#196)

Mozmill 2.0.7 and 2.0.8 have been released

The Firefox Automation team would like to announce the release of Mozmill 2.0.7 and Mozmill 2.0.8. Both versions had to be released in such a short time frame to ensure continuing support for Firefox. Some latest changes done for Firefox Nightly broke Mozmill, or at least made it misbehaving. If you run tests with Mozmill ensure to upgrade to the latest version. You can do this via PyPI, or simply download the already pre-configured environment.

Changes in 2.0.7

  • Bug 1066295 – testMultipleLoads.js is failing due to new HTTP Cache v2
  • Bug 1000098 – Fix testPageLoad.js test for invalid cert page
  • Bug 1065436 – Disable e10s until full support landed
  • Bug 1062773 – Disconnect errors invalidate the report
  • Bug 999393 – Expose assert and expect by default in sub modules
  • Bug 970820 – Mozmill aborts with socket.timeout when trying to send the report

Changes in 2.0.8

  • Bug 1067939 – JSBridge and Mozmill broken due to ‘let’ changes in bug 1001090

Please keep in mind that Mozmill 2.0 does not support electrolysis (e10s) builds of Firefox yet. We are working hard to get full support for e10s added, and hope it will be done until the next version bump mid of October.

Thanks everyone who was helping with those releases!

Firefox Automation report – week 29/30 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 29 and 30.

Highlights

During week 29 it was time again to merge the mozmill-tests branches to support the upcoming release of Firefox 31.0. All necessary work has been handled on bug 1036881, which also included the creation of the new esr31 branch. Accordingly we also had to update our mozmill-ci system, and got the support landed on production.

The RelEng team asked us if we could help in setting up Mozmill update tests for testing the new update server aka Balrog. Henrik investigated the necessary tasks, and implemented the override-update-url feature in our tests and the mozmill-automation update script. Finally he was able to release mozmill-automation 2.6.0.2 two hours before heading out for 2 weeks of vacation. That means Mozmill CI could be used to test updates for the new update server.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 29 and week 30.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 29 and week 30.

Firefox Automation report – week 27/28 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 27 and 28.

Highlights

Henrik continued his work on our QA specific PuppetAgain setup. One of the blockers for us was bug 997721, which is the full proxy support on OS X and Linux. By week 27 we were finally able to get this finished. Further Henrik also got the manifest for installing Java done.

On TPS we also made progress. So Cosmin got the Pulse listener script implemented for the Coversheet CI, which triggers TPS tests whenever new nightly builds of Firefox have been made available. Further a couple of fixes for Mozrunner were necessary given that the 6.0 release caused a couple of regressions for TPS. As result we agreed on to pin Python package dependencies to specific versions of mozrunner and related packages.

One big thing for our team is also to assist people in the decision, if automated tests are possible for certain Firefox features. The questions mainly come up for tests, which cannot be implemented for any of our developer driven test frameworks due to limitations on buildbot (no network access allowed, restart of the application, and others…). To be more successful in the future, Henrik started a discussion on the dev-quality mailing list. We hope to get the proposed process established for bug verification.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 27 and week 28.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 27 and week 28.

Firefox Automation report – week 1/2 2014

I promised to keep up with our updates over the last week but given a major breakage in the freshly released version of Mozmill 2.0.4, I had a full week of work to get the fix out. I promise that during this week I will write reports for the weeks in January.

Highlights

With the new year our team has been reorganized and we are part of the Mozilla QA team again. That means we will have a way closer relationship to any feature owner, and also working towards in bringing more automation knowledge to everyone. The goals for our team are getting worked out and I will present those in one of my following blog posts. As of now you can find our team page on the Mozilla wiki under Firefox Automation.

Since the landing of all the new features for Mozmill-CI on our staging machine before Christmas, we have no longer experienced any crash of the Jenkins master. Given that Henrik pushed all the changes to our production system. We are totally happy that the incremental updates made our system that stable, and that Mozilla QA doesn’t have cope with outages.

Henrik and Jarek were both working on the mozfile 1.1 release to make it more stable in terms of removing files when those are still in use or don’t have the right permissions set.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 1 and week 2.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the first Firefox Automation meeting of week 2.

Results of our first “Ask an Expert” automation session

It’s was clearly a wow! The way how our first “Ask an Expert” session went was promising. I haven’t thought that we would be able to fill a complete hour with discussions right away – but finally we ended up with additional 15 minutes to get the next steps set for Mike Kaply’s last question how to run automated tests for autoconf.

So what is that amazing on such a session? Given that in the past we as team were mostly working on Bugzilla to get a plan how to solve a specific problem, a live discussion is indeed a lot more helpful. Every attendee can directly participate, ask if there are unclear steps or if there are better ways to accomplish the wanted goal. The discussion is ongoing until you really met your expectation, or the time is up. That’s the difference to threaded conversations via email, where you should expect delays in the turnaround. You will lose valuable time which is even harder when you already have a strong timeline to finish up a feature.

The Automation Development team strongly believes that we all can work together to improve the Test Automation area at Mozilla even more. To achieve that goal we will continue in having live discussions each week and to help others in increasing their skills in automation.

And as a hint for all MozCamp attendees: If you are around keep an eye out for our “Make Sure Your Code Works” hacking session on Saturday at 3:05pm. I will post details about it before MozCamp but just as a heads-up so you can mark it in your calendar.

New ‘in-qa-testsuite’ flag available for Mozilla QA driven test frameworks

Given the importance of automation in QA the Automation Development team had a discussion lately how to improve tracking of QA covered automated testcases in Bugzilla. In the past we haven’t had a way to mark code in bugs for new features or regression fixes as being covered by Mozmill automation.

As we have agreed on it is important to scale our visibility and make it easier for both developers and QA to determine if a feature has automated tests, whether in the tree, in the Mozmill tests repository, or any other framework we would have to create in the future. A whiteboard entry didn’t make much sense for us, and to better align with the existent ‘in-testsuite’ flag, we decided on the ‘in-qa-testsuite‘ flag.

So if you think that a feature needs coverage by Mozmill, make sure to set this flag and assign myself (use “:whimboo” as name). I will then ensure the request will be handled appropriately and the test gets implemented.

Further documentation on automation frameworks and this flag will be available soon. I will blog about once it is ready.