Tag Archives: mozqa

Firefox Automation report – week 27/28 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 27 and 28.

Highlights

Henrik continued his work on our QA specific PuppetAgain setup. One of the blockers for us was bug 997721, which is the full proxy support on OS X and Linux. By week 27 we were finally able to get this finished. Further Henrik also got the manifest for installing Java done.

On TPS we also made progress. So Cosmin got the Pulse listener script implemented for the Coversheet CI, which triggers TPS tests whenever new nightly builds of Firefox have been made available. Further a couple of fixes for Mozrunner were necessary given that the 6.0 release caused a couple of regressions for TPS. As result we agreed on to pin Python package dependencies to specific versions of mozrunner and related packages.

One big thing for our team is also to assist people in the decision, if automated tests are possible for certain Firefox features. The questions mainly come up for tests, which cannot be implemented for any of our developer driven test frameworks due to limitations on buildbot (no network access allowed, restart of the application, and others…). To be more successful in the future, Henrik started a discussion on the dev-quality mailing list. We hope to get the proposed process established for bug verification.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 27 and week 28.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 27 and week 28.

Firefox Automation report – week 1/2 2014

I promised to keep up with our updates over the last week but given a major breakage in the freshly released version of Mozmill 2.0.4, I had a full week of work to get the fix out. I promise that during this week I will write reports for the weeks in January.

Highlights

With the new year our team has been reorganized and we are part of the Mozilla QA team again. That means we will have a way closer relationship to any feature owner, and also working towards in bringing more automation knowledge to everyone. The goals for our team are getting worked out and I will present those in one of my following blog posts. As of now you can find our team page on the Mozilla wiki under Firefox Automation.

Since the landing of all the new features for Mozmill-CI on our staging machine before Christmas, we have no longer experienced any crash of the Jenkins master. Given that Henrik pushed all the changes to our production system. We are totally happy that the incremental updates made our system that stable, and that Mozilla QA doesn’t have cope with outages.

Henrik and Jarek were both working on the mozfile 1.1 release to make it more stable in terms of removing files when those are still in use or don’t have the right permissions set.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 1 and week 2.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the first Firefox Automation meeting of week 2.

Results of our first “Ask an Expert” automation session

It’s was clearly a wow! The way how our first “Ask an Expert” session went was promising. I haven’t thought that we would be able to fill a complete hour with discussions right away – but finally we ended up with additional 15 minutes to get the next steps set for Mike Kaply’s last question how to run automated tests for autoconf.

So what is that amazing on such a session? Given that in the past we as team were mostly working on Bugzilla to get a plan how to solve a specific problem, a live discussion is indeed a lot more helpful. Every attendee can directly participate, ask if there are unclear steps or if there are better ways to accomplish the wanted goal. The discussion is ongoing until you really met your expectation, or the time is up. That’s the difference to threaded conversations via email, where you should expect delays in the turnaround. You will lose valuable time which is even harder when you already have a strong timeline to finish up a feature.

The Automation Development team strongly believes that we all can work together to improve the Test Automation area at Mozilla even more. To achieve that goal we will continue in having live discussions each week and to help others in increasing their skills in automation.

And as a hint for all MozCamp attendees: If you are around keep an eye out for our “Make Sure Your Code Works” hacking session on Saturday at 3:05pm. I will post details about it before MozCamp but just as a heads-up so you can mark it in your calendar.

New ‘in-qa-testsuite’ flag available for Mozilla QA driven test frameworks

Given the importance of automation in QA the Automation Development team had a discussion lately how to improve tracking of QA covered automated testcases in Bugzilla. In the past we haven’t had a way to mark code in bugs for new features or regression fixes as being covered by Mozmill automation.

As we have agreed on it is important to scale our visibility and make it easier for both developers and QA to determine if a feature has automated tests, whether in the tree, in the Mozmill tests repository, or any other framework we would have to create in the future. A whiteboard entry didn’t make much sense for us, and to better align with the existent ‘in-testsuite’ flag, we decided on the ‘in-qa-testsuite‘ flag.

So if you think that a feature needs coverage by Mozmill, make sure to set this flag and assign myself (use “:whimboo” as name). I will then ensure the request will be handled appropriately and the test gets implemented.

Further documentation on automation frameworks and this flag will be available soon. I will blog about once it is ready.

Mozmill 1.5.12 dropped support for Firefox 3.6

Some days ago we noticed that with the Mozmill 1.5.12 release we accidentally dropped the support for Firefox 3.6 or the Gecko 1.9.2 platform. It has been introduced with the fix for the global per compartment issue on bug 751424, especially with the addition of the ‘outer-window-destroyed’ observer notification listener.

Given the EOL of Firefox 3.6 and the Gecko 3.6 platform we do not intent to push another hotfix release for Mozmill 1.5 at this time. If you have to test applications which are based on Gecko 1.9.2 you should use Mozmill 1.5.11.

If you feel that this doesn’t solve your problem and we should re-add support for Firefox 3.6, please get in contact with us and together we can figure out what will be the best strategy for you.

Automation Development Team and Q2 Goals

Some changes to our team happened lately I want to quickly talk about before starting to dive into our team goals for this quarter. So what’s going on?

Automation is a key part when it comes to qualifying new and existing code. Over the last years it’s getting more and more important at Mozilla for developers and QA to proof the quality of Firefox on a fast pace without the hassle of manual testing. With the change in the Firefox release process to a rapid train model a ton of different builds have to be tested by QA on a regular basis. Doing all that manually is impossible. A test framework which has a high impact for QA is Mozmill and the tests and automation tools written around by the Automation Services team over the last couple of months help a lot. But at the current state it’s not enough. We have to be better and cover way more of the manual testing with automated tests.

To make that possible the Automation Services team is now part of the Automation and Tools Engineering team (aka A-team) and has been renamed to Automation Development. We, which means Dave Hunt and myself, are part of the crew who will be responsible for any kind of automation question. For now we can only cover the Mozmill and Selenium frameworks but with additional team members added hopefully soon, more and more harnesses can be covered. Another key aspect for our crew will be mentorship and education, which means that we are eager to bring you into the automation land and to know all the stuff to write automated tests on your own. Sounds interesting? It definitely is, and I like it a lot!

So what will we work on during the current quarter?

There is a lot of work which has to be done for Mozmill to finally get out version 2.0 and to update our automation scripts and Mozmill tests, including a new API to write the tests. So those will be our prioritized projects.

  1. Mozmill 2.0: A new major version of Mozmill has to be released to fix a lot of our problems we still have on the 1.5 branch, and which can’t be easily solved there. The development is already active for a lot of months but we were never able to finish it up. Given that this new version is a blocker for us to create more reliable Mozmill tests, lets get it hopefully done by this quarter. If you have Python or Javascript skills stop by and let us know. We could need any help we can get.
  2. Automation Scripts: Those scripts drive our Mozmill test-runs for different versions of Firefox. They have been written in the early stages around Mozmill 1.0 and need a complete refactor to work with Mozmill 2.0. Also a couple of features will have to be moved to MozBase so that even other harnesses can benefit from. This is all Python related work.
  3. Test API refactor: In parallel to the above mentioned projects we will also have to update our API, used by Mozmill tests, to make it easier to write new and maintain existing tests. This is absolutely important especially for new members who start with Mozmill. If you have Javascript skills and want to improve them, let us know too.

Beside those projects we will also continue our work in the following areas:

  • Mozmill CI: More and more tests are getting covered by our new Mozmill CI system, which I will talk about more in detail soon. So we have to move this instance to an ESX cluster to have a ton of machines for test execution, include handling of the DTC (default to compatible) test-run, and make sure that the upcoming Firefox Beta Nightly builds are handled too.
  • Mozmill Tests: We take a stab in handling and coordinating the creation of new Mozmill tests and fixing the broken ones. The reason for this change I will also explain in a separate blog post shortly. But to say in general we need more test coverage and skipped/failing tests over a long time are bad practice and counter-productive. So lets get all of them re-enabled again and new tests written.
  • Addons: There will not be that much time for us this quarter to continue developing the add-ons we own. But we will make sure to fix breakage for MemChaser and Nightly Tester Tools if those occur. If you want to help out, you are more then welcome!
  • WebApps Testing: While this is not on our official goals for this quarter we have to assist the WebApps QA team to figure out the right test harnesses to create automated functional tests for open web apps. It will be a challenge and we are kinda interested to see that upcoming.

As you can see there are plenty of projects to work on. Any of those are public and are waiting for your contribution. If you are interested in working with us and to raise your personal skills, please contact us via our public mailing list or #automation IRC channel.

Mozmill 1.5.12 released

Something we have learned over the weekend was: Never say never. After we have released the 1.5.11 release of Mozmill on April 19th, we were sure to not have to ship any other release off this branch. We want to concentrate our work on Mozmill 2.0 and get it out as soon as possible.

But things can change, and sometimes faster as expected. So by last Friday (it was already late in the evening) Mike Conley contacted me on IRC and mentioned that Mozmill will be broken starting with tomorrows build. The reason for it was the ‘Global-Per-Compartment’ patch which has been landed in Nightly builds of Firefox and Thunderbird at that time. Great! Why do things like those happen just before when I’m already in my weekend?

Anyway, given the importance of this bug (it broke the whole test infrastructure of Thunderbird which mostly relies on Mozmill) I was talking immediately to Clint Talbert. Given the good bug report from John Schoenick we were directly aware of what the problem was and agreed on a new API to get the issue fixed.

So I have spent a good portion of my Saturday and Sunday evening to put together a patch which removed the broken code in waitForPageLoad() and replaces it with the new API. So by now we do no longer add a custom ‘mozmillDocumentLoaded’ property to all windows, but have a nice windowMap object that handles all the loaded states of each window on its own. I think that’s great and modifying the DOM of each window was even a bad idea since the beginning of the project. Good to see that killed.

That means early today we got all patches landed and I was able to release Mozmill 1.5.12 to PyPI and even the uploaded extension on addons.mozilla.org has been approved in the meantime. If you haven’t updated yet and you make use of Nightly builds, you should do it now!

Lets cross the fingers that no other unexpected issues will arise and force another 1.5 branch release.

Nightly Tester Tools 3.3 released

Having a good experience when testing Firefox Nightly builds should be self-evident. Therefor the QA Automation Services team is maintaining the Nightly Tester Tools extension, which brings a couple of useful features to ease the testing efforts.

While the extension works fine most of the time, we unfortunately fail at times when a version bump for Firefox happens. Exactly this action repeats in a 6 week interval and is the release cycle of Firefox.

So what happens here? The extension registers binary components for the crashme feature in the manifest file and that’s why it is not handled as a ‘compatible by default’ extension by the add-ons manager. So right after a version bump the extension gets marked as incompatible until the compatibility information on addons.mozilla.org has been updated. This results in an unavailability of the extension in Nightly builds for a couple of hours or even a day, depending on when one of us has time to actually do the version bump. That had to be changed, and thankfully Szabolcs Hubai has spent some time on it to fix the problem and to make our extension ‘compatible by default’. From now on you will never see the Nightly Tester Tools disabled after a version bump of Firefox, which is great. I hope you all will enjoy it!

With the 3.3 release we also ship some other fixes which you can read about in the changelog.

You want to help in the development of the extension? Feel free to contact us as usual via MozIRC or our mailing list.

Review of Automation Services goals in Q1 2012

Now, three weeks after Q1 of 2012 has been passed by I finally have the time to give some details about our work happened during the first three months.

As usual we set a couple of goals we wanted to see by the end of the quarter. All of them were kinda important so we did our best to get all of them done – and we succeeded even if it was in the last minute! So everyone of us was happy about the goal achievements.

Below you will find an overview of each of those projects. For more detailed information I will follow-up with additional blog posts by next week.

Mozmill CI system for release, daily, and l10n builds

It has been taken a while for us to get to a state when we absolutely had to work on a continuous integration solution. Seeing that more and more tests are getting automated with Mozmill and that we still require manual interaction, or crontab jobs and tasks on Windows to get tests executed, we had to find a better way in getting tests run. After a bit of investigation we started by using Jenkins (formerly known as Hudson). And since beginning of this month we now have a system which runs our Mozmill tests for all the supported daily and beta candidate builds, and for each tinderbox build based on l10n check-ins on whichever branch. As trigger we make use of Mozilla Pulse which informs us directly when builds have been made available. Further we also had to ensure that tests can be triggered by on-demand, because update tests cannot be covered by Pulse yet and sometimes we have to re-run tests against a specific build. All in all the system works great but still needs a lot of improvements on which we will work on in the next couple of months.

Automation system for the Add-ons compatibile by default feature

With the change in Firefox 11 to make all add-ons compatible by default, we have been asked to create a system to automatically test add-on compatibility for the top 100 add-ons. As result a new testrun for Mozmill has been created which runs our endurance, update, and endurance tests again. That way we want to make sure that there are no performance regressions involved and updates of Firefox will be applied without any failures.

MemChaser – An extension to visualize memory and GC/CC related activities

Given the amount of manual testing for GC/CC related issues in Firefox by end of 2011, I had the idea to create an extension which makes it easy for users to track memory related activities. That means that all the important information is displayed directly in the add-on bar and don’t have to be retrieved from ‘about:memory‘ or the Error Console (which constantly wipes out old entries). The idea got approved as goal so Dave Hunt, David Guo, and myself were heavily working on getting this extension released on addons.mozilla.org as soon as possible. Meanwhile even more features have been implemented and more are about to come in the next couple of months. If you want to follow the development of the extension check the Github repository.

Mozmill Dashboard – Validation method for uploaded test reports

During a security review of our dashboard by end of 2011 it has been turned out that arbitrary data can be uploaded to our dashboard. The reason was that we haven’t checked the data which gets uploaded before storing it in our database. So we have had to implement a validation method for the CouchDB database. Now the report data gets checked and if the method fails the document will not be stored.

As mentioned before I will follow-up with detailed information later. So please watch out for upcoming blog posts in the next days.

Mozmill 1.5.11 released

Pushing out releases are fun! Especially if new regressions for patches landed a month ago will be identified 15 minutes after the release actually happened. Exactly that fooled us this time, and we even had to pull down Mozmill 1.5.10 from PyPI. As a Mozmill user you will probably have noticed that we skipped the announcement for Mozmill 1.5.10.

As by late yesterday Mozmill 1.5.11 is now available for everyone on PyPI. This version will be the last one we expect to see for the 1.5 branch. Current work will only happen on Mozmill 2.0, so it can be released by end of this quarter. If you are interested in helping us, please let us know and we definitely will find something for you.

As usual I would like to point out new features and bug fixes which are part of version 1.5.11:

  • Bug 686320: JSBridge is not limited to a single port (24242) anymore but probes for a free one now. This allows you to run multiple concurrent testruns with Mozmill at the same time. Keep in mind that the focus manager has to be put into testing mode for this.
  • Bug 696834: Securable modules have not been imported as singletons so that sharing data between modules didn’t work.
  • Bug 709773: The Mozmill API has been updated internally to let customers override the manifest (-m) and tests (-t) settings provided by the command line options.
  • Bug 727660: HTTP proxies are now supported for sending reports to a remote database.
  • Bug 740773: With the update of our Mozmill Dashboard invalid reports will be declined. Mozmill hasn’t shown any failure message yet which was kinda unhelpful in tracking down broken reports.

With the changes as mentioned above we can now go ahead and work on the transition of our Firefox tests to Mozmill 2.0.