What to do when Firefox crashes under test automation with Selenium

If you have the task to create automated tests for websites you will most likely make use of Selenium when it comes to testing UI interactions. To execute the tests for the various browsers out there each browser vendor offers a so called driver package  which has to be used by Selenium to run each of the commands. In case of Firefox this will be geckodriver.

Within the last months we got a couple of issues reported for geckodriver that Firefox sometimes crashes while the tests are running. This feedback is great, and we always appreciate because it helps us to make Firefox more stable and secure for our users. But to actually being able to fix the crash we would need some more data, which was a bit hard to retrieve in the past.

As first step I worked on the Firefox crash reporter support for geckodriver and we got it enabled in the 0.19.0 release. While this was fine and the crash reporter created minidump files for each of the crashes in the temporarily created user profile for Firefox, this data gets also removed together with the profile once the test has been finished. So copying the data out of the profile was impossible.

As of now I haven’t had the time to improve the user experience here, but I hope to be able to do it soon. The necessary work which already got started will be covered on bug 1433495. Once the patch on that bug has been landed and a new geckodriver version released, the environment variable “MINIDUMP_SAVE_PATH” can be used to specify a target location for the minidump files. Then geckodriver will automatically copy the files to this target folder before the user profile gets removed.

But until that happened a bit of manual work is necessary. Because I had to mention those steps a couple of time and I don’t want to repeat that in the near future again and again, I decided to put up a documentation in how to analyze the crash data, and how to send the data to us. The documentation can be found at:

https://firefox-source-docs.mozilla.org/testing/geckodriver/doc/geckodriver/CrashReports.html

I hope that helps!

Element interactability checks with geckodriver and Firefox 58

When you are using Selenium and geckodriver to automate your tests in Firefox you might see a behavior change with Firefox 58 when using the commands Element Click or Element Send Keys. For both commands we have enabled the interactability checks by default now. That means that if such an operation has to be performed for any kind of element it will be checked first, if a click on it or sending keys to it would work from a normal user perspective at all. If not a not-interactable error will be thrown.

If you are asking now why this change was necessary, the answer is that we are more WebDriver specification conformant now.

While pushing this change out by default, we are aware of corner cases where we accidentally might throw such a not-interactability error, or falsely assume the element is interactable. If you are hitting such a condition it would be fantastic to let us know about it as best by filing an geckodriver issue with all the required information so that it is reproducible for us.

In case the problem causes issues for your test suites, but you totally want to use Firefox 58, you can use the capability moz:webdriverClick and turn off those checks. Simply set it to False, and the former behavior will happen. But please note that this workaround will only work for Firefox 58, and maybe Firefox 59, because then the old and legacy behavior will be removed.

That’s why please let us know about misbehavior when using Firefox 58, so that we have enough time to get it fixed for Firefox 59, or even 58.

Thanks!

Firefox Automation report – Q3 2015

It’s time for another Firefox Automation report! It’s incredible how fast a quarter passes by without that I have time to write reports more often. Hopefully it will change soon – news will be posted in a follow-up blog post.

Ok, so what happened last quarter for our projects.

Mozharness

One of my deliverables in Q3 was to create mozharness scripts for our various tests in the firefox-ui-tests repository, so that our custom runner scripts can be replaced. This gives us a way more stable system and additional features like crash report handling, which are necessary to reach the tier 2 level in Treeherder.

After some refactoring of the firefox ui tests, scripts for the functional and update tests were needed. But before those could be implemented I had to spent some time in refactoring some modules of mozharness to make them better configurable for non-buildbot jobs. All that worked pretty fine and finally the entry scripts have been written. Something special for them is that they even have different customers, so extra configuration files had to be placed. In detail it’s us who run the tests in Jenkins for nightly builds, and partly for release builds. On the other side Release Engineering want to run our update tests on their own hardware when releases have to be tested.

By the end of September all work has been finished. If you are interested in more details feel free to check the tracking bug 1192369.

Mozmill-CI

Our Jenkins instance got lots of updates for various new features and necessary changes. All in all I pushed 27 commits which affected 53 files.

Here a list of the major changes:

  • Refactoring of the test jobs has been started so that those can be used for mozharness driven firefox-ui-tests later in Q4. The work has not been finished and will be continued in Q4. Especially the refactoring for report submission to Treeherder even for aborted builds will be a large change.

  • A lot of time had to be spent in fixing the update tests for all the changes which were coming in with the Funsize project of Release Engineering. Due to missing properties in the Mozilla Pulse messages update tests could no longer be triggered for nightly builds. Therefore the handling of Pulse messages has been completely rewritten to allow the handling of similar Pulse messages as sent out from TaskCluster. That work was actually not planned and has been stolen me quite some time from other projects.

  • A separation of functional and remote tests didn’t make that much sense. Especially because both types are actually functional tests. As result they have been merged together into the functional tests. You can still run remote tests only by using --tag remote; similar for tests with local testcases by using `–tag local.

  • We stopped running tests for mozilla-esr31 builds due to Firefox ESR31 is no longer supported.

  • To lower the amount of machines we have to maintain and to getting closer what’s being run on Buildbot, we stopped running tests on Ubuntu 14.10. Means we only run on Ubuntu LTS releases from now on. Also we stopped tests for OS X 10.8. The nodes will be re-used for OS X 10.11 once released.

  • We experienced Java crashes due to low memory conditions of our Jenkins production master again. This was kinda critical because the server is not getting restarted automatically. After some investigation I assumed that the problem is due to the 32bit architecture of the VM. Given that it has 8GB of memory a 64bit version of Ubuntu should have been better used. So we replaced the machine and so far everything looks fine.

  • Totally surprising we had to release once more a bugfix release of Mozmill. This time the framework didn’t work at all due to the enforcement of add-on signing. So Mozmill 2.0.10.2 has been released.

mozdownload 1.18 released

Today we have released mozdownload 1.18 to PyPI. The reason why I think it’s worth a blog post is that with this version we finally added support for a sane API. With it available using the mozdownload code in your own script is getting much easier. So there is no need to instantiate a specific scraper anymore but a factory scraper is doing all the work depending on the options it gets.

Here some examples:

from mozdownload import FactoryScraper
scraper = FactoryScraper('release', version='40.0.3', locale='de')
scraper.download()
from mozdownload import FactoryScraper
scraper = FactoryScraper('candidate', version='41.0b9', platform='win32')
scraper.download()
from mozdownload import FactoryScraper
scraper = FactoryScraper('daily', branch='mozilla-aurora')
scraper.download()

If you are using mozdownload via its API you can also easily get the remote URL and the local filename:

from mozdownload import FactoryScraper
scraper = FactoryScraper('daily', branch='mozilla-aurora')
print scraper.url
print scraper.filename

Hereby the factory class is smart enough to only select those passed-in options which are appropriate for the type of scraper. If you have to download different types of builds you will enjoy that feature given that only the scraper type has to be changed and all other options could be still passed-in.

We hope that this new feature will help you by integrating mozdownload into your own project. There is no need anymore by using its command line interface through a subprocess call.

The complete changelog can be found here.

Mozmill 2.0 released!

Believe it or don’t believe it. But this announcement is for real! Really, you can trust me! If you still think it’s a joke I can say: even with the expected release of Mozmill 2.0 was like Duke Nukem Forever over the past 3 years, we finally finished all bits and got it released yesterday. Check it out yourself on pypi: http://pypi.python.org/pypi/mozmill.

So the whole process of working on a new release of Mozmill might have been started when we branched for the final 1.5 release, which was on Aug 23rd, 2010. Since then various people spent more or less time on its code to get it improved and lesser buggy. Also given some major refactoring at the beginning of the development of version 2.0, and lot of other feature requests from QA and other teams, we were working mostly double-tracked. Means, we have not only implemented new features for 2.0 but also had to backport those onto the hotfix-1.5 branch. That caused a lot of work for us. And because we weren’t working full-time on that project either, the time went by so fast.

It’s interesting to see that on Oct 10th, 2011, we released an RC1 for Mozmill 2.0 because we thought it would be the right time to get it out soon. But since then an amazing number of additional 276 changesets have been landed, and another nearly 2 years passed by. Amazing how your own schedule can be busted. But there was no way around.

In the last couple of months our Automation Development team has fully taken-over the project, and promised to get it finished and released. It was a kinda tricky and hard task, especially with all the additional bugs we have identified, which not only caused broken tests but also crashes and dataloss. Thanks to the help of our awesome contributors, who spend tons of hours to put a nice and polished version of Mozmill together. We wouldn’t have been here without your help. A big big thanks!

One of the things we are most proud of is the backward compatibility of Mozmill 2.0. We made it that as a Mozmill 1.5 user you shouldn’t see any difference. Any method and property available in Mozmill 1.5 is still present in 2.0, but has been marked as deprecated. With that you will be able to run all of your tests with both versions, and you can transition over step by step. There are indeed some new ways how to work with elements, which are explained on our MDN Mozmill page.

One very impressive improvement we have seen a couple of days ago, is the speed how Mozmill tests are getting executed. Compared to Mozmill 1.5 we can see a drop of the duration by a factor of 2.5! That’s two and a half times faster! I think already because of that you should not miss to at least test Mozmill 2.0!

As usual please give us feedback via email or on IRC about your own experiences and please report any bug you have discovered. If you like hacking and want to help us with the implementation of new features or fixing broken behavior, then have a look at our Mozmill repository on Github.

Soon I will come up with our plan for the future of Mozmill. And what I can say as of now, there is a lot to come…

Mozdownload 1.9 released

Oh my! It happened again. And here it is. Another release of mozdownload with a couple of new features included. As a regular user please let us know if all is working well, or if you want to see some new features included.

Below you can find an overview about the main features in mozdownload 1.9:

  • Tinderbox builds can be downloaded by timestamp now
  • For linux64 the 64-bit and not the 32-bit installer gets downloaded
  • Support for the stub installer on Windows has been added
  • Better error messages are displayed

All details for this release can be seen in the changelog.

Thanks again to everyone who was involved in this release!

Mozmill 1.5.22 and 2.0rc5 released

As of today we can announce a double release of Mozmill versions. While for Mozmill 1.5 it might hopefully be the last update, our final Mozmill 2.0 release is coming really close. The now released RC5 should be the last release candidate.

Mozmill 1.5.22

Another update of Mozmill on our 1.5 branch was necessary given that the Thunderbird folks detected a memory leak (bug 876399) in our module loading code via a sandbox. This is now fixed and different tests no longer share global objects anymore.

Beside that the following improvements are also included:

  • In some cases we failed to correctly select an item from a menulist. As it has been turned out, a focus issue was the problem (871441)

  • about:newtab is now preloading its content, and causes our waitForPageLoad() method to fail when opening a new tab. That only happened with Mozmill 1.5 given for Mozmill 2.0 the whole code has been rewritten (887258)

  • At least on OS X Firefox shows a system dialog now, when a test tries to enable the geolocation feature for a page. To stop that from happening a new default preference has been added to Mozmill, which stopped that from appearing (897891)

As mentioned above it will most likely be the very last release of Mozmill 1.5. Further changes will completely go into Mozmill 2.0, which has been designed to be 100% backward compatible.

Mozmill 2.0rc5

It has passes a fair amount of time since our last RC4 for Mozmill has been released mid of June. That was more as we have anticipated, but it was worth the time. In all those weeks we fixed 32 bugs for Mozmill and 4 bugs for JSBridge.

The most important changes are:

  • JSBridge forced single and multiple conversions of strings to unicode where it was not necessary. As result sent data was not correctly transmitted or completely dismissed. That was a pain given that test results were inaccurate (761603)

  • In some cases Mozmill failed to send data during the shutdown of the application via the bridge to the Python process. As result we hit an assertion and with debug builds the application crashed due to an invalid access to an object in our socket implementation via NSPR (865690)

  • Another problem in our socket implementation was the missing handling of failed send requests. Before the patch has been landed, JSBridge simply ignored that and we hit a dataloss issue when too much information had to be send through the bridge (764640)

  • Our local HTTP server based on httpd.js was instantiated once for each test. This caused a lot of trouble and overhead. Now it is created on startup of the application and keeps running until shutdown. With that you will also be able to use it while Mozmill is in manual mode (881657)

  • Various changes to the restart logic have been included. With that we also got our state machine implemented for restart tests (895657).

  • Specifically for Metro Firefox we landed support for new touch events (880426)

Full details about the underlying changes in RC5 can be found in the changelog for Mozmill 2.0rc5 and JSBridge 3.0rc5

MozInstall 1.0 released

A crucial part of test automation is the fact that you can install and uninstall the application under test. While the installer part for Gecko based applications was already included in MozInstall we missed the uninstaller feature. I have added this feature now for the 1.0 release. On Windows it will first try to run the Firefox uninstaller in silent mode to remove any trace even from the registry. If files remain or if you are on another platform the specified folder will just be wiped out.

This work is part of our overall code rewrite of the Mozmill Automation scripts for Mozmill 2.0, whereby a lot of useful code will be merged with existing mozbase packages, or new packages like the mozdownload one will be created.

Keep in mind that this release changes the API of several functions. We are aware of that and hope that everyone of you will be able to adapt to it easily. It was necessary before more and more projects rely on this package. For any details please see the appropriate README file. Otherwise here some examples:

CLI:

$ cd /Volumes/mozilla/builds/
$ mozinstall firefox-13.0-en-US.dmg
/Volumes/mozilla/builds/FirefoxAurora.app/Contents/MacOS/firefox
$ mozuninstall firefox/

With those two commands the specified DMG image of Firefox 13 will be installed into a sub folder of the current working directory and removed right after. The installer will print the location of the actual binary to stdout.

API:

import mozinstall
path = mozinstall.install('firefox-13.0-en-US.exe)
print mozinstall.get_binary(path, 'firefox')
mozinstall.uninstall(path)

This code is the equivalent to the CLI example above. It installs Firefox into a sub folder of the current working directory, and returns the path. With the call to get_binary and by specifying the type of application, the actual binary will be returned. Replace ‘firefox’ with ‘thunderbird’ if you want to install a Thunderbird build. Finally the uninstaller expects the path Firefox has been installed into, and removes the application.

Mozmill 1.5.12 released

Something we have learned over the weekend was: Never say never. After we have released the 1.5.11 release of Mozmill on April 19th, we were sure to not have to ship any other release off this branch. We want to concentrate our work on Mozmill 2.0 and get it out as soon as possible.

But things can change, and sometimes faster as expected. So by last Friday (it was already late in the evening) Mike Conley contacted me on IRC and mentioned that Mozmill will be broken starting with tomorrows build. The reason for it was the ‘Global-Per-Compartment’ patch which has been landed in Nightly builds of Firefox and Thunderbird at that time. Great! Why do things like those happen just before when I’m already in my weekend?

Anyway, given the importance of this bug (it broke the whole test infrastructure of Thunderbird which mostly relies on Mozmill) I was talking immediately to Clint Talbert. Given the good bug report from John Schoenick we were directly aware of what the problem was and agreed on a new API to get the issue fixed.

So I have spent a good portion of my Saturday and Sunday evening to put together a patch which removed the broken code in waitForPageLoad() and replaces it with the new API. So by now we do no longer add a custom ‘mozmillDocumentLoaded’ property to all windows, but have a nice windowMap object that handles all the loaded states of each window on its own. I think that’s great and modifying the DOM of each window was even a bad idea since the beginning of the project. Good to see that killed.

That means early today we got all patches landed and I was able to release Mozmill 1.5.12 to PyPI and even the uploaded extension on addons.mozilla.org has been approved in the meantime. If you haven’t updated yet and you make use of Nightly builds, you should do it now!

Lets cross the fingers that no other unexpected issues will arise and force another 1.5 branch release.

MemChaser 0.3 is out

Late on Thursday we got our MemChaser 0.3 release out the door. Compared to the last releases this version comes with a ton of new features and bug fixes. Most of them have been requested by users and from the JS team so we put some extra focus on those.

As you can see in the image new ui elements have been added. The memory and garbage collector related items are equivalent to the buttons in ‘about:memory‘ and help those users who want to trigger a memory clean-up more often but don’t want to keep the before mentioned page open. The reaction of Firefox and the updated memory and GC/CC values can be seen in the MemChaser widget right after the triggered action has been finished.

Also we have updated our code which retrieves GC/CC related information to use the new Garbage Collector API. This allows us to get all the information and not have to scrape the console messages anymore for particular elements. That makes our life a lot easier and as benefit we can even store the complete set of data to the log file for further investigation. This log file is now a valid JSON file and will make it easier for you to analyze the data. If you wondered (with the former versions of MemChaser) where the log files have been stored, you can now open the containing folder via the new panel element too. If you are not satisfied with the location go ahead and change the target folder in the add-on preferences to any folder you like.

To help new users of the extension to understand the entries in the widget, tooltips have been added. Those will inform you about the background of the individual values. This was a request by a couple of users so we wanted to have this included too.

If you are interested in all the changes happened in the 0.3 release you can find the complete list as usual on our MemChaser issue tracker.

This version of MemChaser has still been built on top of the Add-on SDK 1.5, because we wanted to get out all of those new features also to our Firefox 10.0 user base which is about 28% of all our users. This was more important to us compared to fixes we would have benefit from the SDK 1.6. But at this time we want to announce that version 0.3 will be the last release which officially supports Firefox 10. With version 0.3.1 which will be released in the next one or two weeks a version bump to the SDK 1.6 will be done to fix the before mentioned issues like memory leaks.

If you are interested in helping us in the implementation of new features or fixing bugs, please check our issue tracker for open issues and contact us on IRC in the #automation channel.