Hello from Platforms Operations! Once a month we highlight one of our projects to help the Mozilla community discover a useful tool or an interesting contribution opportunity.
This month’s project is firefox-ui-tests!
What are firefox-ui-tests?
Firefox UI tests are a test suite for integration tests which are based on the Marionette automation framework and are majorly used for user interface centric testing of Firefox. The difference to pure Marionette tests is, that Firefox UI tests are interacting with the chrome scope (browser interface) and not content scope (websites) by default. Also the tests have access to a page object model called Firefox Puppeteer. It eases the interaction with all ui elements under test, and especially makes interacting with the browser possible even across different localizations of Firefox. That is a totally unique feature compared to all the other existing automated test suites.
Where Firefox UI tests are used
As of today the Firefox UI functional tests are getting executed for each code check-in on integration and release branches, but limited to Linux64 debug builds due to current Taskcluster restrictions. Once more platforms are available the testing will be expanded appropriately.
But as mentioned earlier we also want to test localized builds of Firefox. To get there the developer, and release builds, for which all locales exist, have to be used. Those tests run in our own CI system called mozmill-ci which is driven by Jenkins. Due to a low capacity of test machines only a handful of locales are getting tested. But this will change soon with the complete move to Taskcluster. With the CI system we also test updates of Firefox to ensure that there is no breakage for our users after an update.
What are we working on?
The current work is fully dedicated to bring more visibility of our test results to developers. We want to get there with the following sub projects:
- Bug 1272228 – Get test results out of the by default hidden Tier-3 level on Treeherder and make them reporting as Tier-2 or even Tier-1. This will drastically reduce the number of regressions introduced for our tests.
- Bug 1272145 – Tests should be located close to the code which actually gets tested. So we want to move as many Firefox UI tests as possible from testing/firefox-ui-tests/tests to individual browser or toolkit components.
- Bug 1272236 – To increase stability and coverage of Firefox builds including all various locales, we want to get all of our tests for nightly builds on Linux64 executed via TaskCluster.
How to run the tests
The tests are located in the Firefox development tree. That allows us to keep them up-to-date when changes in Firefox are introduced. But that also means that before the tests can be executed a full checkout of mozilla-central has to be made. Depending on the connection it might take a while… so take the chance to grab a coffee while waiting.
Now that the repository has been cloned make sure that the build prerequisites for your platform are met. Once done follow these configure and build steps to build Firefox. Actually the build step is optional, given that the tests also allow a Firefox build as downloaded from mozilla.org to be used.
When the Firefox build is available the tests can be run. A tool which allows a simple invocation of the tests is called
mach and it is located in the root of the repository. Call it with various arguments to run different sets of tests or a different binary. Here some examples:
# Run integration tests with the Firefox you built
# Run integration tests with a downloaded Firefox
./mach firefox-ui-functional --binary %path%
# Run update tests with an older downloaded Firefox
./mach firefox-ui-update --binary %path%
There are some more arguments available. For an overview consult our MDN documentation or run eg.
mach firefox-ui-functional --help.
Useful links and references
- Project URL: https://wiki.mozilla.org/Auto-tools/Projects/Firefox_UI_Tests
- User documentation: https://developer.mozilla.org/en-US/docs/Mozilla
- Firefox Puppeteer documentation: http://firefox-puppeteer.readthedocs.io/en/latest/
How to get involved
If the above sounds interesting to you, and you are willing to learn more about test automation, the firefox-ui-tests project is definitely a good place to get started. We have a couple of open mentored bugs, and can create even more, depending on individual requirements and knowledge in Python.