As blogged by Release Engineering last week the Firefox 3.5 builds have been stopped. That means there are no more daily nightly builds available Mozilla QA will have to test with Mozmill. According to this action we also have turned off our daily test-runs for the latest Firefox 3.5.x builds on all platforms. At the same time the mozilla-1.9.1 branch of our mozmill-tests repository has been closed and as a good result, test failures which couldn’t be fixes due to limitations in Mozmill don’t exist anymore.
Today the builds for the new rapid release cycle have been pushed. With this change we have to update some of our automation tools and the way how we operate with the mozmill-tests repository. The following changes
coming into effect by now:
- We have two new named branches:
- aurora for builds on the Aurora channel
- beta for builds on the Beta channel
- From now on new Mozmill tests will be first checked into the aurora branch. The reason is because we want to have our new tests stable, and there will be too many changes happening for Nightly builds which could cause our tests to fail. Whenever a merge for Firefox happens we will do the same for our new tests. That means we merge all changes from aurora to default. That way we can ensure to test all new features from the last Firefox release in Nightly builds on a daily basis. The same strategy will apply to the beta channel, which we will merge all the tests into once a merge for Firefox happened.
- There will be situations which will cause us to check-in new tests on default first, but those should be really rarely and we will handle it on a case-by-case basis.
That’s why please take care that you always work on the aurora branch when you create new Mozmill tests.
Any questions or proposals? Feel free to ask.
Tags: automation, firefox, mozilla, mozmill, testing
The Mozilla QA Automation team proudly presents the next version of the MozMill Crowd extension which has been released today.
With Mozmill Crowd 0.1.2 we now support the Endurance Tests, which have been used a lot in the last couple of weeks to find memory leaks in Firefox. Now as part of the crowd extension, those tests are as easy to execute as all the other supported test-runs. Only if you would like to run those tests with a given set of add-ons installed, you will have to run the tests outside of the extension. Dave Hunt wrote an excellent article about this topic recently.
Further we were also able to remove the dependency for the Python headers on Linux. That means there is no pre-requisite anymore which has to be fulfilled before Mozmill Crowd will work. Only on OS X you will have to install the Developer Tools because VirtualEnv requires it to work. We hope to fix that by not using VirtualEnv in one of the upcoming releases.
A complete list of fixed bugs you can find on the tracking bug for the Mozmill Crowd 0.1.2 release.
Important Note: The extension is still not able to detect if an updated test environment has been made available. If you already have used MozMill Crowd before make sure that you manually delete the ‘mozmill-env.zip’ file and ‘mozmill-env’ folder in the selected storage folder. Triggering a new test-run will automatically download and install the newest version of the environment.
Also please check out our tomorrows testday which is completely devoted to the MozMill Crowd extension and the test execution in Firefox 4.
Tags: automation, crowd, firefox, mozilla, mozmill, mozqa, testing
Since I have been started to work on Mozmill tests for Firefox early 2009, our QA Automation Team has been grown and got a lot of work done. A fair amount of functional tests have been created, followed by a separate test-run of automated software update tests. Those two kinds of tests have been proven to reduce the time for release testing for QA drastically, and as result we have added even more test-runs like for add-ons, l10n, and lately for endurance tests.
As more and more different runs are getting added the structure of the whole repository, we have agreed on in 2009, doesn’t make sense anymore. Also looking ahead to our upcoming API refactoring which will land soon, we had to chose a new structure of the whole repository. A discussion was necessary to make sure the new hierarchy will fit our needs for the next couple of months or even years. And I can say for sure that we will add a lot more of test-runs and those need the right place to live.
To get this step accomplished, I worked on top of the specs and got the new structure implemented today. If you are interested in check bug 637306. So without reading through the bug you may ask, what has been changed? Lets compare in detail the structure before and after my work.
Before:
- addons (Tests for add-ons)
- firefox (Folder of mixed tests for Firefox)
- crashTests (Crash tests)
- enduranceTests (Endurance tests)
- helperClasses (Tests for shared code)
- l10nTests (l10n tests)
- restartTests (Functional restart tests)
- softwareUpdate (Update tests)
- test-files (Test data served via httpd.js)
- test* (Folders which contain functional tests)
- shared-modules (Shared code divided into components)
- templates (Simple templates for starters)
After:
- data (Test data served via httpd.js)
- lib (Shared code divided into components)
- tests (Tests for shared code)
- tests (Folder for the supported test-runs)
- addons (Tests for add-ons)
- crash (Crash tests)
- endurance (Endurance tests)
- functional (Functional tests for Firefox)
- restartTests (Functional restart tests)
- l10n (l10n tests)
- update (Update tests)
- templates (Simple templates for starters)
As you can see sections are clearly defined and it’s easy to find the code you are looking for. It’s the perfect base for our future work, you will hear about in the next couple of weeks.
Tags: automation, firefox, mozmill, mozqa, testing
As Clint Talbert has already written on Friday last week, Mozmill 1.5.2 has finally been released. A lot of great new stuff is now part of Mozmill core and we are happy to use it in our tests and automation scripts in the next couple of weeks.
The following changes will have to be made:
- With the nearly complete refactoring of the Mozmill Python modules our Mozmill wrapper used by the automation scripts is no longer necessary and needs to be removed. For me it means I have to complete rewrite the automation scripts. With those changes implemented we will no longer be backward compatible to former Mozmill releases.
- Mozmill can now run a list of specified tests from the command line. We want to use that feature to allow users of the Mozmill Crowd extension to run selected tests in a given order. That means that the automation scripts will have to support this feature, and I will take care about in the rewrite.
- We have disabled the compatibility check for add-ons in Firefox 4. That was necessary because lot of add-ons aren’t compatible yet, and weren’t correctly installed from the command line when specified via the –addons option. Now we also want to support it in our automation scripts.
- Failures for fatal and non-fatal assertions are now using the same structure, which is important for our Mozmill Dashboard and especially for the l10n testrun results which mostly contain results of non-fatal assertions.
I’m looking forward to see all the mentioned enhancements active in March. Those are a big step forward, also for our users of the Mozmill Crowd extension.
Tags: automation, firefox, mozmill, QA, testing
A week ago I had the pleasure to attend the FOSDEM (Free and Open source Software Developers’ European Meeting) in Brussels. As every year Mozilla has been setup its own booth and also got assigned an own developer room for talks. I have used this opportunity to spread the word about our crowd-testing initiative with Mozmill Crowd. Thanks to the committee my talk also got approved and I was able to demonstrate our latest activities in-front of a couple of people. If you weren’t able to attend, you can check my slides below or on Slideshare:
I have to thank everyone for the amazing feedback. A lot ideas have been collected during those 2 days, which now need to be injected into our roadmap. One of the most important items is definitely the way how we offer a testing solution for add-ons. Because of the Jetpack team already offers unit tests via the Addon SDK framework, we should work together to get more add-on authors to create tests for their add-ons, but also to being able to collect more test results from our testing community. It’s absolutely an item for me after Firefox 4 has been released and we have finished our refactoring of the shared modules used by our Mozmill tests. So assume updates by next quarter.
Beside my own activities I was able to visit some other booth and finally got in contact with Bernhard Wiedemann from the openSUSE project, who now also runs our Mozmill tests on a daily basis on their machines. I’m kinda inspired by some of the features they are using, especially the creation of a video for the complete test-run. Features like those would have been already helped us a lot. It’s nothing more frustrating to not being able to reproduce an issue because of lack of information.
A big thanks goes also to the planning team of Mozilla who took care of anything related over this weekend. Thanks for the accommodation, the dinner and bowling party on Saturday and all the other small but necessary stuff. As always it was great to get in contact with our and external communities.
Oh, and some pictures have been uploaded to Flick.
Tags: automation, crowd, event, firefox, fosdem, mozilla, mozmill, mozqa, testing
After talking to one of my colleagues at Mozilla, we have decided to push another minor release of the Mozmill-Crowd extension as soon as possible. The main task was to improve the user experience when using the extension. The following fixes have been incorporated:
- Save the screenshots which are automatically taken by the l10n test-run into the screenshots folder of the extension storage. That allows you to access those a way easier, compared to before when you had to locate those first in the temporary folder.
- Disable the slow script warning dialog which was unnecessary before, because we already know that the test-run will take longer than the 20s which are set as default.
- After a successful test-run show the logging information in the output section of the extension main window.
All those changes were part of Bug 627108. If you wanna give the extension a try, you can find the Mozmill Crowd 0.1.1 release on AMO.
Tags: automat, crowd, extension, firefox, moz, mozilla, mozmill, testing
Firefox 4, the next major version of Mozilla’s web browser, will include hundreds of new features and enhancements. A lot of those will happen in the background and will primarily not be visible at the front-end. A very good example for such a feature is the support for graphics hardware acceleration. It will drastically improve the rendering of pages, videos, or the Firefox UI itself. But noticeable it is only because it is that fast. Alongside those back-end features, also the Firefox UI has been vastly changed. Mainly you will notice the reduced amount of visible elements in the browsers chrome, or the newly introduced Firefox button on Windows and Linux at the top-left side of the window. To ensure that those new features or landed changes work as expected, Mozilla QA and all our testers have to constantly run manual tests for those areas which cannot be automatically tested by a Mochitest. To make the life of all of us easier, we have been actively working on getting even those areas populated with functional automated tests by using our own testing framework called Mozmill.
While all that testing with Mozmill is working fine on an already setup machine, we got a lot of feedback from testers that the process to create the test environment is kinda complicated. That’s why we have decided to create an extension, which will automate all those steps. The tester can lean back and do not worry about this process anymore. Alongside that important feature, we can now even distribute our tests across the globe and get it run by everyone. With the test results collected we have a lot of opportunities to work with the data. Regressions can be identified right at the time when they appear the first time, even if it is restricted to localized Firefox builds. Further we could analyze the performance of executed tests across all the different machines which could give us ideas of existent bottlenecks to fix. And there are even more areas we want to work on, once this process has been established.
How it works?
The Mozmill-Crowd extension is a simple wrapper around the Mozmill testing framework. It ships with its own UI to allow a customization of the test-runs. Testers can select the version of the browser to test, or choose one of the supported test-runs. For the execution the Mozmill command line interface is used. It’s implemented in Python and gets automatically installed as part of the testing environment, which the extension setup itself. The combination of the command line tool and the extension allows us to execute even tests which need a restart of the browser. All the tests are executed in another Firefox instance which uses a fresh profile. That means your daily profile is safe and will not suffer from any data loss caused by an running test.
Installation and Setup
To install Mozmill Crowd for Firefox simply open the add-ons page on addons.mozilla.org and click the Install button. After a restart of Firefox the extension is installed and its menu entry has been added to the Tools menu. Clicking on it will open the Mozmill-Crowd window as shown below:
The settings for test-runs has to be done at the top of the window. Already populated with default values you should see the currently running browser selected in the application dropdown, and the General Test-run as the to test-run to execute. You can switch the application as necessary by adding another installed Firefox versions. To switch the test-run select one of the following options:
- General Test-run: Executes a good portion of our manual Litmus BFT tests (Basic Functional Tests). Those cover areas like the Password Manager, Tabbed Browsing, Search, and a couple of others. All tests can be also run in localized builds.
- L10n Test-run: Tests especially created for localizers. It has checks implemented for multiple usage of access keys and cropped elements. At the moment only the preferences window is checked.
- Add-ons Test-run: Tests written by add-on authors to ensure the add-on is working properly with Firefox.
Beside the above mentioned settings which will be changed most often, others are available in the preferences of the add-on.
To finalize the setup and to be able to run the tests, the test environment has to be prepared. At the moment this step is part of the test-run itself and will be executed as the first task when the “Start Test-run” button is clicked. If the setup hasn’t been run before a dialog will appear and ask you to select a folder. This selected folder is used for all the extension related files, including the test environment. Create a new one or select an existing folder on your hard-drive to continue. In the next step the test environment gets downloaded and prepared in the specified folder. Additional modules, i.e. Mozmill itself, will be downloaded and installed automatically.
Note: Due to current limitations the extension executes the process via a blocking call. That means the browser instance which has been used to start the setup process will not react and also not update the screen until the test-run has been completed. That’s intended at the moment but will be changed in the next version of the extension. The slow script warning can also be ignored.
Test Execution
To execute a test-run simply select its entry in the dropdown and also choose the application under test. A click on “Start Test-run” will start the tests. With the setup step already executed, a script from the test environment will automatically pull the latest version of the automation scripts and all available Mozmill tests before it starts the browser, and runs the selected test-run. Once all the tests have been executed, the application is shutdown.
Note: Similar to the environment setup step the browser itself will be blocked until the test-run has been finished.
Sending Test Results
The extension supports sending of test results to a document based database, i.e. CouchDB. Per default the sending is disabled, but can easily be enabled in the preferences dialog of the extension. We would really appreciate your willingness to send reports to our server. Therefore please obey the following two important information:
- The application under test always has to be the front-most application. Don’t switch to another application while the tests are executed.
- Don’t kill the application during the test-run. Due to a bug in Mozmill it would cause a lot of false positive failures in our database.
If you are curious about your own test results simply head to our Mozmill dashboard which lists all the uploaded reports for the general test-run. Your result should appear at the top of the list.
L10n Test-run
With the Firefox 4.0 RC1 release coming up soon our primarily focus is targeted at localizers at the moment. That means if you are leading an l10n team or are a member of one, please run our l10n test-run in your locale of Firefox. Even with only tests for the preferences window available, we already have covered a couple of bugs in different locales of Firefox 4.0 beta releases. If failures have been found, screenshots are saved in the temporary folder of your user account. Those can be used to easily identify the broken UI elements. As an example have a look at bug 614938.
Thanks everyone for your feedback!
Tags: automation, extension, firefox, mozmill, mozqa, testing
After some discussion last week how to track bugs of the different projects inside Mozilla QA we came to the conclusion that a new product on Bugzilla is necessary. Similar to other groups at Mozilla, QA would also benefit from it when having all their projects under one single umbrella, instead of in dozen different components. Finally we came up with the “Mozilla QA” product.
By today the new “Mozilla QA” product has been created on Bugzilla under the Others section. At the same time 5 components have been created, which mostly fit the active projects of the QA Test Automation team. If you want to see details, check Bug 612578. Once a new component has been created, lot of you already know what happens. Right, normally hundreds of bugs have to be moved to the new location. I did the same a couple of minutes ago for our Mozmill Tests. Finally an amount of 508 bugs have been landed in the components under the Mozmill QA product. That move also cleaned-up all Firefox, Toolkit, and Core components we have used before to track the creation of Mozmill tests.
The old “Mozmill Tests” component under Testing has already been deleted. That means for any new issues please file bugs under “Mozilla QA > Mozmill Tests”.
If you want to watch ongoing work, you can subscribe to the following QA aliases:
- automation-team-dashboard@mozilla-qa.bugs – Team dashboard (starts soon)
- mozmill-automation@mozilla-qa.bugs – Automation scripts (updates, add-ons)
- mozmill-crowd-extension@mozilla-qa.bugs – The Mozmill Crowd extension
- mozmill-result-dashboard@mozilla-qa.bugs – Our Mozmill Result dashboard
Tags: automation, bugzilla, firefox, mozilla, mozmill, testing
Already two weeks ago, on November 3rd, the Selenium meetup #3 of the London Selenium user group has been taken place. Organized by Dave Hunt, we had 2.5h of interesting talks and discussions at the Google office in London.
The overall topic for this evening was how Mozilla is using Selenium and other tools to ensure the quality of their products. Therefore I have been asked to do a presentation about our client-side ui testing framework Mozmill. I was pleased to do so, because I thought that not that many people from that group have already seen how client tests for Firefox look like. Especially not when those get run directly from an extension in the crowd. But another reason also was our ongoing work for Mozmill itself and that we are thinking about using parts of Webdriver, which is a component of the upcoming Selenium 2.0 release. We already have contacts to the developers from Google, and I wanted to finally meet them and have a discussion to get an impression about the status of Webdriver.
I already arrived on Nov 2nd and had nearly one more day to finish up my slides and the extension itself. Everything went very well, even with a change in Minefield which has been broken the IPC-PIPE component my extension is relying on. By using an older Minefield build everything was ready a couple of hours before the meetup. The remaining time I mostly spent together with Simon Stewart, the leading person for the Webdriver development. He gave me a great insight in how native events are implemented, but also pointed out some open questions and still missing features. Personally I really missed the native events on OS X feature. I hope that it will find its way into Webdriver. But overall it looks great and once the focus issues are solved, we can try to investigate which parts we can re-use in Mozmill. Thanks Simon and Eran! Oh, and you are still welcome to do the same with our code. 
Finally at 7pm the first chunk of people appeared and we had a welcome snack to create the first contacts. Half an hour later we have started with the first presentation in a really nice and already crowded lecture room. I believe about 50 interested people were attending the meetup – even with the underground strike happening in London this day. So Stephen Donner was heating-up everyone with technical details about how Selenium is used at Mozilla and how tests are getting run. Right after Simon Stewart entered the stage and while getting applause he talked about the current status of Selenium 2. After a following short break with more pizza and coke we continued with the second round of presenters. That means David Burns started to talk about how to use Selenium for client-side performance monitoring. He showed some interesting examples with Firebug’s YSlow add-on. Once he had finished his Q&A part, the stage was ready for myself. In the following 20 minutes people were able to see which topics Mozilla QA is working on, how strong we rely on and are proud of our community, and finally how the current state of the Mozmill Crowd extension is working. People were amazed of the demo, even it didn’t worked out that well because of a broken screen assignment between my MBP and the two big screens behind me. But that didn’t matter. I got lot of great feedback and questions afterward. Thanks everyone!
If you are interested in details of my talk, check my slideshare presentation:
Tags: firefox, london, mozilla, mozmill, selenium, slides, testing
Recent Comments