Mozmill 2.0 released!

Believe it or don’t believe it. But this announcement is for real! Really, you can trust me! If you still think it’s a joke I can say: even with the expected release of Mozmill 2.0 was like Duke Nukem Forever over the past 3 years, we finally finished all bits and got it released yesterday. Check it out yourself on pypi:

So the whole process of working on a new release of Mozmill might have been started when we branched for the final 1.5 release, which was on Aug 23rd, 2010. Since then various people spent more or less time on its code to get it improved and lesser buggy. Also given some major refactoring at the beginning of the development of version 2.0, and lot of other feature requests from QA and other teams, we were working mostly double-tracked. Means, we have not only implemented new features for 2.0 but also had to backport those onto the hotfix-1.5 branch. That caused a lot of work for us. And because we weren’t working full-time on that project either, the time went by so fast.

It’s interesting to see that on Oct 10th, 2011, we released an RC1 for Mozmill 2.0 because we thought it would be the right time to get it out soon. But since then an amazing number of additional 276 changesets have been landed, and another nearly 2 years passed by. Amazing how your own schedule can be busted. But there was no way around.

In the last couple of months our Automation Development team has fully taken-over the project, and promised to get it finished and released. It was a kinda tricky and hard task, especially with all the additional bugs we have identified, which not only caused broken tests but also crashes and dataloss. Thanks to the help of our awesome contributors, who spend tons of hours to put a nice and polished version of Mozmill together. We wouldn’t have been here without your help. A big big thanks!

One of the things we are most proud of is the backward compatibility of Mozmill 2.0. We made it that as a Mozmill 1.5 user you shouldn’t see any difference. Any method and property available in Mozmill 1.5 is still present in 2.0, but has been marked as deprecated. With that you will be able to run all of your tests with both versions, and you can transition over step by step. There are indeed some new ways how to work with elements, which are explained on our MDN Mozmill page.

One very impressive improvement we have seen a couple of days ago, is the speed how Mozmill tests are getting executed. Compared to Mozmill 1.5 we can see a drop of the duration by a factor of 2.5! That’s two and a half times faster! I think already because of that you should not miss to at least test Mozmill 2.0!

As usual please give us feedback via email or on IRC about your own experiences and please report any bug you have discovered. If you like hacking and want to help us with the implementation of new features or fixing broken behavior, then have a look at our Mozmill repository on Github.

Soon I will come up with our plan for the future of Mozmill. And what I can say as of now, there is a lot to come…

Automation Development report – week 38 2013

The last week was more dedicated for maintenance. We were working on features for mozmill-ci to let us more easily debug failing tests, especially with all the cpu load spikes we have seen across our Windows machines. But we also got the final Mozmill 2.0rc6 release out. Continue reading to find details.


One of our biggest issues last week was definitely the drastic increase of the CPU load across all of our Windows nodes in the mozmill-ci cluster. It magically started when QA did the first ondemand tests for the upcoming Firefox 25.0b1 release. Especially the Windows 8.1 machines started to sporadically disconnect from the Jenkins master over the day, caused by some blue screen crashes related to the kernel. At the end this happened hourly or even more often, when the machines had to run tests. Observing the task manager has been shown that specifically the Windows Defender real-time protection is causing a high load, which e.g. caused a slow-down for installing Firefox from about 6 seconds to 4 minutes!! Disabling the real-time protection helped us across all Windows nodes, but not for Windows 8.1, which was still crashing and rebooting. But also other processes like svchost for network services or the desktop manager showed a high CPU load. Those we weren’t able to figure out yet. Stopping to run tests on Windows 8.1 preview release seemed to be the only option for us. Now we are waiting for VMs with the final version of Windows 8.1 installed. Lets see if that will work better.

Beside that our production CI instance got a couple of nice new features. At first Henrik added support for the mozmill-automation package, which comes with the latest versions of the mozmill-environment. With it we have a defined release schema now, and do no longer pull automation changes from the Mercurial repository for each change. This allows us to deeply test the scripts before the release, and will not cause that massive test failures when a mistake was made. Further Dave landed features like timestamp output in the console log, listing of injected environment variables, and HTTP logging for our update tests. Especially the latter should help us a lot while investigating update test failures. Last but not least Henrik also landed ANSI color support for the console, which will be used by Mozmill 2.0 to show colored output for test results.

Beside all that we also released the next release candidate for Mozmill, which is v2.0rc6. Read this thread for more details. Just to add, the previously setup mozmill-ci instance for Mozmill 2.0 was of big help here, given that we discovered a couple of unknown issues and were able to fix them all in a quick turnaround. With that release we also pushed mozmill-automation 2.0rc6 and an updated mozmill-env.

For Gaia we were able to finalize a very important step, we worked on for a long time and which will help everyone working on that product. It’s the move of the gaia-ui-tests from the separate github repository into the gaia main repository. With it we have always tests, which are synced with the latest feature additions and bug fixes in Gaia. Also developers can more easily run those tests, and we can detect regressions before they actually land in the repository.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad.

Automation Development report – week 37 2013

Another week passed by and there are only two weeks remaining until the end of Q3. That means we are working eagerly to get the remaining goals finished up.


As of last week we are finally running our Mozmill l10n tests, which discover duplicated access keys and cut-off elements, in our mozmill-ci production environment. That’s a big step forward because with that we no longer have any other test machines which run Mozmill tests on a production like system. Everything is combined in a single continuous integration system. To make that happen Henrik had to fix a couple of dependencies across mozmill-ci, mozdownload, and down to pulsetranslator. As of now we will not regularly check the test results but will report them once in a while. Reason for that is the lack of human resources or any automation, which syncs the results with the l10n dashboard.

With yesterdays release of Firefox 24.0 we got another extended support release (ESR). That means we have to cover both 24.0.x ESR and 17.0.x ESR releases for the next 12 weeks. The necessary jobs for mozmill-ci have been created and our Pulse listeners adjusted.

We finally updated our mozmill environment to make use of the new and shiny mozmill-automation package. That allows us to do signed off releases, instead of direct repository checkouts for mozmill-ci, which caused a bit of breakage since it exists. We have new packages available for download for Mozmill 1.5 and 2.0rc5.

To let us have an easier transition from Mozmill 1.5 to the upcoming version 2.0, Henrik setup a new mozmill-ci instance, which runs the same kind of tests as production but with Mozmill 2.0rc5. The setup was already helpful given that we discovered a regression which didn’t let us run Mozmill on OS X 10.6. With all the remaining blockers landed we will have another release candidate. It will be out this week.

In terms of Mozmill tests for Firefox we finally were able to separate out tests, which make use of remote test data, to a separate testrun. With that step our functional testrun only uses local resources served via httpd.js. Big thanks here goes to one of our new contributors with the name Balazs Juhasz.

Not to leave gaia alone I also want to mention some of the highlights in that area. The gaia-ui-tests for example are using mozdownload now to download b2g desktop builds for testing. Support for mobile builds will hopefully land soon too. Beside that Dave pushed two software releases of b2gperf 0.7 (with logging included) and b2gpopulate 0.9 (with a fix for nightly builds).

Last but not least we have to give credits to Johannes, who finished the patch to get the first automated tests for mozdownload via mozhttpd landed. It’s the base for all the other tests we need to get full test coverage across all the different scrapers.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad.

Automation Development report – week 36 2013

It looks like since everyone is back from the summer vacations we keep our productivity kinda high over the last couple of weeks. During the last week we finished again a couple of important tasks.


As announced in last weeks report we were going to upgrade our Mozmill continuous integration system to the latest LTS release of Jenkins. That task has been done now, after a full week of observation on our staging instance. Everything looked fine, so Henrik pushed the upgrade to production on Monday. Since then the system hasn’t shown any instability, but gave us a fantastic improvement in terms of memory usage. The mean heap usage dropped from about 1.6 GB to only 800 MB, which is only 50% compared to the formerly usage! Also with other updates for used plugins we were able to close some long outstanding issues.

As requested by Mozilla QA a while ago we were finally able to get our Windows 8.1 enabled and activated for testing. The crashes we have seen here frequently when logging into those machines were really related to the remote desktop service. With a hint from Clint Talbert to use VNC instead (what we actually do already on all the buildbot machines), we were able to stop the crashes by simply using UltraVNC now. Since the tests are getting run on that platform no specific regressions have been observed. So as of now all looks good for a release on Windows 8.1.

Given that the release of Mozmill 2.0 is really close, we started to work on a transition plan, which would allow us to easily flip between Mozmill 1.5 and 2.0 in our mozmill-ci system. To accomplish that we have decided to release mozmill-automation as a package on PyPI from now on. That means the appropriate version will be pre-installed in the mozmill-environment, which the Jenkins jobs are using. A bit of work has to be done on the CI side too, but once that happened we will have a second CI instance, which will run Mozmill 2.0 in parallel to our production system. If all is green we will release 2.0 next week.

Henrik is also working to get the mozmill-l10n tests moved to our production mozmill-ci. As of now those were running on an old Mac Mini located outside the datacenter. All the code it was using was outdated too, so after lots of updates we pushed the changes to staging last week. By next week I will give some more details about it, probably by a separate blog post.

But also on the Gaia front we were able to make some good progress. Dave found the time to finally create a command line interface tool to directly communicate with Gaia via simple commands. You should really check this out!

A big step for us was also the fact that Dave was able to get the Gaia-UI endurance tests landed into the main Mozilla Gaia repository for the v1-train! With that landing the Jenkins instance has been updated to run the tests with nightly builds from the new location now. If you are interested in running those tests on your own, you shouldn’t miss to read the blog post from Rob in how to run the Gaia-UI endurance tests.

In terms of Eideticker Dave also was able to make progress and setup the first working version of the Eideticker CI. Over the curse of the next couple of week it will get further improvements, so we will have a trusted source of test results.

Last but not least Dave also pushed a new version of Marionette Client 0.5.37, which is able to forward command line arguments to the application under test by simply specifying them via –app-arg. That new feature was necessary for us to get failures in B2G desktop better investigated.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad.

Automation Development report – week 35 2013

Our last week was again very productive, and a lot of smaller and larger tasks have been finished. We are getting closer to reach our quarterly goals. Our next big step is really to get the final version of Mozmill 2.0 out, and the Metro support finished. Then we can get started with implementing ui driven tests as requested by Mozilla QA.


With the combined work with our Softvision team and a couple of contributors we were able to release two versions of Mozmill in one go. See my blog post for release details of Mozmill 1.5.22 and 2.0rc5. Hopefully the 1.5.22 release will mark the end of life for the 1.5 branch, and that the 2.0rc5 was the last release candidate build for Mozmill 2.0. More to that topic soon.

To be able to move our Mozmill l10n tests from an older Mac Mini, which is currently running in the MV network, to the more stable and IT supported network in the SCL3 datacenter, we had to fix a couple of issues in mozdownload. So as result we have released mozdownload 1.9 last week. That fixed the last remaining blocker, and we will have the l10n tests running on staging soon.

Over the last couple of weeks we were monitoring the memory usage of Jenkins a lot. That happened not only in the browser but also more important for the whole system on the server side too. As we noticed the Java process has a really high heap memory usage. Beside that we had a couple of annoying issues around, which caused some of our testruns to early abort. So Henrik put a lot of work in getting Jenkins upgraded to the latest LTS (Long Term Support) version. Given that this was a big jump for our really outdated Jenkins version a lot of work had to be spent in testing everything. Also because nearly all of the Jenkins plugins, which we make use of, had updates available and had a slightly different format of the XML configuration, all our static files had to be updated too. But by early last week we were able to push those bits to our staging server, and running that version of Jenkins for a week, hasn’t brought up any kind of regression. That’s great! To finish up the work, Henrik published all bits and bytes to our production machine yesterday. We will continue observing the memory usage, and let you know if the situation got better.

Beside that we made a couple of changes to our list of available nodes for the mozmill-ci cluster. As of last week we brought our Windows 8.1 nodes (3x 32bit, 3x 64bit) online for preliminary testing. Given that all went well and no further issues have been seen, they are now fully integrated into jobs triggered by Pulse notifications. Further we have replaced our existent Ubuntu 12.10 nodes with the current release of Ubuntu 13.04. That means we run our tests on Ubuntu 12.04 (LTS) and Ubuntu 13.04 for both 32bit and 64bit now.

As requested by Mozilla QA we have to run more tests for localized beta and release candidate builds of Firefox. That will help us to finally stop releasing broken l10n builds like Firefox 24.0b1 to our users. For all locales covered by our tests, we do now check for start-up failures like the XML parsing error for the main window. So Henrik pushed a patch to production, which enabled tests to be run for the first 10 most used locales of Firefox.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad.