Firefox Automation report – week 51/52 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 51 and 52 of 2014. I’m sorry for this very late post but changes to our team, which I will get to in my next upcoming post, caught me up with lots of more work and didn’t give me the time for writing status reports.

Highlights

Henrik started work towards a Mozmill 2.1 release. Therefore he had to upgrade a couple of mozbase packages first to get latest Mozmill code on master working again. Once done the patch for handling parent sections in manifest files finally landed, which was originally written by Andrei Eftimie and was sitting around for a while. That addition allows us to use mozhttpd for serving test data via a local HTTP server. Last but not least another important feature went in, which let us better handle application disconnects. There are still some more bugs to fix before we can actually release version 2.1 of Mozmill.

Given that we only have the capacity to fix the most important issues for the Mozmill test framework, Henrik started to mass close existing bugs for Mozmill. So only a hand-full of bugs will remain open. If there is something important you want to see fixed, we would encourage you to start working on the appropriate bug.

For Mozmill CI we got the new Ubuntu 14.10 boxes up and running in our staging environment. Once we can be sure they are stable enough, they will also be enabled in production.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 51 and week 52.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meeting of week 51 and week 52.

Firefox Automation report – week 49/50 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 49 and 50.

Highlights

During the first week of December the all-hands work week happened in Portland. Those were some great and inspiring days, full of talks, discussions, and conversations about various things. Given that I do not see my colleagues that often in real life, I have taken this opportunity to talk to everyone who is partly or fully involved in projects of our automation team. There are various big goals in front of us, so clearing questions and finding the next steps to tackle ongoing problems was really important. Finally we came out with a long list of todo items and more clarity about so far unclear tasks.

In week 50 we got some updates landed for Mozmill CI. Given a regression from the blacklist landing, our l10n tests haven’t been executed for any locale of the Firefox Developer Edition. Since the fix landed, we have seen problems with access keys in nearly each locale for a new test, which covers the context menu of web content.

Also we would like to welcome Barbara Miller in our team. She joined us as an intern via the FOSS outreach program as driven by Gnome. She will be with us until March and will mainly work on testdaybot and the conversion of Mozmill tests to Marionette. The latter project is called m21s and details can be found on its project page. Soon I will post more details about it.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 49 and week 50.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meeting of week 48. Due to the Mozilla all-hands workweek there was no meeting in week 49.

Firefox Automation report – week 47/48 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 47 and 48.

Highlights

Most of the work during those two weeks made by myself were related to get [Jenkins](http://jenkins-ci.org/ upgraded on our Mozmill CI systems to the most recent LTS version 1.580.1. This was a somewhat critical task given the huge number of issue as mentioned in my last Firefox Automation report. On November 17th we were finally able to get all the code changes landed on our production machine after testing it for a couple of days on staging.

The upgrade was not that easy given that lots of code had to be touched, and the new LTS release still showed some weird behavior when connecting slave nodes via JLNP. As result we had to stop using this connection method in favor of the plain java command. This change was actually not that bad because it’s better to automate and doesn’t bring up the connection warning dialog.

Surprisingly the huge HTTP session usage as reported by the Monitoring plugin was a problem introduced by this plugin itself. So a simple upgrade to the latest plugin version solved this problem, and we will no longer get an additional HTTP connection whenever a slave node connects and which never was released. Once we had a total freeze of the machine because of that.

Another helpful improvement in Jenkins was the fix for a JUnit plugin bug, which caused concurrent builds to hang, until the former build in the queue has been finished. This added a large pile of waiting time to our Mozmill test jobs, which was very annoying for QA’s release testing work – especially for the update tests. Since this upgrade the problem is gone and we can process builds a lot faster.

Beside the upgrade work, I also noticed that one of the Jenkins plugins in use, it’s actually the XShell plugin, failed to correctly kill the running application on the slave machine in case of an job is getting aborted. The result of that is that following tests will fail on that machine until the not killed job has been finished. I filed a Jenkins bug and did a temporary backout of the offending change in that plugin.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 47 and week 48.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 47 and week 48.

Firefox Automation report – week 45/46 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 45 and 46.

Highlights

In our Mozmill-CI environment we had a couple of frozen Windows machines, which were running with 100% CPU load and 0MB of memory used. Those values came from the vSphere client, and didn’t give us that much information. Henrik checked the affected machines after a reboot, and none of them had any suspicious entries in the event viewer either. But he noticed that most of our VMs were running a very outdated version of the VMware tools. So he upgraded all of them, and activated the automatic install during a reboot. Since then the problem is gone. If you see something similar for your virtual machines, make sure to check that used version!

Further work has been done for Mozmill CI. So were finally able to get rid of all the traces for Firefox 24.0ESR since it is no longer supported. Further we also setup our new Ubuntu 14.04 (LTS) machines in staging and production, which will soon replace the old Ubuntu 12.04 (LTS) machines. A list of the changes can be found here.

Beside all that Henrik has started to work on the next Jenkins v1.580.1 (LTS) version bump for the new and more stable release of Jenkins. Lots of work might be necessary here.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 45 and week 46.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 45 and week 46.

Firefox Automation report – week 43/44 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 43 and 44.

Highlights

In preparation for the QA-wide demonstration of Mozmill-CI, Henrik reorganized our documentation to allow everyone a simple local setup of the tool. Along that we did the remaining deployment of latest code to our production instance.

Henrik also worked on the upgrade of Jenkins to latest LTS version 1.565.3, and we were able to push this upgrade to our staging instance for observation. Further he got the Pulse Guardian support implemented.

Mozmill 2.0.9 and Mozmill-Automation 2.0.9 have been released, and if you are curious what is included you want to check this post.

One of our major goals over the next 2 quarters is to replace Mozmill as test framework for our functional tests for Firefox with Marionette. Together with the A-Team Henrik got started on the initial work, which is currently covered in the firefox-greenlight-tests repository. More to come later…

Beside all that work we have to say good bye to one of our SoftVision team members.October the 29th was the last day for Daniel on the project. So thank’s for all your work!

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 43 and week 44.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 43 and week 44.

Firefox Automation report – week 41/42 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 41 and 42.

With the beginning of October we also have some minor changes in responsibilities of tasks. While our team members from SoftVision mainly care about any kind of Mozmill tests related requests and related CI failures, Henrik is doing all the rest including the framework and the maintenance of Mozmill CI.

Highlights

With the support for all locales testing in Mozmill-CI for any Firefox beta and final release, Andreea finished her blacklist patch. With that we can easily mark locales not to be tested, and get rid of the long white-list entries.

We spun up our first OS X 10.10 machine in our staging environment of Mozmill CI for testing the new OS version. We hit a couple of issues, especially some incompatibilities with mozrunner, which need to be fixed first before we can get started in running our tests on 10.10.

In the second week of October Teodor Druta joined the Softvision team, and he will assist all the others with working on Mozmill tests.

But we also had to fight a lot with Flash crashes on our testing machines. So we have seen about 23 crashes on Windows machines per day. And that all with the regular release version of Flash, which we re-installed because of a crash we have seen before was fixed. But the healthy period did resist long, and we had to revert back to the debug version without the protect mode. Lets see for how long we have to keep the debug version active.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 41 and week 42.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 41 and week 42.

Firefox Automation report – week 39/40 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 39 and 40.

Highlights

One of our goals for last quarter was to get locale testing enabled in Mozmill-CI for each and every supported locale of Firefox beta and release builds. So Cosmin investigated the timing and other possible side-effects, which could happen when you test about 90 locales across all platforms! The biggest change we had to do was for the retention policy of logs from executed builds due to disk space issues. Here we not only delete the logs after a maximum amount of builds, but also after 3 full days now. That gives us enough time for investigation of test failures. Once that was done we were able to enable the remaining 60 locales. For details of all the changes necessary, you can have a look at the mozmill-ci pushlog.

During those two weeks Henrik spent his time on finalizing the Mozmill update tests to support the new signed builds on OS X. Once that was done he also released the new mozmill-automation 2.0.8.1 package.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 39 and week 40.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 39 and week 40.

Firefox Automation report – week 37/38 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 37 and 38.

Highlights

After 7 months without a new release we finally were able to release mozdownload 1.12 with a couple of helpful enhancements and fixes.

We released Mozmill 2.0.7 and mozmill-automation 2.0.7 mainly for adding support of the v2 signed Firefox application bundles on OS X. Sadly we quickly had to follow-up with an appropriate 2.0.8 release for both tools, because a let change in the JS Engine caused a complete bustage of Mozmill. More details can be found in my appropriate blog post.

We were even able to finally release Memchaser 0.6, which is fixing a couple of outstanding bugs and brought in the first changes to fully support Australis.

One of our goals was to get the failure rate of Mozmill tests for release and beta candidate builds under 5%. To calculate that Cosmin wrote a little script, which pulls the test report data for a specific build from out dashboard and outputs the failure rate per executed testrun. We were totally happy to see that the failure rate for all Mozmill tests was around 0.027%!

During the merge process for the Firefox 32 release Andrei has seen some test inconsistencies between our named branches in the Mozmill-Tests repository. Some changes were never backported, and only present on the default branch for a very long time. He fixed that and also updated our documentation for branch merges

Something else worth for highlighting is also bug 1046645. Here our Mozmill tests found a case when Firefox does not correctly show the SSL status of a website if you navigate quickly enough. The fix for this regression caused by about:newtab made it even into the release notes

Last but not least Andreea started planning our Automation Training day for Q3. So she wrote a blog post about this event on QMO.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 37 and week 38.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 37 and week 38.

Firefox Automation report – week 35/36 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 35 and 36.

Highlights

Due to a lot of Mozmill test failures related to add-on installation and search, we moved from the addons-dev.allizom.org website to the staging website located at addons.allizom.org. Since then we experiencing way lesser test failures, which are most likely network related.

In order to keep up with all test failures, and requests for new tests we started our Bug triage meeting on a weekly basis on Friday. For details we have created an etherpad.

If you are interested in helping us with Mozmill tests, you can now find a list of mentored and good first bugs on the bugsahoy web page.

Because of the app bundle changes on OS X, which were necessary due to the v2 signing requirements, we had to fix and enhance a couple of our automation tools. Henrik updated mozversion, mozmill, and mozmill-automation. We were close to releasing Mozmill 2.0.7.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 35 and week 36.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 35 and week 36.

Erster offizieller Halbmarathon

Nachdem ich mir bereits vor mehr als einem Jahr vorgenommen hatte, einen Halbmarathon zu laufen, hat es nun doch bis letzte Woche gedauert. Eine wirklich lange Zeit, aber der Grund waren einfach Knieprobleme, die mich letztes Jahr zu einer mehrmonatigen Pause beim Laufen zwangen. Aufgeben wollte ich trotzdem nicht und habe deshalb die Auszeit genutzt, und meine Füße im normalen Alltag komplett an das Barfußlaufen zu gewöhnen. Für jedmögliche Gelegenheit gibt es nun spezielle, sogenannte Barfußschuhe in meinem Schuhschrank. Durch deren ständiges Tragen konnte ich die Muskulatur in meinen Füßen wunderbar stärken und auch den Laufstil anpassen. Vom vormals Hinterfußläufer bin ich nun zum Mittelfuß- aber auch schon zum Teil Vorderfußläufer geworden. Diese Veränderung hat durch die natürliche Dämpfung des Fußes wesentlich geringere Belastungen für das Knie zur Folge, und lässt mich erfolgreich mit dem Training fortsetzen. Wie froh ich darüber bin, muss ich sicherlich nicht gesondert erwähnen. Seit ca. April dieses Jahres bin ich nun wieder wöchentlich aktiv und habe die gelaufenen Distanzen mit den Barfußschuhen kontinuierlich erweitert. Am 05.10.2014 hatte ich dann meine erste Halbmarathondistanz von 21.1km erfolgreich absolviert und war mir sicher, den offiziellen Lauf auch bewältigen zu können.

Starter des Marathons und Halbmarathons – (c) Bild von mopo24.de

Der 16. Dresdner Morgenpost Marathon fand am 19.10.2014 hier in Dresden statt. Dieses Mal mit einer komplett geänderten Streckenführung und einem getrennten Start der Marathon- und Halbmarathonläufer vom 10km Feld. Das war sicherlich hilfreich, aber aufgrund der riesigen Anzahl von Läufern – allein 2255 für den Halbmarathon – gab es trotzdem ein arges Gedränge am Start. Glücklicherweise konnte die breite Ostra-Allee und der weitere Weg über den Theaterplatz und die Augustusbrücke, die Läufermasse etwas auseinander ziehen, sodass spätestens in der Neustadt ein angenehmes Laufen möglich war. Auch das Wetter spielte wieder mit, das mit ca. 20 Grad und viel Sonne ein eher sommerliches Ambiente schuf.

Dieses Mal musste ich nicht allein rennen, sondern hatte Anita als Begleitung dabei. Für sie war es bereits der zweite Halbmarathon. Ich wollte den Lauf unterhalb von 2h beenden, und von ihrer Seite war ein Pace von 5:30 min/km geplant. Da beide nicht weit auseinander liegen, schloss ich mich ihr also an. Anita musste mich aber gerade zu Beginn gut drosseln. Wie immer wollte mich die schnellere Masse mitziehen. Nach den ersten Kilometern waren wir aber gut in unserem Tempo drin, und ich habe dann auch öfters selbst mal die Zeit kontrolliert. Alles lief ziemlich gut, bis bei Kilometer 13 dann die ersten Probleme mit dem linken Fuß begannen. Das ständige Laufen auf Asphalt ist mit Barfußschuhen dann doch etwas anderes, als mein Training auf gemischtem Boden. Aber es half nichts. Mein Wille war da, es zu schaffen! Deshalb wurde mit zusammengebissenen Zähnen weitergerannt. Wenn ich die geplante Zeit nicht schaffe, dann wollte ich wenigstens das Ziel erreichen. Glücklicherweise ließen die Schmerzen auf den letzten beiden Kilometern etwas nach. Ein Abgleich mit der Zeit bei Kilometer 20 zeigte aber auch, dass wir schon lange nicht mehr den 5:30 min/km Pace hatten. Um ihn doch noch zu schaffen, nahm ich bei etwas mehr als 500m vorm Ziel all meine Kräfte zusammen und sprintete ins Ziel. Das Resultat war unglaublich – mit 5:30 min/km Pace war ich somit exakt nach 1:56:50 anggekommen. Ein einfach herrliches Gefühl sich selbst noch etwas mehr als die 2h unterboten zu haben.

Selbstportrait kurz vor dem Endspurt – (c) V. Lichanov

Ganz besonders freut es mich natürlich wieder, dass die mit dem Lauf verbundene Spendenaktion zusammen mit dem PTV Sachsen so erfolgreich war. Aufgrund all der engagierten Spender, die meinen Lauf unterstützt haben, kamen stolze 219 EUR für einen wohltätigen Zweck zusammen. Sicherlich wird ein Großteil wieder an Aktionen für Kinder aus sozial schwächeren Familien genutzt werden. Somit vielen lieben Dank an Euch alle!!!

Nach diesem Lauf und dem Erreichen einer neuen Distanz und Zeit, liegen nun neue Herausforderungen vor mir. Sicherlich werde ich jetzt versuchen, Zeiten für die 10km und Halbmarathon-Distanz zu verbessern, aber ein kompletter Marathon-Lauf wäre auch ein klasse Ziel – vielleicht ja schon für das nächste Jahr zur gleichen Zeit!

Mozilla, Photography and the Daily Life