I promised to keep up with our updates over the last week but given a major breakage in the freshly released version of Mozmill 2.0.4, I had a full week of work to get the fix out. I promise that during this week I will write reports for the weeks in January.
With the new year our team has been reorganized and we are part of the Mozilla QA team again. That means we will have a way closer relationship to any feature owner, and also working towards in bringing more automation knowledge to everyone. The goals for our team are getting worked out and I will present those in one of my following blog posts. As of now you can find our team page on the Mozilla wiki under Firefox Automation.
Since the landing of all the new features for Mozmill-CI on our staging machine before Christmas, we have no longer experienced any crash of the Jenkins master. Given that Henrik pushed all the changes to our production system. We are totally happy that the incremental updates made our system that stable, and that Mozilla QA doesn’t have cope with outages.
If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the first Firefox Automation meeting of week 2.
Wow, somehow I totally missed to send out reports for our automation work. Most likely that happened because of the amount of work I had in the past couple of weeks. So for now lets do a final update before the title will be updated to ‘Firefox Automation report’ by the year 2014.
We have released Mozmill 2.0.3 to fix a couple of issues (see dependencies on bug 950831 seen with Firefox Metro builds and our Firefox shutdown code. We pushed those changes together with the releases of mozmill-automation 2.0.3 and the new mozmill-environment files to our mozmill-ci staging instance for baking.
Henrik was able to finish the work in setting up our new mozmill-ci staging instance in the SCL3 datacenter. Please see bug 947108 for details. With it we have the identical environment as the production instance and can see regressions immediately and not when we merge to production, which was pretty bad in the past couple of week. So RIP old staging server!
One of our goals for quarter 3 in 2013 was to setup a web based configutation tool for ondemand testruns in mozmill-ci, which can be used by QA people to trigger testruns for beta and release builds. Cosmin jumped on it and got the first version implemented. You can find a running instance on Github for now. Later we want to make the tool available via http://www.mozqa.com.
To make our mozmill-ci system more stable, Henrik pushed a large set of new features and fixes to the staging instance. Our plan was to let it bake over the Christmas holidays with the hope that Jenkins will run way more stable now.
If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the last Automation Development meeting of week 51.
It’s getting closer to Christmas. So here the 2nd last automation development report for 2013. Also please note that all team members including myself, who were dedicated to Firefox Desktop automation, have been transitioned back from the A-Team into the Mozilla QA team. This will enable us to have a better relationship with QA feature owners, and get them trained for writing automated tests for Firefox. Therefor my posts will be named “Firefox Automation” in the future.
To get rid of failed attempts to remove files after a testrun with Mozmill, Henrik was working on a new version of mozfile, which includes a method called remove() now. It should be used by any code given that it tries multiple times to remove files or directories if access is getting denied on Windows systems.
We are still working on the remaining issues with Mozmill 2.0 and are hoping to get them fixed as soon as possible. So that an upgrade to the 2.0.x branch can happen.
This is again an overview about the highlights of our work in week 47 and 48 this year. Sorry for the delay of a couple of days but some critical work was holding me off from writing this post.
Henrik and Dave were working on a couple of Mozmill-CI updates, which have been pushed to production. The first batch of features and bug fixes landed in week 47. It included the monitoring plugin for Jenkins, which hopefully will help us to figure out the reasons of Java crashes. Also we can finally run project branches via our CI now, even it can only be done manually as of now. It is important for our upcoming tests for the Holly branch (Firefox Nightly without the Australis feature). The second batch landed by last week was intended to upgrade our system to Mozmill 2.0.1. Sadly we failed in it due to a couple of other failures, which we haven’t seen before on our staging server. So we partly reverted back the latest commits for production and we all are working hard on getting those issues fixed.
With the detected failures by upgrading to Mozmill 2.0.1, Henrik has noticed that one of those existed because of an incompatibility of mozbase with Python 2.6. See bug 944361 for details. To solve this we upgraded our OS X 10.6 boxes to Python 2.7.3, so all machines are running the same version of Python now. As a very nice side-effect we noticed a speed improvement by running our Mozmill tests of about 25%!
To be prepared for executing the first Metro tests with Mozmill we had to prepare the mozmill-tests repository for handling multiple applications. Therefore Andreea refactored nearly the whole repository. You can find details on bug 927397.
Yesterday we tried to upgrade our mozmill-ci cluster to the previously released Mozmill 2.0.1. Sadly we failed on the OS X 10.6 machines and had to revert this change. After some investigation I found out that incompatibility issues between Python 2.6 and 2.7.3 were causing this problem in mozprofile. Given the unclear status of Python 2.6 support in mozbase, and a talk in the #ateam IRC channel, I have been advised to upgrade those machines to Python 2.7. I did so after some testing, also because all other machines are running Python 2.7.3 already. So I didn’t expect any fallout. First post upgrade tests have proven this.
The interesting fact I would like to highlight here is that we can see speed improvements by running our tests now. Previously a functional testrun on 10.6 has been taken about 15 minutes. Now after the upgrade it went down to 11 minutes only. That’s an improvement of nearly 27% with Mozmill 1.5.24. With Mozmill 2.0.1 there is a similar drop which is from 8 minutes to 6 minutes.
Given all that and the upcoming upgrade (hopefully soon) of our mozmill-ci system to Mozmill 2.0.1 we will see an overall improvement of 60% (15 minutes -> 6 minutes) per testrun!! This is totally stunning and allows us to run 2.5 times more tests in the same timespan. With it we can further increase our coverage for locales from 20 to 40 for beta and release candidate builds as next step.
After the last report over a two week cycle, I have to follow-up with another one for the weeks 45 and 46. Due to my move I had limited availability and to fix some important other stuff. So hopefully this is the last report over a two weeks period in the near future.
To be able to release Mozmill 2.0.1 as soon as possible Henrik had to fix a lot of existent bugs for mozprofile in conjunction with its add-on manager class. Those fixes were necessary because our restart tests were broken due to an inappropriate clean-up of add-ons after closing Firefox. At the end 10 bugs have been fixed.
We are close to get our first Metro tests running with Mozmill on Windows 8 and 8.1. But before we can really do that, the whole mozmill-tests repository has to be refactored to support multiple applications. Here both Andreea and Andrei are doing most of the work.
Dave and Henrik were both working on a couple of Mozmill-CI issues, which will help us to better diagnose the memory issues and random crashes of the Jenkins Java process. Everything has been merged to our staging server and has to bake a bit before a push to production will happen.
Henrik added a new feature to Mozmill-CI which allows us to run tests for builds off project branches. This will come into play really soon when we have to execute daily tests for the upcoming holly branch, which is mozilla-central without the Australis UX.
As of now we have released a new version of mozdownload. The 1.10 release contains a couple of new features, bug fixes, and absolutely to mention a suite of tests. A big thank you goes here to Johannes and Jarek. Both spend a lot of time again, to make it a successful and lesser bug prone release.
Here some major items to highlight:
- Addition of tests for all types of scrapers and a lot for command line options
- Output of all candidate builds founds if build number has been specified
- Add support for tinderbox stub installer
- Allow to download files with different extensions than exe
A full list of features can be found in the changelog
Given that I was partly away in the last two weeks I haven’t had the time to write another summary report of our work yet. So lets combine the last two weeks in a single blog post this time.
To be able to continue our work to speed-up testruns in Mozmill CI Henrik upgraded our Mozmill-CI system to Jenkins 1.509.4. This step was necessary so that we can investigate the slowness in sending out emails for finished testruns, which could take up to 25 minutes! Beside that Henrik also disabled the sending of emails for successful testruns given that those were mostly blocking us, and no-one really uses them. That was already a real boost and our system was able to execute all of the 245 update tests for the 25.0 release in 25 minutes – across all platforms!
If you are interested in our goals you might want to have a look at Henrik’s post about the Automation Development team goals for Q4.
As noted in the last couple of Automation Development reports, our Windows 8.1 64bit nodes were really unstable and restarted at random times. With a lot of research Henrik was able to identify the problem. Finally it turned out a bug in VMWare vSphere you could circumvent by tweaking the CPUID mask of the machine. All the details about investigation and problem solving can be found on bug 916746.
Together with Adrian from the IT team Henrik setup another slave node for each supported platform of the Mozmill-CI production instance. That means for the 16 different platforms (including 32bit vs. 64bit) we are running 64 slaves now. That is a lot, and the need for Puppet support is more important than ever before. Manually keeping those machines up-2-date is a real pain for me.
More and more we are facing a really nasty behavior of Jenkins when it cannot delete the workspace for all kinds of jobs on Windows. It forces us to log into the machines, delete the whole Jenkins folder, and re-connect the slave. As of now we have no idea what causes this problem, but it might be related to a restart of the slave and some permission problems. Has anyone else seen the same and shed some light on us?
Speaking about Mozmill we are still blocked on some addon-on related issues in mozprofile before we can release version 2.0.1. Henrik is heavily working on those bugs and hopefully we will have a new mozprofile release next week.
Dave participated in the Mozilla Festival this year, and while he was really busy over that weekend, he found the time to get started on the Mozmill CI configuration generator for ondemand tests. We will collect the requirements for the first version soon, and get started to further hack on it.
Letzten Sonntag war es wieder mal soweit. Der alljährliche Dresden Marathon inclusive Alternativstrecken (Halbmarathon und 10km-Lauf) fand statt. Auch ich war wieder mit dabei, musste mich aber jedoch schon vorab von meinem Ziel, einen Halbmarathon zu laufen, verabschieden. Aufgrund Zeitmangels in den letzten Monaten hatte ich nur ein einziges Mal die Chance, diese Distanz zu trainieren und hatte sie auch nur knapp verfehlt. Somit blieb für mich der 10km Lauf übrig. Immerhin die Möglichkeit meine Bestzeit vom letzten Jahr zu übertrumpfen?
Die Tage zuvor waren grau mit teilweise Regen. Düstere Gedanken machten sich breit, da ich nicht wirklich gern im Regen laufe, zumindest nicht zu dieser Jahreszeit mit ca. 10°C Außentemperatur. Aber wir hatten Glück und der Sonntag startete mit strahlendem Sonnenschein und wunderbaren Temperaturen. Aber auch die Masse an teilnehmenden Läufern war beträchtlich. Mehr als 9000 Laufbegeisterte jedes Alters fanden sich ein, um ein gemeinschaftliches Laufgefühl auszuleben. Es war einfach toll.
Des Rennen an sich verlief ganz gut. Vor Allem aufgrund des großen Nachrückens im Startbereich, stand ich relativ weit vorn und die Menge an Läufern verteilte sich gut. Damit konnte ich schon hier mein Tempo von ca. 5:00 min/km durchziehen. Dies setzte sich dann gut über die nächsten Kilometer fort. Vorbei an der Semperoper und den großen Trauben anfeuernder Menschen, entlang des Terassenufers bis hinauf zum Sachsenplatz. Dort gab es wie jedes Jahr die Auftrennung zwischen (Halb-)Marathon und 10km. Ab hier wollte ich eigentlich das Tempo erhöhen, aber aufgrund des mangelnden Trainings der letzten Wochen, war ich nicht wirklich dazu in der Lage. Ein aufkommendes Seitenstechen setzte Riegel vor und ich fiel auf mein normales Tempo zurück. Besser so gut ankommen, als sich kilometerlang damit rumzuquälen. Ich war mächtig erstaunt, als ich kurz danach ein junges, zierliches Mädchen überholt habe, die vollen Eifers nicht viel langsamer als ich gerannt ist. Hut ab!
Weiter ging es nun über die Carolabrücke entlang des Rosengartens bis kurz vor die Albertbrücke, dort hinunter zum Elbufer und entlang des Elbradweges bis zum Palaisgarten. Beim letzten Überqueren der Elbe über die Augustusbrücke beschleunigte ich nun doch noch etwas, um wenigstens auf den letzten Kilometer noch eine bessere Zeit herauszuholen. Die jubelnden Massen im Verlauf der Zielgeraden machen es dann doch etwas einfacher, noch einmal Höchstleistung zu geben.
Letztendlich erreichte ich das Ziel in einer guten Verfassung mit 49:38min. Somit leider kein neuer Streckenrekord für mich selbst, aber doch eine Zeit, mit der ich zufrieden bin. Mit Platz 46 in meiner Altersgruppe mit 154 Läufern war dies somit im ersten Drittel und etwa gleichauf mit der Gesamtzahl männlicher Läufer (Platz 389 von 1328). Ich bin zufrieden.
Noch glücklicher bin ich, da ich zusammen mit meinen Spendern, aufgrund einer mit dem Lauf verbundenen Spendenaktion des PTV-Sachsen e.V. eine Spendensumme von 215 EUR erlaufen konnte. Die Summe kommt wieder Kindern von sozial schwachen Familien zu Gute. Also vielen, vielen Dank für eure Unterstützung.
Wer als Laufbegeisterter nun noch genaue Informationen zu meinem Lauf sehen möchte, der sollte bitte nachfolgend die Garmin-Daten anschauen:
We were surprised to see a bug for the httpd.js component fixed after more than a year, which drastically improved the execution time of our Mozmill tests. Given that we are still not able to upgrade our CI system to Mozmill 2.0, another release off the 1.5 branch had to go out. So we made Mozmill 1.5.24 available for everyone.
With the help of IT, Henrik was able to make the new Windows 8.1 nodes available for our Mozmill-CI production system. That means we have 4 nodes for the 32bit version and another 4 nodes for 64bit. Sadly we are still facing bad “A critical MSR modification” system crashes for the 64bit nodes, where we think it might be related to our vSphere software, which doesn’t have full support for Windows 8.1 yet. If you have ideas please let us know on bug 916746.
For more granular updates of each individual team member please visit our weekly team etherpad.
If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from this weeks Automation Development meeting.