In this post you can find an overview about the work happened in the Firefox Automation team during week 9 and 10. I for myself was a week on vacation. A bit of relaxing before the work on the TPS test framework should get started.
In preparation to run Mozmill tests for Firefox Metro in our Mozmill-CI system, Andreea has started to get support for Metro builds and appropriate tests included.
With the help from Henrik we got Mozmill 2.0.6 released. It contains a helpful fix for waitForPageLoad(), which let you know about the page being loaded and its status in case of a failure. This might help us to nail down the intermittent failures when loading remote and even local pages. But the most important part of this release is indeed the support of mozcrash. Even that we cannot have a full support yet due to missing symbol files for daily builds on ftp.mozilla.org, we can at least show that a crash was happening during a testrun, and let the user know about the local minidump files.
For more granular updates of each individual team member please visit our weekly team etherpad for week 9 and week 10.
If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 9 and week 10.
The current work load is still affecting my time for getting out our automation status reports. The current updates are a bit old but still worth to mention. So lets get them out.
As mentioned in my last report, we had issues with Mozmill on Windows while running our restart tests. So during a restart of Firefox, Mozmill wasn’t waiting long enough to detect that the old process is gone, so a new instance has been started immediately. Sadly that process failed with the profile already in use error. Reason here was a broken handling of process.poll() in mozprocess, which Henrik fixed in bug 947248. Thankfully no other Mozmill release was necessary. We only re-created our Mozmill environments for the new mozprocess version.
With the ongoing pressure of getting automated tests implemented for the Metro mode of Firefox, our Softvision automation team members were concentrating their work on adding new library modules, and enhancing already existent ones. Another big step here was the addition of the Metro support to our Mozmill dashboard by Andrei. With that we got new filter settings to only show results for Metro builds.
When Henrik noticed one morning that our Mozmill-CI staging instance was running out of disk space, he did some investigation and has seen that we were using the identical settings from production for the build retention policy. Without having more then 40GB disk space we were trying to keep the latest 100 builds for each job. This certainly wont work, so Cosmin worked on reducing the amount of builds to 5. Beside this we were also able to fix our l10n testrun for localized builds of Firefox Aurora.
Given that we stopped support for Mozmill 1.5 and continued with Mozmill 2.0 already a while ago, we totally missed to update our Mozmill tests documentation on MDN. Henrik was working on getting all of it up-to-date, so new community members wont struggle anymore.
For more granular updates of each individual team member please visit our weekly team etherpad for week 7 and week 8.
If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the Firefox Automation meetings of week 7 and week 8.
A lot of things were happening in weeks 5 and 6, and we made some good progress regards the stability of Mozmill.
The unexpected and intermittent Jenkins crashes our Mozmill CI system was affected with are totally gone now. Most likely the delayed creation of jobs made that possible, which also gave Jenkins a bit more breath and not bomb it with hundreds of API calls.
For the upcoming release of Mozmill 2.0.4 a version bump for package dependencies was necessary for mozdownload. So we released mozdownload 1.11. Sadly a newly introduced regression in packaging caused us to release mozdownload 1.11.1 a day later.
After a lot of work for stability improvements we were able to release Mozmill 2.0.4. This version is one with the largest amount of changes in the last couple of months. Restarts and shutdowns of the application is way better handled by Mozmill now. Sadly we noticed another problem during restarts of the application on OS X (bug 966234) which forced us to fix mozrunner.
Henrik released mozrunner 5.34 which includes a fix how mozrunner retrieves the state of the application during a restart. It was failing here by telling us that the application has quit while it was still running. As result Mozmill started a new Firefox process, which was not able to access the still used profile. A follow-up Mozmill release was necessary, so we went for testing it.
As another great highlight for community members who are usually not able to attend our Firefox Automation meetings, we have started to record our meetings now. So if you want to replay the meetings please check our archive.
For more granular updates of each individual team member please visit our weekly team etherpad for week 5 and week 6.
If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the Firefox Automation meetings of week 5 and week 6.
Due to the high work load and a week of vacation I was not able to give some updates for work done by the Firefox Automation team. In the next days I really want to catch up with the reports, and bring you all on the latest state.
After the staging system for Mozmill CI has been setup by IT and Henrik got all the VMs connected, also the remaining Mac Minis for OS X 10.6 to 10.9 have been delivered. That means our staging system is complete and can be used to test upcoming updates, and to investigate failures.
For the production system the Ubuntu 13.04 machines have been replaced by 13.10. It’s again a bit late but other work was stopping us from updating earlier. The next update to 14.04 should become live faster.
Beside the above news we also had 2 major blockers. First one was a top crasher of Firefox caused by the cycle collector. Henrik filed it as bug 956284 and together with Tim Taubert we got it fixed kinda quick. The second one was actually a critical problem with Mozmill, which didn’t let us successfully run restart tests anymore. As it has been turned out the zombie processes, which were affecting us for a while, kept the socks server port open, and the new Firefox process couldn’t start its own server. As result JSBridge failed to establish a connection. Henrik got this fixed on bug 956315
For more granular updates of each individual team member please visit our weekly team etherpad for week 3 and week 4.
If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the Firefox Automation meetings of week 3 and week 4.
I promised to keep up with our updates over the last week but given a major breakage in the freshly released version of Mozmill 2.0.4, I had a full week of work to get the fix out. I promise that during this week I will write reports for the weeks in January.
With the new year our team has been reorganized and we are part of the Mozilla QA team again. That means we will have a way closer relationship to any feature owner, and also working towards in bringing more automation knowledge to everyone. The goals for our team are getting worked out and I will present those in one of my following blog posts. As of now you can find our team page on the Mozilla wiki under Firefox Automation.
Since the landing of all the new features for Mozmill-CI on our staging machine before Christmas, we have no longer experienced any crash of the Jenkins master. Given that Henrik pushed all the changes to our production system. We are totally happy that the incremental updates made our system that stable, and that Mozilla QA doesn’t have cope with outages.
Henrik and Jarek were both working on the mozfile 1.1 release to make it more stable in terms of removing files when those are still in use or don’t have the right permissions set.
For more granular updates of each individual team member please visit our weekly team etherpad for week 1 and week 2.
If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the first Firefox Automation meeting of week 2.
Wow, somehow I totally missed to send out reports for our automation work. Most likely that happened because of the amount of work I had in the past couple of weeks. So for now lets do a final update before the title will be updated to ‘Firefox Automation report’ by the year 2014.
We have released Mozmill 2.0.3 to fix a couple of issues (see dependencies on bug 950831 seen with Firefox Metro builds and our Firefox shutdown code. We pushed those changes together with the releases of mozmill-automation 2.0.3 and the new mozmill-environment files to our mozmill-ci staging instance for baking.
Henrik was able to finish the work in setting up our new mozmill-ci staging instance in the SCL3 datacenter. Please see bug 947108 for details. With it we have the identical environment as the production instance and can see regressions immediately and not when we merge to production, which was pretty bad in the past couple of week. So RIP old staging server!
One of our goals for quarter 3 in 2013 was to setup a web based configutation tool for ondemand testruns in mozmill-ci, which can be used by QA people to trigger testruns for beta and release builds. Cosmin jumped on it and got the first version implemented. You can find a running instance on Github for now. Later we want to make the tool available via http://www.mozqa.com.
To make our mozmill-ci system more stable, Henrik pushed a large set of new features and fixes to the staging instance. Our plan was to let it bake over the Christmas holidays with the hope that Jenkins will run way more stable now.
For more granular updates of each individual team member please visit our weekly team etherpad for week 51 and week 52.
If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the last Automation Development meeting of week 51.
It’s getting closer to Christmas. So here the 2nd last automation development report for 2013. Also please note that all team members including myself, who were dedicated to Firefox Desktop automation, have been transitioned back from the A-Team into the Mozilla QA team. This will enable us to have a better relationship with QA feature owners, and get them trained for writing automated tests for Firefox. Therefor my posts will be named “Firefox Automation” in the future.
With the latest release of Firefox 26.0 a couple of merges in our mozmill-tests repository had to be done. Work involved here was all done by Andreea and Andrei on bug 945708
To get rid of failed attempts to remove files after a testrun with Mozmill, Henrik was working on a new version of mozfile, which includes a method called remove() now. It should be used by any code given that it tries multiple times to remove files or directories if access is getting denied on Windows systems.
We released Mozmill 2.0.2 with a couple of minor fixes and the above mentioned file removal fixes. Beside that mozmill-automation 2.0.2 and new mozmill-environments have been released.
We are still working on the remaining issues with Mozmill 2.0 and are hoping to get them fixed as soon as possible. So that an upgrade to the 2.0.x branch can happen.
For more granular updates of each individual team member please visit our weekly team etherpad for week 49 and week 50.
If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the last two Automation Development meetings of week 49 and week 50.
This is again an overview about the highlights of our work in week 47 and 48 this year. Sorry for the delay of a couple of days but some critical work was holding me off from writing this post.
Henrik and Dave were working on a couple of Mozmill-CI updates, which have been pushed to production. The first batch of features and bug fixes landed in week 47. It included the monitoring plugin for Jenkins, which hopefully will help us to figure out the reasons of Java crashes. Also we can finally run project branches via our CI now, even it can only be done manually as of now. It is important for our upcoming tests for the Holly branch (Firefox Nightly without the Australis feature). The second batch landed by last week was intended to upgrade our system to Mozmill 2.0.1. Sadly we failed in it due to a couple of other failures, which we haven’t seen before on our staging server. So we partly reverted back the latest commits for production and we all are working hard on getting those issues fixed.
With the detected failures by upgrading to Mozmill 2.0.1, Henrik has noticed that one of those existed because of an incompatibility of mozbase with Python 2.6. See bug 944361 for details. To solve this we upgraded our OS X 10.6 boxes to Python 2.7.3, so all machines are running the same version of Python now. As a very nice side-effect we noticed a speed improvement by running our Mozmill tests of about 25%!
Henrik pushed a couple of releases to pypi, which include mozprofile 0.17 with a lot of add-on manager related bug fixes, mozdownload 1.10 (see details), and mozmill 2.0.1 (see details)
To be prepared for executing the first Metro tests with Mozmill we had to prepare the mozmill-tests repository for handling multiple applications. Therefore Andreea refactored nearly the whole repository. You can find details on bug 927397.
For more granular updates of each individual team member please visit our weekly team etherpad for week 47 and week 48.
If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the last two Automation Development meetings of week 47 and week 48.
Yesterday we tried to upgrade our mozmill-ci cluster to the previously released Mozmill 2.0.1. Sadly we failed on the OS X 10.6 machines and had to revert this change. After some investigation I found out that incompatibility issues between Python 2.6 and 2.7.3 were causing this problem in mozprofile. Given the unclear status of Python 2.6 support in mozbase, and a talk in the #ateam IRC channel, I have been advised to upgrade those machines to Python 2.7. I did so after some testing, also because all other machines are running Python 2.7.3 already. So I didn’t expect any fallout. First post upgrade tests have proven this.
The interesting fact I would like to highlight here is that we can see speed improvements by running our tests now. Previously a functional testrun on 10.6 has been taken about 15 minutes. Now after the upgrade it went down to 11 minutes only. That’s an improvement of nearly 27% with Mozmill 1.5.24. With Mozmill 2.0.1 there is a similar drop which is from 8 minutes to 6 minutes.
Given all that and the upcoming upgrade (hopefully soon) of our mozmill-ci system to Mozmill 2.0.1 we will see an overall improvement of 60% (15 minutes -> 6 minutes) per testrun!! This is totally stunning and allows us to run 2.5 times more tests in the same timespan. With it we can further increase our coverage for locales from 20 to 40 for beta and release candidate builds as next step.
After the last report over a two week cycle, I have to follow-up with another one for the weeks 45 and 46. Due to my move I had limited availability and to fix some important other stuff. So hopefully this is the last report over a two weeks period in the near future.
To be able to release Mozmill 2.0.1 as soon as possible Henrik had to fix a lot of existent bugs for mozprofile in conjunction with its add-on manager class. Those fixes were necessary because our restart tests were broken due to an inappropriate clean-up of add-ons after closing Firefox. At the end 10 bugs have been fixed.
We are close to get our first Metro tests running with Mozmill on Windows 8 and 8.1. But before we can really do that, the whole mozmill-tests repository has to be refactored to support multiple applications. Here both Andreea and Andrei are doing most of the work.
Dave and Henrik were both working on a couple of Mozmill-CI issues, which will help us to better diagnose the memory issues and random crashes of the Jenkins Java process. Everything has been merged to our staging server and has to bake a bit before a push to production will happen.
Henrik added a new feature to Mozmill-CI which allows us to run tests for builds off project branches. This will come into play really soon when we have to execute daily tests for the upcoming holly branch, which is mozilla-central without the Australis UX.
Dave released gaiatest 0.19, b2gpopulate 0.11, and b2gperf 0.12 to resolve leaks in performance tests.
For more granular updates of each individual team member please visit our weekly team etherpad for week 45 and week 46.
If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the last two Automation Development meetings of week 45 and week 46.