Given that I was partly away in the last two weeks I haven’t had the time to write another summary report of our work yet. So lets combine the last two weeks in a single blog post this time.
To be able to continue our work to speed-up testruns in Mozmill CI Henrik upgraded our Mozmill-CI system to Jenkins 1.509.4. This step was necessary so that we can investigate the slowness in sending out emails for finished testruns, which could take up to 25 minutes! Beside that Henrik also disabled the sending of emails for successful testruns given that those were mostly blocking us, and no-one really uses them. That was already a real boost and our system was able to execute all of the 245 update tests for the 25.0 release in 25 minutes – across all platforms!
If you are interested in our goals you might want to have a look at Henrik’s post about the Automation Development team goals for Q4.
As noted in the last couple of Automation Development reports, our Windows 8.1 64bit nodes were really unstable and restarted at random times. With a lot of research Henrik was able to identify the problem. Finally it turned out a bug in VMWare vSphere you could circumvent by tweaking the CPUID mask of the machine. All the details about investigation and problem solving can be found on bug 916746.
Together with Adrian from the IT team Henrik setup another slave node for each supported platform of the Mozmill-CI production instance. That means for the 16 different platforms (including 32bit vs. 64bit) we are running 64 slaves now. That is a lot, and the need for Puppet support is more important than ever before. Manually keeping those machines up-2-date is a real pain for me.
More and more we are facing a really nasty behavior of Jenkins when it cannot delete the workspace for all kinds of jobs on Windows. It forces us to log into the machines, delete the whole Jenkins folder, and re-connect the slave. As of now we have no idea what causes this problem, but it might be related to a restart of the slave and some permission problems. Has anyone else seen the same and shed some light on us?
Speaking about Mozmill we are still blocked on some addon-on related issues in mozprofile before we can release version 2.0.1. Henrik is heavily working on those bugs and hopefully we will have a new mozprofile release next week.
Dave participated in the Mozilla Festival this year, and while he was really busy over that weekend, he found the time to get started on the Mozmill CI configuration generator for ondemand tests. We will collect the requirements for the first version soon, and get started to further hack on it.