Automation Development report – week 38 2013

The last week was more dedicated for maintenance. We were working on features for mozmill-ci to let us more easily debug failing tests, especially with all the cpu load spikes we have seen across our Windows machines. But we also got the final Mozmill 2.0rc6 release out. Continue reading to find details.

Hightlights

One of our biggest issues last week was definitely the drastic increase of the CPU load across all of our Windows nodes in the mozmill-ci cluster. It magically started when QA did the first ondemand tests for the upcoming Firefox 25.0b1 release. Especially the Windows 8.1 machines started to sporadically disconnect from the Jenkins master over the day, caused by some blue screen crashes related to the kernel. At the end this happened hourly or even more often, when the machines had to run tests. Observing the task manager has been shown that specifically the Windows Defender real-time protection is causing a high load, which e.g. caused a slow-down for installing Firefox from about 6 seconds to 4 minutes!! Disabling the real-time protection helped us across all Windows nodes, but not for Windows 8.1, which was still crashing and rebooting. But also other processes like svchost for network services or the desktop manager showed a high CPU load. Those we weren’t able to figure out yet. Stopping to run tests on Windows 8.1 preview release seemed to be the only option for us. Now we are waiting for VMs with the final version of Windows 8.1 installed. Lets see if that will work better.

Beside that our production CI instance got a couple of nice new features. At first Henrik added support for the mozmill-automation package, which comes with the latest versions of the mozmill-environment. With it we have a defined release schema now, and do no longer pull automation changes from the Mercurial repository for each change. This allows us to deeply test the scripts before the release, and will not cause that massive test failures when a mistake was made. Further Dave landed features like timestamp output in the console log, listing of injected environment variables, and HTTP logging for our update tests. Especially the latter should help us a lot while investigating update test failures. Last but not least Henrik also landed ANSI color support for the console, which will be used by Mozmill 2.0 to show colored output for test results.

Beside all that we also released the next release candidate for Mozmill, which is v2.0rc6. Read this thread for more details. Just to add, the previously setup mozmill-ci instance for Mozmill 2.0 was of big help here, given that we discovered a couple of unknown issues and were able to fix them all in a quick turnaround. With that release we also pushed mozmill-automation 2.0rc6 and an updated mozmill-env.

For Gaia we were able to finalize a very important step, we worked on for a long time and which will help everyone working on that product. It’s the move of the gaia-ui-tests from the separate github repository into the gaia main repository. With it we have always tests, which are synced with the latest feature additions and bug fixes in Gaia. Also developers can more easily run those tests, and we can detect regressions before they actually land in the repository.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad.