Tag Archives: mozmill

Firefox Automation report – week 25/26 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 25 and 26.

Highlights

June the 11th was actually the last Automation Training day for our team in Q3. About the results you can read here. We will implement some changes for the next quarter, when we most likely want to host 2 of them.

Henrik finally got the time to upgrade our Mozmill-CI systems to the lastest LTS version of Jenkins. There were a bit of changes necessary but in general all went fine this time, and we can see some great improvements. Especially the long delays when sending out job results seem to be gone.

Further Henrik investigated the slow behavior with the mozmill-ci production master, when it is under load, e.g. QA runs ondemand update tests for releases of Firefox. The main problem stays with Java, which is taking up about 100% of the CPU. Because of this the integrated web server cannot serve pages in a timely manner. Adding a 2nd CPU to this node gave us way better response times.

Given that the new version of Ubuntu came out already in April, we want to have our Mozmill tests also run on that platform version. So we got new VM spun-up by IT, which we now have to puppetize and bring online. But this may still take a bit, given the remaining blockers for using PuppetAgain.

While talking about Puppet we got the next big change reviewed and landed. With bug 1021230 we now have our own user account, which can be customized to our needs. And that’s what we totally need, given that our infrastructure is so different from the Releng one.

Also for TPS we made progress, so the new TPS-CI production machine came online. Yet it cannot replace the current CI due to still a fair amount of open blockers, but hopefully by end of July we should be able to turn the switch.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 25 and week 26.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 25 and week 26.

Firefox Automation report – week 23/24 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 23 and 24.

Highlights

To continue the training for Mozilla related test frameworks, we had the 3rd automation training day on June 4th. This time lesser people attended, but we were still able to get a couple of tasks done on oneanddone.

Something which bothered us already for a while, is that for our mozmill-tests repository no push_printurl hook was setup. As result the landed changeset URL gets not printed to the console during its landing. Henrik fixed that on bug 1010563 now, which allows an easier copy&paste of the link to our bugs.

Our team started to work on the new continuous integration system for TPS tests. To be able to manage all the upcoming work ourselves, Henrik asked Jonathan Griffin to move the Coversheet repository from his own account to the Mozilla account. That was promptly done.

In week 24 specifically on June 11th we had our last automation training day for quarter 2 in 2014. Given the low attendance from people we might have to do some changes for future training days. One change might be to have the training on another day of the week. Andreea probably will post updates on that soon.

Henrik was also working on getting some big updates out for Mozmill-CI. One of the most important blockers for us was the upgrade of Jenkins to the latest LTS release. With that a couple of issues got fixed, including the long delays in sending out emails for failed jobs. For more details see the full list of changes.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 23 and week 24.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 23 and week 24.

Firefox Automation report – week 21/22 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 21 and 22.

Highlights

To assist everyone from our community to learn more about test automation at Mozilla, we targeted 4 full-day automation training days from mid of May to mid of June. The first training day was planned for May 21rd and went well. Lots of [people were present and actively learning more about automation[https://quality.mozilla.org/2014/05/automation-training-day-may-21st-results/). Especially about testing with Mozmill.

To support community members to get in touch with Mozmill testing a bit easier, we also created a set of one-and-done tasks. Those start from easy tasks like running Mozmill via the Mozmill Crowd extension, and end with creating the first simple Mozmill test.

Something, which hit us by surprise was that with the release of Firefox 30.0b3 we no longer run any automatically triggered Mozmill jobs in our CI. It took a bit of investigation but finally Henrik found out that the problem has been introduced by RelEng when they renamed the product from ”’firefox”’ to ”’Firefox”’. A quick workaround fixed it temporarily, but for a long term stable solution we might need a frozen API for build notifications via Mozilla Pulse.

One of our goals in Q2 2014 is also to get our machines under the control of PuppetAgain. So Henrik started to investigate the first steps, and setup the base manifests as needed for our nodes and the appropriate administrative accounts.

The second automation training day was also planned by Andreea and took place on May 28th. Again, a couple of people were present, and given the feedback on one-and-done tasks, we fine-tuned them.

Last but not least Henrik setup the new Firefox-Automation-Contributor team, which finally allows us now to assign people to specific issues. That was necessary because Github doesn’t let you do that for anyone, but only known people.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 21 and week 22.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 21 and week 22.

Firefox Automation report – week 19/20 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 19 and 20.

Highlights

When we noticed that our Mozmill-CI production instance is quickly filling up the /data partition, and having nearly no space left to actually run Jenkins, Henrik did a quick check, and has seen that the problem were the update jobs. Instead of producing log files with about 7MB in size, files with more than 100MB each were present. Inspecting those files revealed that the problem were all the SPDY log entries. As a fix Henrik reduced the amount of logging information, so it is still be helpful but it’s not exploding.

In the past days we also have seen a lot of JSBridge disconnects while running our Mozmill tests. Andrei investigated this problem, and it turned out that the reduced delay for add-on installations were the cause of it. Something is most likely messing up with Mozmill and our SOCKS server. Increasing the delay for that dialog fixed the problem for now.

We are using Bugsahoy for a long time now, but we never actually noticed that the Github implementation was somewhat broken when it comes to filtering for languages. To fix that Henrik added all the necessary language mappings. After updating the Github labels for all of our projects, we were seeing a good spike of new contributors interested to work with us.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 19 and week 20.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 19 and week 20.

Firefox Automation report – week 17/18 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 17 and 18.

Highlights

In one of the last automation reports I mentioned that we wanted to finish up the already in-progress Mozmill tests for Firefox Metro. With lots of work mostly done by Daniel we were finally able to close all the bugs as fixed. Now everyone is back on top of our new quarterly goals.

To allow new contributors of the mozmill-tests repository to pick appropriate reviewers for their patches more easily, Henrik requested to setup default reviewers on Bugzilla. The changes are now in place and the list points to Andreea, Andrei, and Henrik.

To follow our goals in educating more people about automation, we also have to reach out more on Github to actually find interested people. Therefore Henrik requested a Github mirror of our mozmill-tests repository. It’s available at https://github.com/mozilla/qa-mozmill-tests now.

With the last release of Mozmill 2.0.6 we got the mozcrash packages included. So far we aren’t able to process stacktraces, given that the necessary crashreporter symbols were not available for daily builds of Firefox on the FTP server. This has been fixed now for all kinds of daily builds.

As seen for the last beta and final releases of Firefox we still have a good amount of broken configuration files for ondemand update tests in Mozmill CI. Those are failures done by the person who runs those tests, but also by broken behavior in the mciconf tool. Given that, we want to put more focus on the full automation stack for update tests now. To get started we need appropriate notifications send out by Mozilla Pulse.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 17 and week 18.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 17 and week 18.

Firefox Automation report – week 15/16 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 15 and 16.

Highlights

To be able to support the new update path for Firefox Beta users via a final release candidate on the beta channel, we had to update our update testruns with Mozmill in the mozmill-automation repository. Further we also had to add support for this update path in Mozmill CI. Cosmin spend his time on those patches, and finally we got those out and active. That means from now on our software update tests can handle Firefox X.0b9 -> Firefox X.0RC -> Firefox %X+1%.0b1 updates. More information can be found on bug 960781.

We also pushed some other updates for Mozmill-CI to our production system, which includes timestamps for the log output of the pulse listener, and the addition of the application name for the product to test. The latter was necessary for MetroFirefox. But even after we stopped support for it, we wanted to see it landed given that we could add other products more easily in the future.

Also Henrik pushed a change for Mozmill-CI, which removed support for running localized versions of Firefox Nightly builds. This was necessary so that our systems can be better utilized for those Aurora, and Beta builds – where actually most of the localization work is happening.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 15 and week 16.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 15 and week 16.

Firefox Automation report – week 13/14 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 13 and 14.

Highlights

Finally we were able to upgrade our mozmill-ci production system to Mozmill 2.0.6. The only caveat is that we had to disable one test for cleaning history to prevent the always occurring Flash crash on Windows.

This week Henrik was able to land the first fixes for broken TPS tests. Together with Andrei we were able to fix 4 of them.

Also we had our first Automation Training days. The first one on Monday was a huge success, and we have seen lots of new faces. Sadly the second one on the following Wednesday we haven’t announced separately, so lesser people joined our training. In the future we will make sure, to announce each Automation Training day separately. Also there will be only a single one in a full week, so it will allow people to do some homework until the next session.

In week 14 we released version 2.0.6.1 of our mozmill-automation package. It was necessary to allow QA to run the Mozmill update tests with overriding the update channel, so update tests from a release candidate to the next beta build can be performed.

After we discovered even more Flash crashes of the same type, we decided to use the debug version of Flash on our Windows systems. Main reason is that those builds don’t crash. Sadly, given that they would give us way more information. But Adobe should at least know where the crash is happening. Also given that we do not have any private data on our test machines, we decided to actually upload a minidump from one of those crashes to Bugzilla. This information actually helped Adobe a lot! So we will continue doing that.

Last but not least Henrik was able to fix another 7 bugs for TPS. Those were broken tests, a broken behavior of the TPS test extension, or enhancements.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 13 and week 14.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 13 and week 14.

Firefox Automation report – week 11/12 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 11 and 12.

Highlights

After Henrik got started on updating the TPS test framework for Firefox Sync tests, he got it into a state that it is working with the new Firefox Accounts feature coming with Firefox 29.0. For details please see bug 966434. Fixing the backward compatibility for the old Sync authentication is his next step.

With the release of Mozmill 2.0.6 and the initial support of handling crashes, we found a major Adobe Flash crash on our Windows nodes when clearing cookies. We started to investigate and reported the problem appropriately, so that Adobe is aware of it.

For other crashes which are related to Firefox directly we would still have to retrieve and print the stack to the console. But we cannot do this until the crashreporter symbols are available for Nightly builds on ftp.mozilla.org. We will continue implementing this feature into Mozmill once the former bug has been fixed.

Given that the release of Firefox 28 was upcoming, Andreea has taken care of all the mozmill-tests related branch merges.

To be prepared for our first Automation Training days, Cosmin updated our Mozmill Crowd extension to ease the first steps for new contributors to our projects. Now the extension is working again for all the latest Firefox releases.

Even with the Firefox Metro projects seeing canceled, we want to finalize our nearly ready Mozmill tests for Firefox Metro and get them landed. That way we can pick up development, if necessary in the future. So Daniel was able to get additional 4 tests finished by this week.

In week 12 Henrik finished the basic implementation work for TPS in being able to switch between Firefox Accounts and the old Sync authentication. Both methods are working now and can be used for testing. As next step we have to investigate and fix all the remaining failing TPS tests.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 11 and week 12.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 11 and week 12.

Firefox Automation report – week 9/10 2014

In this post you can find an overview about the work happened in the Firefox Automation team during week 9 and 10. I for myself was a week on vacation. A bit of relaxing before the work on the TPS test framework should get started.

Highlights

In preparation to run Mozmill tests for Firefox Metro in our Mozmill-CI system, Andreea has started to get support for Metro builds and appropriate tests included.

With the help from Henrik we got Mozmill 2.0.6 released. It contains a helpful fix for waitForPageLoad(), which let you know about the page being loaded and its status in case of a failure. This might help us to nail down the intermittent failures when loading remote and even local pages. But the most important part of this release is indeed the support of mozcrash. Even that we cannot have a full support yet due to missing symbol files for daily builds on ftp.mozilla.org, we can at least show that a crash was happening during a testrun, and let the user know about the local minidump files.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 9 and week 10.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda, the video recording, and notes from the Firefox Automation meetings of week 9 and week 10.

Firefox Automation report – week 7/8 2014

The current work load is still affecting my time for getting out our automation status reports. The current updates are a bit old but still worth to mention. So lets get them out.

Highlights

As mentioned in my last report, we had issues with Mozmill on Windows while running our restart tests. So during a restart of Firefox, Mozmill wasn’t waiting long enough to detect that the old process is gone, so a new instance has been started immediately. Sadly that process failed with the profile already in use error. Reason here was a broken handling of process.poll() in mozprocess, which Henrik fixed in bug 947248. Thankfully no other Mozmill release was necessary. We only re-created our Mozmill environments for the new mozprocess version.

With the ongoing pressure of getting automated tests implemented for the Metro mode of Firefox, our Softvision automation team members were concentrating their work on adding new library modules, and enhancing already existent ones. Another big step here was the addition of the Metro support to our Mozmill dashboard by Andrei. With that we got new filter settings to only show results for Metro builds.

When Henrik noticed one morning that our Mozmill-CI staging instance was running out of disk space, he did some investigation and has seen that we were using the identical settings from production for the build retention policy. Without having more then 40GB disk space we were trying to keep the latest 100 builds for each job. This certainly wont work, so Cosmin worked on reducing the amount of builds to 5. Beside this we were also able to fix our l10n testrun for localized builds of Firefox Aurora.

Given that we stopped support for Mozmill 1.5 and continued with Mozmill 2.0 already a while ago, we totally missed to update our Mozmill tests documentation on MDN. Henrik was working on getting all of it up-to-date, so new community members wont struggle anymore.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 7 and week 8.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the Firefox Automation meetings of week 7 and week 8.