Tag Archives: firefox automation

Firefox Automation report – week 7/8 2014

The current work load is still affecting my time for getting out our automation status reports. The current updates are a bit old but still worth to mention. So lets get them out.

Highlights

As mentioned in my last report, we had issues with Mozmill on Windows while running our restart tests. So during a restart of Firefox, Mozmill wasn’t waiting long enough to detect that the old process is gone, so a new instance has been started immediately. Sadly that process failed with the profile already in use error. Reason here was a broken handling of process.poll() in mozprocess, which Henrik fixed in bug 947248. Thankfully no other Mozmill release was necessary. We only re-created our Mozmill environments for the new mozprocess version.

With the ongoing pressure of getting automated tests implemented for the Metro mode of Firefox, our Softvision automation team members were concentrating their work on adding new library modules, and enhancing already existent ones. Another big step here was the addition of the Metro support to our Mozmill dashboard by Andrei. With that we got new filter settings to only show results for Metro builds.

When Henrik noticed one morning that our Mozmill-CI staging instance was running out of disk space, he did some investigation and has seen that we were using the identical settings from production for the build retention policy. Without having more then 40GB disk space we were trying to keep the latest 100 builds for each job. This certainly wont work, so Cosmin worked on reducing the amount of builds to 5. Beside this we were also able to fix our l10n testrun for localized builds of Firefox Aurora.

Given that we stopped support for Mozmill 1.5 and continued with Mozmill 2.0 already a while ago, we totally missed to update our Mozmill tests documentation on MDN. Henrik was working on getting all of it up-to-date, so new community members wont struggle anymore.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 7 and week 8.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the Firefox Automation meetings of week 7 and week 8.

Join our first Automation Training days on March 24/26

Building software is fun. Spending countless hours or even days on something to get it finally working. Helping someone who uses your software to speed-up the daily workflow. All that is fantastic and every developer loves that. But don’t let the dark side come up, when customers are pointing you to a lot of software defects. Are you still proud and will you continue the work as how you did it before?

Most likely not. Or well, lets say at least not when quality is what you want to ship. So you will start to think about how to test your application. You can do a lot of manual testing based on test plans or just do exploratory testing. That will work as long as your application is not complex enough, and can be done in a couple of minutes. But once you have to do the same kind of steps over and over again for each release, you will find it boring and loose the interest or concentration on it. Failures will happen so that issues slip through your testing, and bugs becoming part of the new release.

That’s indeed something you eventually want to kill in the future. But how? There is an easy answer to this question! Use test automation! Create tests for each feature you implement, or regression you got fixed. Over time the enlarged suite of tests will make you happy, given that you have to spend nearly no time on manual tests, and have results in a split of the time needed before. Releasing new versions of the application can be done much faster.

At some point, when your application is large enough, you might even not work alone anymore on that product. There will be other developers, or even software testers whose job is to plan and execute the testing strategy. Given that in the past there was not such a high demand on automation knowledge for them, the requirements for jobs have been changed in the past months. So lesser companies will hire engineers for quality assurance who do not have a coding background. This is hard for them, given that it can take ages to find a new position. Something has to change for them.

We, the Firefox Automation team at Mozilla want to help out here. Given our knowledge in automation for various Mozilla related projects, our goal is to support interested people in gaining their knowledge in software development and especially test automation. Therefor we are planning to have automation trainings on a regular basis. And all based on our own projects, so you will have the chance to practice all the new things, which you have learned. All that indeed depends on the acceptance for that offer, and the number of participants.

The first two training days will happen on March 24th and 26th, and will mainly take place in our #automation channel on IRC. Given that we have no idea how many of you will join us during that day, and what your knowledge is, we will start with the basics. That means we will guide you through courses of Javascript, Python, HTML, or CSS. We will collect your feedback and extend the etherpad for automation trainings to finally have a wonderful list of getting started tutorials.

For those of you who already have more experience, we will offer tasks to work on depending on your skills and directions. Please see the before mentioned etherpad for areas of work and appropriate mentors. We will guarantee that it will be fun!

We would love to see you next week, and if you have questions, don’t hesitate to ask here, or in the automation mailing list.

Firefox Automation report – week 5/6 2014

A lot of things were happening in weeks 5 and 6, and we made some good progress regards the stability of Mozmill.

Highlights

The unexpected and intermittent Jenkins crashes our Mozmill CI system was affected with are totally gone now. Most likely the delayed creation of jobs made that possible, which also gave Jenkins a bit more breath and not bomb it with hundreds of API calls.

For the upcoming release of Mozmill 2.0.4 a version bump for package dependencies was necessary for mozdownload. So we released mozdownload 1.11. Sadly a newly introduced regression in packaging caused us to release mozdownload 1.11.1 a day later.

After a lot of work for stability improvements we were able to release Mozmill 2.0.4. This version is one with the largest amount of changes in the last couple of months. Restarts and shutdowns of the application is way better handled by Mozmill now. Sadly we noticed another problem during restarts of the application on OS X (bug 966234) which forced us to fix mozrunner.

Henrik released mozrunner 5.34 which includes a fix how mozrunner retrieves the state of the application during a restart. It was failing here by telling us that the application has quit while it was still running. As result Mozmill started a new Firefox process, which was not able to access the still used profile. A follow-up Mozmill release was necessary, so we went for testing it.

As another great highlight for community members who are usually not able to attend our Firefox Automation meetings, we have started to record our meetings now. So if you want to replay the meetings please check our archive.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 5 and week 6.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the Firefox Automation meetings of week 5 and week 6.

Firefox Automation report – week 3/4 2014

Due to the high work load and a week of vacation I was not able to give some updates for work done by the Firefox Automation team. In the next days I really want to catch up with the reports, and bring you all on the latest state.

Highlights

After the staging system for Mozmill CI has been setup by IT and Henrik got all the VMs connected, also the remaining Mac Minis for OS X 10.6 to 10.9 have been delivered. That means our staging system is complete and can be used to test upcoming updates, and to investigate failures.

For the production system the Ubuntu 13.04 machines have been replaced by 13.10. It’s again a bit late but other work was stopping us from updating earlier. The next update to 14.04 should become live faster.

Beside the above news we also had 2 major blockers. First one was a top crasher of Firefox caused by the cycle collector. Henrik filed it as bug 956284 and together with Tim Taubert we got it fixed kinda quick. The second one was actually a critical problem with Mozmill, which didn’t let us successfully run restart tests anymore. As it has been turned out the zombie processes, which were affecting us for a while, kept the socks server port open, and the new Firefox process couldn’t start its own server. As result JSBridge failed to establish a connection. Henrik got this fixed on bug 956315

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 3 and week 4.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the Firefox Automation meetings of week 3 and week 4.

Firefox Automation report – week 1/2 2014

I promised to keep up with our updates over the last week but given a major breakage in the freshly released version of Mozmill 2.0.4, I had a full week of work to get the fix out. I promise that during this week I will write reports for the weeks in January.

Highlights

With the new year our team has been reorganized and we are part of the Mozilla QA team again. That means we will have a way closer relationship to any feature owner, and also working towards in bringing more automation knowledge to everyone. The goals for our team are getting worked out and I will present those in one of my following blog posts. As of now you can find our team page on the Mozilla wiki under Firefox Automation.

Since the landing of all the new features for Mozmill-CI on our staging machine before Christmas, we have no longer experienced any crash of the Jenkins master. Given that Henrik pushed all the changes to our production system. We are totally happy that the incremental updates made our system that stable, and that Mozilla QA doesn’t have cope with outages.

Henrik and Jarek were both working on the mozfile 1.1 release to make it more stable in terms of removing files when those are still in use or don’t have the right permissions set.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad for week 1 and week 2.

Meeting Details

If you are interested in further details and discussions you might also want to have a look at the meeting agenda and notes from the first Firefox Automation meeting of week 2.