My 15th Bugzilla account anniversary

Exactly 15 years ago at “2003-06-05 09:51:47 PDT” my journey in Bugzilla started. At that time when I created my account I would never have imagined where all these endless hours of community work ended-up. And even now I cannot predict how it will look like in another 15 years…

Here some stats from my activities on Bugzilla:

Bugs filed 4690
Comments made 63947
Assigned to 1787
Commented on 18579
QA-Contact 2767
Patches submitted 2629
Patches reviewed 3652


Automation Survey Follow-up

As promised in my last post about the automation survey results I wanted to come up with a follow-up to clarify our next steps in being more open for our activities, discussions, and also quarterly goals. Sorry, that it has been taken a bit longer but end of the quarter and especially the year is mostly packed with stuff to finish up. Also the all-hands work week in Orlando beginning of December hold me off from doing a lot real work.

So lets get started with the mailing list topic first. As we have seen most people kinda like to get our news via the automation mailing list. But given the low usage of that list in the last months it was a bit surprising. Nearly all the time I sent emails myself (not to count in Travis results). That means we want to implement a change here. From now on we won’t use the list but instead utilize the list. Also because this is the recommended list for the Engineering Productivity team we are all part of, and discussions will reach a larger audience. So please subscribe to this list via Google Groups or Email.

For status updates about our current activities we started to use last quarter. It seems to work pretty well for us and everyone else is welcome to also post updates to our automation project section. If you are interested in those updates then read through that list or simply subscribe the page in your RSS reader.

Please also note that from now on there will be no Firefox Automation reports anymore. Instead I will reduce the amount of different contents, and only write about projects I worked on. So keep an eye out to not miss those!

Survey about sharing information inside the Firefox Automation team

Within the Firefox Automation team we were suffering a bit in sharing information about our work over the last couple of months. That mainly happened because I was alone and not able to blog more often than once in a quarter. The same applies to our dev-automation mailing list which mostly only received emails from Travis CI with testing results.

Given that the team has been increased to 4 people now (beside me this is Maja Frydrychowicz, Syd Polk, and David Burns, we want to be more open again and also trying to get more people involved into our projects. To ensure that we do not make use of the wrong communication channels – depending where most of our readers are – I have setup a little survey. It will only take you a minute to go through but it will help us a lot to know more about the preferences of our automation geeks. So please take that little time and help us.

The survey can be found here and is open until end of November 2015:

Thank you a lot!

Join our first Automation Training days on March 24/26

Building software is fun. Spending countless hours or even days on something to get it finally working. Helping someone who uses your software to speed-up the daily workflow. All that is fantastic and every developer loves that. But don’t let the dark side come up, when customers are pointing you to a lot of software defects. Are you still proud and will you continue the work as how you did it before?

Most likely not. Or well, lets say at least not when quality is what you want to ship. So you will start to think about how to test your application. You can do a lot of manual testing based on test plans or just do exploratory testing. That will work as long as your application is not complex enough, and can be done in a couple of minutes. But once you have to do the same kind of steps over and over again for each release, you will find it boring and loose the interest or concentration on it. Failures will happen so that issues slip through your testing, and bugs becoming part of the new release.

That’s indeed something you eventually want to kill in the future. But how? There is an easy answer to this question! Use test automation! Create tests for each feature you implement, or regression you got fixed. Over time the enlarged suite of tests will make you happy, given that you have to spend nearly no time on manual tests, and have results in a split of the time needed before. Releasing new versions of the application can be done much faster.

At some point, when your application is large enough, you might even not work alone anymore on that product. There will be other developers, or even software testers whose job is to plan and execute the testing strategy. Given that in the past there was not such a high demand on automation knowledge for them, the requirements for jobs have been changed in the past months. So lesser companies will hire engineers for quality assurance who do not have a coding background. This is hard for them, given that it can take ages to find a new position. Something has to change for them.

We, the Firefox Automation team at Mozilla want to help out here. Given our knowledge in automation for various Mozilla related projects, our goal is to support interested people in gaining their knowledge in software development and especially test automation. Therefor we are planning to have automation trainings on a regular basis. And all based on our own projects, so you will have the chance to practice all the new things, which you have learned. All that indeed depends on the acceptance for that offer, and the number of participants.

The first two training days will happen on March 24th and 26th, and will mainly take place in our #automation channel on IRC. Given that we have no idea how many of you will join us during that day, and what your knowledge is, we will start with the basics. That means we will guide you through courses of Javascript, Python, HTML, or CSS. We will collect your feedback and extend the etherpad for automation trainings to finally have a wonderful list of getting started tutorials.

For those of you who already have more experience, we will offer tasks to work on depending on your skills and directions. Please see the before mentioned etherpad for areas of work and appropriate mentors. We will guarantee that it will be fun!

We would love to see you next week, and if you have questions, don’t hesitate to ask here, or in the automation mailing list.

Automation Development report – week 35 2013

Our last week was again very productive, and a lot of smaller and larger tasks have been finished. We are getting closer to reach our quarterly goals. Our next big step is really to get the final version of Mozmill 2.0 out, and the Metro support finished. Then we can get started with implementing ui driven tests as requested by Mozilla QA.


With the combined work with our Softvision team and a couple of contributors we were able to release two versions of Mozmill in one go. See my blog post for release details of Mozmill 1.5.22 and 2.0rc5. Hopefully the 1.5.22 release will mark the end of life for the 1.5 branch, and that the 2.0rc5 was the last release candidate build for Mozmill 2.0. More to that topic soon.

To be able to move our Mozmill l10n tests from an older Mac Mini, which is currently running in the MV network, to the more stable and IT supported network in the SCL3 datacenter, we had to fix a couple of issues in mozdownload. So as result we have released mozdownload 1.9 last week. That fixed the last remaining blocker, and we will have the l10n tests running on staging soon.

Over the last couple of weeks we were monitoring the memory usage of Jenkins a lot. That happened not only in the browser but also more important for the whole system on the server side too. As we noticed the Java process has a really high heap memory usage. Beside that we had a couple of annoying issues around, which caused some of our testruns to early abort. So Henrik put a lot of work in getting Jenkins upgraded to the latest LTS (Long Term Support) version. Given that this was a big jump for our really outdated Jenkins version a lot of work had to be spent in testing everything. Also because nearly all of the Jenkins plugins, which we make use of, had updates available and had a slightly different format of the XML configuration, all our static files had to be updated too. But by early last week we were able to push those bits to our staging server, and running that version of Jenkins for a week, hasn’t brought up any kind of regression. That’s great! To finish up the work, Henrik published all bits and bytes to our production machine yesterday. We will continue observing the memory usage, and let you know if the situation got better.

Beside that we made a couple of changes to our list of available nodes for the mozmill-ci cluster. As of last week we brought our Windows 8.1 nodes (3x 32bit, 3x 64bit) online for preliminary testing. Given that all went well and no further issues have been seen, they are now fully integrated into jobs triggered by Pulse notifications. Further we have replaced our existent Ubuntu 12.10 nodes with the current release of Ubuntu 13.04. That means we run our tests on Ubuntu 12.04 (LTS) and Ubuntu 13.04 for both 32bit and 64bit now.

As requested by Mozilla QA we have to run more tests for localized beta and release candidate builds of Firefox. That will help us to finally stop releasing broken l10n builds like Firefox 24.0b1 to our users. For all locales covered by our tests, we do now check for start-up failures like the XML parsing error for the main window. So Henrik pushed a patch to production, which enabled tests to be run for the first 10 most used locales of Firefox.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad.

Mozdownload 1.9 released

Oh my! It happened again. And here it is. Another release of mozdownload with a couple of new features included. As a regular user please let us know if all is working well, or if you want to see some new features included.

Below you can find an overview about the main features in mozdownload 1.9:

  • Tinderbox builds can be downloaded by timestamp now
  • For linux64 the 64-bit and not the 32-bit installer gets downloaded
  • Support for the stub installer on Windows has been added
  • Better error messages are displayed

All details for this release can be seen in the changelog.

Thanks again to everyone who was involved in this release!

Automation Development report – week 33 2013

Starting from today, the Automation Development team wants to publish weekly reports about the work happened in the previous week. Reason is that similar to other groups we want to offer everyone more insight into our projects, and that we might find even more community members who are interested to help us and learn new stuff at the same time. If you have comments or proposals to the style and contents please let us know.


Beginning of last week we promoted Andrei Eftimie, who is working for our team as a Softvision contractor from Romania, as peer for our Mozmill Tests repository. That happens in regard of all his contribution in the last couple of months. So if you are looking for a reviewer of your patch, you can flag him from now on. Congratulations again.

Dave Hunt got the performance tests running on the “Leo” devices now. Those are testing the cold load times and fps of the applications under test.

Over the past couple of weeks we detected a lot of application disconnects while running the Mozmill tests for Firefox under the Mozmill-CI system. As it has been turned out notifications (bubbles) from application updates could be the problem, and especially the new update overlay screen for Windows 8 updates. The latter completely blocked our testing. So Henrik Skupin re-configured all existing nodes and templates by disabling any automatic update checks (Bug 900860). Since that has been done, no further application disconnects have been observed. Lets cross the fingers that this will stay that way!

We noticed a massive disconnect of slave nodes in our Mozmill-CI cluster early last week. At the same time the response time of our Jenkins master was drastically slow. Henrik investigated the problem and identified that our current VM is not able to handle the memory load with 2GB of memory. So we increased the memory to 4GB, which indeed stopped the swapping process of consuming about 60% cpu load nearly all the time. Henrik will continue to watch the current memory consumption over the next weeks. Further we will upgrade our Jenkins installation to the latest LTS version soon, once all blockers are fixed.

Individual Updates

For more granular updates of each individual team member please visit our weekly team etherpad.

Join our “Make sure your code works” session @MozCamp

Woot!!1! MozCamp EU 2012 is approaching really fast. Only two days left until a couple of hundred supporters of Mozilla will meet in Warsaw to plan and work on the future of the open web. It’s exciting to be part of it and to see you all again!

My task together with Rob Wood – we both are from the Automation Development team – will be a hacking session about WebAPI test automation. WebAPI’s are a crucial part of the Firefox OS (B2G) initiative, Mozilla is putting a lot of work in to bring the open web even to mobile devices.

If you are interested in raising your skills in automation and want to help us to strengthen the quality of Firefox OS please join us on Saturday at 3:05pm in B2G room 1. We still have a couple of seats left. 🙂 To be best prepared please read through our session details and try to setup your machine before you leave for MozCamp. We have setup links to get all necessary tools and the prepared Ubuntu VM. If you aren’t able to do that, don’t worry, we will have USB thumb drives available for you to grab all the data you will need.

As a sneak-peak we will also give you a link to our introduction slides so that you can change your mind if you are unsure about your attendance:

We are looking forward and can’t await to meet-up with you all. So see you @MozCamp on Saturday!

Update: We are planning to send out prizes for the best hand-crafted test(s) some days after MozCamp when we have reviewed all the written tests. So please be aware of code correctness, coverage, and style while implementing the tests.

Results of our first “Ask an Expert” automation session

It’s was clearly a wow! The way how our first “Ask an Expert” session went was promising. I haven’t thought that we would be able to fill a complete hour with discussions right away – but finally we ended up with additional 15 minutes to get the next steps set for Mike Kaply’s last question how to run automated tests for autoconf.

So what is that amazing on such a session? Given that in the past we as team were mostly working on Bugzilla to get a plan how to solve a specific problem, a live discussion is indeed a lot more helpful. Every attendee can directly participate, ask if there are unclear steps or if there are better ways to accomplish the wanted goal. The discussion is ongoing until you really met your expectation, or the time is up. That’s the difference to threaded conversations via email, where you should expect delays in the turnaround. You will lose valuable time which is even harder when you already have a strong timeline to finish up a feature.

The Automation Development team strongly believes that we all can work together to improve the Test Automation area at Mozilla even more. To achieve that goal we will continue in having live discussions each week and to help others in increasing their skills in automation.

And as a hint for all MozCamp attendees: If you are around keep an eye out for our “Make Sure Your Code Works” hacking session on Saturday at 3:05pm. I will post details about it before MozCamp but just as a heads-up so you can mark it in your calendar.

Weekly “Ask an Expert” Q&A sessions for Test Automation

As you might have noticed in the last couple of months a group of people inside the Automation and Tools Team (A-Team) have been formed to help in making the automation expertise better at Mozilla. We are known as the Automation Development team and currently consists of Dave Hunt, Rob Wood, and myself. If you haven’t met us yet or even never heard our names, please check our team member section where we offer some more information about us and nice pictures. Better now? 🙂

At Mozilla we have dozen of test frameworks in use; like Mochitests, Selenium, Mozmill; which mostly dependents on the product under test. With this large matrix it’s sometimes not that easy to figure out when a test can be automated and which framework is best to use for. Even there can be overlaps because multiple frameworks would fit for a particular test. Confusion is pre-programmed.

So the Automation Development team wants to bring some light into the darkness by offering help for those questions. Therefore we will start to host office hour like sessions each week on Thursday at 12:00pm UTC. This will be done via our Vidyo infrastructure and you will only have to install the Vidyo desktop client as suggested when trying to enter the room the first time. For details and up-2-date information please visit our meeting page.

Given that we don’t exist that long yet and are 3 people only, we will start with the Mozmill, Selenium, and B2G testing frameworks. We will extend our coverage to other frameworks once we have the knowledge and man power, and know that sessions like those are helpful and accepted by others. Also please note that the current time has been chosen to start improving our relationship with community members in Europe first – here most of them are located for the above mentioned frameworks. In the future we are planning to have another session for US folks.

We are kinda excited about this opportunity and we would like to invite everyone who has interest to join and actively participate in the discussion.