MemChaser – An extension to track Firefox’s memory activities

Today I want to announce a new extension for Firefox. It’s been developed by the Mozilla QA Automation Services team, which exists to support the Mozilla QA community by developing tools and frameworks for ease of testing. And this time, in case of the extension, we even target (web) developers, and our huge Firefox user base. So what is it about?

Mid of December 2011 bug 711900 has been filed. It complained about an increased garbage and cycle collector activity in Firefox 10, which was not existent in Firefox 9. So Mozilla QA has been asked to analyze that behavior and find possible use cases for reproduction. While working on it, we noticed that observing the related information about memory usage and garbage/cycle collector activity wasn’t that comfortable. For the memory you have had to constantly watch and refresh ‘about:memory‘ in a separate tab, and the latter even forced us to always have the Error Console open and a specific hidden preference set.

To make it easier for us I had the idea of creating an extension, which would be able to present all the necessary information at a prominent place and would be visible all the time. There should be no need of switching tabs or scraping the Error Console for output. Scraping was even hard because all entries constantly get wiped out when a maximum threshold of messages has been reached. Further copying out this data is a hassle. So lets create an extension with logging capabilities…

Some days later after an internal discussion across teams we started the development and called it the MemChaser project. Based on the fantastic help from members of the SpiderMonkey team – lets name Nicholas Nethercote and Andrew McCreight – we got a very early prototype setup kinda quickly. But as is turned out having an extension, which needs a restart to get installed, is pretty useless if the user is already facing a strange memory related issue. So one of the bigger tasks was to get familiar with the Add-on SDK project aka Jetpack and to transform the extension into a restartless one. As it turned out the work wasn’t that hard but gave us well better structured code and a nice toolchain to test and build the extension. And finally with all the features implemented we were able to release the MemChaser extension on addons.mozilla.org.

The current set of features is not that big but covers all the requirements we got from the QA team. We will make sure to enhance the extension drastically in the next couple of months so that it doesn’t only display current values in the Add-on bar. But what is it doing right now?

As you can see in the image a couple of values are displayed. But what do those stand for? Here the answer:

  • Resident: Memory mapped by the process that is present in physical memory. It’s normally the memory usage you will see in the systems process manager.
  • GC: Duration of the last Garbage Collector activity and in brackets the time when the second last activity happened.
  • CC: Duration of the last Cycle Collector activity and in brackets the time when the second last activity happened.
  • Logging: Starts and stops the internal logger, which writes out the current memory usage and GC/CC activities to a log file, which can be found in the memchaser sub directory of your Firefox profile.

All those information already helps a lot to keep track of memory related activities and enables someone to see if Firefox is using too much memory at all or is having long GC/CC cycles which could freeze the UI of Firefox for short moments.

As I have mentioned above we will continuously enhance the features of the extension. We already have some great ideas and I will certainly blog about those more in the next couple of days, once we have updated our documentation and set milestones for the next releases.

For now please install the extension in your daily profile and watch the memory activities. If you recognize strange behavior turn on logging and let us know by contacting us directly.

Mozmill 1.5.9 released


As noted in my yesterdays blog post about the freeze of Mozmill in the JSBridge module when Python 2.7.2 is used, there was an urgent need for a new version of Mozmill.

Late yesterday we were able to finally release Mozmill 1.5.9 on PYPI. It only includes the fix for bug 722707 and allows everyone who is running on Python 2.7.2 to use Mozmill again. If you are affected please upgrade immediately.

Mozmill Crowd 0.1.5 released

Yesterday we had to release Mozmill Crowd 0.1.5 due to changes in our reporting infrastructure. Given the fallout of one of our servers our Mozmill Dashboard is now located at http://mozmill-crowd.blargon7.com/. This move needed an update of the default report URL. Now you can send reports of your test results again.

We have to admit that not much work happened on the extension in the last couple of months. Reasons are mostly other important projects and mainly the development of the new pre-configured environment for Mozmill. There are still some outstanding improvements to implement before we can release a new version of the environment.

If you are interested to help out please get in contact with us via our team page or any other public channel.

Mozmill 1.5.8 has been released

The final bits of Mozmill 1.5.8 are now available through Pypi. Install or upgrade Mozmill by running:
pip install --upgrade mozmill==1.5.8

We had to release this version to fix a recent regression which has been identified in Mozmill 1.5.7, and which caused a hard-stop when an user restart has been requested. The issue did only exist on OS X.

Beside this fix we now enabled the new boolean preference called ‘focusmanager.testmode’ by default. It will allow us to run multiple Mozmill tests concurrently on the same machine. Support for the focusmanager testing mode has already been made available on Nightly and Aurora builds. Also there is a good chance to get this patch landed in an upcoming Firefox 10.0.1 release. I will follow-up with another blog post with more details once the final landing has happened and we can actually make use of it across our supported branches.

Suppress socket warnings when using rdiff-backup

Since I have been using rdiff-backup to backup my Xen domU machines, I get a daily message from cron which informs me that some sockets couldn’t be backed-up because the path is too long:

SpecialFileError var/spool/postfix/public/flush Socket error: AF_UNIX path too long

After some digging I have found that this is related to the length limitation of the AF_UNIX constant in one of the kernel header files, and which only allows 107 characters. Depending on where you want to backup to the length can exceed.

The solution is simple. Why do you want to backup sockets at all? Those are bound to the current system and will be executed when an application gets started. So I don’t see a reason to backup those files. Therefor just use the --exclude-sockets option for rdiff-backup.

Please note that you will have to run rdiff-backup twice before the failure message from above will no longer appear.

How to fix the OOM killer crashes under Linux

Linux! Linux is great. Linux is Open Source. Any nerd wants to run Linux. But is any part of Linux really that great? This was a good question I wasn’t really able to answer until yesterday. Now I have mixed feelings but understanding the following problem better, gives even a bit more safety, also for my personal life.

During the whole last year I had a lot of situations when one of my virtual machines on the server died due to an OOM killer process. Those crashes were not predictable and happened randomly. Sometimes it didn’t happen for weeks but there were also situations when it crashed after 1 day again. Given that a good list of customers are hosting their websites on it, raises a lot of trouble for me. I did a lot of work in trying to fix particular running services on that host, but nothing helped to stop those crashes. Recently I have even doubled the memory for that machine but without success. It always ran into an out of memory crash.

Given all my former research and attempts to fix the problem, I wasn’t sure what else I could do. But thankfully I have found a website which has the explanation and even offered steps to solve the problem.

So what’s happened? The reason can be explained shortly: The Linux kernel likes to always allocate memory if applications asking for it. Per default it doesn’t really check if there is enough memory available. Given that behavior applications can allocate more memory as really is available. At some point it can definitely cause an out of memory situation. As result the OOM killer will be invoked and will kill that process:

Jun 11 11:35:21 vsrv03 kernel: [378878.356858] php-cgi invoked oom-killer: gfp_mask=0x1280d2, order=0, oomkilladj=0
Jun 11 11:36:11 vsrv03 kernel: [378878.356880] Pid: 8490, comm: php-cgi Not tainted 2.6.26-2-xen-amd64 #1

The downside of this action is that all other running processes are also affected. As result the complete VM didn’t work and needed a restart.

To fix this problem the behavior of the kernel has to be changed, so it will no longer overcommit the memory for application requests. Finally I have included those mentioned values into the /etc/sysctl.conf file, so they get automatically applied on start-up:

vm.overcommit_memory = 2
vm.overcommit_ratio = 80

The results look good so far and I hope it will stay that way. The lesson I have learned is to not trust any default setting of the Linux kernel. It really can result in a crappy and unstable behavior.

Mozmill 1.4.1 released

Today Mozilla proudly presents the next bug fix release for Mozmill. A couple of improvements have been made into this release. The most important part is definitely the support of Firefox 3.7, which allows us to run Mozmill tests against Minefield builds in the near future. But we also support Open Solaris now and offer a much easier way to setup Mozmill on Windows. There is no dependency for pywin32 anymore. And with all the other fixes we really have a shiny new release.

A list of fixes, which have been made it into this release, can be found on Bugzilla. Lets give an explanation for some of those:

  • Bug 542000: With former Mozmill releases there were needs to install the Python for Windows extensions (pywin32). If you have used the MozillaBuild environment, changes to the registry were needed. All that work hasn’t to be done from now on. Installing MozillaBuild is enough to prepare the system for Mozmill. A big thanks goes to Atul Varma who removed that dependency and made our life more compelling.
  • Bug 543501: One of our upcoming projects is the execution of Mozmill tests against add-ons. Therefore we have to make sure that the usage of Mozmill will be as easy as possible. That’s why the misleading –plugin command line option has been changed to –addons. Make sure to use this option from now on.
  • Bug 548446: Mozmill had problems with installing some of the existing extensions because it was not able to find the extension id. Thanks goes to Jonathan!
  • Bug 544896: If you have long test-runs it was possible that the software update dialog came up and removed the focus from the window currently under test. That’s why the automatic update for the application has been disabled. If you want to run tests against that feature make sure to enable the preference app.update.enabled before.
  • Bug 558404 and Bug 559152: Some controller functions have been updated which missed some details about failures. Now we always have a stacktrace with full information available.

The new version of the add-on can be found on addons.mozilla.org. It’s under review right now but can already be installed. For all the others who are using the command line client of Mozmill, you can simply run the “easy_install -U mozmill” command to update to the new 1.4.1 release.

If you have questions don’t hesitate to send your feedback to the mozmill-dev mailing list or directly contact us on IRC in the #qa channel.

Synchronizing Nokia N900 with Mac OS X

About 2 weeks ago I got my new Nokia N900 device for testing purposes. During the next couple of months I will have to run some dogfooding against the mobile Firefox aka. Fennec. To have it handy all the time I will have to use it as my mobile phone. Therefore it’s necessary to synchronize all the relevant information, e.g. Contacts, Calendars, and Tasks, between OS X and the N900.

There is no official plug-in yet which will allow the synchronization with iSync. But after a quick check I have found a plug-in which is still in a beta state but works quiet well for all of my data, and that in both directions. You can download the software from this website. The installation is fast and you will be able to start the first sync process nearly immediately.

Using your Fritz!Box to send a fax

Even when having an ISDN phone connection at home you will be able to send a fax. There is no analog connection needed. The only requirement is an installed Fritz!Box Fon from AVM which acts as your telephone system. Via its configuration web interface you can already specify a number to receive faxes. But also sending a fax is possible in some simple steps.

Download and install Fritz!Fax on your Windows machine. A printer driver is installed automatically which is used to send b/w or colored faxes. Works like a charm.

Thanks AVM even it is an unsupported software.

Synchronizing a Mercurial repository with Git

Some weeks ago a member of the Mozilla community asked me if we can also use Github for our Mozmill test repository which was at this time only available at hg.mozilla.org/qa/mozmill-tests/. I agreed but only with the requirement that there has to be a simple way to synchronize both remote repositories. Then Mikeal Rogers who leads the Mozmill project pointed me to Hg-Git which is a tiny Mercurial plugin and implements the Git protocol.

Yesterday I was finally able to play around with this plugin and promptly run in a couple of problems due to wrong application dependencies. Most of my applications have been installed via MacPorts while for Mozmill and hg-git setuptools has been used. As result Mercurial wasn’t able to find the hg-git plugin. It has been taken nearly half an hour before I figured out whats going on. So for any future reference and as a tutorial for others I will give instructions below how to get this plugin working on your Mac.

To install other software on your Mac it’s mostly helpful to do this via MacPorts. The installation package and needed instructions can be found on their website. Just download the appropriate build for your version of OS X and install it.

Now use MacPorts to install Python and setuptools. The command below can take a while to execute depending on the power of your machine:
$ sudo port install python25 py-setuptools

With setuptools installed you can use easy_install to install all the remaining packages:
$ sudo easy_install mercurial hg-git

To activate the hg-git plugin the following lines have to be added to the .hgrc file in your home directory:
[extensions]
hgext.bookmarks =
hggit =

And that’s all! Now you can use hg-git to synchronize your local copy of the repository with the remote one via hg or git. As example I will use our Mozmill test repository which has two branches (default and mozilla1.9.1).

Create a local copy of the remote repository and change into its folder.
$ hg clone http://hg.mozilla.org/qa/mozmill-tests
destination directory: mozmill-tests
requesting all changes
adding changesets
adding manifests
adding file changes
added 251 changesets with 840 changes to 132 files (+1 heads)
updating to branch default
98 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd mozmill-tests/

Because we have two branches we will have to create two bookmarks. Both are used to map those branches to separate branches in the git repository. Then push all the data to the github repository:
mozmill-tests$ hg bookmark -r default master
mozmill-tests$ hg bookmark -r mozilla1.9.1 1.9.1
mozmill-tests$ hg push git+ssh://git@github.com/whimboo/mozmill-tests.git
pushing to git+ssh://git@github.com/whimboo/mozmill-tests.git
importing Hg objects into Git
creating and sending data
git::refs/heads/mozilla-1.9.1 => GIT:70300757
git::refs/heads/master => GIT:092e7cf3

If you want to pull updates from the Github repository to include those into your local version just run a normal pull command in hg with the appropriate URL:
mozmill-tests$ hg pull git+ssh://git@github.com/whimboo/mozmill-tests.git
pulling from git+ssh://git@github.com/whimboo/mozmill-tests.git
importing Hg objects into Git
Total 0 (delta 0), reused 0 (delta 0)
importing Git objects into Hg
(run 'hg update' to get a working copy)

If you do not want to add the path of the remote github repository each time you can simply add it to the .hg/hgrc file:
cat .hg/hgrc
mozmill-tests$ cat .hg/hgrc
[paths]
default = http://hg.mozilla.org/qa/mozmill-tests
git = git+ssh://git@github.com/whimboo/mozmill-tests.git

And run this command afterward:
mozmill-tests$ hg pull -u git

As you can see it’s really simple to keep your Mercurial and Github repository in sync. Personally I will move to Github now which makes handling patches on different branches much easier.