Memchaser 0.6 has been released

The Firefox Automation team would like to announce the release of memchaser 0.6. After nearly a year of no real feature updates, but also some weeks of not being able to run Memchaser in Firefox Aurora (34.0a2) at all (due to a regression), we decided to release the current state of development as a public release. We are aware that we still do not fully support the default Australis theme since Firefox 29.0, but that’s an issue, which takes some more time to finish up.

Changes in 0.6

  • Upgrade to Add-on SDK 1.17 (#201)
  • Fix test_start_stop_logging for ‘File not found’ error (#199)
  • contentURL expects a string in Panel() and Widget() constructors (#198)
  • Bring the minimizeMemory() implementation up to date (#193)
  • Use ‘residentFast’ instead of ‘resident’ for memory reporter (#192)
  • Use postMessage for widget and panel communication instead port object (#185)
  • Support incremental cycle collection statistics (#187)
  • Added a usage section to the README (#178)
  • Change require statements and update the SDK to 1.14 (#174)
  • Simplify .travis.yml (#172)
  • Use . (dot) to include files while invoking /bin/sh (#169)
  • Use failonerror attribute in exec tasks (#168)

For all the details about the 0.6 release please check our issue tracker on Github.

MemChaser 0.3 is out

Late on Thursday we got our MemChaser 0.3 release out the door. Compared to the last releases this version comes with a ton of new features and bug fixes. Most of them have been requested by users and from the JS team so we put some extra focus on those.

As you can see in the image new ui elements have been added. The memory and garbage collector related items are equivalent to the buttons in ‘about:memory‘ and help those users who want to trigger a memory clean-up more often but don’t want to keep the before mentioned page open. The reaction of Firefox and the updated memory and GC/CC values can be seen in the MemChaser widget right after the triggered action has been finished.

Also we have updated our code which retrieves GC/CC related information to use the new Garbage Collector API. This allows us to get all the information and not have to scrape the console messages anymore for particular elements. That makes our life a lot easier and as benefit we can even store the complete set of data to the log file for further investigation. This log file is now a valid JSON file and will make it easier for you to analyze the data. If you wondered (with the former versions of MemChaser) where the log files have been stored, you can now open the containing folder via the new panel element too. If you are not satisfied with the location go ahead and change the target folder in the add-on preferences to any folder you like.

To help new users of the extension to understand the entries in the widget, tooltips have been added. Those will inform you about the background of the individual values. This was a request by a couple of users so we wanted to have this included too.

If you are interested in all the changes happened in the 0.3 release you can find the complete list as usual on our MemChaser issue tracker.

This version of MemChaser has still been built on top of the Add-on SDK 1.5, because we wanted to get out all of those new features also to our Firefox 10.0 user base which is about 28% of all our users. This was more important to us compared to fixes we would have benefit from the SDK 1.6. But at this time we want to announce that version 0.3 will be the last release which officially supports Firefox 10. With version 0.3.1 which will be released in the next one or two weeks a version bump to the SDK 1.6 will be done to fix the before mentioned issues like memory leaks.

If you are interested in helping us in the implementation of new features or fixing bugs, please check our issue tracker for open issues and contact us on IRC in the #automation channel.

MemChaser 0.2 released

Exactly one month after we have released our initial version of MemChaser, version 0.2 has been made publicly available. You can install the add-on as usual from addons.mozilla.org or if it is already installed, simply check for updates within the Add-ons Manager.

A couple of subtle changes have been made which will give a better experience for users. So we have combined the formerly two widgets in the Add-on bar into a single one to prevent other extensions from inserting their widgets in-between. At the same time the width has been reduced to not waste too much space (keep in mind that we are still blocked on a SDK bug which doesn’t let the widget size adapt to the content length dynamically).

Given the recent landing of the incremental garbage collector in Nightly builds of Firefox we had to do some extra unplanned updates to support this new feature. When you are running a Nightly build and an incremental GC happens, the ‘GC‘ label will be replaced with ‘iGC‘.

Further we have improved the parsing of the GC/CC console messages to allow us to add more interested entries dynamically. So this time we have added the reason and type (global or compartmental) for a GC, and the number of collected entries from the cycle collector. All three will be written together with all the other data to the log file and are not visible in the add-on bar.

As reported by an user MemChaser v0.1.1 was affected by some memory leaks which were caused by the Add-on SDK 1.4. Given the tireless efforts of the SDK devs nearly all of those leaks (except for widgets) have been fixed in version 1.5, which has been recently released and is now used by MemChaser 0.2.

Beside all those visible changes we also made some improvements to our development process. So we have added the Add-on SDK as a submodule to our own repository on Github. Also to get more traction we made the decision to move the MemChaser repository to the Mozilla account. And last but not least we implemented the Travis CI integration to get our code automatically tested with each check-in.

If you are interested in further information about the changes feel free to check our list of issues for 0.2.

We would also like to welcome everyone who has interests to help us with coding, ui work, or documentation. There are a lot of features we want to implement and need talented people for. Stop by on one of our channels and get in contact with us.

MemChaser – An extension to track Firefox’s memory activities

Today I want to announce a new extension for Firefox. It’s been developed by the Mozilla QA Automation Services team, which exists to support the Mozilla QA community by developing tools and frameworks for ease of testing. And this time, in case of the extension, we even target (web) developers, and our huge Firefox user base. So what is it about?

Mid of December 2011 bug 711900 has been filed. It complained about an increased garbage and cycle collector activity in Firefox 10, which was not existent in Firefox 9. So Mozilla QA has been asked to analyze that behavior and find possible use cases for reproduction. While working on it, we noticed that observing the related information about memory usage and garbage/cycle collector activity wasn’t that comfortable. For the memory you have had to constantly watch and refresh ‘about:memory‘ in a separate tab, and the latter even forced us to always have the Error Console open and a specific hidden preference set.

To make it easier for us I had the idea of creating an extension, which would be able to present all the necessary information at a prominent place and would be visible all the time. There should be no need of switching tabs or scraping the Error Console for output. Scraping was even hard because all entries constantly get wiped out when a maximum threshold of messages has been reached. Further copying out this data is a hassle. So lets create an extension with logging capabilities…

Some days later after an internal discussion across teams we started the development and called it the MemChaser project. Based on the fantastic help from members of the SpiderMonkey team – lets name Nicholas Nethercote and Andrew McCreight – we got a very early prototype setup kinda quickly. But as is turned out having an extension, which needs a restart to get installed, is pretty useless if the user is already facing a strange memory related issue. So one of the bigger tasks was to get familiar with the Add-on SDK project aka Jetpack and to transform the extension into a restartless one. As it turned out the work wasn’t that hard but gave us well better structured code and a nice toolchain to test and build the extension. And finally with all the features implemented we were able to release the MemChaser extension on addons.mozilla.org.

The current set of features is not that big but covers all the requirements we got from the QA team. We will make sure to enhance the extension drastically in the next couple of months so that it doesn’t only display current values in the Add-on bar. But what is it doing right now?

As you can see in the image a couple of values are displayed. But what do those stand for? Here the answer:

  • Resident: Memory mapped by the process that is present in physical memory. It’s normally the memory usage you will see in the systems process manager.
  • GC: Duration of the last Garbage Collector activity and in brackets the time when the second last activity happened.
  • CC: Duration of the last Cycle Collector activity and in brackets the time when the second last activity happened.
  • Logging: Starts and stops the internal logger, which writes out the current memory usage and GC/CC activities to a log file, which can be found in the memchaser sub directory of your Firefox profile.

All those information already helps a lot to keep track of memory related activities and enables someone to see if Firefox is using too much memory at all or is having long GC/CC cycles which could freeze the UI of Firefox for short moments.

As I have mentioned above we will continuously enhance the features of the extension. We already have some great ideas and I will certainly blog about those more in the next couple of days, once we have updated our documentation and set milestones for the next releases.

For now please install the extension in your daily profile and watch the memory activities. If you recognize strange behavior turn on logging and let us know by contacting us directly.

How to fix the OOM killer crashes under Linux

Linux! Linux is great. Linux is Open Source. Any nerd wants to run Linux. But is any part of Linux really that great? This was a good question I wasn’t really able to answer until yesterday. Now I have mixed feelings but understanding the following problem better, gives even a bit more safety, also for my personal life.

During the whole last year I had a lot of situations when one of my virtual machines on the server died due to an OOM killer process. Those crashes were not predictable and happened randomly. Sometimes it didn’t happen for weeks but there were also situations when it crashed after 1 day again. Given that a good list of customers are hosting their websites on it, raises a lot of trouble for me. I did a lot of work in trying to fix particular running services on that host, but nothing helped to stop those crashes. Recently I have even doubled the memory for that machine but without success. It always ran into an out of memory crash.

Given all my former research and attempts to fix the problem, I wasn’t sure what else I could do. But thankfully I have found a website which has the explanation and even offered steps to solve the problem.

So what’s happened? The reason can be explained shortly: The Linux kernel likes to always allocate memory if applications asking for it. Per default it doesn’t really check if there is enough memory available. Given that behavior applications can allocate more memory as really is available. At some point it can definitely cause an out of memory situation. As result the OOM killer will be invoked and will kill that process:

Jun 11 11:35:21 vsrv03 kernel: [378878.356858] php-cgi invoked oom-killer: gfp_mask=0x1280d2, order=0, oomkilladj=0
Jun 11 11:36:11 vsrv03 kernel: [378878.356880] Pid: 8490, comm: php-cgi Not tainted 2.6.26-2-xen-amd64 #1

The downside of this action is that all other running processes are also affected. As result the complete VM didn’t work and needed a restart.

To fix this problem the behavior of the kernel has to be changed, so it will no longer overcommit the memory for application requests. Finally I have included those mentioned values into the /etc/sysctl.conf file, so they get automatically applied on start-up:

vm.overcommit_memory = 2
vm.overcommit_ratio = 80

The results look good so far and I hope it will stay that way. The lesson I have learned is to not trust any default setting of the Linux kernel. It really can result in a crappy and unstable behavior.