Category Archives: Software

Crusial SSD bug and firmware fix

About 2.5 years ago I got a Thinkpad X230 as working machine. It’s quiet a decent one given the hardware specs and since then I was happy with it even I’m not able to use the touchpad at all. The quality is just so poor compared to an Apple Macbook which I owned before. But anyway that’s another topic…

What I want to talk about here is more my experience with the hard drive. When I was asked what kind of drive I wanna have I told that it should be a SSD drive. Benefits are the speed improvements and no risks in a hard-drive crash when traveling or walking around while the laptop is turned on. Especially the speed is amazing and I’m happy to see that my installed Ubuntu starts in less than 20s and copying small files goes with an amazing speed too.

So all was working fine since then but a couple of months ago I noticed the first glitches. Sometimes when I take my laptop out of the docking station or put it back into it the system seem to hang for a short moment. Something similar also happened while walking around and e.g. playing a movie. Usually it took up to 5s and then all was working fine again. But then one more issue appeared which let my LVM partitions remount in read-only state. This was quite frustrating because an additional fsck would have to be run during each reboot due to some inconsistencies of my ext4 journal data. Even more frustrating was that in some irregular intervals my LightDM display manager froze and needed a restart by logging into my machine via SSH from a different machine. If I had no other machine available a reboot of my laptop was necessary due no keyboard press was accepted.

All this came to a final end by last Friday when I wanted to participate in a debugging theme night in the local CCC (chaos computer club) hacker space. Once arrived there I wanted to boot up my laptop but I haven’t seen more than a black screen after the initial BIOS routines were done. It didn’t went away after a couple of cold restarts. So actively participating was only a dream. Also the first question: “Looks like your SSD died. How is your backup doing?”, which I actually got from different people was kinda expected. :) Thankfully I was able to answer that it is only 2 days old… But even with this a lot of work would be ahead of me. Anyway, I will have to think about if I wanna go back there after that bad omen…

Being back at home and having the machine turned off for a while it actually booted again. Hurray! I was confronted with all the orphaned nodes again across the partitions which I had to repair first, but then the system started up as usual. After creating another backup I did some more investigations to see if the hard-drive is reporting any kind of failure. But all looked fine as before. All entries of the smart status didn’t show any failure. So I did a long test with ‘smartctl’ and once I got the full report, I saw a line which told that my SSD might be affected by a firmware bug which let it stop working until the the next reboot. And during a reboot even a BSOD could happen. The problem will start to occur after the first 5000 hours on-time usage and will will continue to happen even with a drive reset. A full description can be found at tomshardware website.

After reading the details and the announcements from Crucial about this problem I look at the firmware of my SSD drive and noticed that it is years old. It’s version was “000F” which looks to be the second version of the firmware of this drive. Since then numerous updates have been released. So it was clear to me that an update of the firmware was necessary. I finally did that with a bootable CD-R and all went smoothly. The necessary restart didn’t show a problem, and even I was confronted with orphaned nodes (most likely from the last shutdown) the system works perfectly. I did various tests for all those situations which caused my machine to fail in the past. All of them passed. No more hanging videos while walking around, and also no read-only remounted file systems after taking the machine out of the docking station or putting it back into. Also no freeze of LightDM has been seen so far!

I for myself will learn from it and will check for firmware updates way more often. Also including the hard-drive now.

One more thing you might wanna do – if you own a SSD and run Linux – is to update several mounting parameters to extend your disk lifetime.

Memchaser 0.6 has been released

The Firefox Automation team would like to announce the release of memchaser 0.6. After nearly a year of no real feature updates, but also some weeks of not being able to run Memchaser in Firefox Aurora (34.0a2) at all (due to a regression), we decided to release the current state of development as a public release. We are aware that we still do not fully support the default Australis theme since Firefox 29.0, but that’s an issue, which takes some more time to finish up.

Changes in 0.6

  • Upgrade to Add-on SDK 1.17 (#201)
  • Fix test_start_stop_logging for ‘File not found’ error (#199)
  • contentURL expects a string in Panel() and Widget() constructors (#198)
  • Bring the minimizeMemory() implementation up to date (#193)
  • Use ‘residentFast’ instead of ‘resident’ for memory reporter (#192)
  • Use postMessage for widget and panel communication instead port object (#185)
  • Support incremental cycle collection statistics (#187)
  • Added a usage section to the README (#178)
  • Change require statements and update the SDK to 1.14 (#174)
  • Simplify .travis.yml (#172)
  • Use . (dot) to include files while invoking /bin/sh (#169)
  • Use failonerror attribute in exec tasks (#168)

For all the details about the 0.6 release please check our issue tracker on Github.

mozdownload 1.12 has been released

The Firefox Automation team would like to announce the release of mozdownload 1.12. Without any other release of our universal download tool in the last 7 months, a couple of nice new features and bug fixes will make this release even more useful. You can upgrade your installation easily via pip, or by downloading it from PyPI.

Changes in 1.12

  • Display selected build when downloading (#149)
  • Add support for downloading B2G desktop builds (#104)
  • Download candidate builds from candidates/ and not nightly/ (#218)
  • Add Travis CI build status and PyPI version badges to README (#220)
  • Add Python 2.6 to test matrix (#210)
  • Fix broken download of mac64 tinderbox builds (#144)
  • Allow download even if content-length header is missing (#194)
  • Convert run_tests script to Python (#168)
  • Ensure that –date option is a valid date (#196)

Mozmill 2.0.7 and 2.0.8 have been released

The Firefox Automation team would like to announce the release of Mozmill 2.0.7 and Mozmill 2.0.8. Both versions had to be released in such a short time frame to ensure continuing support for Firefox. Some latest changes done for Firefox Nightly broke Mozmill, or at least made it misbehaving. If you run tests with Mozmill ensure to upgrade to the latest version. You can do this via PyPI, or simply download the already pre-configured environment.

Changes in 2.0.7

  • Bug 1066295 – testMultipleLoads.js is failing due to new HTTP Cache v2
  • Bug 1000098 – Fix testPageLoad.js test for invalid cert page
  • Bug 1065436 – Disable e10s until full support landed
  • Bug 1062773 – Disconnect errors invalidate the report
  • Bug 999393 – Expose assert and expect by default in sub modules
  • Bug 970820 – Mozmill aborts with socket.timeout when trying to send the report

Changes in 2.0.8

  • Bug 1067939 – JSBridge and Mozmill broken due to ‘let’ changes in bug 1001090

Please keep in mind that Mozmill 2.0 does not support electrolysis (e10s) builds of Firefox yet. We are working hard to get full support for e10s added, and hope it will be done until the next version bump mid of October.

Thanks everyone who was helping with those releases!

Mozdownload 1.10 released

As of now we have released a new version of mozdownload. The 1.10 release contains a couple of new features, bug fixes, and absolutely to mention a suite of tests. A big thank you goes here to Johannes and Jarek. Both spend a lot of time again, to make it a successful and lesser bug prone release.

Here some major items to highlight:

  • Addition of tests for all types of scrapers and a lot for command line options
  • Output of all candidate builds founds if build number has been specified
  • Add support for tinderbox stub installer
  • Allow to download files with different extensions than exe

A full list of features can be found in the changelog

Mozmill 1.5.12 released

Something we have learned over the weekend was: Never say never. After we have released the 1.5.11 release of Mozmill on April 19th, we were sure to not have to ship any other release off this branch. We want to concentrate our work on Mozmill 2.0 and get it out as soon as possible.

But things can change, and sometimes faster as expected. So by last Friday (it was already late in the evening) Mike Conley contacted me on IRC and mentioned that Mozmill will be broken starting with tomorrows build. The reason for it was the ‘Global-Per-Compartment’ patch which has been landed in Nightly builds of Firefox and Thunderbird at that time. Great! Why do things like those happen just before when I’m already in my weekend?

Anyway, given the importance of this bug (it broke the whole test infrastructure of Thunderbird which mostly relies on Mozmill) I was talking immediately to Clint Talbert. Given the good bug report from John Schoenick we were directly aware of what the problem was and agreed on a new API to get the issue fixed.

So I have spent a good portion of my Saturday and Sunday evening to put together a patch which removed the broken code in waitForPageLoad() and replaces it with the new API. So by now we do no longer add a custom ‘mozmillDocumentLoaded’ property to all windows, but have a nice windowMap object that handles all the loaded states of each window on its own. I think that’s great and modifying the DOM of each window was even a bad idea since the beginning of the project. Good to see that killed.

That means early today we got all patches landed and I was able to release Mozmill 1.5.12 to PyPI and even the uploaded extension on has been approved in the meantime. If you haven’t updated yet and you make use of Nightly builds, you should do it now!

Lets cross the fingers that no other unexpected issues will arise and force another 1.5 branch release.

Nightly Tester Tools 3.3 released

Having a good experience when testing Firefox Nightly builds should be self-evident. Therefor the QA Automation Services team is maintaining the Nightly Tester Tools extension, which brings a couple of useful features to ease the testing efforts.

While the extension works fine most of the time, we unfortunately fail at times when a version bump for Firefox happens. Exactly this action repeats in a 6 week interval and is the release cycle of Firefox.

So what happens here? The extension registers binary components for the crashme feature in the manifest file and that’s why it is not handled as a ‘compatible by default’ extension by the add-ons manager. So right after a version bump the extension gets marked as incompatible until the compatibility information on has been updated. This results in an unavailability of the extension in Nightly builds for a couple of hours or even a day, depending on when one of us has time to actually do the version bump. That had to be changed, and thankfully Szabolcs Hubai has spent some time on it to fix the problem and to make our extension ‘compatible by default’. From now on you will never see the Nightly Tester Tools disabled after a version bump of Firefox, which is great. I hope you all will enjoy it!

With the 3.3 release we also ship some other fixes which you can read about in the changelog.

You want to help in the development of the extension? Feel free to contact us as usual via MozIRC or our mailing list.

MemChaser 0.2 released

Exactly one month after we have released our initial version of MemChaser, version 0.2 has been made publicly available. You can install the add-on as usual from or if it is already installed, simply check for updates within the Add-ons Manager.

A couple of subtle changes have been made which will give a better experience for users. So we have combined the formerly two widgets in the Add-on bar into a single one to prevent other extensions from inserting their widgets in-between. At the same time the width has been reduced to not waste too much space (keep in mind that we are still blocked on a SDK bug which doesn’t let the widget size adapt to the content length dynamically).

Given the recent landing of the incremental garbage collector in Nightly builds of Firefox we had to do some extra unplanned updates to support this new feature. When you are running a Nightly build and an incremental GC happens, the ‘GC‘ label will be replaced with ‘iGC‘.

Further we have improved the parsing of the GC/CC console messages to allow us to add more interested entries dynamically. So this time we have added the reason and type (global or compartmental) for a GC, and the number of collected entries from the cycle collector. All three will be written together with all the other data to the log file and are not visible in the add-on bar.

As reported by an user MemChaser v0.1.1 was affected by some memory leaks which were caused by the Add-on SDK 1.4. Given the tireless efforts of the SDK devs nearly all of those leaks (except for widgets) have been fixed in version 1.5, which has been recently released and is now used by MemChaser 0.2.

Beside all those visible changes we also made some improvements to our development process. So we have added the Add-on SDK as a submodule to our own repository on Github. Also to get more traction we made the decision to move the MemChaser repository to the Mozilla account. And last but not least we implemented the Travis CI integration to get our code automatically tested with each check-in.

If you are interested in further information about the changes feel free to check our list of issues for 0.2.

We would also like to welcome everyone who has interests to help us with coding, ui work, or documentation. There are a lot of features we want to implement and need talented people for. Stop by on one of our channels and get in contact with us.

MemChaser – An extension to track Firefox’s memory activities

Today I want to announce a new extension for Firefox. It’s been developed by the Mozilla QA Automation Services team, which exists to support the Mozilla QA community by developing tools and frameworks for ease of testing. And this time, in case of the extension, we even target (web) developers, and our huge Firefox user base. So what is it about?

Mid of December 2011 bug 711900 has been filed. It complained about an increased garbage and cycle collector activity in Firefox 10, which was not existent in Firefox 9. So Mozilla QA has been asked to analyze that behavior and find possible use cases for reproduction. While working on it, we noticed that observing the related information about memory usage and garbage/cycle collector activity wasn’t that comfortable. For the memory you have had to constantly watch and refresh ‘about:memory‘ in a separate tab, and the latter even forced us to always have the Error Console open and a specific hidden preference set.

To make it easier for us I had the idea of creating an extension, which would be able to present all the necessary information at a prominent place and would be visible all the time. There should be no need of switching tabs or scraping the Error Console for output. Scraping was even hard because all entries constantly get wiped out when a maximum threshold of messages has been reached. Further copying out this data is a hassle. So lets create an extension with logging capabilities…

Some days later after an internal discussion across teams we started the development and called it the MemChaser project. Based on the fantastic help from members of the SpiderMonkey team – lets name Nicholas Nethercote and Andrew McCreight – we got a very early prototype setup kinda quickly. But as is turned out having an extension, which needs a restart to get installed, is pretty useless if the user is already facing a strange memory related issue. So one of the bigger tasks was to get familiar with the Add-on SDK project aka Jetpack and to transform the extension into a restartless one. As it turned out the work wasn’t that hard but gave us well better structured code and a nice toolchain to test and build the extension. And finally with all the features implemented we were able to release the MemChaser extension on

The current set of features is not that big but covers all the requirements we got from the QA team. We will make sure to enhance the extension drastically in the next couple of months so that it doesn’t only display current values in the Add-on bar. But what is it doing right now?

As you can see in the image a couple of values are displayed. But what do those stand for? Here the answer:

  • Resident: Memory mapped by the process that is present in physical memory. It’s normally the memory usage you will see in the systems process manager.
  • GC: Duration of the last Garbage Collector activity and in brackets the time when the second last activity happened.
  • CC: Duration of the last Cycle Collector activity and in brackets the time when the second last activity happened.
  • Logging: Starts and stops the internal logger, which writes out the current memory usage and GC/CC activities to a log file, which can be found in the memchaser sub directory of your Firefox profile.

All those information already helps a lot to keep track of memory related activities and enables someone to see if Firefox is using too much memory at all or is having long GC/CC cycles which could freeze the UI of Firefox for short moments.

As I have mentioned above we will continuously enhance the features of the extension. We already have some great ideas and I will certainly blog about those more in the next couple of days, once we have updated our documentation and set milestones for the next releases.

For now please install the extension in your daily profile and watch the memory activities. If you recognize strange behavior turn on logging and let us know by contacting us directly.

Mozmill 1.5.9 released

As noted in my yesterdays blog post about the freeze of Mozmill in the JSBridge module when Python 2.7.2 is used, there was an urgent need for a new version of Mozmill.

Late yesterday we were able to finally release Mozmill 1.5.9 on PYPI. It only includes the fix for bug 722707 and allows everyone who is running on Python 2.7.2 to use Mozmill again. If you are affected please upgrade immediately.