As of now we have released a new version of mozdownload. The 1.10 release contains a couple of new features, bug fixes, and absolutely to mention a suite of tests. A big thank you goes here to Johannes and Jarek. Both spend a lot of time again, to make it a successful and lesser bug prone release.
Here some major items to highlight:
- Addition of tests for all types of scrapers and a lot for command line options
- Output of all candidate builds founds if build number has been specified
- Add support for tinderbox stub installer
- Allow to download files with different extensions than exe
A full list of features can be found in the changelog
Something we have learned over the weekend was: Never say never. After we have released the 1.5.11 release of Mozmill on April 19th, we were sure to not have to ship any other release off this branch. We want to concentrate our work on Mozmill 2.0 and get it out as soon as possible.
But things can change, and sometimes faster as expected. So by last Friday (it was already late in the evening) Mike Conley contacted me on IRC and mentioned that Mozmill will be broken starting with tomorrows build. The reason for it was the ‘Global-Per-Compartment’ patch which has been landed in Nightly builds of Firefox and Thunderbird at that time. Great! Why do things like those happen
just before when I’m already in my weekend?
Anyway, given the importance of this bug (it broke the whole test infrastructure of Thunderbird which mostly relies on Mozmill) I was talking immediately to Clint Talbert. Given the good bug report from John Schoenick we were directly aware of what the problem was and agreed on a new API to get the issue fixed.
So I have spent a good portion of my Saturday and Sunday evening to put together a patch which removed the broken code in waitForPageLoad() and replaces it with the new API. So by now we do no longer add a custom ‘mozmillDocumentLoaded’ property to all windows, but have a nice windowMap object that handles all the loaded states of each window on its own. I think that’s great and modifying the DOM of each window was even a bad idea since the beginning of the project. Good to see that killed.
That means early today we got all patches landed and I was able to release Mozmill 1.5.12 to PyPI and even the uploaded extension on addons.mozilla.org has been approved in the meantime. If you haven’t updated yet and you make use of Nightly builds, you should do it now!
Lets cross the fingers that no other unexpected issues will arise and force another 1.5 branch release.
Having a good experience when testing Firefox Nightly builds should be self-evident. Therefor the QA Automation Services team is maintaining the Nightly Tester Tools extension, which brings a couple of useful features to ease the testing efforts.
While the extension works fine most of the time, we unfortunately fail at times when a version bump for Firefox happens. Exactly this action repeats in a 6 week interval and is the release cycle of Firefox.
So what happens here? The extension registers binary components for the crashme feature in the manifest file and that’s why it is not handled as a ‘compatible by default’ extension by the add-ons manager. So right after a version bump the extension gets marked as incompatible until the compatibility information on addons.mozilla.org has been updated. This results in an unavailability of the extension in Nightly builds for a couple of hours or even a day, depending on when one of us has time to actually do the version bump. That had to be changed, and thankfully Szabolcs Hubai has spent some time on it to fix the problem and to make our extension ‘compatible by default’. From now on you will never see the Nightly Tester Tools disabled after a version bump of Firefox, which is great. I hope you all will enjoy it!
With the 3.3 release we also ship some other fixes which you can read about in the changelog.
You want to help in the development of the extension? Feel free to contact us as usual via MozIRC or our mailing list.
Exactly one month after we have released our initial version of MemChaser, version 0.2 has been made publicly available. You can install the add-on as usual from addons.mozilla.org or if it is already installed, simply check for updates within the Add-ons Manager.
A couple of subtle changes have been made which will give a better experience for users. So we have combined the formerly two widgets in the Add-on bar into a single one to prevent other extensions from inserting their widgets in-between. At the same time the width has been reduced to not waste too much space (keep in mind that we are still blocked on a SDK bug which doesn’t let the widget size adapt to the content length dynamically).
Given the recent landing of the incremental garbage collector in Nightly builds of Firefox we had to do some extra unplanned updates to support this new feature. When you are running a Nightly build and an incremental GC happens, the ‘GC‘ label will be replaced with ‘iGC‘.
Further we have improved the parsing of the GC/CC console messages to allow us to add more interested entries dynamically. So this time we have added the reason and type (global or compartmental) for a GC, and the number of collected entries from the cycle collector. All three will be written together with all the other data to the log file and are not visible in the add-on bar.
As reported by an user MemChaser v0.1.1 was affected by some memory leaks which were caused by the Add-on SDK 1.4. Given the tireless efforts of the SDK devs nearly all of those leaks (except for widgets) have been fixed in version 1.5, which has been recently released and is now used by MemChaser 0.2.
Beside all those visible changes we also made some improvements to our development process. So we have added the Add-on SDK as a submodule to our own repository on Github. Also to get more traction we made the decision to move the MemChaser repository to the Mozilla account. And last but not least we implemented the Travis CI integration to get our code automatically tested with each check-in.
If you are interested in further information about the changes feel free to check our list of issues for 0.2.
We would also like to welcome everyone who has interests to help us with coding, ui work, or documentation. There are a lot of features we want to implement and need talented people for. Stop by on one of our channels and get in contact with us.
Today I want to announce a new extension for Firefox. It’s been developed by the Mozilla QA Automation Services team, which exists to support the Mozilla QA community by developing tools and frameworks for ease of testing. And this time, in case of the extension, we even target (web) developers, and our huge Firefox user base. So what is it about?
Mid of December 2011 bug 711900 has been filed. It complained about an increased garbage and cycle collector activity in Firefox 10, which was not existent in Firefox 9. So Mozilla QA has been asked to analyze that behavior and find possible use cases for reproduction. While working on it, we noticed that observing the related information about memory usage and garbage/cycle collector activity wasn’t that comfortable. For the memory you have had to constantly watch and refresh ‘about:memory‘ in a separate tab, and the latter even forced us to always have the Error Console open and a specific hidden preference set.
To make it easier for us I had the idea of creating an extension, which would be able to present all the necessary information at a prominent place and would be visible all the time. There should be no need of switching tabs or scraping the Error Console for output. Scraping was even hard because all entries constantly get wiped out when a maximum threshold of messages has been reached. Further copying out this data is a hassle. So lets create an extension with logging capabilities…
Some days later after an internal discussion across teams we started the development and called it the MemChaser project. Based on the fantastic help from members of the SpiderMonkey team – lets name Nicholas Nethercote and Andrew McCreight – we got a very early prototype setup kinda quickly. But as is turned out having an extension, which needs a restart to get installed, is pretty useless if the user is already facing a strange memory related issue. So one of the bigger tasks was to get familiar with the Add-on SDK project aka Jetpack and to transform the extension into a restartless one. As it turned out the work wasn’t that hard but gave us well better structured code and a nice toolchain to test and build the extension. And finally with all the features implemented we were able to release the MemChaser extension on addons.mozilla.org.
The current set of features is not that big but covers all the requirements we got from the QA team. We will make sure to enhance the extension drastically in the next couple of months so that it doesn’t only display current values in the Add-on bar. But what is it doing right now?
As you can see in the image a couple of values are displayed. But what do those stand for? Here the answer:
- Resident: Memory mapped by the process that is present in physical memory. It’s normally the memory usage you will see in the systems process manager.
- GC: Duration of the last Garbage Collector activity and in brackets the time when the second last activity happened.
- CC: Duration of the last Cycle Collector activity and in brackets the time when the second last activity happened.
- Logging: Starts and stops the internal logger, which writes out the current memory usage and GC/CC activities to a log file, which can be found in the memchaser sub directory of your Firefox profile.
All those information already helps a lot to keep track of memory related activities and enables someone to see if Firefox is using too much memory at all or is having long GC/CC cycles which could freeze the UI of Firefox for short moments.
As I have mentioned above we will continuously enhance the features of the extension. We already have some great ideas and I will certainly blog about those more in the next couple of days, once we have updated our documentation and set milestones for the next releases.
For now please install the extension in your daily profile and watch the memory activities. If you recognize strange behavior turn on logging and let us know by contacting us directly.
As noted in my yesterdays blog post about the freeze of Mozmill in the JSBridge module when Python 2.7.2 is used, there was an urgent need for a new version of Mozmill.
Late yesterday we were able to finally release Mozmill 1.5.9 on PYPI. It only includes the fix for bug 722707 and allows everyone who is running on Python 2.7.2 to use Mozmill again. If you are affected please upgrade immediately.
Yesterday we had to release Mozmill Crowd 0.1.5 due to changes in our reporting infrastructure. Given the fallout of one of our servers our Mozmill Dashboard is now located at http://mozmill-crowd.blargon7.com/. This move needed an update of the default report URL. Now you can send reports of your test results again.
We have to admit that not much work happened on the extension in the last couple of months. Reasons are mostly other important projects and mainly the development of the new pre-configured environment for Mozmill. There are still some outstanding improvements to implement before we can release a new version of the environment.
If you are interested to help out please get in contact with us via our team page or any other public channel.
The final bits of Mozmill 1.5.8 are now available through Pypi. Install or upgrade Mozmill by running:
pip install --upgrade mozmill==1.5.8
We had to release this version to fix a recent regression which has been identified in Mozmill 1.5.7, and which caused a hard-stop when an user restart has been requested. The issue did only exist on OS X.
Beside this fix we now enabled the new boolean preference called ‘focusmanager.testmode’ by default. It will allow us to run multiple Mozmill tests concurrently on the same machine. Support for the focusmanager testing mode has already been made available on Nightly and Aurora builds. Also there is a good chance to get this patch landed in an upcoming Firefox 10.0.1 release. I will follow-up with another blog post with more details once the final landing has happened and we can actually make use of it across our supported branches.
Since I have been using rdiff-backup to backup my Xen domU machines, I get a daily message from cron which informs me that some sockets couldn’t be backed-up because the path is too long:
SpecialFileError var/spool/postfix/public/flush Socket error: AF_UNIX path too long
After some digging I have found that this is related to the length limitation of the AF_UNIX constant in one of the kernel header files, and which only allows 107 characters. Depending on where you want to backup to the length can exceed.
The solution is simple. Why do you want to backup sockets at all? Those are bound to the current system and will be executed when an application gets started. So I don’t see a reason to backup those files. Therefor just use the –exclude-sockets option for rdiff-backup.
Please note that you will have to run rdiff-backup twice before the failure message from above will no longer appear.
Linux! Linux is great. Linux is Open Source. Any nerd wants to run Linux. But is any part of Linux really that great? This was a good question I wasn’t really able to answer until yesterday. Now I have mixed feelings but understanding the following problem better, gives even a bit more safety, also for my personal life.
During the whole last year I had a lot of situations when one of my virtual machines on the server died due to an OOM killer process. Those crashes were not predictable and happened randomly. Sometimes it didn’t happen for weeks but there were also situations when it crashed after 1 day again. Given that a good list of customers are hosting their websites on it, raises a lot of trouble for me. I did a lot of work in trying to fix particular running services on that host, but nothing helped to stop those crashes. Recently I have even doubled the memory for that machine but without success. It always ran into an out of memory crash.
Given all my former research and attempts to fix the problem, I wasn’t sure what else I could do. But thankfully I have found a website which has the explanation and even offered steps to solve the problem.
So what’s happened? The reason can be explained shortly: The Linux kernel likes to always allocate memory if applications asking for it. Per default it doesn’t really check if there is enough memory available. Given that behavior applications can allocate more memory as really is available. At some point it can definitely cause an out of memory situation. As result the OOM killer will be invoked and will kill that process:
Jun 11 11:35:21 vsrv03 kernel: [378878.356858] php-cgi invoked oom-killer: gfp_mask=0x1280d2, order=0, oomkilladj=0
Jun 11 11:36:11 vsrv03 kernel: [378878.356880] Pid: 8490, comm: php-cgi Not tainted 2.6.26-2-xen-amd64 #1
The downside of this action is that all other running processes are also affected. As result the complete VM didn’t work and needed a restart.
To fix this problem the behavior of the kernel has to be changed, so it will no longer overcommit the memory for application requests. Finally I have included those mentioned values into the /etc/sysctl.conf file, so they get automatically applied on start-up:
vm.overcommit_memory = 2
vm.overcommit_ratio = 80
The results look good so far and I hope it will stay that way. The lesson I have learned is to not trust any default setting of the Linux kernel. It really can result in a crappy and unstable behavior.