Release testing which has to be done by QA right before a new release of Firefox will be offered to our users is still an area where lot of manual work is involved. That means we run Smoketests and the Basic Functional Tests (BFT’s) against the build candidate. As I have already written there is ongoing work with Mozmill to get those work fully automated in the future. But that are not the only tests we have to run…
Since ever Firefox is supporting automatic updates we also have to check that each and every user will get the right update package for the installed version of Firefox. Most of our users should run the latest version of Firefox but there are also cases where people don’t update immediately or even don’t want to upgrade to the next major version of Firefox. Given that updates have to be delivered to each of the supported branches (e.g. Firefox 3.0.0.x and Firefox 3.5.x) and also as major update for upgrading to the next major version. We also have a community beta program running where users can help testing beta versions of the next Firefox version. Those users will get a separate update offer on another update channel.
Finally there are 4 different channels we have to test for en-US and some of our P1 localized builds. In detail those are in the right order:
- betatest: This channel makes sure that updates which will be delivered to beta users will pass.
- beta: Beta testers will get their updates on that channel.
- releasetest: This channel tests the update snippets which have been pushed to our official download mirrors.
- release: Default channel for all Firefox installations to get the next version.
For each of those mentioned channels we offer partial and complete updates. The former one will be used if the latest minor version of Firefox is in use, e.g. a user wants to update from 3.5.4 to 3.5.5, while the latter one is for all other versions of the same branch. If an update fails to apply which could happen due to different reasons like a download problem, users will not get stuck on their installed version. In such a case a fallback update will be downloaded which is identical to the complete update. If that fails too the same process will be started again after a given time.
Until now these tests had to be done manually by us. An example can be seen in the update section of the Firefox 3.5.5 test plan. So we normally tests updates on all supported platforms, for each update type (minor, major), and make sure that fallback updates will pass.
With the new software update tests for Mozmill which I have finished two days ago, we can easily automate this process now. The only manual steps which have to be done is to prepare the tests by downloading the necessary builds for all the platforms and place them in their own folders. Once that is done the automated test can be started. It will use all builds within a given folder and runs tests updates for the specified channel. The results are printed in wiki format to the console and only have to be copied to the appropriate Wiki page.
If you are interested in running those update tests you have to install Mozmill on your machine and clone our Mozmill test repository. Detailed steps can be found in the Mozmill test creation tutorial on QMO.
This is a big step forward in a direction where we can run update checks against each localized build of Firefox and can make sure that updates are successfully applied and don’t fail or cause any sort of failure. And it will give QA more time to focus on other topics.
If you are interested and want to know more about Mozmill then join us in #QA on IRC or subscribe to the Mozmill developer list.
(No Ratings Yet)
Loading ...Tags: automation, mozilla, mozmill, QA, software, testing, update
That’s pretty cool! Years ago I was fascinated by watching minimalism pictures but today I noticed something which is much much better. Keith Loutit from Sydney uses the same technique to create own videos with the same effect. I cannot get enough from it… See on of his shorts movies to get fascinated too:
Bathtub IV from Keith Loutit on Vimeo.
(No Ratings Yet) Loading ...Tags: minimalism, video
Nearly 3 month after we have released Mozmill 1.2 we are close to our next release of Mozmill. Lots of bugs have been fixed and even a couple of new features were implemented. A nearly complete list you can find on Bugzilla.
Everyone who is using Mozmill regularly is welcome to help us in testing the beta version. As long as no big issues will come up the release of Mozmill 1.3 will happen next week.
If you want to test the extension please download it from Github.
Users of the pyPI packages only have to run “easy_install -U mozmill” to get the latest packages for Mozmill, JSBridge, and Mozrunner.
Thanks in advance.
(No Ratings Yet) Loading ...Working on two or even more machines in parallel would require a KVM switch or just a neat software like Teleport which lets you share your keyboard and mouse simply over the network. One big advantage against Synergy is that this solution comes as a prefpane and embeds the configuration UI and the backend within one application. Further once a Mac has been enabled for sharing other Mac’s will automatically find it in the network. Anyone who has security concerns will be happy that the connection can be encrypted.
After a couple of hours using it I will definitely not miss it anymore!
(No Ratings Yet) Loading ...As a couple of you already know Mozilla QA runs its own set of automated functional tests which are separated from the tests in the automated test suite. The main goal for us is to shorten the test duration for manual functional tests so those tests will be run more often. There are Smoketests, BFT’s (basic functional tests), and FFT (full functional tests) available on Litmus which get partly run by QA during release testing or at any time by contributors. Given the fact that all those tests need a lot of time to execute manually, we are working on getting most of those tests automated.
There is a question which I get asked very often from developers: “Why do we need Mozmill tests when we already have a suite of automated tests available?” The clear answer is that those tests are used to simulate user actions on UI elements the same way as when a user would sit in-front of the computer. That means that for example clicks on hidden or disabled elements shouldn’t trigger the execution of the underlying command. That’s the difference to Mochitests which always trigger the command when the synthesizeMouse function is used to click on an element. Another really helpful feature is the capability to run restart tests of any sort. That’s not possible with the existing test harnesses which makes Mozmill test unique.
Seeing the importance of those tests we want to have a full suite of BFT tests for Mozmill by end of Q2 in 2010. The total number of 196 doable tests, except the ones which require OS level interaction, would allow us to run 82% of the tests automated. At the moment 65 of the tests have been already finished and can be run with Mozmill against builds from the 1.9.1 and 1.9.2 branch. For detailed information about the current state and actual work please check the Google spreadsheet.
To get more tests automated the following goals have been set by the QA execution team for Q4 in 2009:
- Firefox 3.6 will be released this quarter. To enhance our testing we want to automate all of the tests in 4 BFT subgroups prioritized as P1. In general these subgroups are: Awesomebar, Add-ons Manager, Download-Manager, and Tabbed Browsing. This will incite us to write the next 40 tests. A list of all available subgroups and their prioritization can be found in the Feature Ownership document.
- For us who are working on release testing, software update tests have to be performed for the betatest, beta, releasetest, and release channels. Given the manual work which have to be performed here automation will help a lot. I will finalize my software update tests so they can be run by everyone.
- Running Mozmill tests you will get results reported in the terminal. Even with the integrated capability to send those reports to a server we don’t have a web frontend to display those results. We want to use Brastacks to visualize Mozmill results similar to the Fennec test results. This work will be a joined effort with Testdev.
If anyone is interested in helping us to write or maintain Mozmill tests you can read more about it in the test creation tutorial or simply join us on IRC and get in contact with whimboo or aakashd. But you can also send me a mail or comment on this blog post. Thanks!
(No Ratings Yet) Loading ...Are you interested to work for a global and open company? The Mozilla Corporation – vendor of the free and open source web browser Firefox – is seeking for a full-time QA Execution Engineer. If you will take the advantage to work with engaged and talented people and to stay in touch with our world-wide and powerful community go ahead and send your resume…
Mozilla Corporation is seeking a QA Engineer who is responsible for test execution of Firefox and Thunderbird products. You will be developing feature test cases and run regression testing during releases. You will also coordinate testing with the open source community and continually evangelize Firefox testing to the world. Development of browsers is accelerating; we’re looking for engineers who can keep us ahead of the curve.
Responsibilities:
- Create test plans and test cases from functional specifications
- Develop and execute test automation scripts
- Execute black box tests on a monthly release cycle
- Maintain test documentation and test cases
- Coordinate projects within the open source community
- Confirm, create steps to reproduce, identify regression ranges and clarify bug and feature requestion
- Monitor user feedback
Requirements:
- Strong knowledge of internet and browser technologies
- Strong understanding of test methodologies and test case development
- Strong problem solving and resolution skills
- Programming experience with in C, C++, JavaScript, Python, Perl, and XML
- Knowledge of Windows / Mac / Linux environments
- Experience with software testing lifecycle process, release management and bug life cycle
- Strong verbal and written communication skills
- Flexibility in dynamic software environments
- BS or MS in Computer Science or related field
Skills Desired:
- Knowledge of web automation tools like Selenium, Watir, Silk, Quicktest Pro, and WinRunner
- Working knowledge of web security
- Experience with browser extensions and plugins
- Experience with virtual machines, and multiple build environments
- Experience with testing open source products and open source community participation
See also the full job description on jobvite.com.
(No Ratings Yet) Loading ...Some weeks ago a friend of mine pointed me to an upcoming photo challenge about image manipulation created. It has been initiated by the biggest German computer magazine c’t. My first thought was to attend this challenge but I recognized that I don’t have any great images on my box and no time to work on that. But surprisingly a friend was visiting me over those days and we had some great shots in the Saxon Switzerland.
Given those pictures Silvia made a fake image in parallel without knowing about this challenge. Once I had seen the result I pushed here to subscribe to the challenge. After some talks and further processing steps of the image we have decided to take the challenge just for fun. About 3 weeks ago the first results have been posted to the online gallery and a public survey has been started. 139 of the best pictures can be voted by everyone and our faked image is in this list too.
Here a former version and not the final one on my Flickr account:
Surprisingly a former colleague and a friend of mine informed me today that we have made it to even show up on spiegel.de. Woot – there are only 13 images posted. So isn’t it a call that we have good chances?
Please help us and vote for our image when you like it. We appreciate your help.
(No Ratings Yet) Loading ...Tags: Personal, photography
Over the last weekend Mozilla Camp Europe 2009 has been taken place in Prague. About 150 people from l10n, qa, dev, and advocacy were invited to join this conference which Mozilla Europe is organizing each year.
Given my project to get manual Litmus tests automated with Mozmill I have prepared some slides with a special focus on l10n. But sadly I wasn’t able to join the conference because of sickness. I have to say a big thanks to my colleague Marcia Knous and also to one of our main contributors for Sunbird tests Merike Sell who both hold the session. As informed at the end of the session via IRC the talk was a great success and a lot of questions were ask.
Due to the amount of sessions not everyone was able to join the Mozmill session. Also given all the people who weren’t be able to come I have uploaded my slides for all of you now. Please check the embedded Slideshare content below:
Feedback
Because I haven’t got any feedback from localizers so far I’m anxious to hear what you think about the usefulness of Mozmill and testing with localized builds. Given by the current number we have over 70 official locales available which are not tested by automated tests and require manual testing from localizers and contributors on a regular basis. With all the 250 BFT and another 750 FFT tests enabled in Litmus manual testing is a time taking action. Running all the tests with Mozmill will take much lesser time, could be run more often, and could cover all platforms which will result in a higher quality of Firefox and helps us to minimize any new regressions for our huge user base.
Please check the following questions I’m interested in getting an answer:
- How often does your l1on team run Litmus tests against your locale whether those are BFT/FFT or the localizer test-run?
- Would you like to see much of those tests automated and are you interested in running those tests on your local machine for each major and stability release?
- Are you interested to help QA in writing Mozmill tests so we have most of them available as soon as possible?
- Do you have further ideas how Mozmill can be used in the l10n area additionally to the points I have pointed out in my slides?
Thanks in advance for your feedback!
(No Ratings Yet) Loading ...The last days I searched for a way how to modify some of the default keybindings for my Mac OS X installation. While searching the web I found the blog of Llew Mason who wrote a nice article about it some years ago.
(No Ratings Yet) Loading ...Tags: keyboard, mac os x, navigation, software
In the last weeks we discovered a couple of changes in the upcoming 1.9.2 code base which forced us to branch our mozmill-test repository. At first we weren’t sure if we should create separate repositories for the 1.9.1 and 1.9.2 branches or handle everything through multiple heads in the same repository. We made the decision to use the first way (see bug 509290) which finally ended up as the wrong way due to some limitations of hgweb. Means we had to revert all the changes and are now using named branches in the same repository.
If you want to use Mozmill to run tests against Shiretoko / Firefox 3.5.x or Namoroka then you can switch between the different branches with the commands seen below.
$ hg clone http://hg.mozilla.org/qa/mozmill-tests
$ CD mozmill-tests
$ hg branches
default 108:d8529dd87a77
mozilla1.9.1 106:221ecb1a8ea9
$ hg branch
default
$ hg up -C mozilla1.9.1
$ hg branch
mozilla1.9.1
$ hg up -C default
$ hg branch
default
Given the output default is our current trunk version of the Mozmill tests while the mozilla1.9.1 branch is for Firefox 3.5.x.
(No Ratings Yet) Loading ...
Recent Comments