Now that I have my Firefox testing environment, it’s time for my first task: Performance Testing! For this, I needed to know what to test (in this case, nightly builds of Minefield and Chromium), how to test (the Chrome Experiments) and what to determine (speed, smoothness, and responsiveness). There are quite a few experiments, but as a class of 12 we split the 120 up fairly easily. I was tasked with #71-80.
To begin testing, I had to try and think about what variables could be eliminated. I decided to launch all experiments directly from the command line in an attempt to standardize application startup. For this I wrote a batch file script with a few parameters to help with this. I also used Windows’ built in PerfMon tool to assist with performance measurement so that I could be free to stress-test the application. With this combination, I felt I could gather information quickly and efficiently, and Perfmon also provided lots of great graphs for later analysis. It almost seemed like overkill at the time, but it helped give extremely measurable results.
Given the target implementation platform (Chrome) for these experiments, I wasn’t surprised to find that a few of them weren’t performing quite as well on Minefield as on Google’s browser… yet. What did surprise me was the creativity and range of the experiments.
Now that I had my data, it was time to analyze further and see if I could spot any large discrepancies, and to try and figure out why they occurred. My testing measures focused on the Working Set (Memory) and CPU utilization for the browser process. The jsCanvasBike, Animated Harmonograph and Liquid Particles seemed to require the most tuning, so my attention turned now to profiling rather than monitoring. As perfmon shows, not all of these problems were visible simply by monitoring the CPU.
Luckily, the Windows SDK comes with their own profiling tool (xperf) which features extremely detailed kernel debugging on stack traces. Mozilla even has its own help page with instructions tailored specifically for use with Firefox. The results can look a little daunting, but it appears to be a very powerful tool and I was glad to have so many resources available. I feel as if I’ve entered a whole new realm of testing, but there is still a great deal more to experiment with.
Leave a Reply