Blog

  • An Introduction to HTML5 Canvas

    One of the great new elements in HTML5 is the Canvas elements. This is an element that takes everything that was about web display and exponents it. It has opened up the opportunity for JavaScript graphics APIs like C3DL to be made and used in much the same way the DirectX and OpenGL APIs exist for their desktop counterparts. It allows for the dynamic modification of video or other elements on the page at a pixel-by-pixel level as seen by Processing.js. With clever use, it can even be used to enhance video experience as seen by Popcorn.js and the upcoming effects extension Candy.js. All of this though, is above my head right now. Right now things are simple: Hello World on Canvas.

    To use canvas effectively, you need to draw to it. To draw to it, you need to get the context you with to draw with. In most cases right now it will be 2D drawings (through which 3D can be simulated) but soon the webgl context will be standard implementation as well and will be possible to render true 3D images (C3DL uses webgl context). But I digress: back to Hello World on Canvas.

    First, you have to get the drawing context from the canvas element (if available):

    var canvas = document.getElementById("canvas");
    if (canvas.getContext) { // Canvas Support
    var ctx = canvas.getContext("2d");
    // Work with context
    }

    This will give you everything you need to get ready to draw! Like all computer graphics, canvases work on a Cartesian Coordinate System, using x and y to specify points. Lines are drawn between 2 points like so:

    ctx.beginPath(); // Tell the context you are drawing a path instead of a shape
    ctx.moveTo(135,120); // Set the starting point (Point A)
    ctx.lineTo(135,70); // Establish a line between Point A and this point (Point B)
    ctx.stroke(); // Draw the line

    Pretty easy to draw a line. Chaining lines together is possible too, just add more calls to ctx.lineTo() before drawing the line. If you want to make a closed shape made up of lines, that too is easy. Rather that drawing the last line to close a figure, there is a handy function to automatically connect a line from the last line drawn to where you started drawing. See the below code to draw a triangle:

    ctx.beginPath(); // Tell the context you are drawing a path instead of a shape
    ctx.moveTo(135,120); // Set the starting point (Point A)
    ctx.lineTo(135,70); // Establish a line between Point A and this point (Point B)
    ctx.lineTo(185,70); // Establish a line between Point B and this point (Point C)
    ctx.closePath(); // Connect Point C to Point A
    ctx.stroke(); // Draw the lines

    Canvas also supports the drawing of things more complex than straight lines. An arc can be drawn by specifying a radius, the degree at which to start drawing (0 degrees, 90 degrees), the number of radians to draw and whether to draw clockwise or not. The below code draws a complete circle (0 to Math.PI*2 radians, or 0 to 360 degrees) with a radius of 25 pixels located at point 100,95. The circle is drawn clockwise (last parameter is true).

    ctx.beginPath();
    ctx.arc(100,95,25,0,Math.PI*2,true);
    ctx.stroke();

    Curves can also be drawn in both the quadratic and cubic (bezier) varieties. These curves are similar to normal arcs, but have one or two control points respectively to skew the curve. The following code draws a quadratic curve from Point A (135,120) to Point B (135, 95) with a control point at 180, 82.5:

    ctx.beginPath();
    ctx.moveTo(135,120);
    ctx.quadraticCurveTo(180,82.5,135,95);
    ctx.lineTo(150,120);
    ctx.stroke();

    A bezier curve is drawn similarly (use bezierCurveTo rather than quadraticCurveTo) with a secondary control point being specified just after the first control point. Up to now all this has been line-based drawings. But shouldn’t there be an easier way to draw a rectangle than by drawing 4 lines? There is!

    ctx.strokeRect(60, 10, 10, 50);

    This one line will draw a rectangle with the top-left corner at point (60,10) and give the rectangle a width of 10 and a height of 50. As this is not a point-to-point drawing this will be drawn without need to specify beginPath(), moveTo() or stroke() like in the earlier examples. But what if you wanted to fill in the enclosed area? Simple: rather than call strokeRect(), call fillRect() and the interior area will be filled! The same can be applied to filling line drawings by substituting the call to stroke() with fill().

    Now that we have some lines and shapes on the canvas, we need to give them colour. Computer graphics quantify a colour by specifying the amount of red, green and blue in the colour on a scale of 0 to 255. Optionally, a fourth value, alpha, can be used to specify the transparency of the colour (0 is transparent, 1 is opaque, anywhere in between is translucent). This is done on the canvas by setting one of two properties: strokeStyle or fillStyle. Stroking also allows for another property, lineWidth, to be given to modify the weight of the line. These must be specified before drawing to have any effect:

    // Stroke a semi-transparent red arc
    ctx.strokeStyle = "rgba(255,0,0,0.5)";
    ctx.lineWidth = 7;
    ctx.beginPath();
    ctx.arc(100,95,25,0,Math.PI*2,true);
    ctx.stroke();

    // Fill an opaque rectangle (no alpha specified means 100% opaque)
    ctx.fillStyle = "rgb(255,0,0)";
    ctx.fillRect(10, 10, 10, 50);

    Note that the alpha doesn’t have to be specified, and that when you don’t you only say “rgb” instead of “rgba”. While these shape and line functions allow for great flexibility, it’s difficult to write with them. Luckily, the Canvas API also allows for text to be written in either the “Stroke” or “Fill” methods listed above. Only one reference point is needed as well, which is specified in the text-writing function itself. What does need to be specified though, is the font.

    // Stroke "Hello World" in grey at coordinates 10,100
    ctx.strokeStyle = "rgba(128,128,128,1)";
    ctx.lineWidth = 3;
    ctx.font = "40pt Arial";
    ctx.strokeText("Hello World", 10, 200);

    // Paint “Hello World” in black at coordinate 10,150
    ctx.fillStyle = “rgba(0,0,0,1)”;
    ctx.font = “20px Arial”;
    ctx.fillText(“Hello World”, 10, 150);

    With this basic functionality it’s possible to make some nifty results:

    Canvas Demo

    See how this was done. Canvas also offers more advanced functionality such as saving/restoring the canvas, image exportation/importation and manipulation, and transformation effects such as rotation and skewing. A comprehensive documentation of usage and compatibility is covered on Mozilla Developer Central, including an overview and independent focuses on graphics and text. With this sort of dynamic control, developers have been hard at work experimenting with the possibilities of canvas for a while now and there is no sign of ingenuity stopping. The Times, They Are A-Changin’.

  • Graceful fallback for HTML5 Video

    While HTML5 has been around for a few years now (Firefox support began in 2007) and modern browser support is nearly universal, some browsers will still throw up their hands and not know how to handle those <video> tags when parsing a page. If video IS supported, the waters are further muddied in that different browsers support different codecs. There isn’t even one universally-supported format either: Safari only supports h.264 while Firefox supports VP8 and Ogg Vorbis. For all of these cases, graceful handling is important.

    Unrecognized tags are ignored, so to gracefully handle when video tags aren’t recognized just pretend they aren’t there while writing markup. The below code will output a text warning if the browser doesn’t support video:

    <video id=”vid1″ name=”vid1″ width=”480″ height=”267″  controls >
    <p>Your browser does not support video</p>
    </video>

    Here we are, the most basic, stripped down video tags you can get. It doesn’t even know what video to play, but if a browser doesn’t know what a “video” is, it certainly knows what a “p” tag is and will output what’s between them. You can put nearly anything in there and video will take precedence over it if HTML5 video is supported (except <script>, those will still run), so fallback to a Flash player is easy and simple. Just put the following between the <video> tags and you are guaranteed to provide the opportunity for a media-rich experience to any user.

    <!– “Fallback” to using Flash if video is not supported –>
    <!– Required for Mozilla < 3.0, Opera < 10.5, Safari < 3.1 –>
    <object classid=”clsid:D27CDB6E-AE6D-11cf-96B8-444553540000″ width=”550″ height=”400″>
    <param name=”movie” value=”../Escape.swf”>
    <!– Required by IE browsers –>
    < embed src=”../Escape.swf” width=”550″ height=”400″ > </embed>
    </object>

    In the event that video isn’t supported, the “Escape.swf” Flash movie will play instead. How to do this too varies by browser, but copying and pasting the above code will work in all major browsers. Internet Explorer requires the tag, while Firefox and other browsers require the <object> tag, with each ignoring the other tag. When using the <object> tag, the UUID for the installed Flash plugin must be given. This is an unchanging value unique to Adobe’s plugin and can be copied without modification as well.

    Given that all but one of the 5 major browsers currently support HTML5 video (with the fifth one, IE, implementing it in the current beta), lets assume the browser knows about the video element. How to account for if the browser doesn’t support the video format? The HTML5 video tag makes this easy: just specify a list of possible video sources and the browser will try each in sequence until it finds one compatible. Each of these are specified in a <source> tag, which are children of the <video> tag.

    <video id=”vid1″ name=”vid1″ width=”480″ height=”267″ controls >
    <!– Default video and encoding –>
    <source src=”http://upload.wikimedia.org/wikipedia/commons/7/79/Big_Buck_Bunny_small.ogv” type=”video/ogg”>
    <!– “Fallback” codec, will play when Ogg isn’t supported (Safari) –>
    <source src=”http://mirrors.creativecommons.org/movingimages/Building_on_the_Past.mp4” type=”video/mp4″>
    </video>

    Putting it all together we get a fail-safe system to fallback to Flash plugins (or a warning message) if <video> isn’t recognized, or we get a format-by-format fallback sequence until the browser can find a <source> that it knows how to work with:

    <video id=”vid1″ name=”vid1″ width=”480″ height=”267″ controls >
    <!– Default video and encoding –>
    <source src=”http://upload.wikimedia.org/wikipedia/commons/7/79/Big_Buck_Bunny_small.ogv” type=”video/ogg”>
    <!– “Fallback” codec, will play when Ogg isn’t supported (Safari) –>
    <source src=”http://mirrors.creativecommons.org/movingimages/Building_on_the_Past.mp4” type=”video/mp4″>
    <!– “Fallback” to using Flash if video is not supported –>
    <!– Required for Mozilla < 3.0, Opera < 10.5 –>
    <object classid=’clsid:D27CDB6E-AE6D-11cf-96B8-444553540000′ width=”550″ height=”400″>
    <param name=”movie” value=”../Escape.swf”>
    <!– Required by IE browsers –>
    < embed src=”../Escape.swf” width=”550″ height=”400″ > </embed>
    </object>
    </video>

    With this we have a completely safe cross-browser way of implementing functionality (as best possible) promised by the HTML5 standard. The standard doesn’t go so far as to say how UI elements such as play/pause/stop controls must be rendered. Standardizing that though, is a subject for another blogging.

  • Open Source Development: The HTML5 Open Video Player

    After much thought I’ve decided to work on developing a video player using JavaScript and HTML5. Up until recently Flash players have been the way to get a rich web experience but the advent of HTML5 may change all that in a plugin-free way. You can find information on the HTML 5 Open Video Player for now on our wiki, with more sources of information announced in the coming days on my colleague Kevin’s and my blogs.

    We have three months in which to bring this to a fairly stable state. Release features below are subject to change (or may be covered by other project members), but dates are not. Details are still being worked out, but a (tentative) release plan is:

    Version 0.1 (Oct 22, 2010) – In cooperation with Kevin to plan to build the core framework which includes the basic functionality of Play, Pause, Stop, Volume, Time display as well as to have CSS-customizable controls, Flash fallback and a re-sizable player. We also plan to have hard-coded English subtitles for our sample video.

    Version 0.2 (Nov 16, 2010) – With the core built, my solo work will be to complete subtitle support by pulling subtitles from TTXT format and to incorporate internationalization into both subtitles and the video controls. Stat tracking will also be generated (though kept local to the user’s machine) along with source hiding to protect against downloads. Screen-reader accessibility will also be implemented at this stage in the project.

    Version 0.3 (Dec 7, 2010) – Further functionality like the ability to “Like” a video and  watermarks over the video will be introduced as will ratings information on the side.

    As a browser-based application we luckily don’t have to worry about hardware limitations, but browser compatibility is of concern as accessibility of information is of paramount importance. We want this to be accessible, something which will become more difficult on non-standards-compliant browsers. Success will be determined by usability and accessibility of each deliverable. Success will also be determined by the modularity of the product so as to aid in future extensibility.

    Never having worked with HTML5, a lot of research will be required to complete this project, but a series of tech demos and prototypes from which code and knowledge can be taken away will assist in completion of this project. As neither myself nor Kevin as extensive experience with audio/video, we’ll be learning a lot this project. Luckily, we’ll be in communication with knowledgeable Mozilla community members including Brett Gaylor and Anna Sobiepanek.

    During development, I also plan to spend time contributing to a build and test system for the C3DL java script library. If any developers are interested in contributing to our project we welcome all, especially any with expertise in audio/video or JSON.

  • My First Bug

    After identifying and analyzing various performance issues I noted related to hardware acceleration in Firefox 4 nightlies, it was now time to make my findings public. Bugzilla would be the end result, but there were quite a few steps in between. First, there was consultation. I’d never filed a bug before and as punctual as I try to be, I had the great luck of a one-time schedule conflict occurring when we were covering bug filing in class. I already knew a bit from earlier classes and my own playing around with Bugzilla, but like most of my colleagues I consulted with Dave on IRC before submitting. That’s when things got interesting.

    IRC is yet another new thing to me in DPS909. I’ve used chat clients like MSN, but within 5 minutes of conversing with Dave he had pulled in a Senior Software Developer from Mozilla to #seneca where we were talking. After a short bit, when we figured that the performance issue was related to Direct2D, he pulled in a “Platform Graphics Guy” at Mozilla. Bam, bam bam. Within 10 minutes of consulting with Dave on IRC I’d spoken with and been helped by two extremely knowledgeable guys located in various places around North America. The connectivity enabled by IRC still boggles me.

    With all of their help, I now had some guidance as to how to go about filing this performance bug. I knew it needed to be filed in the first place. I knew it was in some way related to Direct2D. And I knew to CC Dave on it and add the “[chromeexperiments]” tag to it. With all this new information to process, I consider myself lucky to have remembered 75% of it (thanks for adding the [chromeexperiments] tag, Dave!). After filing the bug, the Open Source wheels churned to life.

    Just as Linus’ Law applies to bug identification and resolution, it seems to also apply to bug spotting in a different light. Between the 4 viewers on Bugzilla and another two on IRC, those six sets of eyes were able to spot a duplicate bug, this one from four months prior. A dialog is now going and some of the best programmers in the world are now looking at resolving this with collated effort. Release early, release often indeed.

  • Getting ready to file my first bug using xperf

    If you’ll remember from my last blog entry, I’ve been stress-testing Minefield using Chromeexperiments in an attempt to identify bugs and bottlenecks. Without being able to identify any crashes or unusual non-performance related behaviour, I then had the task of investigating profiling tools to find the issue. And then things got messy.

    I thought things would be as simple the tool, running the test and the tool would tell you exactly what was wrong. That assumed two things: 1) That I knew how to use the tool and 2) That the tool knew how to identify problems. Turns out I started with incorrect assumptions on both. I had never used a profiler before, and the Windows SDK-supplied xperf had a lot of options. Mozilla was great at giving me the setup, but there was more to learn. Firstly, that profilers are not sentient and that their results still require analysis.

    The idea behind a profiler is that it generally runs in the background, taking samples of CPU and memory states (or other conditions still, like registry accesses or file io) periodically while all processes are running. Good ones like xperf can be configured to hook into kernel events to create a stack trace to show state at every function call.

    This tool was great for discounting some of my earlier theories by using the Windows-default perfmon tool. The  ability of xperf to monitor both CPU and memory during stack-walking disproved some of my earlier causal theories and showed me that in most cases, CPU and memory usage was eaten up within DirectDraw drivers.

    To figure out where the information I wanted was required a fair bit of googling. Even something so simple as viewing a stack walk took a bit of detective work though for those looking for the Coles notes, here’s how to capture both Heap and CPU state for each stack frame (courtesy of Mozilla and MSDN links above):

    xperf -on latency -stackwalk profile
    call xperf -start heapsession -heap -PidNewProcess “%platform% %siteToGo% %args%” -stackwalk HeapAlloc+HeapRealloc -BufferSize 512 -MinBuffers 128 -MaxBuffers 512

    %platform% would be the absolute path for the executable to run (in this case, Minefield)

    %siteToGo% is the web site you wish to test (saves on collecting unneccessary data from loading home page or google)

    %args% would be any other arguments you wish to pass to the application. In this case, I opened Minefield with “-P Testing -no-remote” (no quotes) to use my test profile. The test profile was a very stripped-down profile to eliminate any background work in Firefox which may have thrown in extra data. The complete script can be found here.

    Various options can be specified in the “On” parameter when calling xperf as Richard Russell shows, but I went with the latency option suggested above by Mozilla. The result were large (1.7 GB for 5 minutes) files, so perhaps for CPU monitoring I would have done well to stick with the “PROC_THREAD” option. Most of the size was heap data, the result of the second call to xperf but every bit helps. Once finished, I  stopped the profiler instances and merged the results:

    xperf -stop heapsession -d heap.etl
    xperf -d main.etl
    xperf -merge main.etl heap.etl result.etl

    Once the report was compiled, I loaded it and saw some complex graphs. It took some getting used to where everything was, but after a long while I learned the following best practices:

    To load symbols (so as to view symbolic function names), enter them under “Trace -> Configure Symbol Paths” and then select “Trace -> Load Symbols” to actually load the symbols. From here, the app has probably become non-responsive for a short time while it associates everything and you’re left still looking at the same graphs. Here’s where the magic happens:

    Right click on a graph (“CPU Sampling by Thread” and “Heap Total Allocation Size” work well) and select “Summary Table” or “Simple Summary Table”. From here there was a tree-view allowing a drill down from process to thread to dll to individual function calls. Richard Russell’s blog  entry was so helpful, I’ll link it here as well, when I say how much it helped describe using the GUI.

    In the end, xperf helped identify some choke-points on some of the experiments I was testing, specifically isolating the need to test both with and without hardware acceleration. Speaking of which, hardware acceleration in Minefield and the upcoming Firefox 4 can be adjusted by use of the following options (enter about:config into your Firefox address bar to adjust):

    gfx.direct2d.disabled
    gfx.direct2d.force-enabled
    gfx.font_rendering.directwrite.enabled
    layers.accelerate-all
    layers.accelerate-none

    The first 3 are for DirectDraw, which handles 2D graphics and text, while the last 2 are for Direct3D, which handles 3D drawing. This is a lot to work with to test hardware acceleration, so Joe the graphics guru from Mozilla has an easier way to test hardware acceleration (Windows only at present).

    At around  848 words this seems to be by far my largest blog post, which given that I’m blogging about what I’ve learned through bug filing must be a good thing. Ah yes, bug filing: the ends to these means. That’s the next entry in this blog post queue.

  • Performance testing Minefield and Chrome using Chrome Experiments

    Now that I have my Firefox testing environment, it’s time for my first task: Performance Testing! For this, I needed to know what to test (in this case, nightly builds of Minefield and Chromium), how to test (the Chrome Experiments) and what to determine (speed, smoothness, and responsiveness). There are quite a few experiments, but as a class of 12 we split the 120 up fairly easily. I was tasked with #71-80.

    To begin testing, I had to try and think about what variables could be eliminated. I decided to launch all experiments directly from the command line in an attempt to standardize application startup. For this I wrote a batch file script with a few parameters to help with this. I also used Windows’ built in PerfMon tool to assist with performance measurement so that I could be free to stress-test the application. With this combination, I felt I could gather information quickly and efficiently, and Perfmon also provided lots of great graphs for later analysis. It almost seemed like overkill at the time, but it helped give extremely measurable results.

    Given the target implementation platform (Chrome) for these experiments, I wasn’t surprised to find that a few of them weren’t performing quite as well on Minefield as on Google’s browser… yet. What did surprise me was the creativity and range of the experiments.

    Now that I had my data, it was time to analyze further and see if I could spot any large discrepancies, and to try and figure out why they occurred. My testing measures focused on the Working Set (Memory) and CPU utilization for the browser process. The jsCanvasBike, Animated Harmonograph and Liquid Particles seemed to require the most tuning, so my attention turned now to profiling rather than monitoring. As perfmon shows, not all of these problems were visible simply by monitoring the CPU.

    Luckily, the Windows SDK comes with their own profiling tool (xperf) which features extremely detailed kernel debugging on stack traces. Mozilla even has its own help page with instructions tailored specifically for use with Firefox. The results can look a little daunting, but it appears to be a very powerful tool and I was glad to have so many resources available. I feel as if I’ve entered a whole new realm of testing, but there is still a great deal more to experiment with.

  • Setting up a Firefox testing environment

    In light of jumping into open source development, I first decided to explore Firefox and to try and learn a bit about it, its architecture and how to accommodate a testing environment for it.

    First thing was first: downloading a prerelease version of Firefox. I had two choices: download the current (but less stable) nightly build or the more stable but less current beta. I chose to test a nightly build, but could have chosen both and simply installed to different directories. Installation was simple and straight-forward and just like release versions of Firefox. One thing of note was that pre-release versions were installed in a separate location from release versions, under the name Minefield. Genius, this way I could have both installed concurrently!

    From here, I came into an issue:my release version of Firefox had been heavily customized by me, but what if I wanted to test the default configuration? After some research I changed the properties of the shortcut for Minefield to add the command-line argument -P (apparently identical to the -ProfileManager argument), which opened up a profile manager for me to create a secondary profile. Firefox allows for different profiles (which group configuration, extensions and more) to be used to run Firefox, but there is always a default. Upon making my new profile (named Testing), Firefox opens normally. I then closed Firefox and re-modified the command line in the shortcut to include the new profile name in quotes after the -P like so:

    This was all fine and well, except that Firefox is designed so that by default only one instance can be run at once. By that I mean if run my release version, minimize and attempt to open Minefield, another instance of the already-running release version pops up. Through lecture with David Humphrey, I learned that the way to resolve this was simply to add the -no-remote command line option with the -P option (after the profile name) to allow multiple instances to be run under different profiles. With this little tweak, I now have completely independent testing environments setup for concurrently testing various Firefox builds.

  • Hello (Open Source) world!

    As a 7th semester student in Seneca’s Bachelor of Software Development program, I’ve long considered starting up a blog for documenting and generally sharing the projects I am currently involved with. Though fitting for a software student with reluctance to buy a laptop, this web developer has also been slow to make the leap into web 2.0. It’s amazing how a class, in this case DPS909, can change things about you.

    Like my aforementioned old-fashioned ways, I’ve long wondered how large projects (and even business models) can grow and thrive while remaining so disconnected and independent. Reading Eric Raymond’s paper The Cathedral and the Bazaar actually showed me that open source is neither. Following the story of fetchmail, it demonstrated how each project is disconnected only geographically, with the internet serving to connect all contributors and bring their independent selves to work together.

    Another thought of mine which is slowly being changed is how businesses work. How could Mozilla possibly turn a profit when it gives away products like Firefox for free? Many websites which don’t sell a product can turn a profit by advertising, but Firefox uses none. A recent article in the NY Times showed me how, through a “community development model” and partnerships with companies like Google. The film Revolution OS brought together both of the above for me by telling the story of the genesis of the Free Software Foundation, Linux and the first commercial enterprises.

    Having watched studied these, I feel I have a much deeper understanding of Open Source philosophy and its affect on development, distribution, and rights, not to mention the business inherent in today’s world.