Last week, my former employer theOTHERmedia open-sourced the last project I worked on there: Helium. It’s a web application that lets you deploy JavaScript packages from Git and load them on-demand into any website by including a single script tag. There’s been a lot of innovation in JavaScript deployment recently, and Helium fits a particular set of needs that I think most web agencies will be all too used to struggling with.

First, some background about the problems we were trying to solve. theOTHERmedia is a design and development agency with dozens of clients and a lot of code to keep under control. Their JavaScript stack is based around YUI and Ojay, as well as other common tools like Google Maps, Analytics etc. As part of client work, we inevitably ended up producing a lot of reusable components that didn’t really fit into the Ojay core but nevertheless needed to be shared between client projects.

Because they had no obvious place to live, these little libraries ended up being copy-pasted from project to project, producing a maintenance headache. If I fixed a bug in one copy, I’d have to track down all the other copies to patch them. Also, some clients required some small customisations: too small to merit generalising the code to accommodate them, but large enough that managed branching, merging and cherry-picking between branches would really help with maintenance.

The main day-to-day problems I’d run into were that these projects became hard to maintain, hard to track down, and that it wasn’t clear how to set them up. Developers would copy a component into their project and come to me when it didn’t work: inevitably some dependency or config setting was missing. I wanted to make it as transparent as possible to load any JavaScript object you liked into a site without running into these sorts of problems, and I wanted each library to be maintained in one repository and farmed out to client sites as required. I should be able to apply a bug fix and see it show up in all our clients’ sites immediately.

The solution to the maintenance problem was obvious: these libraries should have been under their own version control, and the need to fork libraries for certain clients and keep the forks up-to-date with bug fixes made Git the perfect choice for this. Git makes branching and merging easy, and lets you cherry-pick changes between branches if you don’t want to do a full merge. Problem one solved.

The second problem is deployment: we need to get the libraries out of Git, build them and distribute them to client sites. Clients need to specify which version (i.e. which branch or tag) of each library they want to use. And, dependencies should be handled automatically: I’m a big fan of package managers like aptitude on Ubuntu or RubyGems in the Ruby community as they make it easy to get software running without worrying about its internal dependencies.

This is where an number of other tools come in. Helium is really a wrapper around a set of smaller components that were all designed to be flexible enough that they can be trivially integrated. The main “front end” of Helium is powered by JS.Packages, a pure-JavaScript dependency manager that ships with JS.Class. It works at a very high level, letting you load JavaScript objects by name rather than by URL, figuring out dependencies for you and downloading files in parallel where possible. For example, it lets me write this:

require('YAHOO.util.Selector', 'GMap2', function() {
    var links = YAHOO.util.Selector.query('a'),
        map = new GMap2(container);
    
    // ...
});

This is very powerful, but it requires some configuration to tell JS.Packages where to find objects, what other objects they depend on and so on. To make the above work, you need the following configuration:

JS.Packages(function() { with(this) {
    file('http://yui.yahooapis.com/2.8.0/build/yahoo-dom-event/yahoo-dom-event.js')
        .provides('YAHOO',
                  'YAHOO.util.Dom',
                  'YAHOO.util.Event');
    
    file('http://yui.yahooapis.com/2.8.0/build/selector/selector-min.js')
        .provides('YAHOO.util.Selector')
        .requires('YAHOO.util.Dom');
    
    loader(function(cb) {
        var url = 'http://www.google.com/jsapi?key=' + Helium.GOOGLE_API_KEY;
        load(url, cb);
    })  .provides('google.load');
    
    loader(function(cb) { google.load('maps', '2.x', {callback: cb}) })
        .provides('GMap2', 'GClientGeocoder',
                  'GEvent', 'GLatLng', 'GMarker')
        .requires('google.load');
}});

Aside from the fact that writing this is fairly tedious, I didn’t want these config files to live in the client projects where they’d need just as much maintenance as the original libraries, especially since different branches of a library often have different dependencies.

Now most of this code tends to be of the file/provides/requires variety rather than the custom loader function variety, and this is code that’s very easy to generate. The final piece of the puzzle, Jake, is what ties this all together.

Jake is a build tool for JavaScript projects that mostly handles code concatenation and minification for larger JavaScript projects. The neat thing about it is that it lets you embed metadata (such as dependency information) and has an event system to let you know when build files are generated and where they live on disk. This means you can use it to build a manifest of all the files a project build generates, which JavaScript objects each file contains and which objects it depends on, just by placing some metadata in each project’s build configuration. We’ve now moved this data out of the client projects and into a central location: each library manages its own dependency data, and Helium uses Jake to extract this and build a manifest of all the libraries on your server and what their dependencies are.

Putting it all together, the whole process pulls projects out of Git, uses Jake to build every branch of each project and extract its dependency data, then uses this to generate a JS.Packages manifest: it’s this file that client sites include in their head section:

<script type="text/javascript"
        src="http://helium.yourcompany.com/js/helium.js">
        </script>

You then need to tell Helium which branch/tag of each library to use, and thereafter you can use require() to load any object you like. On the client side, Helium essentially acts as a versioning layer on top of JS.Packages.

<script type="text/javascript">
Helium.use('yui', '2.7.0');
Helium.use('ojay', '0.4.1');
</script>

The intent is for Helium users to host their own installations of it, since providing a single package manifest for all the JavaScript libraries on the web would produce a huge file and obviate the benefit of on-demand loading. If you want to check it out, the best place to start is the documentation on GitHub, which covers a ton of stuff I’ve not even mentioned here. In particular it explains how to make your own JavaScript projects Helium-deployable, a process I’ve tried hard to make as easy as possible. I’d love to see this adopted as a package distribution system for JavaScript, so if you have any feedback on how it can be improved, get over to GitHub and let us know!