Building JavaScript projects with Make

As a long-time Ruby and JavaScript user, I’ve seen my share of build tools. Rake, Jake, Cake, Grunt, Gulp, the Rails asset pipeline… I’ve even invented one or two of my own. I’ve always wondered why every language ecosystem feels the need to invent its own build tools, and I’ve often felt like they get in my way. Too often, Rake tasks are just wrappers around existing executables like rspec or cucumber, or require custom glue code to hook a tool into the build system – witness the explosion of grunt-* packages on npm. I find Grunt particularly problematic; its configuration is verbose and indirect, and its plugins bake in assumptions that you cannot change. For example, grunt-contrib-handlebars currently depends on handlebars~1.1.2, when the current version of Handlebars is 1.3.0. It seems strange to have your build tools choose which version of your libraries you must use.

Most of the tasks we need to build our projects can be run using existing executables in the shell. CoffeeScript, Uglify, Browserify, testing tools, PhantomJS, all have perfectly good executables already and it seems silly to require extra tooling and plugins just so I can glue them together using Grunt. In the Node community we talk a lot about ‘the Unix way’, but the current crop of build tools don’t seem Unixy to me: they’re highly coupled, obfuscatory and verbose, wrapping particular versions of executables in hundreds of lines of custom integration code and configuration.

Fortunately, Unix has had a perfectly good build tool for decades, and it’s called Make. Though widely perceived as being only for C projects, Make is a general-purpose build tool that can be used on any kind of project. I’m currently using it to build my book, generating EPUB, MOBI and PDF from AsciiDoc and checking all the code in the book is successfully tested before any files are generated. (I’ve put my build process on GitHub if you’re interested.) While writing the book, I decided to use Make for the book’s example projects themselves, and found it to be a very quick way to get all the usual JavaScript build steps set up.

In this article I’m going to build a project that uses CoffeeScript source code, Handlebars templates, and is tested using jstest on PhantomJS. To start with, let’s cover the project’s source files. The contents of these files aren’t too important, the important thing is what we need to do with the files to make them ready to run. There’s a couple of .coffee files in the lib directory, just a small Backbone model and view:

# lib/concert.coffee

Concert = Backbone.Model.extend()

window.Concert = Concert

# lib/concert_view.coffee

ConcertView = Backbone.View.extend
  initialize: ->
    @render()
    @model.on "change", => @render()

  render: ->
    html = Handlebars.templates.concert(@model.attributes)
    @$el.html(html)

window.ConcertView = ConcertView

(You could use CommonJS modules instead of window globals and build the project with Browserify; hopefully having read this it’ll be obvious how to add this to your project.)

And, we have a template for displaying a little information about a concert:

<!-- templates/concert.handlebars -->

<div class="concert">
  <h2 class="artist">{{artist}}</h2>
  <h3 class="venue">{{venueName}}, {{cityName}}, {{country}}</h3>
</div>

There are also a couple of test suites in spec/*.coffee that test the template and view from above:

# spec/concert_template_spec.coffee

JS.Test.describe "templates.concert()", ->
  @before ->
    @concert =
      artist:    "Boredoms",
      venueName: "The Forum",
      cityName:  "Kentish Town",
      country:   "UK"

    @html = $(Handlebars.templates.concert(@concert))

  @it "renders the artist name", ->
    @assertEqual "Boredoms", @html.find(".artist").text()

  @it "renders the venue details", ->
    @assertEqual "The Forum, Kentish Town, UK", @html.find(".venue").text()

# spec/concert_view_spec.coffee

JS.Test.describe "ConcertView", ->
  @before ->
    @fixture = $(".fixture").html('<div class="concert"></div>')

    @concert = new Concert
      artist:    "Boredoms",
      venueName: "The Forum",
      cityName:  "Kentish Town",
      country:   "UK"

    new ConcertView(el: @fixture.find(".concert"), model: @concert)

  @it "renders the artist name", ->
    @assertEqual "Boredoms", @fixture.find(".artist").text()

  @it "updates the artist name if it changes", ->
    @concert.set "artist", "Low"
    @assertEqual "Low", @fixture.find(".artist").text()

Now, to get from these files to working code, we need to do a few things:

Compile all the CoffeeScript to JavaScript
Compile all the Handlebars templates to a JS file
Combine the app’s libraries and compiled source code into a single file
Minify the bundled app code using Uglify
Run the tests after making sure all the files are up to date

Make is a great tool for managing these tasks, as they are essentially relationships between files. Make does not have any baked-in assumptions about what type of project you have or what languages you’re using, it’s simply a tool for organising sets of shell commands. It has a very simple model: you describe which files each build file in your project depends on, and how to regenerate build files if they are out of date. For example, if file a.txt is generated by concatenating b.txt and c.txt, then we would write this rule in our Makefile:

a.txt: b.txt c.txt
	cat b.txt c.txt > a.txt

The first line says that a.txt (the target) is generated from b.txt and c.txt (the dependencies). a.txt will only be rebuilt if one of its dependencies was changed since a.txt was last changed; Make always tries to skip unnecessary work by checking the last-modified times of files. The second line (the recipe) says how to regenerate the target if it’s out of date, which in this case is a simple matter of piping cat into the target file. Recipe lines must begin with a tab, not spaces; I deal with this by adding the following to my .vimrc:

autocmd filetype make setlocal noexpandtab

Let’s start by installing the dependencies for this project. Add the following to package.json and run npm install:

{
  "dependencies": {
    "backbone":      "~1.1.0",
    "underscore":    "~1.5.0"
  },

  "devDependencies": {
    "coffee-script": "~1.7.0",
    "handlebars":    "~1.3.0",
    "jstest":        "~1.0.0",
    "uglify-js":     "~2.4.0"
  }
}

This installs all the build tools we’ll need, and all of them have executables that npm places in node_modules/.bin.

Let’s write a rule for building our Handlebars templates. We want to compile all the templates – that’s templates/*.handlebars – into the single file build/templates.js. Here’s a rule for this:

PATH  := node_modules/.bin:$(PATH)
SHELL := /bin/bash

build/templates.js: templates/*.handlebars
	mkdir -p $(dir $@)
	handlebars templates/*.handlebars > $@

The first line adds the executables from npm to the Unix $PATH variable so that we can refer to, say handlebars by its name without typing out its full path. (Installing programs ‘globally’ just means installing them into a directory that is usually listed in $PATH by default.) The first line of the recipe uses mkdir to make sure the directory we’re compiling the templates into already exists; $@ is a special Make variable that contains the pathname of the target we’re trying to build, and the dir function takes the directory part of that pathname.

The rule duplicates the names of the source and target files, and we often use variables to remove this duplication:

PATH  := node_modules/.bin:$(PATH)
SHELL := /bin/bash

template_source := templates/*.handlebars
template_js     := build/templates.js

$(template_js): $(templates_source)
	mkdir -p $(dir $@)
	handlebars $(templates_source) > $@

With this Makefile, running make in the shell will generate build/templates.js, or do nothing if that file is already up to date.

$ touch templates/concert.handlebars 

$ make
mkdir -p build/
handlebars templates/*.handlebars > build/templates.js

$ make
make: `build/templates.js' is up to date.

Next up, we need to compile our CoffeeScript. We want to say that every file lib/foo.coffee generates a corresponding file build/lib/foo.js, and likewise every file spec/foo_spec.coffee generates build/spec/foo_spec.js. In Make, we can use the wildcard function to find all the names of the CoffeeScript files in lib and spec, and generate lists of JavaScript files from those names using pattern substitution. In Make, the expression $(files:%.coffee=build/%.js) means for every filename in the list files, replace %.coffee with build/%.js, for example replace lib/foo.coffee with build/lib/foo.js. We also use a pattern-based rule to describe how to compile any CoffeeScript file to its JavaScript counterpart. Here are the rules:

source_files := $(wildcard lib/*.coffee)
build_files  := $(source_files:%.coffee=build/%.js)

spec_coffee  := $(wildcard spec/*.coffee)
spec_js      := $(spec_coffee:%.coffee=build/%.js)

build/%.js: %.coffee
	coffee -co $(dir $@) $<

We need to generate the names of all the generated JavaScript files because later targets will depend on them. If a target simply depended on build/*.js but we’d not built those files yet, the build wouldn’t work correctly. With this configuration, Make sets the variables to these values:

source_files := lib/concert.coffee lib/concert_view.coffee
build_files  := build/lib/concert.js build/lib/concert_view.js
spec_coffee  := spec/concert_template_spec.coffee spec/concert_view_spec.coffee
spec_js      := build/spec/concert_template_spec.js build/spec/concert_view_spec.js

So, Make now knows the names of all the generated files before they exist. The recipe for CoffeeScript states the any file build/foo/bar.js is generated from foo/bar.coffee, and uses coffee -co to make coffee compile each file into a given output directory. We use $@ as before to get the name of the current target, and $< gives the name of the first dependency of the current target. These variables are essential when dealing with pattern-based rules like this.

Pattern rules are invoked if Make has not been told explicitly how to build a particular file. If you run make build/lib/concert.js you’ll see that it generates the named file from the pattern rule.

Now that we’ve compiled all our code, we can concatenate and minify it. We want to take all the files in build_files, and template_js, and any third party libraries we need, and use Uglify to compress them. A rule for this is straightforward; note how app_bundle depends on the upstream files so that if any of them change, Make knows it needs to rebuild app_bundle.

app_bundle := build/app.js

libraries  := vendor/jquery.js \
              node_modules/handlebars/dist/handlebars.runtime.js \
              node_modules/underscore/underscore.js \
              node_modules/backbone/backbone.js

$(app_bundle): $(libraries) $(build_files) $(template_js)
	uglifyjs -cmo $@ $^

Here we’ve used another of Make’s automatic variables: $^ is a list of all the dependencies, separated with spaces. It’s handy when all your recipe does is combine all the dependencies into an aggregate file.

It’s customary to make a target called all that depends on all your project’s compiled files, and make this the first rule in the Makefile so that running make will run this rule. The all rule is also what’s called ‘phony’: all is not the name of an actual file, it’s just the name of a task we want to run, and so Make should not look for a file called all and check its last-modified time before proceeding. Targets are marked as phony by making them dependencies of the special .PHONY target.

.PHONY: all

all: $(app_bundle)

And finally, we need a task to run our tests. Let’s create a little web page for doing that:

<!-- test.html -->

<!doctype html>
<html>
  <head>
    <meta charset="utf-8">
    <title>jstest</title>
  </head>
  <body>

    <div class="fixture"></div>

    <script src="./build/app.js"></script>

    <script src="./node_modules/jstest/jstest.js"></script>
    <script src="./build/spec/concert_template_spec.js"></script>
    <script src="./build/spec/concert_view_spec.js"></script>

    <script>
      JS.Test.autorun()
    </script>

  </body>
</html>

and a PhantomJS script for launching this page and displaying the results:

// phantom.js

var JS = require("./node_modules/jstest/jstest")

var reporter = new JS.Test.Reporters.Headless({})
reporter.open("test.html")

and to top it all off, a Make task that runs the tests after making sure all the files are up to date (this task is also phony since it does not generate any files):

test: $(app_bundle) $(spec_js)
	phantomjs phantom.js

It’s also customary to add a phony task called clean that deletes any generated files from the project, putting it back in its ‘clean’ state:

clean:
	rm -rf build

So, the whole finished Makefile looks like this, containing instructions for compiling all the source code, building a single app bundle, and running the tests:

PATH  := node_modules/.bin:$(PATH)
SHELL := /bin/bash

source_files    := $(wildcard lib/*.coffee)
build_files     := $(source_files:%.coffee=build/%.js)
template_source := templates/*.handlebars
template_js     := build/templates.js
app_bundle      := build/app.js
spec_coffee     := $(wildcard spec/*.coffee)
spec_js         := $(spec_coffee:%.coffee=build/%.js)

libraries       := vendor/jquery.js \
                   node_modules/handlebars/dist/handlebars.runtime.js \
                   node_modules/underscore/underscore.js \
                   node_modules/backbone/backbone.js

.PHONY: all clean test

all: $(app_bundle)

build/%.js: %.coffee
	coffee -co $(dir $@) $<

$(template_js): $(template_source)
	mkdir -p $(dir $@)
	handlebars $(template_source) > $@

$(app_bundle): $(libraries) $(build_files) $(template_js)
	uglifyjs -cmo $@ $^

test: $(app_bundle) $(spec_js)
	phantomjs phantom.js

clean:
	rm -rf build

If you run make test, you’ll see all the compiled files are generated on the first run, but not regenerated afterward. They will only be regenerated if the source files change and you run a task that depends on them. Expressing the dependencies between files lets Make save you a lot of time waiting for things to unnecessarily recompile.

$ make test 
coffee -co build/lib/ lib/concert.coffee
coffee -co build/lib/ lib/concert_view.coffee
mkdir -p build/
handlebars templates/*.handlebars > build/templates.js
uglifyjs -cmo build/app.js vendor/jquery.js node_modules/handlebars/dist/handlebars.runtime.js \
                           node_modules/underscore/underscore.js node_modules/backbone/backbone.js \
                           build/lib/concert.js build/lib/concert_view.js build/templates.js
coffee -co build/spec/ spec/concert_template_spec.coffee
coffee -co build/spec/ spec/concert_view_spec.coffee
phantomjs phantom.js
Loaded suite: templates.concert(), ConcertView

....

Finished in 0.015 seconds
4 tests, 4 assertions, 0 failures, 0 errors

We’ve got rather a lot done with very little configuration, and no need for plugins to glue the programs we want to use into Make. We can use whatever programs we want, Make will happily execute whatever we tell it to without us needing to write any glue code between Make and the compiler tools themselves. That means fewer things to install, audit and keep up to date, and more time getting on with your project.

You can add whatever build steps you like to these recipes, so long as you follow the pattern of describing relationships between files. You can add in other testing tools like JSHint if you like, and even make them into dependencies of other tasks so that the downstream tasks won’t run unless your tests are good. That’s how I build my book: the dependencies are set up so that the book won’t build unless all the example tests pass first.

Having all these steps automated is a big time-saver, and setting them up so quickly without needing to install any tools beyond what comes with Unix means you’ve no excuse not get your project organised. It also encourages you to write functionality you need as generic scripts, rather than hiding it away inside plugins that only work with a particular build system. You save time and the whole community benefits. Not bad for a build tool from the eighties.

The If Works

by James Coglan

Building JavaScript projects with Make