Do I need DI?

This article is based on a talk of the same name, on the topic of dependency injection, which I gave at Eurucamp 2012. I thought I would write it up since it’s of some relevance to the current discussion about TDD, and I frequently find myself wanting to reiterate that talk’s argument. It is only related to current events insofar as I find them emblematic of most conversations about TDD; since the talk is nearly two years old it does not directly address the current debate but tries to illustrate a style of argument I wish were more common.

DHH’s keynote at RailsConf 2014 and his ensuing declaration that TDD is dead have got the Ruby community talking testing again. My frustration with the majority of talks and writing on testing, TDD, agile development, software architecture and object-oriented design is that a great deal of it consists of generalised arguments, with very little in the way of concrete case studies. Despite the fact that I’m a perfectly capable programmer, and the fact I’ve written automated tests for all sorts of different kinds of software, I’ve sat through any number of talks on these topics that went right over my head. The impression I’m left with is that talks on TDD often only make sense if the audience already understands and agrees with the speaker’s argument. They are often littered with appeals to authority figures and books that only land if the listener knows of the cited authority or has read the particular book. At worst, they are presented as ‘movements’, wherein all the old ways of thinking must be discarded in favour of the new one true way.

Now, grand general assertions make for entertaining and engaging talks, but they are unhelpful. Real-world software development is an exercise in design, and to design things based on general rules without reference to the context of the problem at hand is to not practice design properly. We need to learn to make decisions based on the properties of the problem we’re dealing with, not by appealing to advice handed down without any context.

This is why JavaScript Testing Recipes almost entirely concerns itself with the application of a few simple techniques to a wide array of real programs, including discussion of cases where those techniques are not applicable or are too costly due to the requirements of a particular problem. I wish more talks were like this, and it’s what I was hoping to do when I wrote this talk originally.

So, this article is not really about dependency injection per se, but about how you might decide when to use it. DI, much like TDD or any other programming support tool, is not de-facto good or bad; but has pros and cons that determine when you might decide to use it. It is not even good or bad in the context of any defined problem; it is a design tool and its usefulness varies according to the mental model of the person using it.

When we ask whether an object-oriented program is well-designed, we often use proxies for this question, like asking whether the code obeys the ‘SOLID principles’:

  • Single responsibility principle
  • Open/closed principle
  • Liskov substitution principle
  • Interface segregation principle
  • Dependency inversion principle

We might look at each class in the system, decide whether or not it obeys each of these principles, and if most of the classes in the system are SOLID then we declare the system to be well-designed. However, this approach encourages us to factor our code in a certain way without regard for what that code means, what it models, how that model is realised in code, and who it is intended to be used by. We should not treat these principles as normative rules, but as observations: programs that are easy to maintain tend to exhibit these properties more often than not, but they are not a barometer against which to accept or reject a design.

Across the fence from the people advocating for SOLID, and TDD, and various prescriptive architectural styles, are the people who want to tell you off for following these rules, assessing your designs by whether you’re practicing ‘true’ OO, or becoming an architecture astronaut, reinventing Java’s morass of AbstractBeanListenerAdapterFactory classes.

I don’t find the rhetoric of either of these camps helpful: both offer decontextualised normative advice based on statistical observations of systems they happen to have worked on or heard of. They don’t know the system you’re working on.

Your job is not to keep Alan Kay happy. Your job is to keep shipping useful product at a sustainable pace, and it’s on you to choose practices that help you do that.

So let’s talk about DI, and when to use it. To begin with, what is dependency injection? Take a look at the following code for a putative GitHub API client:

class GitHub::Client
  def get_user(name)
    u = URI.parse("https://api.github.com/users/#{name}")
    http = Net::HTTP.new(u.host, u.port)
    http.use_ssl = true
    response = http.request_get(u.path)
    if response.code == '200'
      data = JSON.parse(response.body)
      GitHub::User.new(data)
    else
      raise GitHub::NotFound
    end
  end
end

In order to get the account data for a GitHub user, this class needs to make an HTTP request to api.github.com. It does this by creating an instance of Net::HTTP, using that to make a request, checking the request is successful, parsing the JSON data that comes back, and constructing a user object with that data.

An advocate for TDD and DI might quibble with the design of this class: it’s too hard to test because there’s no way, via the class’s interface to intercept its HTTP interactions with an external server – something we don’t want to depend on during our tests. A common response to this in Ruby is that you can stub out Net::HTTP.new to return a mock object, but this is often problematic: stubbing out singleton methods does not just affect the component you’re testing, it affects any other component of the system or the test framework that relies on this functionality. In fact, DHH’s pet example of bad DI uses the Time class, which is particularly problematic to stub out since testing frameworks often need to do time computations to enforce timeouts and record how long your tests took to run. Globally stubbing out Date is so problematic in JavaScript that JSTR dedicates a six-page section to dealing with dates and times.

Dependency injection is one technique for solving this problem: instead of the class reaching out into the global namespace and calling a singleton method to fabricate an HTTP client itself, the class is given an HTTP client as an argument to its constructor. That client offers a high-level API that makes the contract between the class and its dependency clearer. Say we want the GitHub::Client class to only deal with knowing which paths to find data at, and how to map the response data to domain objects. We could push all the concerns of parsing URIs, making requests, checking response codes and parsing the body – that is, all the concerns not specifically related to GitHub – down into a high-level HTTP client, and pass that into the GitHub::Client constructor.

class GitHub::Client
  def initialize(http_client)
    @http = http_client
  end

  def get_user(name)
    data = @http.get("/users/#{name}").parsed_body
    GitHub::User.new(data)
  end
end

This makes the class easier to test: rather than globally stubbing out Net::HTTP.new and mocking its messy API, we can pass in a fake client during testing and make simple mock expectations about what should be called.

So dependency injection means passing in any collaborators an object needs as arguments to its constructor or methods. At its most basic, this means removing any calls to constructors and other singleton methods and passing in the required objects instead. But, this is often accompanied by refactoring the objects involved to make their interactions simpler and clearer.

So that’s what DI is, but what is it for? Many advocates will tell you that it exists to make testing easier, and indeed it does often accomplish that. But testability is not usually an end-user requirement for a component; being testable is often a proxy for being usable. A class is easy to test if it’s easy to set up and use, without requiring a complex environment of dependencies in order for the class to work properly. Applying dependency injection solely to achieve better tests is a bad move: design is all about dealing with constraints, and we must take all the applicable constraints into consideration when designing a class’s API. The patterns we use must emerge in response to the program’s requirements, rather than being dogmatically imposed in the name of ‘best practices’.

This is best illustrated by a concrete example. I maintain a library called Faye, a pub/sub messaging system for the web. It uses a JSON-based protocol called Bayeux between the client and the server; the client begins by sending a handshake message to the server:

{
  "channel":  "/meta/handshake",
  "version":  "1.0",
  "supportedConnectionTypes": ["long-polling", "callback-polling", "websocket"]
}

and the server replies with a message containing an ID for the client, and a list of transport types the server supports:

{
  "channel":    "/meta/handshake",
  "successful": true,
  "clientId":   "cpym7ufcmkebx4nnki5loe36f",
  "version":    "1.0",
  "supportedConnectionTypes": ["long-polling", "callback-polling", "websocket"]
}

Once it has a client ID, the client can tell the server it wants to subscribe to a channel by sending this message:

{
  "channel":      "/meta/subscribe",
  "clientId":     "cpym7ufcmkebx4nnki5loe36f",
  "subscription": "/foo"
}

and the server acknowledges the subscription:

{
  "channel":    "/meta/subscribe",
  "clientId":   "cpym7ufcmkebx4nnki5loe36f",
  "successful": true
}

After the client has subscribed to any channels it is interested in, it sends a ‘connect’ message that tells the server the it wants to poll for new messages:

{
  "channel":  "/meta/connect",
  "clientId": "cpym7ufcmkebx4nnki5loe36f"
}

When the server receives a message on any channel the client is subscribed to, it sends a response to the connect message with the new message attached. The client then sends another connect message to poll for more messages. (When using a socket-based connection, new messages can be pushed to the client immediately, without a pending connect request; the connect messages then act as a keep-alive heartbeat mechanism.)

So Faye is fundamentally a client-server system rather than a true peer-to-peer one. The architecture as we understand it so far is very simple:

                          +--------+
                          | Client |
                          +---+----+
                              |
                              V
                          +--------+
                          | Server |
                          +--------+

However there is more to it than that. Notice the supportedConnectionTypes field in the handshake messages: the client and server allow these messages to be sent and received via a number of different transport mechanisms, including XMLHttpRequest, JSON-P, EventSource and WebSocket. So we can add another layer of indirection to our architecture:

                          +--------+
                          | Client |
                          +---+----+
                              |
                              V
    +-----------+-------------+----------------+-------+
    | WebSocket | EventSource | XMLHttpRequest | JSONP |
    +-----------+-------------+----------------+-------+
                              |
                              V
                          +--------+
                          | Server |
                          +--------+

We now have a question: the Server class allows for multiple implementations of the same concept – sending messages over an HTTP connection – to co-exist, rather than having one mechanism hard-coded. Above, we allowed our GitHub::Client class to take an HTTP client as a parameter, letting us change which HTTP client we wanted to use. Surely we have another opportunity to do the same thing here, to construct servers with different kinds of connection handlers, including the possibility of using a fake connection handler to make testing easier.

xhr_server         = Server.new(XMLHttpRequestHandler.new)
jsonp_server       = Server.new(JSONPHandler.new)
eventsource_server = Server.new(EventSourceHandler.new)
websocket_server   = Server.new(WebSocketHandler.new)

If we’re doing DI by the book, this seems like the right approach: to separate the concern of dealing with the abstract protocol embodied in the JSON messages from the transport and serialization mechanism that delivers those messages. But it doesn’t meet the constraints of the problem: the server has to allow for clients using any of these transport mechanisms. The whole point of having transport negotiation is that different clients on different networks will have different capabilities, and the central server needs to accommodate them all. So, although there is a possibility for building classes that all implement an abstract connection handling API in different ways (and this is indeed how the WebSocket portion of the codebase deals with all the different WebSocket protocol versions – see the websocket-driver gem), the server must use all of these connection handlers rather than accepting one of them as a parameter.

What about the client side? The Client class uses only one of the four transport classes at a time, to mediate its communication with the server. The client depends on having a transport mechanism available, but the choice of which transport to use can be deferred until runtime. This seems like a more promising venue to apply DI. So, should we make its API look like this?

var url    = 'http://www.example.com/faye',
    client = new Client(new WebSocketTransport(url));

Again, the answer is no. Allowing Client to accept a transport implementation as a constructor parameter gives the system that creates the client (whether that’s a user directly writing Faye client code or part of an application that makes use of Faye) a free choice over which transport to use. But, the choice of transport in this situation is not a free choice; it’s determined by the capabilities of the host browser runtime, the transports that the server says it supports, and which transports we can detect as actually working over the user’s network connection to the server – even if the client and server claim to support WebSocket, an intermediate proxy can stop WebSocket from working.

It is actually part of the client’s responsibility to choose which transport to use, based on automated analysis of these constraints at runtime. The same application, running on different browsers on different networks, will require different transports to be used, even throughout the lifetime of a single client instance. If the Client constructor took a transport as an argument, then the user of the Client class would have to conduct all this detective work themselves. Hiding this work is a core problem the client is designed to solve, so it should not accept a transport object via injection. Even though it would improve testability, it would result in the wrong abstraction being presented to the user.

Finally, let’s look at the internal structure of the Server class. It’s actually composed of multiple layers, and until version 0.6 those layers looked something like this:

    +--------------+
    |  RackServer  |    --> * WebSocket, XHR, JSON-P
    +-------+------+
            |                           |   message
            V                           V   objects
    +---------------+
    | BayeuxHandler |   --> * Bayeux message protocol
    +---------------+       * Connections / subscriptions
                            * Message queues

The RackServer deals with the front-end HTTP handling logic, processing all the different transport types and extracting JSON messages from them. Those JSON messages are then handed off to the BayeuxHandler, which contains all the transport-independent protocol logic. It processes the contents of the various JSON messages I showed you above, stores client IDs and their subscriptions and message queues, and routes incoming messages to all the subscribed clients. Now, some of the concerns of this class can be factored out:

    +--------------+
    |  RackServer  |    --> * WebSocket, XHR, JSON-P
    +-------+------+
            |                           |   message
            V                           V   objects
    +---------------+
    | BayeuxHandler |   --> * Bayeux message protocol
    +---------------+     +-------------------------------+
                          | * Connections / subscriptions |
                          | * Message queues              |
                          +-------------------------------+
                                        STATE

The BayeuxHandler class actually deals with two things: implementing the protocol as described by the JSON messages, and storing the state of the system: which clients are active, which channels they are subscribed to, and which messages are queued for delivery to which clients. There are many potential ways of implementing this state storage without changing the details of the protocol, and so in version 0.6 this concern was extract into an Engine class:

    +--------------+
    |  RackServer  |    --> * WebSocket, XHR, JSON-P
    +-------+------+
            |                           |   message
            V                           V   objects
    +---------------+
    | BayeuxHandler |   --> * Bayeux message protocol
    +---------------+
            |
            V
    +--------------+
    |    Engine    |    --> * Subscriptions
    +--------------+        * Message queues

There are two engine implementations available: in-memory storage, and a Redis database using Redis’s pub/sub mechanism for IPC.

                      +--------------+
                      |  RackServer  |
                      +-------+------+
                              |
                              V
                      +---------------+
                      | BayeuxHandler |
                      +---------------+
                              |
                    +---------+---------+
                    |                   |
            +--------------+     +-------------+
            | MemoryEngine |     | RedisEngine |
            +--------------+     +-------------+

Finally: we have an honest candidate for dependency injection: whether the stack uses the in-memory or Redis engine makes absolutely no difference to the rest of the stack. It’s a contained implementation detail; given any object that implements the Engine API correctly, the BayeuxHandler and all of the components that sit above it will not be able to tell the difference. The choice of engine is a free choice that the user can make entirely on their own terms, without being bound by environmental constraints as we saw with client-side transport negotiation.

However, we don’t have multiple choices at the BayeuxHandler layer of the stack: there is only one Bayeux protocol, it’s an open standard, and there aren’t multiple competing implementations of this component. It’s just in-process computation that takes messages extracted by the RackServer, validates them, determines their meaning and delegates any state changes to the engine.

So, BayeuxHandler can be parameterized on which engine object it uses, but the Server will always construct a BayeuxHandler (as well as any transport-handling objects it needs). A highly simplified version of RackServer than only deals with HTTP POST would look like this, taking an engine as input and handing it down to the BayeuxHandler:

class RackServer
  def initialize(engine)
    @handler = BayeuxHandler.new(engine)
  end

  def call(env)
    request  = Rack::Request.new(env)
    message  = JSON.parse(request.params['message'])
    response = @handler.process(message)
    [
      200,
      {'Content-Type' => 'application/json'},
      [JSON.dump(response)]
    ]
  end
end

A user would then start up the server like so:

server = RackServer.new(RedisEngine.new)

thin = Rack::Handler.get('thin')
thin.run(server, :Port => 80)

Now, in this scenario we get good tests, that make mock expectations about one object’s contract with another without stubbing globally visible bindings, as a side effect of the code fulfilling its design requirements, and of it being easy to use at the right level of abstraction for the problem it solves. Here are a couple of tests that assert the server tells the engine to do the right things, and that the server relays information generated by the engine back to the client.

require './rack_server'
require 'rack/test'

describe RackServer do
  include Rack::Test::Methods

  let(:engine) { double('engine') }
  let(:app)    { RackServer.new(engine) }

  describe 'handshake' do
    let(:message) { {
      'channel' => '/meta/handshake',
      'version' => '1.0',
      'supportedConnectionTypes' => ['long-polling']
    } }

    it 'tells the engine to create a new client session' do
      expect(engine).to receive(:create_client).and_return 'new_client_id'
      post '/bayeux', :message => JSON.dump(message)
    end

    it 'returns the new client ID in the response' do
      engine.stub(:create_client).and_return 'new-client-id'
      post '/bayeux', :message => JSON.dump(message)
      expect(JSON.parse(last_response.body)).to include('clientId' => 'new-client-id')
    end
  end
end

Even when we do decide to use dependency injection, we face some trade-offs. By making the external interface for constructing an object more complicated, we gain some flexibility but lose some convenience. For example, a convenient way to read files looks like this:

File.read('path/to/file.txt')

while a flexible way to read files might look like this:

FileInputStream fis = new FileInputStream("path/to/file.text");
DataInputStream dis = new DataInputStream(fis);
BufferedReader br = new BufferedReader(dis);
// ...

Yes, I have picked extreme examples and I’ve probably got the Java version wrong. The important point is, you can build the former API on top of the latter, but not the other way around. You can wrap flexible building blocks in a convenience API – just look at jQuery or any UI toolkit – but it’s much harder or impossible to go the other way around. Suppose you want to use the file I/O code independently of the string encoding code? File.read(path) does not expose those building blocks so you’re going to need to find them somewhere else.

When I was at Songkick, we built the clients for our backend services much like the GitHub example above: each client is instantiated with an HTTP client adapter, letting us easily switch which HTTP library we used (yes, we had to do this in production and it took an afternoon), as well as letting us pass in a fake for testing. But for calling these clients from a controller, we wrapped them in a convenience module that constructed canonical instances for us:

module Services
  def self.github
    @github ||= GitHub::Client.new(HttpClient.new('https://api.github.com'))
  end

  # and so on for other services
end

So controller code just made a call like Services.github.get_user('jcoglan'), which is pretty much as convenient as ActiveRecord::Base.find().

To summarise, there is certainly a place for DI, or for any architectural technique, but you must let the requirements of the problem – not just testing concerns – drive the patterns you use. In the case of DI, I reach for it when:

  • There are multiple implementations of an abstract API that a caller might use
  • The code’s client has a free choice over which implementation to use, rather than environmental factors dictating this choice
  • To provide plugin APIs, as in the case of Faye’s engine system or the protocol handlers in websocket-driver

Beyond DI, I would like to see much more of our design discussion focus on just that: design, in context, with case studies, without deferring to generalisation. Design patterns must be derived from the program’s requirements, not imposed in an attempt to shoe-horn the program into a predefined shape. Ultimately, rather than focussing on testing for its own sake, we must focus on usability. Testability will soon follow.

I’m not helping

I just arrived home from the Realtime Conference in Lyon, France. It was a terrific event full of interesting people talking about a diverse range of topics in beautiful surroundings, and I’m hugely grateful to the organisers for inviting me. I could wax lyrical about some of the technical things I learned there but I’m going to focus on the topic of Adam Brault’s incredibly honest closing speech, and a topic I rarely discuss here: people.

RealtimeConf was just one in a series of events I’ve attended recently where I got to talk to people about their experience of community events and interpersonal relationships. I hope I can write something about community in general in the future, but this post is about me. I apologise in advance for the grotesque level of self-indulgence here but I need to put this on the record. I have not named anyone who informed this article through one-on-one conversation with me; I am tremendously thankful for their input and if you recognize yourself in this story and I have misrepresented you, please let me know and I will make corrections.

So. I don’t know of any way to say this without seeming boastful or pretentious, but: I have somehow become moderately well-known in the JavaScript and Ruby communities. What I mean by that is that it’s not infrequently that I meet people who ‘know me from the internet’, because of my open-source projects, my blog, conference talks, or my Twitter feed. Actually, particularly my Twitter feed. People tell me they like my rants, or ‘diatribes’ as one friend put it. And indeed I’ve met various great people, and through some slightly bizarre circumstances, made some in-real-life friends because people find my verbose streams of incoherence on matters technological entertaining.

But on the other hand, I’ve been called out by several people for being too negative, too angry, for swearing too much, for disparaging other’s work without constructive feedback, for posting too frequently, and generally making people feel bad. And before anyone feels obliged to leap to my defence or tell me you can’t please everyone: these are all valid criticisms. I’m a nasty person sometimes. People see different sides of you, and have differing opinions of those sides, but the aforelisted behaviours are matters of observable fact. I do those things.

Okay, James, enough with the navel-gazing. Why does any of this matter?

The thing about being well-known is it creeps up on you. It’s very hard to tell, when you walk into a conference hall full of strangers, how many people know who you are. It still comes as a surprise to me when people have heard of me or know my work, and I still feel as anonymous walking into a venue as I did at the very first developer conf I went to in 2007. But the illusion of anonymity soon fades when people come and introduce themselves.

I’ve met a lot of new people recently. RealtimeConf was my third conf in as many weeks, I’m going to another next week, and at the beginning of March I did /dev/fort. And several of those people have had a reaction somewhere in the region of: “wow, you’re a nice person in real life”. Now I won’t claim that’s always the case either, but it’s certainly disappointing to find out that people’s expectation, based solely on following me on Twitter, is that I’ll be a sarcastic, negative troll who’s no fun to be around.

While it sucks to find out people are apprehensive about meeting me, the straw that broke the camel’s back is something another one of the speakers told me yesterday: they were nervous about giving their talk knowing I was in the audience. My reputation for mouthing off about the minutiae of software made someone worry their material would risk setting me off. It’s one thing to know I might not get to meet someone because I put them off, it’s another to know they’re having a worse time on stage on my account.

Public speaking is regularly listed as many people’s greatest fear. As Adam Brault said in his talk, “your heroes are all scared, too.” It’s so true. I don’t know anyone who isn’t at least nervous about speaking in public. I know some remarkable speakers who have almost paralysing fears about what will happen on stage. I’ve seen friends pretty much pass out from exhaustion after presenting. Me, I feel nauseous for about an hour beforehand, I have to pee every five minutes, I sweat like crazy and I’m scared I’ll go into panic attack on stage. Oh and I stuttered as a child and it comes back when I’m not confident in what I’m saying. I don’t know why any of us choose to do it sometimes, and I have huge respect for anyone that even tries. I want the person on stage to know that, and not instead feel scared that I’m looking for ways to trip them up.

I’m ecstatic if I get one laugh during a whole talk. It’s the only way you know the audience is on your side.

Anyway, to get to the point. I make software and give it away and answer people’s questions about it because those are the best ways I have of helping people. I want people to expect that I’ll want to be helpful in real life, but that’s not the signal I send out sometimes. I’m sorry about that. Context matters, and I’m sure I’ll still indulge the people I know are comfortable with my snarkiness when I know I’m talking directly to them, just as good friends are allowed to insult each other without getting hurt. But I ought to remember I’ve not met most of the people who follow me, and some of them look up to me for my programming work. If you only know me from Twitter, you’re mostly seeing my angry side.

One of the biggest lessons I’ve taken from talking to people about community is that it pays to be up-front and explicit about what you expect from people and what they should expect from you, even when (or maybe especially when) those expectations seem obvious. In that spirit, I’m asking anyone that follows me online to tell me when I’m overstepping the line. Don’t hold back because I’m an ‘expert’ on something or because you think I’ll get defensive. Several people have done this in person in recent weeks and I was grateful to everyone one of them. I’m tired of being the angry programmer from the internet, so I have to change people’s expectations. Telling me when I’m getting it wrong helps that.

Hopefully next time I meet one of you, you won’t be so surprised.

Cross-platform JavaScript testing

Last week I gave a talk at the London Ajax User Group on testing JavaScript software across different platforms. I wanted to share the talk with a wider audience so what follows is an essay version of the talk; it’s what I planned on saying before I got nervous, fluffed my lines and spoke too fast.

I want to start with a quick history lesson. Cast your mind back to 2006. We had a few browsers out in the wild, not as many as are in mainstream use as today but enough to give us a headache, especially with IE6 still dominating the stats. The problem we had back then was that all the browsers behaved in slightly (sometimes vastly) different ways and had different scripting APIs. Standards were being slowly rolled out but the overhead of dealing with browser quirks was still very high.

So, around 2005 and 2006 we see two projects that try to fix the situation: Prototype and jQuery. They both aimed to normalize and improve the scripting API across all browsers, so you could be more productive and be more confident that your code would work across the board. They’ve both been really successful and since then we’ve seen a lot more projects that have their own take on how we should tame the browsers; projects like YUI, MooTools and Dojo.

Skip forward five years, and the landscape looks quite different. We’ve got a few more browsers in mainstream use, especially with the mobile web becoming a mainstream platform, but we’ve also got JavaScript being used all over the place. People are finally taking it seriously for server-side work, it’s being embedded in databases, it’s everywhere. Node has kick-started this, but there’s a ton of frameworks based on Rhino.

So what’s the problem this time? Well, it’s not so obvious. These platforms all have different APIs but that’s largely because they’re used for different things; you probably don’t care that your web front-end won’t run on CouchDB. But one of the promises we’ve heard for years and years about server-side JavaScript is that it’ll amplify code reuse. You’ll be able to share business logic between the client and the server. Remember ‘write once, run anywhere’? We were actually going to make that work.

But what we’re actually seeing is needless specialization. Application frameworks for Node, testing libraries for Node, template languages for Node, API clients for Node. I don’t mean to pick on Node; it’s been the same with jQuery, although fortunately jQuery became so pervasive that depending on it wasn’t so much of a problem. But we’re seeing the same pattern all over again with Node and this seems like a wasted opportunity to me. Node’s popularity is bringing client-side developers onto the server, and it would be great of this helped unify our efforts.

Now I didn’t want to single any particular project out for criticism, because really I’m just as guilty as anybody else. I maintain a project called Faye which bills itself as a pub/sub messaging server for Node. But the truth is, most of the internal logic about managing clients and subscriptions is just pure JavaScript, Node is just used to make this logic accessible over the Internet. You could probably tear the Node part out and run the server in a browser, if you felt like it.

So we have a problem with portabilty, we’re wasting time reinventing the same wheels on every platform we migrate to. And to solve this, to help us share more code, I don’t think we need to normalize APIs like we did in the browser. Variety and innovation are good things, and people should have some choice about which platforms they target. But for those of us that want to write cross-platform code, I think we’re going to need testing tools that work everywhere.

Now, some back story. Since 2007 I’ve been working on this project called JS.Class. It’s an object system for JavaScript that’s based on Ruby, so it gives you classes and mixins and a few of Ruby’s object methods. It comes with a class library of common object-oriented idioms and data structures, and it tries to establish some conventions for things like how to compare objects for equality, how to sort them, store them in hash tables, and what-have-you.

OrderedHash = new JS.Class('OrderedHash', {
  include: JS.Enumerable,
  
  initialize: function() {
    // ...
  },
  
  forEach: function(callback, context) {
    // ...
  }
})

It lets you make a class like this. Say I want a new data structure, which we’ll call an ordered hash. I can create it, add the Enumerable methods to it, give an initializer, tell the Enumerable module how to iterate over it, and add other methods. Pretty standard stuff.

Then some time in around 2009, I added JS.Packages. It’s a package manager, and the aim is to seperate out the logic of how to load code and dependencies from the code itself. So you say I’ve got this file /lib/ordered_hash.js, it defines OrderedHash, and it depends on JS.Class and JS.Enumerable. You can use local files or files off other domains. Then when you need to use an object you just require() it and JS.Packages will make sure it’s loaded then run your code.

JS.Packages(function() { with(this) {
  
  file('/lib/js/ordered_hash.js')
    .provides('OrderedHash')
    .requires('JS.Class', 'JS.Enumerable')
  
  file('http://cdn.google.com/jquery.js')
    .provides('jQuery', '$')
}})

JS.require('jQuery', 'OrderedHash', function() {
  var links = $('a'),
      hash  = new OrderedHash()
  
  // ...
})

The reason it focuses on object names rather than file paths is because I also wanted to use it to load code from libraries with their own loaders. For example, Google has this thing where you include a seed file, and then use the google.load() function to load more components. Using JS.Packages, you can have custom loader functions that bridge to other platforms’ loading systems, so I can use the same system to load all the objects I want to use. This abstraction also means I can use the same configuration to load code on non-browser platforms; JS.Packages can figure out how to load external files based on the environment.

JS.Packages(function() { with(this) {
  
  file('https://www.google.com/jsapi?key=INSERT-YOUR-KEY')
    .provides('google.load')
  
  loader(function(onload) {
    google.load('maps', '2', {callback: onload})
  })
    .provides('google.maps')
    .requires('google.load')
}})

JS.require('google.maps', function() {
  var node = document.getElementById("map"),
      map  = new google.maps.Map2(node)
})

Finally there’s this autoload() function, which lets you say if I require an object whose name matches this regex, try to load it from this directory. It’ll turn the object name into a path and then try to load that file for you. You can also use the matches from the regex to generate a dependency, for example if I’m testing and I load the TwitterSpec, I probably want to load the Twitter class as well, and I can use the match from the regex to specify that.

JS.Packages(function() { with(this) {
  
  autoload(/^(.*)Spec$/, {
           from: 'test/specs',
           require: '$1' })
  
  // e.g. TwitterSpec
  //      defined in test/specs/twitter_spec.js
  //      requires Twitter
}})

So I’d been using this for months and all was good with the world until this landed in my inbox:

Does it work on Node?

And it turned out that it didn’t. There are some environment differences in Node that meant that JS.Class wouldn’t run. I patched them up and had a play around with it and everything looked okay. But still, all I could legitimately say at the time was, “I don’t know.” All my tests at the time were written with Scriptaculous, which I loved because it’s really simple to get going with, but it meant my tests were confined to the browser. I needed something new.

So I had a look around to see what was out there, and there are a few really nice frameworks around that work on a few platforms. But none of them seemed to Just Work out of the box on all the platforms people were asking me to support. A few of them you can hack to override their platform bindings but it’s not a great first-time experience. So I did what any self-respecting nerd with a GitHub account does, and wrote a whole new testing library.

The next version of JS.Class will ship with a package called JS.Test. It’s a testing library, and it looks pretty much like most other testing libraries you’ve seen, but with a few explicit goals. First, it should run everywhere, without modification or configuration. I shouldn’t have to tell it how to load files or print results, it should figure that out based on the environment. It should remove as much boilerplate as possible, because setting up a new test suite is always a drag and I constantly forget how to do it. And, it should hide any platform differences – you should just be able to use a single API to write, load and execute your tests across all supported platforms.

Now rather than pore over the API details, which aren’t that interesting since they’re similar to stuff you’ve already seen, I thought I’d take you through an example. We’re going to build a little Twitter API client that works in Node and in web browsers.

Twitter has a search API, which you don’t need to sign up or do any OAuth leg-work to use, it’s a great place to get started. It looks like this, you make a request to search.twitter.com/search.json with your query and you get back a JSON document with some tweets matching the search.

$ curl 'http://search.twitter.com/search.json?q=@jcoglan'
    
{
    "results": [{
        "id": 23843942428577792,
        "created_at": "Sat, 08 Jan 2011 20:50:13 +0000",
        "from_user_id": 4393058,
        "from_user": "extralogical",
        "to_user_id": 86308,
        "to_user": "jcoglan",
        "text": "@jcoglan I need to write some JS testing code...",
    }
    ...
    ]
}

Now, let’s say I want to access this with a JavaScript API. I make a new client, tell it to search for something, then process the results with a callback function. This is what we’re going to build.

var client = new Twitter()

client.search('@jcoglan', function(tweets) {
  // tweets == [{to_user: 'jcoglan', ...}, ...]
})

So first we need to set up a project structure for this. JS.Test doesn’t require any particular file layout, this is just how I’ve settled on doing things. You see we have a source directory that contains the project’s source code, we have a vendor directory where I’ve installed JS.Class, and we have a test directory. This test directory needs a few items in it.

twitter/
    source/
        twitter.js            : source code
    test/
        browser.html          : runs in browsers
        console.js            : runs on command line
        run.js                : loads and runs all tests
        specs/
            twitter_spec.js   : test definitions
    vendor/
        jsclass/
            core.js           |
            loader.js         | -> Framework code
            test.js           |
            (etc)             |

browser.html is what we’ll load up in a web browser to run the tests, and console.js is a script we’ll run in the terminal. They both do exactly the same thing, which is to load the JS.Class seed file, and load the test runner. The test runner, that’s run.js, is a cross-platform script that loads the project’s code, loads all the tests, and runs them. The tests themselves live in the specs directory, one spec file for each source file. Again, this is just convention, you can change this easily once you’re familiar with the setup.

I’m going to start with the console setup first, because it’s slightly simpler. As I said the job of console.js is to load the JS.Class seed file and then load the test runner. Here we’re using Node’s require() function to load the files, some platforms use load() but it’s easy to detect what’s available and pick the right one.

// test/console.js

JSCLASS_PATH = 'vendor/jsclass'
require('../' + JSCLASS_PATH + '/loader')
require('./run')

So with that done we move onto the runner file, this is the script that’s used in all environments to load the project and execute its tests. Notice we can do away with require() vs. load() here, since we’ve got JS.Packages loaded now we can use it to load everything. We start with an autoload() statement to tell it where to find Spec objects, then we tell it where our source code is: file source/twitter.js provides Twitter and requires JS.Class. Finally we load JS.Test, load all our specs and tell JS.Test to run the test suite.

// test/run.js

JS.Packages(function() { with(this) {
  autoload(/^(.*)Spec$/, {
           from: 'test/specs',
           require: '$1' })
  
  file('source/twitter.js')
    .provides('Twitter')
    .requires('JS.Class')
}})

JS.require('JS.Test', function() {
  JS.require('TwitterSpec',
             function() { JS.Test.autorun() })
})

Now onto the spec itself. In our spec file, we create a spec. This is almost the same API is Jasmine, or JSpec, or any number of things so it should be familiar. We have a before block that creates a new Twitter client, then a test for it: when I call search() with "@jcoglan", the client should yield tweets mentioning me. That resume() business is there because we’re running an asynchronous test; JS.Test passes this function to the test block, and we call it when the test is ready to continue, passing any assertions we want to make. It you leave the resume argument out, JS.Test assumes it’s a synchronous test and won’t suspend running when the outer test block completes.

// test/specs/twitter_spec.js

TwitterSpec = JS.Test.describe("Twitter", function() {
  before(function() {
    this.client = new Twitter()
  })

  it("yields matching tweets", function(resume) {
    client.search("@jcoglan", function(tweets) {
      resume(function() {
        assertEqual( "jcoglan", tweets[0].to_user )
      })
    })
  })
})

Let’s go and run our test suite. We immediately get a helpful error message from Node: it couldn’t find our source code.

$ node test/console.js

Error: Cannot find module './source/twitter'

Great, let’s go and create a blank file to get rid of this error. Now we’ve created the file, Node finds it but JS.Pacakges starts complaining.

$ mkdir source
$ touch source/twitter.js
$ node test/console.js

Error: Expected package at ./source/twitter.js
       to define Twitter

You said twitter.js would define the Twitter class, but it doesn’t! Better go and add that.

// source/twitter.js

Twitter = new JS.Class('Twitter')

Now we’ve made the package loader happy and we start to get some meaningful output.

$ node test/console.js

Loaded suite Twitter
Started
E
Finished in 0.072 seconds.

1) Error:
test: Twitter returns tweets matching the search:
TypeError: Object #<Twitter> has no method 'search'

1 tests, 0 assertions, 0 failures, 1 errors

We get an error because our Twitter class doesn’t have the method we need. So, let’s go and implement it. I won’t go through the whole TDD cycle here, let’s just assume I prepared some Node code earlier that does what we want.

// source/twitter.js

Twitter = new JS.Class('Twitter', {
  search: function(query, callback) {
    var http   = require('http'),
        host   = 'search.twitter.com',
        client = http.createClient(80, host)

    var request = client.request('GET',
                  '/search.json?q=' + query,
                  {host: host})
    
    request.addListener('response', function(response) {
      var data = ''
      response.addListener('data', function(c) { data += c })
      response.addListener('end', function() {
        var tweets = JSON.parse(data).results
        callback(tweets)
      })
    })
    request.end()
  }
})

And run the test again:

$ node test/console.js 

Loaded suite Twitter
Started
.
Finished in 2.684 seconds.

1 tests, 1 assertions, 0 failures, 0 errors

We’re all good! Except… this won’t run in a browser. All the network code we wrote only works on Node, and we want this to work client-side too. We’re going to need tests for this. Thankfully, JS.Test makes this easy: all we need is a web page that, just like our terminal script, loads the JS.Class seed file, and loads the test runner. All the test code we wrote earlier will work just fine in the browser.

<html>
  <head>
    <meta http-equiv="Content-type" content="text/html">
    <title>Twitter test suite</title>
  </head>
  <body>
    <script src="../vendor/jsclass/loader.js"></script>
    <script src="../test/run.js"></script>
  </body>
</html>

If we load this up in a browser, we see something like this. “ReferenceError: require is not defined”. Okay, it’s hitting our Node implementation where we load the Node HTTP library, we want to avoid that. So what do we do? Easy, just detect whether we’re in a DOM environment and switch to using JSONP to talk to Twitter instead of Node’s HTTP libraries. Again, here’s one I made earlier, this is just the usual JSONP hackery, nothing fancy, and we’ve moved the Node version into the nodeSearch() method that will be called if we’re not in a DOM environment.

Twitter = new JS.Class('Twitter', {
  search: function(query, callback) {
    if (typeof document === 'object')
      this.jsonpSearch(query, callback)
    else
      this.nodeSearch(query, callback)
  },
  
  jsonpSearch: function(query, callback) {
    var script  = document.createElement('script')
    script.type = 'text/javascript'
    script.src  = 'http://search.twitter.com/search.json?' +
                  'callback=__twitterCB__&' +
                  'q=' + query
    
    window.__twitterCB__ = function(tweets) {
      window.__twitterCB__ = undefined
      callback(tweets.results)
    }
    var head = document.getElementsByTagName('head')[0]
    head.appendChild(script)
  },
  
  nodeSearch: function(query, callback) {
    // ...
  }
})

Reload the page, and fantastic – we’ve got a green build. Quick side-note, in the browser UI, JS.Test will print out a tree of all your nested context blocks that you can browse, which can be more useful than the terminal UI for finding errors. It’ll also notify TestSwarm if that’s where you’re running your tests, so you can use it for continuous integration.

The final piece of the process is to refactor. We’ve got a bunch of networking code gunking up our API client. Maybe we should move the networking code into its own module that’s a generic interface for making HTTP calls in any environment we support. Then we could call it like this:

// source/twitter.js

Twitter = new JS.Class('Twitter', {
  search: function(query, callback) {
    var resource = 'http://search.twitter.com' +
                   '/search.json?q=' + query
    
    Twitter.Net.getJSON(resource, callback)
  }
})

Because the logic for how to do networking is now isolated in one module, it’s easier for a user to replace if they want to make the Twitter client run in another environment: they just have to replace the implementation of Twitter.Net.getJSON() with the HTTP code for their platform. It also means that the network is easier to stub out, since we don’t want to rely on the real Internet during testing:

// test/specs/twitter_spec.js

TwitterSpec = JS.Test.describe("Twitter", function() {
  before(function() {
    this.client = new Twitter()
    
    stub(Twitter.Net, "getJSON")
        .given("http://search.twitter.com/...")
        .yields([{to_user: "jcoglan"}])
  })
  
  it("yields matching tweets", function() {
    // ...
  })
})

This approach also means we can run the test in any platform because we don’t need to actually talk to the network. We’d then write unit tests for the Twitter.Net module to make sure it worked on the right platforms.

So what I wanted to get across here isn’t that you should all go and use my code, but try to follow some of the same patterns. If we want highly reusable software we’re going to need to test it everywhere. If you’re building libraries to support your work, and it looks like the abstraction would be useful in other contexts, consider making it available to users of other platforms. If you do have platform-specific code, try to isolate it in one place and hide it behind abstractions. Remember how in Faye I’ve isolated the Node bindings to make it easy to run and test the internal components in other environments. Make it easy to replace the platform bindings, so if someone wants to run it somewhere you didn’t expect it’s easy to swap in new bindings.

And finally, write usage documentation. If someone’s trying to get your code running somewhere new, step-by-step tutorials are great for showing people the ropes and getting them comfortable with how your stuff works, so they feel more confident hacking it to their needs. You’ll be amazed what people do with your code when you make it easy to use and write nice docs for it.

Speaking at RubyConf 2010

A quick bit of self-promotional guff: I’m thrilled to announce I’ll be presenting at RubyConf 2010 in New Orleans this November. I’ll be speaking about the Ruby incarnation of Faye, and more broadly about doing asynchronous programming and testing using Ruby and EventMachine.

I gave a similar talk based on the JavaScript/Node version at the London Ajax User Group a couple months back, if you want a preview.

You can grab a ticket from the conference site, assuming they’ve not sold out by the time you read this. It’s a pretty daunting prospect sharing a bill with Matz and DHH, so fingers-crossed my first proper conference spot goes well.

Evented programming patterns: Object lifecycle

This post is part of a series on event-driven programming. The complete series is:

Earlier in this series I covered a very common pattern in event-driven programming: the Observable object. This technique lets one object notify many others when interesting things happen. JavaScript developers will be very familiar with this: it’s the same pattern that underlies the DOM event model.

I while ago I rewrote the JS.Class package loader and noticed a variation of this pattern emerge, which I’m going to call the object lifecycle. The typical use case is when some part of your code needs to execute once, as soon as some condition becomes true. In the package loader, this looks something like:

thePackage.when('loaded', function() {
  // Run code that relies on thePackage
});

This says: if thePackage is already loaded, then run this callback immediately. Otherwise, wait until thePackage is loaded and then run the callback. The implication is that the package will become loaded, only once, at some point in its life, and as soon as that happens we want to be notified. (I tend to use when for one-shot lifecycle events, and on for multi-fire events.) The implementation is quite similar to the Observable pattern, so you might want to revisit that before reading on.

Obviously, our lifecycle object is going to need to store lists of callbacks, indexed by event name as before. But in this case, if we know the event has already been triggered on that object, we can run the callback immediately and forget about it. When we trigger events, we also want to remove all the old pending callbacks after running them, since they don’t need to be called again.

LifeCycle = {
  when: function(eventType, listener, scope) {
    this._firedEvents = this._firedEvents || {};
    if (this._firedEvents.hasOwnProperty(eventType))
      return listener.call(scope);

    this._listeners = this._listeners || {};
    var list = this._listeners[eventType] = this._listeners[eventType] || [];
    list.push([listener, scope]);
  },
  
  trigger: function(eventType) {
    this._firedEvents = this._firedEvents || {};
    
    if (this._firedEvents.hasOwnProperty(eventType)) return false;
    this._firedEvents[eventName] = true;
    
    if (!this._listeners) return true;
    var list = this._listeners[eventType];
    if (!list) return true;
    list.forEach(function(listener) {
      listener[0].apply(listener[1], args);
    });
    delete this._listeners[eventType];
    return true;
  }
};

Note how the trigger() method checks to see if the event has already been fired: we don’t want the same stage in the lifecycle to be triggered multiple times. It also removes the listeners from the object after calling them, and returns true or false to indicate whether the event fired. This makes it easy to tell whether some action that should only be done once has already happened; for example in my package system I do something like this:

JS.Package = new JS.Class({
  include: LifeCycle,
  
  // various methods
  
  load: function() {
    if (!this.trigger('request')) return;
    // perform download logic...
  }
});

This kills two birds with one stone: it lets other listeners know that the package has been requested and checks whether it’s already been requested, so we don’t try to download the package multiple times.

Naturally, one thing a package system has to deal with is dependencies. Dependencies are just prerequisites: you can’t load a package until all its dependencies are loaded. More precisely, a package is loaded once the browser has downloaded its source code, and it is complete once it is loaded and all its dependencies are complete. To fill in some more of the load() method, this is easily expressed as:

JS.Package.prototype.load = function() {
  if (!this.trigger('request')) return;
  
  when({complete: this._dependencies, load: [this]}, function() {
    this.trigger('complete');
  }, this);

  when({loaded: this._dependencies}, function() {
    loadFile(this._path, function() { this.trigger('load') }, this);
  }, this);
};

This reads quite naturally: if the package has already been requested, do nothing. When the dependencies are complete and this package is loaded, then this package is complete. When the dependencies are loaded, load this package and then trigger its load event. Note how the load event will trigger a complete event if there are no dependencies, and this will ripple down the tree and trigger dependent packages to load.

I’ve used when() above to express groups of prerequisites in a natural way, but we don’t have an implementation for that yet – we only have the when() method for individual objects. So let’s write one. This when() function will need to gather up the list of preconditions, keep a tally of how many have triggered, and when they’re all done we can fire our callback. The first step in the function converts preconditions, which maps event names to lists of objects, into a simple list of object-event pairs. That is it turns {complete: [foo, bar], load: [this]} into [[foo, 'complete'], [bar, 'complete'], [this, 'load']].

var when = function(preconditions, listener, scope) {
  var eventList = [];
  for (var eventType in preconditions) {
    for (var i = 0, n = preconditions[eventType].length; i < n; i++) {
      var object = preconditions[eventType][i];
      eventList.push([object, eventType]);
    }
  }
  
  var pending = eventList.length;
  if (pending === 0) return listener.call(scope);
  
  for (var i = 0, n = pending; i < n; i++) {
    eventList[i][0].when(eventList[i][1], function() {
      pending -= 1;
      if (pending === 0) listener.call(scope);
    });
  }
};

If there are no pending events, we can just call the listener immediately. Otherwise, we set up listeners for all the events, and when each one fires (and remember: some of them may have fired already) we count down how many events we’re waiting for. When this reaches zero, we can carry on with the work we wanted to do.

This pattern is essentially a cross between Observable and Deferrable: we’re deferring an action, but the deferred items – the events – aren’t complex enough to merit their own objects so the implementation is closer to an observable object. The technique lends itself really well to expressing prerequisites in a natural way, even if the work you’re doing is not asynchronous.

I’ll have a couple more articles on event-driven programming next week, and you can catch me speaking at the London Ajax User Group on August 10th where I’ll be talking about Faye, event-driven code and testing.

Talk: Writing a language in 15 minutes

I gave a talk at London Ruby User Group yesterday, based on the work I’ve been doing on Heist, my Scheme interpreter project. I wrote the core of a basic Scheme interpreter in about 15 minutes as a live-coded demo (well, kind of – the coding was pre-recorded so I could focus on talking), which seemed to go down pretty well. If you missed it (or if you were there and want to watch it again in slow motion), here’s the slides and the video (just code, no narrative (sorry)). (Side note: I think Lisp may be affecting my writing style.)

The slides first: lrug-scheme-15.zip. They are S5-format HTML, introducing the Scheme language features I implement during the talk. The video shown below is available at higher resolution from Vimeo.

Scheme interpreter in 15 minutes from James Coglan on Vimeo.

Video is also available from Skills Matter if you want the narrative. The code’s not really visible in this version so combine the audio from this with the above video and you should just about piece things together.

Some relevant links:

  • Heist is my main Scheme interpreter project. It has macros, tail recursion, continuations, and a reasonable chunk of the R5RS spec and its REPL auto-indents your code. It’s about 1000 lines of well-commented Ruby with a few hundred lines of Scheme, including macros for most of the syntax.
  • Stickup is a tiny interpreter for a small subset of Scheme, about 150 lines long. Closer to what I present in the talk, and easier to get your teeth into.
  • Treetop is what I use to generate parsers, it’s super-simple to use and lets you write a parser in no time at all.

Thanks to everyone who came along and had nice things to say about the talks, especially to whoever was telling me about about the trie data structure; Heist’s tab-completion code is now much prettier.