Terminus 0.5: now with Capybara 2.0 support and remote debugging

You might remember from my previous post that Terminus is a Capybara driver that lets you run your integration tests on any web browser on any device. Well, in mid-November Capybara 2.0 came out, and since I was at the excellent RuPy conference at the time, my conf hack project became getting Terminus compatible with this new release.

I almost finished it that weekend, but not quite, and as always once you’re home and back at work you lose focus on side projects. But, for my final release of 2012, I can happily announce Terminus 0.5 is released, and makes Terminus compatible with both Capybara 1.1 and 2.0. It’s mostly a compatibility update but it adds a couple of new features. First, Capybara’s screenshot API is supported when running tests with PhantomJS:

page.save_screenshot('screenshot.png')

And, it supports the PhantomJS remote debugger. You can call this API:

page.driver.debugger

This will pause the test execution, and open the WebKit remote debugger in Chrome so you can interact with the PhantomJS runtime through the WebKit developer tools. When testing on other browsers it simply pauses execution so you can inspect the browser where the tests are running.

As usual, ping the GitHub project if you find bugs.

Happy new year!

Terminus 0.4: Capybara for real browsers

As I occasionally mention, the original reason I built Faye was so I could control web browsers with Ruby. The end result was Terminus, a Capybara driver that controls real browsers. Since the last release, various improvements in Faye – including the extracted WebSocket module, removal of the Redis dependency and overall performance gains – have made various improvements to Terminus possible. Since Faye’s 0.8 release, I’ve been working on Terminus on-and-off and can now finally release version 0.4.

Terminus is a driver designed to control any browser on any device. To that end, this release adds support for the headless PhantomJS browser, as well as Android and IE8. In combination with the performance improvements, this makes Terminus a great option for headless and mobile testing. The interesting thing about Android and IE is that they do not support the document.evaluate() method for querying the DOM using XPath, and Capybara gives XPath queries to the driver to execute. In order to support these browsers, I had to write an XPath library, and in order to get that done quickly I wrote a PEG parser compiler. So that’s now three separate side projects that have sprung out of Terminus – talk about yak shaving.

But the big change in 0.4 is speed: Terminus 0.4 runs the Capybara test suite 3 to 5 times faster than 0.3 did. It does this using some trickery from Jon Leighton’s excellent Poltergeist dirver, which just got to 1.0. Here’s how Terminus usually talks to the browser: first, the browser connects to a running terminus server using Faye, and sends ping messages to advertise its presence:

        +---------+
        | Browser |
        +---------+
             |
             | ping
             V
        +---------+
        | Server  |
        +---------+

When you start your tests, the Terminus library connects to the server, discovers which browsers exist, and sends instructions to them. The browser executes the instructions and sends the results back to the Terminus library via the server.

        +---------+
        | Browser |
        +---------+
            ^  |
   commands |  | results
            |  V
        +---------+           +-------+
        | Server  |< -------->| Tests |
        +---------+           +-------+

As you can guess, the overhead of two socket connections and a pub/sub messaging protocol makes this a little slow. This is where the Poltergeist trick comes in. If the browser supports WebSocket, the Terminus library will boot a blocking WebSocket server in your test process, and wait for the browser to connect to it. It can then use this socket to perform request/response to the browser – it sends a message over the socket and blocks until the browser sends a response. This turns out to be much faster than using Faye and running sleep() in a loop until a result arrives.

        +---------+
        | Browser |< -------------+
        +---------+               |
            ^  |                  | queries
   commands |  | results          |
            |  V                  V
        +---------+           +-------+
        | Server  |< -------->| Tests |
        +---------+           +-------+

The Faye connection is still used to advertise the browser’s existence and to bootstrap the connection, since it’s guaranteed to work whatever browser or network you’re on.

The cool thing about this is that Jon’s code reuses the Faye::WebSocket protocol parser, supporting both hixie-76 and hybi protocols, on a totally different I/O stack. Though Faye::WebSocket is written for EventMachine, I did try to keep the parser decoupled but had never actually tried to use it elsewhere, so it’s really nice to see it used like this.

Anyway, if you’re curious about Terminus you can find out more on the website.

Announcing Canopy, a Treetop-like PEG compiler for JavaScript

This is very brief announcement to say that I’ve just released a new PEG parser-compiler for JavaScript, called Canopy. It generates fast, self-contained parser modules from grammar definition files and runs in all the major browsers and on CommonJS platforms including Node.js, Narwhal and RingoJS.

Why do we need another PEG compiler? We don’t. PEG.js is excellent, actively used and maintained and has a really nice website. Canopy exists mostly for reasons of personal taste: I wanted something more akin to Treetop, which lets you keep the grammar definition separate from any methods you add to the parse tree. In fact, Canopy goes further than this: you have to keep them separate, you cannot have inline JavaScript in the grammar files. JavaScript goes in JavaScript files with all your other JavaScript.

It’s taken rather a while to release. I initially wrote it as part of a huge yak-shaving exercise: I was trying to get Terminus to work in IE, which doesn’t support the document.evaluate() API. I thought it would be a fun idea to reimplement it, so I created a project called Pathology, an ad-hoc, informally specified, bug-ridden slow implementation of half of XPath for IE. Of course this meant parsing XPath queries, which I wasn’t going to do by hand, so I thought it would even more fun to learn how parser compilers work and build one. Pathology never really panned out, although it turns out Android browsers don’t have XPath either so I might revive it, you never know.

I also used Canopy to build Fargo, my fiber-aware version of Scheme. It’s great for getting a new language off the ground quickly.

So, after two years of off-and-on development, and after some interest from a few other people, a couple of months ago I finally got around to documenting it, getting rid of some annoying dependencies (the parsers it generates are now completely self-contained and work on lots of JS platforms), and testing it properly. It’s available by running npm install -g canopy, or from the website, along with the documentation. Let me know what you think.

Terminus 0.3: control multiple browsers with Ruby

As you’ll have noticed if you made it to the end of my last post, there is a new release of Terminus. Terminus is a Capybara driver that is designed to let you control your app in any browser on any device, by sending all driver instructions to be executed on the client side in JavaScript.

This release is the first since Capybara 1.0, and supports the entire Capybara API. This includes:

  • Reading response headers and status codes
  • Handling cookies
  • Running JavaScript and receiving the results
  • Resynchronizing XHR requests (jQuery only)
  • Switching between frames and windows
  • Detecting infinite redirects

This is a superset of the supported features of the Rack::Test and Selenium drivers, and has the added bonus of letting you switch between browsers. When you have multiple browsers connected to your Terminus server, you can select which one you want to control by matching on the browser’s name, OS, version and current URL, for example:

Terminus.browser = {:name => /Safari/, :current_url => /pitchfork.com/}

You can select any browser that is ‘docked’, i.e. idling on the Terminus holding page:

Terminus.browser = :docked

Or simply by selecting one browser from the list:

Terminus.browser = Terminus.browsers.first

All this lets you control multiple browsers at once, for example I’ve been using it to automate some of the Faye integration tests:

#================================================================
# Acquire some browsers and log into each with a username

NAMES = %w[alice bob carol]
BROWSERS = {}
Terminus.ensure_browsers 3

Terminus.browsers.each_with_index do |browser, i|
  name = NAMES[i]
  puts "#{name} is using #{browser}"
  BROWSERS[name] = browser
  Terminus.browser = browser
  visit '/'
  fill_in 'username', :with => name
  click_button 'Go'
end

#================================================================
# Send a message from each browser to every other browser,
# and check that it arrived. If it doesn't arrive, send all
# the browsers back to the dock and raise an exception

BROWSERS.each do |name, sender|
  BROWSERS.each do |at, target|
    next if at == name
    
    Terminus.browser = sender
    fill_in 'message', :with => "@#{at} Hello, world!"
    click_button 'Send'
    
    Terminus.browser = target
    unless page.has_content?("#{name}: @#{at} Hello, world!")
      Terminus.return_to_dock
      raise "Message did not make it from #{sender} to #{target}"
    end
  end
end

#================================================================
# Re-dock all the browsers when we're finished

Terminus.return_to_dock

So what’s not supported? Internet Explorer is still not supported because I cannot find a decent way to run XPath queries on it. I was working on Pathology to solve this but I can’t get it to perform well enough for the workload Capybara throws at it. It might be possible to work around this by monkey-patching Capybara to pass through CSS selectors instead of compiling them to XPath, though. File attachments are not supported for security reasons, and there are still some bugs that show up if you do stuff you’re not supposed to, like using duplicate element IDs. These are particularly apparent on Opera. And finally visiting remote hosts outside your application is supported but is not particularly robust as yet.

You can find out more and see a video of it in action on its new website.

Refactoring towards testable JavaScript, part 3

This article is one in a 3-part series. The full series is:

We finished up the previous article having separated the business logic from the DOM interactions in our JavaScript, and adjusted our unit tests to take advantage of this. In the final part of this series, we’ll take a look at how to take the tests we have and run them across a range of browsers automatically to give us maximum confidence that our code works.

To automate cross-browser testing, I use a much-overlooked tool called TestSwarm. Developed by John Resig for testing jQuery, it takes care of tracking your test status in multiple browsers as you make commits to your project.

To set it up, we need to clone it from GitHub and create a directory within to host revisions of our project.

$ git clone git://github.com/jquery/testswarm.git
$ cd testswarm
$ cp config/config-sample.ini config.ini
$ mkdir -p changeset/jsapp

You’ll need to create a MySQL database for it:

CREATE USER 'testswarm'@'localhost' IDENTIFIED BY 'choose-a-password';
CREATE DATABASE testswarm;
GRANT ALL ON testswarm.* TO 'testswarm'@'localhost';

Then import the TestSwarm schema:

$ mysql -u testswarm -p testswarm < config/testswam.sql
$ mysql -u testswarm -p testswarm < config/useragents.sql

Once you’ve added the database details to config.ini and set up an Apache VHost, you can visit your TestSwarm server and click ‘Signup’. Once you’ve filled in that form you’ll be able to grab your auth key from the database:


$ mysql testswarm -u testswarm -p
mysql> select auth from users;
+------------------------------------------+
| auth                                     |
+------------------------------------------+
| a962c548c22a591e8f150b9d9f6b673b6f212d08 |
+------------------------------------------+

Keep that auth code somewhere as you’ll need it later on. Now before we go any further, to show how TestSwarm works I want to deliberately break our application so that it doesn’t work in Internet Explorer. Do this I’m going to replace jQuery#bind with HTMLElement#addEventListener, and when we push our code to TestSwarm we should see it break.

To get our tests running on TestSwarm, we need a config file. I just grabbed one of the standard Perl scripts from the TestSwarm project and added my own configuration. This tells the script where your TestSwarm server is, where your code should be checkout out, any build scripts you need to run, which files to load, etc. JS.Test includes TestSwarm support baked in so we don’t need to modify our tests at all to make them send reports to the TestSwarm server, we just need to load the same old spec/browser.html file we’ve been using all along. You should configure the Perl script to clone your project into the changeset/jsapp directory we created earlier: this is in TestSwarm’s public directory so web browsers will be able to load it from there. You’ll need to include the auth key we created earlier to submit jobs to the server.

Having created this file, we clone the project on our server somewhere and create a cron job to periodically update our copy and run the TestSwam script: this means that new test jobs will be submitted whenever we commit to the project.

# crontab
* * * * * cd $HOME/projects/jsapp && git pull && perl spec/testswarm.pl

If you now open a couple of browsers and connect to the swarm, you’ll see tests begin to run. If you inspect the test results for our project you should see this:

The green box is Chrome reporting 5 passed tests, and the black box is IE8 reporting 7 errors. If we click through we see what happened:

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)

  • Error:
    FormValidator with valid data displays no errors:
    TypeError: Object doesn’t support this property or method
  • Error:
    FormValidator with an invalid name displays an error message:
    TypeError: Object doesn’t support this property or method
  • Error:
    FormValidator with an invalid email displays an error message:
    TypeError: Object doesn’t support this property or method
  • Error:
    FormValidator with an invalid argument displays an error message:
    TypeError: Object doesn’t support this property or method
  • Error:
    FormValidator with an invalid name displays an error message:
    TypeError: Object doesn’t support this property or method
  • Error:
    FormValidator with an invalid name displays an error message:
    TypeError: ‘submit’ is undefined
  • Error:
    FormValidator with an invalid name displays an error message:
    TypeError: ‘error’ is undefined

5 tests, 4 assertions, 0 failures, 7 errors

“Object doesn’t support this property or method” is IE’s way of saying you’re calling a method that doesn’t exist, in our case addEventListener(). If we make some changes so that we use attachEvent() instead in IE, when TestSwam picks up the change and runs our tests they go green in IE.

You can leave any number of browsers connected to the swarm and they will run tests automatically as you make commits to the project. This is the great advantage of sticking to portable unit tests for your JavaScript, it makes this kind of automation much easier, you run tests in real browsers and don’t need a lot of additional tooling to set up fake environments. I run the JS.Class tests this way and it’s great for making sure my code works across all platforms before shipping a release.

The final big win, having set all this up, is that we can now delete some of our full-stack tests since they just duplicate our unit tests. When we’re testing a full-stack integration, we really just want to test at the level of abstraction the integration works at, i.e. test the glue that binds the pieces together, rather than testing all the different cases of every piece of business logic. In our case, this means testing two broad cases: either the form is valid or it is not valid. We have unit tests that cover in detail what ‘valid’ and ‘invalid’ mean, and we don’t need to duplicate these. We just need to test their effect on the application as a whole: either the form submits or it doesn’t. We can then instantly discard all the integration tests that cover the validation details, leaving the broad cases covered. Doing this rigorously will keep your integration tests to a minimum and keep your build running quickly.

To wrap up this series, I thought I’d mention a couple other things we can do with our tests to get extra coverage. The first is the JS.Test coverage tool. If we write our code using JS.Class, we can make the test framework report which methods were called during the test just by adding cover(FormValidator) to the top of the spec. When we run the tests we get a report:

<code>$ node spec/console.js 
Loaded suite FormValidator

Started
....

  +-----------------------------------+-------+
  | Method                            | Calls |
  +-----------------------------------+-------+
  | Method:FormValidator.validate     | 4     |
  | Method:FormValidator#handleSubmit | 0     |
  | Method:FormValidator#initialize   | 0     |
  +-----------------------------------+-------+

Finished in 0.005 seconds
4 tests, 4 assertions, 0 failures, 0 errors</code>

If any of the methods are not called, the process exits with a non-zero exit status so you can treat your build as failing until all the methods are called during the test.

Finally, I have an ongoing experimental Capybara driver called Terminus that lets you run your Capybara-based tests on remote machines like phones, iPads and so on. If we change our Capybara driver as required, we can open a browser on a remote machine, connect to the Terminus server and run the tests on that machine, or on many machines at once if your tests involve communication between many clients.

Here’s the full list of software we’ve used in this series:

  • Sinatra – Ruby web application framework used to create our application stack
  • jQuery – client side DOM, Ajax and effects library used to handle from submissions
  • JS.Class, JS.Test – portable object system and testing framework for JavaScript
  • Cucumber – Ruby acceptance testing framework used for writing plain-text test scenarios
  • Capybara – web scripting API that can drive many different backends
  • Rack::Test – Capybara backend that talks to Rack applications directly with no wire traffic, suited to doing fast, in-process testing, does not support JavaScript
  • Selenium – Capybara backend that runs using a real browser, slower but supports JavaScript
  • Terminus – Capybara backend that can drive any remote browser using JavaScript
  • PhantomJS – headless distribution of WebKit, scriptable using JavaScript
  • TestSwam – automated cross-browser CI server for tracking JavaScript unit tests across the project history

I’ll leave you with a few points to bear in mind to keep your JavaScript unit-testable:

  • Minimize DOM interaction – write your business logic in pure JavaScript, test it server-side, and use a ‘controller’ layer to bind this logic to your UI.
  • Keep controllers DOM focused – in JavaScript, ‘controllers’ in MVC parlance are basically your event handlers. They should handle user input, trigger actions in your business logic, and update the page as appropriate.
  • If you need a browser, use a real one – in my experience, given how easy it is to test on real browsers and minimize integration tests, fake DOM environments are often more pain than they’re worth. The important thing is to keep your code as portable as possible so you can adapt if you spot more suitable tools.

Terminus driving multiple browsers

Conferences usually prompt me to hack on some loose thread I’ve not picked up in months. At this year’s Scottish Ruby Conference I decided I had to give a lightning talk on Terminus, my Capybara driver for scripting remote browsers. With a little hacking and lot of sitting around waiting for tests to complete, I’ve got it up to date with a lot of the latest Capybara specs and added a really simple API for switching between browsers based on name, OS, version etc.

I’m not putting out a release just yet but I thought I’d share a couple videos of it in action; one’s a hi-res screen capture and one’s a fuzzy shaky mobile capture so you can see it running across multiple machines.

The app it’s running is the Faye example application – a chat app much like Twitter. Terminus logs in as a different user on each browser, then sends messages between all the pairs of browsers and checks that each message arrives on screen in the intended browser.

Sadly, it looks like Internet Explorer support is a pipe dream for now. The XPath queries Capybara spits out are too much for my half-baked XPath engine to deal with, so unless someone comes up with a fast implementation of document.evaluate() for IE I’m leaving it alone.

Anyway, the videos:

The script that runs this is really simple, it’s just some Capybara calls with a little extra to tell Terminus to switch browsers. Check it out on GitHub.

As for what I’ve added to Terminus, here’s a quick run-down:

  • Headers and status code support
  • Multiple windows and iframes
  • Improved concurrency handling for running the same test in multiple browsers
  • Browser selection API
  • Removed the need for you to embed a script in your application
  • Very basic support for scripting remote applications

The addition of status and header support means it actually supports a superset of the behaviour supported by the Selenium driver, albeit considerably slower. The main win is being able to script remote devices – it’s great fun watching it control somebody’s iPad! Would love to see if people come up with novel uses for it.

Terminus: a client-side Capybara driver

Last week the Capybara project released version 0.4. Since the 0.3.9 release, which added support for third-party drivers, I’ve been working on turning Terminus into fully compatible driver for it. It’s still an experiment but it’s in the sort of semi-useful state that means I’m okay throwing it out to see if anyone’s interested in it.

So what is it? Terminus is a Capybara driver where most of the driver functions are implemented in client-side JavaScript. It lets you script any browser on any machine using the Capybara API, without any browser plugins or extensions. ‘Any browser’ really means ‘any browser that supports the document.evaluate() XPath API’ for now, so you can’t use IE, but the ‘any machine’ part is true: any browser that can see your development machine can be controlled using Terminus, including any phone, iPad or desktop machine on your local network.

This is very much still an experiment. Because every little Capybara call has to go over the network (it uses Faye to send commands to browsers) it’s very slow, and the connection sometimes flakes out. Under ideal circumstances though it does pass most of the Capybara test suite. It supports JavaScript, cookies and window switching (assuming the connected browser has these features enabled). The major omissions are:

  • attach_file does not work since file uploads cannot be controlled by JavaScript for security reasons.
  • A few selectors and HTML5 features in the specs don’t work due to browser inconsistencies.
  • Redirects work but infinite redirection cannot be detected.

The original intention was for Terminus to be a tool for automating cross-browser testing, so the lack of IE support is kind of a problem. I actually went some way towards fixing this; I wrote a pretty stupid document.evaluate() replacement called Pathology, and to support that a PEG parser compiler called Canopy so I could write a declarative XPath grammar. Unfortunately these don’t yet perform well enough to support the workload that Capybara throws at a browser.

I also want to add an API for switching browsers based on vendor and OS versions, but this release is just enough to support the required Capybara API.

You can find out more on the project’s GitHub page, and installation is the usual gem install terminus command – I’d love if a few of you could kick its tyres a bit and figure out if it’s actually useful for anything.

Terminus: control your browser from the command line

I’ve been saying for a while that I want to use Faye for automating JavaScript and integration testing, especially now that it has server-side clients. Well I took the first step in that direction this afternoon by hacking together Terminus, a distributed JavaScript console. You just install and run like so:

$ sudo gem install terminus
$ terminus
Terminus running at http://0.0.0.0:7004
Press CTRL-C to exit
>> 

Visiting the aforementioned http://0.0.0.0:7004 will give you a bookmarklet that you can drag up to your bookmarks bar. Running the bookmarklet while Terminus is running will connect the current page to your Terminus session, letting you run JavaScript on that page from the command line. And not just that page: every page you’ve connected will execute every line of script you type in. You may very well become drunk with power.

Anyway, have a play around and see if it’s useful. It’s still alpha quality and needs a fair few things adding, like per-page return values, connection reporting etc. It also needs some polish before you can drop the client script into your own apps and use it to drive tests. One step at a time.