Announcing reStore, a personal remoteStorage server for Node

If you’ve been following my Vault project, you might have heard me talk about how I’m going to support saving your password settings while using the web version. There is currently no means of doing this baked into Vault itself; it relies on your saving your settings somewhere yourself, and doesn’t integrate with the command-line version.

Obviously, this needs fixing, and I’ve actually been working on fixing it since before the first stable Vault release came out. I began looking for some way of supporting this feature that didn’t involve me storing people’s data on my servers, for fairly obvious security and logistical reasons. Back in May last year I did a little digging and stumbled on the remoteStorage project, part of the Unhosted collection of web technologies. It looked like just what I needed to support Vault: it lets web applications delegate storage to a server under the user’s control. The user gives the application their username and server as an email address, the application does an OAuth handshake with the server, and can then use it to store the user’s data.

This has two important benefits: obviously, it gives users freedom of movement between software providers: several standard modules are being worked out for various common things people store, like contacts, photos and so on. But secondly, decoupling storage from software lets organisations use public web applications but keep their data off the public internet: as long as JavaScript in the browser can talk to a server visible on your private network, it just works. This is how I’m planning on making getvau.lt usable by teams and organisations.

So, in support of this aim, I’m releasing the remoteStorage server I’ve been working on for the last few months as open-source software. It’s called reStore, and you can get it from npm:

$ npm install restore

The documentation on GitHub should tell you everything you need to get started. Right now, I want to show you what it’s like to use from an application’s point of view. It’s really very simple to talk to, in fact I have unreleased code for Vault that integrates it seamlessly into the web and command-line versions. This demo uses 5apps.com, another remoteStorage provider:

Vault: syncing data with remoteStorage from James Coglan on Vimeo.

An app starts off by asking the user for your remoteStorage address. Say I give it me@local.dev. The application then goes and queries that server using WebFinger to discover where the authorization and storage endpoints are:

$ curl -iX GET https://local.dev/.well-known/host-meta.json?resource=acct:me@local.dev
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json

{
  "links": [
    {
      "href": "https://local.dev/storage/me",
      "rel": "remotestorage",
      "type": "draft-dejong-remotestorage-00",
      "properties": {
        "auth-method": "http://tools.ietf.org/html/rfc6749#section-4.2",
        "auth-endpoint": "https://local.dev/oauth/me"
      }
    }
  ]
}

So the app now knows the root of my storage tree is at https://local.dev/storage/me, and it redirects me to https://local.dev/oauth/me to authorize it for the directories it wants to store things in. For reStore, that looks like this, for example if the app wants read/write access to my /vault directory:

reStore

The user enters their password if they want to grant access to the directories the app wants, and reStore then redirects to the app’s OAuth callback endpoint, passing a bearer token, say df6VCO3jTVZOCbHUMB2uLmmqL5M=. The app can now use this token to store any data it likes from the directory I authorized:

$ curl -iH 'Authorization: Bearer df6VCO3jTVZOCbHUMB2uLmmqL5M=' \
       -X PUT https://local.dev/storage/me/vault/foo \
       -H 'Content-Type: text/plain' -d 'Hello, world!'                                

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
ETag: 1361834896000
Last-Modified: Mon, 25 Feb 2013 23:28:16 GMT

$ curl -iH 'Authorization: Bearer df6VCO3jTVZOCbHUMB2uLmmqL5M=' \
       -X PUT https://local.dev/storage/me/vault/nested/bar \
       -H 'Content-Type: application/json' -d '{"hello":"world"}'

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
ETag: 1361834933000
Last-Modified: Mon, 25 Feb 2013 23:28:53 GMT

It can list the contents of authorized directories:

$ curl -iH 'Authorization: Bearer df6VCO3jTVZOCbHUMB2uLmmqL5M=' \
       -X GET https://local.dev/storage/me/vault/

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json
Content-Length: 54

{
  "nested/": 1361834933000,
  "foo": 1361834896000
}

$ curl -iH 'Authorization: Bearer df6VCO3jTVZOCbHUMB2uLmmqL5M=' \
       -X GET https://local.dev/storage/me/vault/nested/

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json
Content-Length: 26

{
  "bar": 1361834933000
}

And retrieve individual documents:

$ curl -iH 'Authorization: Bearer df6VCO3jTVZOCbHUMB2uLmmqL5M=' \
       -X GET https://local.dev/storage/me/vault/foo

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Type: text/plain
ETag: 1361834896000
Last-Modified: Mon, 25 Feb 2013 23:28:16 GMT
Content-Length: 13

Hello, world!

$ curl -iH 'Authorization: Bearer df6VCO3jTVZOCbHUMB2uLmmqL5M=' \
       -X GET https://local.dev/storage/me/vault/nested/bar

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json
ETag: 1361834933000
Last-Modified: Mon, 25 Feb 2013 23:28:53 GMT
Content-Length: 17

{"hello":"world"}

When you retrieve a document you get the content-type and last-modified time and etag of the document, and you can then use these with if-unmodified-since when making PUT and DELETE requests. For example, here’s a failed DELETE request with a conflict:

$ curl -iH 'Authorization: Bearer df6VCO3jTVZOCbHUMB2uLmmqL5M=' \
       -X DELETE https://local.dev/storage/me/vault/nested/bar \
       -H 'If-Unmodified-Since: Sat, 25 Feb 2012 23:28:53 GMT'

HTTP/1.1 409 Conflict
Access-Control-Allow-Origin: *

Of course, if you don’t specify the version the document is simply deleted:


$ curl -iH 'Authorization: Bearer df6VCO3jTVZOCbHUMB2uLmmqL5M=' \
       -X DELETE https://local.dev/storage/me/vault/nested/bar

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
ETag: 1361834933000
Last-Modified: Mon, 25 Feb 2013 23:28:53 GMT

$ curl -iH 'Authorization: Bearer df6VCO3jTVZOCbHUMB2uLmmqL5M=' \
       -X GET https://local.dev/storage/me/vault/nested/bar

HTTP/1.1 404 Not Found
Access-Control-Allow-Origin: *

That’s really all there is to it – WebFinger, OAuth and GET/PUT/DELETE. Since this is the first release of reStore it should go without saying it is alpha software, and you should expect it to eat your data – please take regular backups if you use it to store data. I’m releasing it for evaluation and while I work on rolling out for my own uses, and I’d really appreciate any feedback you have.