A recent conversation on Mastodon reminded me of some old JavaScript history. When promises were relatively new and the Promises/A+ spec was being developed, there was a request from people with backgrounds in functional programming to incorporate monads and category theory. This issue generated a heated debate, the two sides so diametrically opposed that they both believed their position was so obviously correct that they could not fathom why anyone could believe anything else. Historically I have tended to side with the spec authors in opposing this request, but I recently ran into a case where it would have been useful for things to go the other way, which is a good prompt for re-evaluating your positions.

To start with, we need to understand what the request actually entails – what do they mean by “incorporating monads and category theory”? I have written before about how promises are the monad of asynchronous programming, but let’s have a quick recap of what functional programmers mean here. Functional languages like Haskell don’t model programs as sequences of instructions to execute, but as sets of equations that define how values in the program relate to one another. It lends itself to an algebraic approach to programming, building heavily on category theory, powerful static typing, and formal proofs to make sure programs are correct.

Two key abstractions in Haskell are the functor and the monad. A functor is any container type that can implement the function fmap, which takes a Functor<A>, a function A -> B, and returns a Functor<B>. An example of a functor in JavaScript is the Array class and its map() method: it takes an array and a function that turns each member of the array into something else, and returns an array of the results.

['category', 'theory'].map((word) => word.length)
// -> [8, 6]

A monad is any container type that implements >>= (pronounced “bind”), which takes a Monad<A>, and a function A -> Monad<B>, and returns Monad<B>. An example of this would be Array.flatMap(), which converts each member to an array, and returns the concatenation of all the results:

['incorporate monads', 'and category theory'].flatMap((str) => str.split(' '))
// -> [ 'incorporate', 'monads', 'and', 'category', 'theory' ]

Here flatMap is taking Array<String>, and a function String -> Array<String> and returning Array<String>. Compare to what map() would do in this situation:

['incorporate monads', 'and category theory'].map((str) => str.split(' '))
// -> [ [ 'incorporate', 'monads' ], [ 'and', 'category', 'theory' ] ]

It returns Array<Array<String>>, rather than Array<String> – it causes the container type Array to nest, while flatMap() flattens the containers. This is a key distinction between the functor fmap and monad >>= functions: the latter flattens one layer of nesting in the container type, and the former does not.

The problem for the functional programmers in the above thread is that Promise.then() does both of these. The function passed to then() is allowed to return either a normal value, or another Promise, and the result is identical. In effect, Promise.then() flattens any amount of nested promises. The main argument for this is convenience, which is to say that if Promise had distinct functor- and monad-flavoured interfaces it would mostly be harder to use. Let’s say that instead of then(), we had Promise.map() and Promise.flatMap() with analogous behaviour to the Array interface. If we have a Promise<String> and want to get the length of said string, we would do:

// Promise<String> -> Promise<Number>
pstr.map((str) => str.length)

It would be an error to use flatMap() here because String.length does not return a Promise, so there would be nothing to flatten. If instead we had a Promise<URL> and we wanted to fetch that URL, we would do:

// Promise<URL> -> Promise<Response>
purl.flatMap((url) => fetch(url))

This works because fetch() returns a Promise. If we instead used map() here we would get a nested promise, just as we get a nested Array if we use map() with a function that returns an Array:

// Promise<URL> -> Promise<Promise<Response>>
let ppresp = purl.map((url) => fetch(url))

To access the response we would then need to “unwrap” two layers of Promise to get at the data we actually care about. Since in this imaginary world the value inside a promise can only be accessed via map() and flatMap(), each of which only removes a single layer of wrapping from its input, we would need to do:

ppresp.map((presp) => presp.map((resp) => /* ... */))

The position of the Promises/A+ authors is that there is very little use for having a nested Promise, since all you ever want to do with a Promise is get the actual value out of it. A nested Array is a useful data structure, whereas nested promises would just retain how many async operations were needed to obtain a value, which isn’t useful, right? Meanwhile, having this map()/flatMap() distinction adds a lot of complexity to the API, creating one case which is an error, and one which does something useless. So on balance it is overwhelmingly more convenient for Promise.then() to flatten everything implicitly.

An alternative would have been to say that the function passed to then() must return a Promise, so then non-Promise values would need to be explicitly wrapped:

// Promise<String> -> Promise<Number>
pstr.then((str) => Promise.resolve(str.length))

This is also an inconvenience that you can argue has little value. Doing this could still lead to nesting promises, which would need to be implicitly flattened, so why bother making programmers type in an extra Promise.resolve()?

The reason then() is a problem for functional programmers is that its behaviour doesn’t fit either the definition of fmap or the definition of >>=. It accepts a function A -> B, and it accepts A -> Promise<B>, and because it implicitly performs any amount of flattening it also conceptually accepts functions that return indefinitely nested promises. If you are building a Haskell-like language that compiles to JavaScript, or just like to program in that style, and especially if you have a static type system, the behaviour of then() is very difficult to fit into your system. It doesn’t have a well-defined type, the ambiguous type of its input function makes it hard to compose with other functions, and so on. It’s a wart that has to be worked around if you want to use functional programming techniques. Because it behaves like a functor or a monad depending on what the input function returns, you cannot actually treat it reliably as either one, and this breaks a lot of patterns of function composition that need to make assumptions about what certain kinds of functions return.

In any case, the functional programmers lost the argument and the spec authors stuck with implicit flattening. But what if they hadn’t? Are there in fact use cases for nested promises that make it worth retaining the ability to create them? I would say that I have almost never needed a nested Promise, until recently when I was writing some slightly unusual concurrency control code.

For a few years now, I have been working on EscoDB, an encrypted document store that’s intended as a backend for password managers, authenticator apps, and other tools that need to maintain data privacy. It is designed around storing the actual data both on local disk and remote object stores with high latency, and goes to great lengths to optimise all the asynchronous I/O and cryptography it needs to do. Since it uses a lot of concurrent programming patterns, it needs abstractions to manage concurrency, one of which is the readers-writer lock.

Whereas a Mutex forces async functions to execute one after another, a RWLock lets either multiple reader functions execute at the same time, or a single writer function. If nobody is modifying a value, then any number of people can read it, but if one person is updating the value, everyone else needs to wait until they’re done. The API lets functions register as either readers or writers, and RWLock will execute them according to these rules. If all the functions are readers, they all execute concurrently:

function sleep (ms) {
  return new Promise((resolve) => setTimeout(resolve, ms))
}

function logger (name) {
  return async () => {
    console.log(name, '-- start')
    await sleep(100)
    console.log(name, '-- end')
  }
}

let rwlock = new RWLock()

rwlock.read(logger('A'))
rwlock.read(logger('B'))
rwlock.read(logger('C'))

/*  ->  A -- start
        B -- start
        C -- start
        A -- end
        B -- end
        C -- end    */

If all the functions are writers, they execute one after another.

rwlock.write(logger('A'))
rwlock.write(logger('B'))
rwlock.write(logger('C'))

/*  ->  A -- start
        A -- end
        B -- start
        B -- end
        C -- start
        C -- end    */

And if they’re a mix of readers and writers, then the readers execute concurrently while the writers wait for all the readers to finish before starting.

rwlock.read(logger('A'))
rwlock.read(logger('B'))
rwlock.write(logger('C'))

/*  ->  A -- start
        B -- start
        A -- end
        B -- end
        C -- start
        C -- end    */

RWLock can be very simply implemented on top of a concurrent Queue, a class which executes async functions with an optional limit on the number that may execute at the same time. It has the following API:

• push(fn): executes the async function fn as soon as it can, under the constraint that no more than limit such functions are running at the same time, and returns a Promise for the result of fn().
• onEmpty(): returns a Promise that is already resolved if the queue is empty, or will resolve as soon as the queue next becomes empty.

Here is the RWLock implementation:

class RWLock {
  constructor () {
    this._inbox = new Queue({ limit: 1 })
    this._read  = new Queue()
    this._write = new Queue({ limit: 1 })
  }

  read (fn) {
    return this._exec(fn, this._read, this._write)
  }

  write (fn) {
    return this._exec(fn, this._write, this._read)
  }

  async _exec (fn, runner, blocker) {
    let { promise } = await this._inbox.push(async () => {
      await blocker.onEmpty()
      return { promise: runner.push(fn) }
    })
    return promise
  }
}

The read() method waits for the _write queue to become empty and then pushes the function onto the _read queue, and the write() method does the inverse. The _exec() method makes use of a nested Promise, not because that value is useful in itself, but because this is necessary for the class to work correctly within JavaScript’s execution model. To see why this is necessary, we need to look at how this construction works, and some other implementations that would not work.

Logically, what _exec() is doing is waiting for one queue to become empty, and then adding fn to the other queue. The precise way it does this is important to making sure everything runs in the right order. The first critical part is that _inbox is a Queue with limit 1, i.e. it executes functions one at a time. Each fn submitted to the RWLock causes an onEmpty() call to execute inside the _inbox limit, i.e. only one onEmpty() request is active at a time.

For example, if the RWLock is currently executing readers, and another reader comes in and we call _write.onEmpty(). The _write queue must be empty since we’re executing readers, so this call returns immediately, and the new function is added to the _read queue. If a writer comes in, we instead call _read.onEmpty(), which will not resolve until all current readers are done. This therefore blocks the _inbox queue and no more incoming functions are dispatched until the current readers finish and the new writer is dispatched.

Then, we do something curious: the _inbox.push() call returns Promise<{ promise: Promise<T> }> where T is the result of fn(). If we removed the { promise: ... } wrapper then the automatic flattening of promises would mean it returns just Promise<T>:

  async _exec (fn, runner, blocker) {
    return await this._inbox.push(async () => {
      await blocker.onEmpty()
      return runner.push(fn)
    })
  }

If we do this, then _inbox, which is a limit-1 queue, will now wait until runner.push(fn) resolves, i.e. until fn() has been executed, before it will process any new functions. If we submit one reader function, then _read starts executing it. If we submit another reader, and the first one is still executing, then _inbox is still blocked, so we won’t push the second function into _read even though this would be safe. This means we now get the same behaviour as a mutex, i.e. completely sequential execution.

rwlock.read(logger('A'))
rwlock.read(logger('B'))
rwlock.read(logger('C'))

/*  ->  A -- start
        A -- end
        B -- start
        B -- end
        C -- start
        C -- end    */

This is incorrect because we’ve lost the ability to do concurrent reads, and we could have just used a mutex, i.e. a single Queue({ limit: 1 }). So, explicitly nesting a promise lets us put fn into a queue, but avoid blocking _inbox on the result of doing so. It’s a way of making one promise not wait on the result of another. Just as the difference between the Array map() and flatMap() methods is that map() does not concatenate the inner arrays, so nesting promises lets us avoid forcing one function to execute after another one, a kind of concatenation in time instead of in space.

But why is _inbox necessary? Consider what happens if we remove it:

  async _exec (fn, runner, blocker) {
    await blocker.onEmpty()
    return runner.push(fn)
  }

This looks like it does the same thing: wait for one queue to become empty and then add fn to the other one. However, with a mixed set of reader and writer functions, it does not behave correctly:

rwlock.read(logger('A'))
rwlock.read(logger('B'))
rwlock.write(logger('C'))

/*  ->  A -- start
        B -- start
        C -- start
        A -- end
        B -- end
        C -- end    */

Here we start the writer function C even though the readers A and B are still running. This is because, when you access the content of a Promise using then() or await, JavaScript always introduces an async microtask delay before the next statement, it never runs the following statement synchronously. That means that all the read() and write() calls above call await blocker.onEmpty() and then experience an async delay before calling runner.push(fn). That means all the calls will view the blocker queue as empty, and therefore execute fn immediately.

Therefore, we need a mutex around the check for blocker being empty, and the action of pushing fn into runner, so that fn is pushed into the appropriate queue before any other caller can check if said queue is empty. That mutex is provided by the _inbox queue. But, we need to avoid _inbox blocking on the execution of fn once it’s in a queue. _inbox just protects the decision about which queue to put fn in and when to do so, but having _inbox blocking on executing fn itself would mean we lose the ability to execute readers concurrently.

So this is not simply an esoteric functional programming problem. It turns out nested promises represent something useful: that one asynchronous function invokes another, but does not block on the completion of the inner one. Flattening promises represents concatenation in time – forcing one thing to execute after another. This is almost always what one wants when using promises normally, but if you’re actively managing concurrency control, orchestrating which things run in parallel and which things run sequentially, then you may have a use for nested promises.