1. Intro

Distributed systems are hard by definition. Thats why its important to not mix the complexity of distributivity with the complexity of the data domain and keep them separated.

In this post we’ll present Syncler – a distributed, optimistic replication backend for KiokuJS. Syncler will keep several replicas of arbitrary data synchronized, in real-time.

Syncler is unobtrusive and does not require from developer to understand all the mechanics of replication.  Replica is simply a set of Joose classes and/or native JavaScript data structures. Application uses accessors to mutate the attributes of classes, and Syncler takes care about all replication details. To be informed about updates in own replica, application can subscribe to various events using the observable pattern.

Syncler has been designed after Bayou (described in this paper). Using the classification of optimistic replication systems from this paper, Syncler can be characterized as the system with:

• operations transfers and semantic conflicts resolution
• many masters (though a single master is being used for updates stabilization)
• real-time updates propagation schedule
• a star topology for updates propagation
• eventual consistency (with very low replicas divergence period due to real-time updates)

To understand the advantages and drawbacks of these design choices please refer to the mentioned papers and other papers on the same topic. Most of the design choices were inherited from Bayou, however Bayou is more focused on off-line usage and Syncler – on real-time synchronization.

In addition to that, Syncler is cross-platform (runs equally well in browsers and NodeJS) and non-blocking.

In fact, Syncler is not a backend per se, but a special role, which can turn any KiokuJS backend into distributed system. Despite that at the moment there’s only one backend (for CouchDB), writing new ones is trivial (backend for Riak looks very promising).

As the transport, Syncler uses Socket.IO. Due to relative instability of most of the Socket.IO transports, at this moment, Syncler work correctly only with websockets.

2. Optimistic replication

Optimistic replication is a special type of replication, in which all masters apply own updates (and “see” the results) immediately. It can be said that every master is optimistic about that his update won’t cause any conflict.

Naturally, because the updates are applied without coordination among all replicas, the state of replicas may slightly diverge in the course of time, but the final state of all replicas is guaranteed to be the same (so called “eventual” or “weak” consistency).

Optimistic replication creates an illusion of zero network latency for own updates and greatly improves the responsiveness of the UI.

3. Applications

The application areas for Syncler is quite wide – it can be used as the basement of any kind of collaborative environment, where is a need to synchronize the operations,  originating from multiple users. Syncler is currently not bound to some specific data domain area.

3.1 Cooperative editing systems.

Probably the most obvious and well-known application area is so called “group-ware”. Typically its the graphical editor which allows several users to simultaneously edit some graphical scene, consisting from various graphical primitives. As the showcase of Syncler, we’ve developed such drawing application “ShareBoard”  (see section 6).

But naturally, the application doesn’t have to focus on graphic edition and can be used in any data domain.

3.2 Online multi-player games

Another very perspective area is the multi-player games, as games can be thought as some special kind of collaboration. This area is gaining interest along with modern browsers gaining visualization capabilities.

Syncler can greatly reduce the “entry barrier” for the creation of a new game as it handles a lower-level question of game scene synchronization, allowing the developers to focus on the gameplay process itself.

4. Architecture and realization

In this section we’ll shortly cover the architecture of the application, based on Syncler along with typical implementation patterns. Everything is still preliminary and subject for refactoring. We’ll use the code fragments from the ShareBoard system.

4.1 Replica

Replica is simply a set of Joose classes, having a `Syncler.Object` role. Currently replica also need to have a special “topic” class, with `Syncler.Topic` role (this requirement will be removed).

For example the board scene is modeled with the following class:

Class('ShareBoard.Model.Board', {

    does            : Syncler.Topic.UUID,

    has : {
        createdBy       : null,
        createdAt       : Joose.I.Now,

        elementsByUUID  : Syncler.I.Object,

        ...
    },

    methods : {

        add: function (element) {
            this.elementsByUUID.set(element.uuid, element)

            return element
        },

        remove : function (element) {
            this.elementsByUUID.remove(element.uuid)
        },

        each : function (func, scope) {
            return this.elementsByUUID.each(func, scope || this)
        },

        ...
    }
})

As you can see, board is generally a collection of board elements. And a base class for board element looks like:

Class('ShareBoard.Model.Board.Element', {

    does         : [
        'Syncler.Object',
        'KiokuJS.Feature.Class.OwnUUID'
    ],

    has : {
        board           : { required : true },

        x               : { required : true },
        y               : { required : true },

        color           : 'blue',

        ....
    },

    methods : {

        getBubbleTarget : function () {
            return this.board
        },

        remove : function () {
            this.board.remove(this)
        },

        moveTo : function (x, y) {
            this.setX(x)
            this.setY(y)
        }
    }
})

There are several subclasses of `Board.Element`, which add additional presentation logic.

4.2 Updates

To update the replica, application just calls the accessors methods. For example to move the board element (graphical primitive) to another position one can call the `moveTo` method, which looks as simple as:

    moveTo : function (x, y) {
        this.setX(x)
        this.setY(y)
    }

The wrappers for native JS data structures (which are required in the absence of proxies) provides obvious accessors for typical opeartions, for example to add/remove the board element to/from board, one can call `add/remove` method,
which updates the key in the underlaying JS Object.

    add: function (element) {
        this.elementsByUUID.set(element.uuid, element)

        return element
    },

    remove : function (element) {
        this.elementsByUUID.remove(element.uuid)
    }

The updates gets applied immediately and will propagate to other replicas in the background. As you can see, all the replication logic is packed into the meta-layer, the application is free from it.

Syncler assumes that any update can be rolled back (thats one of the design decisions of Bayou), thus all updates captures the “precondition” information which is being used for consistency checks.

4.3 Observable

To know about updates in replica, application should subscribe to various mutation events, which will be emitted by the meta-class. For example, to listen to mutation of any attribute in the underlying `model` instance, the base widget class can do the following:

Class('ShareBoard.View.Board.Element', {

    has : {
        boardWidget     : { is : 'rw', required : true },
        model           : { is : 'rw', required : true },

        ...
    },

    methods : {

        initialize : function () {
            this.model.on(
                '/mutation/apply/Syncler.Mutation.Class.Attribute',
                this.onAttributeMutate,
                this
            )
        },

        onAttributeMutate : function (event, mutation) {
            var attributeName       = mutation.attributeName

            if (attributeName == 'status')
                this.updateStyle()
            else
                this.updatePosition()
        },
        ...
    }
})

Another example – the board widget listen to `onNewInstanceOf` event of the replica itself and add widgets as new model elements appears:

        replica.onNewInstanceOf(
            'ShareBoard.Model.Board.Element',
            this.onNewBoardElement,
            this
        )

        onNewBoardElement : function (event, element, e) {
            var widgetClass = Joose.S.strToClass(element.widgetClass)

            var widget = new widgetClass({
                boardWidget : this,
                model       : element
            })

            this.addWidget(widget)

            widget.render()
        }

One more example – the board widget listen to the `remove` event of the underlying set of elements – and removes the widgets:

board.elementsByUUID.on(
    '/mutation/commit/Syncler.Mutation.Object.Remove',
    this.onBoardElementRemove,
    this
)

4.4 MVC

Its worth to note, that Syncler gently but insistently structures the application in MVC pattern.

Any Syncler application naturally has a Model (which is a replica being synchronized). Then, when implementing the UI (View), its not possible to put any Model logic in View, it must only listen the events from Model. The reason is simple – otherwise, UI will ignore the updates from other collaborators.

So, the distributed nature of the application, when there is an external source of model mutations, already implies the separation between Model and View. The only needed step for pure MVC is making the View as much stateless as possible, and moving the processing logic to Controllers. This step is not as obvious as the 1st one though and requires certain qualification level from the programmer.

5. Status and roadmap

Syncler is still on very early development stage. In fact, even the name of the framework isn’t final yet. However, it has an extensive test suite (including a stress load test) so it can already host simple applications ala ShareBoard.

The roadmap items list is quite long, among the other things are:

• Write some simple multi-player game to better research the capabilities of current approach.
• Research the possibilities of using state-transfer algorithms and multi-master storage schemes.
• Implementing a backend for KiokuJS, based on some graph database. This should be a very welcome addition, as graph databases usually have query languages and hopefully some of them even will provide certain form of referential integrity.
• Adopting the other Socket.IO transports.

6. The showcase system – ShareBoard

ShareBoard is the proof of concept showcase of Syncler. Its a simple collaborative graphical editor, which allows several users to edit the same “online white-board” scene.

Try it yourself (in web-socket enabled browser, like Chrome): http://shareboard.joose.it

Tip: After creation of the board – open several browser windows with it side-by-side, or drop the link to you friend.

7. Conclusion

We’ve presented Syncler – a distributed system, implemented as the backend for KiokuJS.

The creation of the showcase system “ShareBoard” has proved, that the demonstrated approach is very simple (so far as this word can be applied to distributed computing) and clean from the architectural point of view.

Syncler is still on very early development stage so further research directions were defined.

This is the 2nd part of the JooseX.CPS tutorial. Before reading it, make sure you’ve groked the 1st part. In the 1st part, we were managing the chains of anonymous functions. Such task can quickly become cumbersome, because the functions are, well, anonymous. Lets see how we can bring this concept to the higher level – out goal is to integrate the continuation passing style right into Joose classes.

Phase 1 – Synchronous

As the show case, lets use the following simple task. You have a class – wrapper around the text file, which provides simple convenience methods, like “read”, “write” etc. The wrapper is being used by Manager, which has one method – `processFile`. This method takes the source file name, reads the file and then saves the upper-cased content of the source file to another file, appending `-res` to the filename. Skim the code below:

require('task-joose-nodejs')

var fs = require('fs')

Class('TextFile', {

    has : {
        fileName    : { required : true },

        data        : null
    },

    methods : {

        asUpperCase : function () {
            return this.data.toUpperCase()
        },

        read : function () {
            return this.data = fs.readFileSync(this.fileName, 'utf8')
        },

        write : function (text) {
            this.data = text

            this.save()
        },

        save : function () {
            this.saveAs(this.fileName)
        },

        saveAs : function (fileName) {
            fs.writeFileSync(fileName, this.data)
        }
    }
})

Class('Manager', {

    my : {

        methods : {
            processFile : function (fileName) {

                var sourceFile = new TextFile({ fileName : fileName })

                sourceFile.read()

                var targetFile = new TextFile({
                    fileName    : fileName.replace(/\.(\w\w\w)$/, '-res.$1')
                })

                targetFile.write(sourceFile.asUpperCase())
            }
        }
    }
})

try {
    Manager.processFile('source.txt')
} catch (e) {

    console.log('Exception caught: %s', e)
}

Everything operates synchronously, any exceptions can be caught with `try/catch` blocks and all is fine. Now save the code in some js file and run it in node. You should see this:

Exception caught: Error: ENOENT, No such file or directory 'source.txt'

Of course, there is no source file. Create the ‘source.txt’ in the same directory as the code file and run it again. This time everything should work and the resulting file ‘source-res.txt’ , containing the uppercased content of the source file should appear.

Now, remove the write permission on the result file and run the script again. You’ll see

Exception caught: Error: EACCES, Permission denied 'source-res.txt'

Note, how we were able to caught the both exceptions from the “processFile” method (1st one thrown from “read” and 2nd  – from “saveAs”) without any additional gymnastic.

Ok, lets re-write the code in asynchronous way, keeping the functionality the same.

Phase 2 – Asynchronous

So here’s how it will looks like (note though that we won’t catch “usual” exceptions – only those reported by NodeJS methods):

require('task-joose-nodejs')

var fs = require('fs')

Class('TextFile', {

    has : {
        fileName    : { required : true },

        data        : null
    },

    methods : {

        asUpperCase : function () {
            return this.data.toUpperCase()
        },

        read : function (callback, errback) {
            var me = this

            fs.readFile(this.fileName, 'utf8', function (err, data) {

                if (err) {
                    errback(err)
                    return
                }

                me.data = data

                callback(data)
            })
        },

        write : function (text, callback, errback) {
            this.data = text

            this.save(callback, errback)
        },

        save : function (callback, errback) {
            this.saveAs(this.fileName, callback, errback)
        },

        saveAs : function (fileName, callback, errback) {
            fs.writeFile(fileName, this.data, function (err) {
                if (err)
                    errback(err)
                else
                    callback()
            })
        }
    }
})

Class('Manager', {

    my : {

        methods : {
            processFile : function (fileName, callback, errback) {

                var sourceFile = new TextFile({ fileName : fileName })

                sourceFile.read(function () {

                    var targetFile = new TextFile({
                        fileName    : fileName.replace(/\.(\w\w\w)$/, '-res.$1')
                    })

                    targetFile.write(sourceFile.asUpperCase(), callback, errback)

                }, errback)
            }
        }
    }
})

Manager.processFile('source.txt', function () {}, function (e) {

    console.log('Exception caught: %s', e)
})

I hear you said “Piece of cake?”. How about such requirements then – “write the content to the target file, and only if the operation completed successfully, remove the source file. If there was an error during `processFile` method, append it a log file. And process the array of filenames in no more than 5 parallel threads”. If you say “thats 2 pieces of cake” on this – then my congratulations, you are a callback’s overlord :). Otherwise, read further.

CPS conversion

So lets convert the classes above to the CPS. First thing to note is – we have “synchronous” (usual) and “asynchronous” methods.

Synchronous methods returns the result immediately, with the `return` statement. In the same way they throw the exceptions immediately, with `throw`.

“Asynchronous” methods returns the result, by passing it to the callback, and they throws the exceptions by passing them to errbacks. The call to callback/errback may happen after some arbitrary delay.

Synchronous and asynchronous methods

Lets split the methods into 2 groups, by their synchronicity:

require('task-joose-nodejs')

var fs = require('fs')

Class('TextFile', {
    trait : JooseX.CPS,
    ...

    // synchronous methods goes here
    methods : {

        asUpperCase : function () {
            return this.data.toUpperCase()
        }
    },

    // asynchronous methods goes here
    continued : {

        // right here
        methods : {

            read : function (callback, errback) {
                ...
            },

            write : function (text, callback, errback) {
                ...
            },

            save : function (callback, errback) {
                ...
            },

            saveAs : function (fileName, callback, errback) {
                ...
            }
        }
    }
})

Class('Manager', {

    my : {

        trait   : JooseX.CPS,

        continued   : {

            methods : {

                processFile : function (fileName, callback, errback) {
                    ...
                }
            }
        }
    }
})

The new class builder `continued` is provided by the trait `JooseX.CPS`. It can contain the following properties: “methods/before/after/override” which has the same semantic as usual builders. All asynchronous methods (or method modifiers) goes into that section.

Magic continuation instance

Next thing to note – every asynchronous method has the “callback” and “errback” arguments. Lets clean the definition and move them to the special `this.CONT` symbol instead, which we’ll made available inside of each asynchronous method. Yes, its the continuation instance from the part 1. This magic symbol is something like `this.SUPER` we are all used to, but its not a function – its an object with own methods and properties.

myAsynchronousMethod : function (p1) {

    // this.SUPER calls the implementation of the current method
    // from the superclass
    this.SUPER

    // this.CONT refers to the current continuation instance
    this.CONT
}

Important notes: Inside of each asynchronous method there is a special `this.CONT` symbol available, which refers to the “current” instance of JooseX.CPS.Continuation. When the method starts, the continuation is empty – it doesn’t contain any tasks.

Then, when inside of any asynchronous method, you call any other asynchronous method – the call is converted into the task for current continuation (with its TRY method). For example:

// asynchronous methods section
continued : {
    methods : {

        save : function () {
            // call to another asynchronous method
            this.saveAs(this.fileName).now()
        },

        saveAs : function () {
            //saveAs definition
            ...
        }
    }
}

// the above gets translated to

// asynchronous methods section
continued : {
    methods : {

        save : function () {
            this.CONT.TRY(function () {
                //saveAs definition
                ...
            }, this, [ this.fileName ]).now()
        },

        saveAs : function () {
            ...
        }
    }
}

Note, that call to TRY contains the object, calling the method, as 2nd argument and arguments for method – as 3rd.

Asynchronous methods are prohibited to return any values with `return`. Instead they should return values using the call to `this.CONT.CONTINUE()` method (or throw the exceptions with `this.CONT.THROW()`). Or they can instead launch the nested continuation – in this case it will be responsible for returning values. In the “synchronous” meaning, asynchronous methods will return the continuation instance they are attached to.

Also, as you remember, continuation tasks won’t launch immediately – they needs to be activated with `now`.

Phase 3 – CPS

The full version of the example above will looks like:

require('task-joose-nodejs')

var fs = require('fs')

Class('TextFile', {

    trait           : JooseX.CPS,

    has : {
        fileName    : { required : true },

        data        : null
    },

    methods : {

        asUpperCase : function () {
            return this.data.toUpperCase()
        }

    },

    continued : {
        methods : {

            read : function () {
                var me          = this
                var callback    = this.CONT.getCONTINUE()
                var errback     = this.CONT.getTHROW()

                fs.readFile(this.fileName, 'utf8', function (err, data) {

                    if (err) {
                        errback(err)
                        return
                    }

                    me.data = data

                    callback(data)
                })
            },

            write : function (text) {
                this.data = text

                this.save().now()
            },

            save : function () {
                this.saveAs(this.fileName).now()
            },

            saveAs : function (fileName) {
                var callback    = this.CONT.getCONTINUE()
                var errback     = this.CONT.getTHROW()

                fs.writeFile(fileName, this.data, function (err) {
                    if (err)
                        errback(err)
                    else
                        callback()
                })
            }
        }
    }
})

Class('Manager', {

    my : {
        trait   : JooseX.CPS,

        continued : {

            methods : {
                processFile : function (fileName) {

                    var sourceFile = new TextFile({ fileName : fileName })

                    sourceFile.read().THEN(function () {

                        var targetFile = new TextFile({
                            fileName    : fileName.replace(/\.(\w\w\w)$/, '-res.$1')
                        })

                        targetFile.write(sourceFile.asUpperCase()).now()

                    }).now()
                }
            }
        }
    }
})

Manager.processFile('source.txt').CATCH(function (e) {

    console.log('Exception caught: %s', e)

}).now()

Lets examine what have changed:
1) In the methods which integrates with “raw” asynchronous methods (“read” and “saveAs”) we’ve just removed the callbacks and errbacks from arguments and get them from the current continuation. Its a small winning as the outer interface of the methods becomes a bit simpler.
2) Methods which calls only other CPS methods (“write” and “save”) are now much cleaner – they don’t contain any callbacks at all.
3) The biggest gain is in the “processFile” method (which also uses only CPS methods). It doesn’t contain any callbacks/errbacks now! And all the errors from it get caught correctly (try to launch the code w/o source file and w/o write permissions to result file).
4) We now catch the exceptions from `Manager.processFile` in much more straightforward syntax.

For consolidation, lets see how the `processFile` method will look like on the JooseX.CPS.Continuation level:

processFile : function (fileName) {

    var sourceFile = new TextFile({ fileName : fileName })

    this.CONT.TRY(function () {
        /* `read` method definition here */,
    }, sourceFile, [])

    this.CONT.THEN(function () {

        var targetFile = new TextFile({
            fileName    : fileName.replace(/\.(\w\w\w)$/, '-res.$1')
        })

        this.CONT.TRY(function () {
            /* `write` definition here */
        }, targetFile, [ sourceFile.asUpperCase() ])

        this.CONT.now()
    })

    this.CONT.now()
}

Some features

The fact that methods don’t launch immediately provides couple of interesting features.

First of all, when you call several “continued” methods in a raw and then activate the continuation – the methods will be executed sequentially. Compare asynchronous and synchronous versions:

asyncMethod : function () {
    this.anotherAsyncMethod1()
    this.anotherAsyncMethod2()
    this.anotherAsyncMethod3()
    this.anotherAsyncMethod4().now()
}

syncMethod : function () {
    this.anotherSyncMethod1()
    this.anotherSyncMethod2()
    this.anotherSyncMethod3()
    this.anotherSyncMethod4()
}

Such pattern works when methods do not depend on the returning values from the earlier methods.

Another feature is that the call to asynchronous method can be used as some sort of “promise” (though JooseX.CPS was not designed to provide this functionality). You can pass the current continuation to some other method which will add some generic step to it. Using JooseX.CPS this way wasn’t intended but its possible.

Some sugar

A JooseX.CPS trait will add a JooseX.CPS.ControlFlow role to your class, which allow you to omit the `this.CONT` when calling various methods of current continuation. So you can call ‘this.CONTINUE()` instead of `this.CONT.CONTINUE()`, `this.getCONTINUE()` etc.

Conclusion

Hopefully this post illustrates the ideas from this one. Note how the meta-layer absorbed all the low-level details and provided a much cleaner abstractions.

JooseX.CPS significantly lowers the overhead of creation and managing the asynchronous interface of your classes. No excuses to write blocking code anymore :)

P.S.

Curious how the “2 pieces of cake” example will looks like in CPS? Here it is (or here download from here) . For brevity we avoided synchronization problems when writing to log file and just output to console. Try to run the code, already having one result file, w/o write permission.

require('task-joose-nodejs')

var fs = require('fs')

Class('TextFile', {

    trait           : JooseX.CPS,

    has : {
        fileName    : { required : true },

        data        : null
    },

    methods : {

        asUpperCase : function () {
            return this.data.toUpperCase()
        }

    },

    continued : {
        methods : {

            read : function () {
                var me          = this
                var callback    = this.CONT.getCONTINUE()
                var errback     = this.CONT.getTHROW()

                fs.readFile(this.fileName, 'utf8', function (err, data) {

                    if (err) {
                        errback(err)
                        return
                    }

                    me.data = data

                    callback(data)
                })
            },

            write : function (text) {
                this.data = text

                this.save().now()
            },

            save : function () {
                this.saveAs(this.fileName).now()
            },

            saveAs : function (fileName) {
                var callback    = this.CONT.getCONTINUE()
                var errback     = this.CONT.getTHROW()

                fs.writeFile(fileName, this.data, function (err) {
                    if (err)
                        errback(err)
                    else
                        callback()
                })
            },

            remove : function () {
                var callback    = this.CONT.getCONTINUE()
                var errback     = this.CONT.getTHROW()

                fs.unlink(this.fileName, function (err) {
                    if (err)
                        errback(err)
                    else
                        callback()
                })
            }
        }
    }
})

Class('Manager', {

    my : {
        trait   : JooseX.CPS,

        continued : {

            methods : {

                log : function (text) {
                    console.log(text)

                    this.CONTINUE()
                },

                processArray : function (fileNames) {

                    Joose.A.each(fileNames, function (fileName) {

                        this.AND(function () {

                            this.processFile(fileName).except(function (e) {

                                this.log("Error processing " + fileName + ": " + e).now()

                            }).now()
                        })

                    }, this)

                    this.ANDMAX(5).now()
                },

                processFile : function (fileName) {

                    var sourceFile = new TextFile({ fileName : fileName })

                    sourceFile.read().andThen(function () {

                        var targetFile = new TextFile({
                            fileName    : fileName.replace(/\.(\w\w\w)$/, '-res.$1')
                        })

                        targetFile.write(sourceFile.asUpperCase()).andThen(function () {

                            sourceFile.remove().now()
                        })
                    })
                }
            }
            // eof methods
        }
        // eof continued
    }
})

var fileNames = [
    'source1.txt',
    'source2.txt',
    'source3.txt',
    'source4.txt',
    'source5.txt',
    'source6.txt',
    'source7.txt'
]

Manager.processArray(fileNames, "log.txt").now()

So, after the recent crack of our server, the post about the JooseX.CPS was lost. It was poorly written anyway, so may be its even better that the server was hacked – because I’m writing a new post. JooseX.CPS is quite important extension, and its being extensively used in many other Joose-based projects, like KiokuDB, Syncler and Symbie. The new post is written in form of tutorial and contains a runnable code. At the end of this post you should grok the continuation passing style :)

Setup

To follow this tutorial you will need:

• NodeJS
• npm
• task-joose-nodejs

The latter can be installed with `npm install task-joose-nodejs`.

Part I – Meet The Continuation

The tutorial will be split into 2 parts. In this part we’ll examine the low-level JooseX.CPS.Continuation class and its features. The second part will describe a JooseX.CPS trait itself, which provides a CPS sugar layer on top of JooseX.CPS.Continuation for you Joose classes.

Just try it, aka “Hello world”

Before we’ll dig in, lets see how the “hello world” will looks like in CPS world:

require('task-joose-nodejs')

TRY(function () {

    console.log('Hello')

    this.CONT.CONTINUE()

}).DELAY(1000).THEN(function () {

    console.log('world!')

}).now()

Try to launch this in your Node. You should see the following with 1s delay between the words:

nickolay@desktop:~/Documents/JooseBlog/CPS-Tutorial$ node hello_world.js
Hello
world!

Compare with

console.log('Hello')
console.log('world!')

You see the idea? With some additional boilerplate between them, the synchronous the statements work asynchronously, thats it. And as we’ll move forward in this tutorial, the amount of boilerplate will reduce, really.

Continuation anatomy

In the first approximation, continuation is an array of tasks, also containing the “catch” and “finally” functions:

CONT = {

    tasks           : [
        {
            func    : function doThis() { ... },
            scope   : {},
            args    : [ ... ]
        },
        {
            func    : function doThat() { ... },
            ...
        },
        ...
    ],

    catchFunc       : function () { ... },
    finallyFunc     : function () { ... },
}

The “task” is simply a function (mandatory), plus a scope for its execution and an array of arguments (both optional). Such continuation roughly corresponds to the following “synchronous” code:

try {

    doThis()
    doThat()

} catch (e) {

    cathcFunc(e)

} finally {

    finallyFunc()
}

Once more – continuation is a single “try/catch” block, containing an array of tasks. You can add tasks to the continuation with the `TRY(func, scope, args)` method. Adding tasks will not launch them immediately – you need to “activate” the continuation manually with its method `now`.

After activation, continuation will launch the first task. It will embed a new, fresh continuation instance into the scope of task, as the `this.CONT`. To pass the control flow from the 1st task to the next one, you need to call the `CONTINUE` method of the embedded continuation. Any parameters, passed to `CONTINUE` will be available as the arguments of the next task. Lets re-write the “Hello world” example, “by hands”:

require('task-joose-nodejs')

var CONT    = new JooseX.CPS.Continuation()

var scope   = { toString : function () { return 'My scope' } }

CONT.TRY(function () {
    console.log("Current scope 1: %s", this)

    console.log('Hello')

    // a new, fresh instance of JooseX.CPS.Continuation has been embedded:
    // this.CONT != CONT
    this.CONT.CONTINUE('world', '!')

}, scope)

CONT.TRY(function (str1, str2) {
    console.log("Current scope 2: %s", this)

    console.log(str1, str2)
})

CONT.now()

Some sugar: there is `THEN` synonym for the `TRY` method, which sounds more naturally for the 2nd and further tasks. Note, that if you have not provided a scope for the task, it will be taken from the previous task.

The call to `CONTINUE` doesn’t have to be done synchronously, you can delay it. To do that, capture it to the closure with the `getCONTINUE` method. Lets add the 1s pause between the outputs, “manually” (w/o the `DELAY()`):

require('task-joose-nodejs')

var CONT    = new JooseX.CPS.Continuation()

var scope   = { toString : function () { return 'My scope' } }

CONT.TRY(function () {
    console.log("Current scope 1: %s", this)

    console.log('Hello')

    // this.CONT != CONT - its a new, fresh instance of JooseX.CPS.Continuation
    // passing the parameters further
    this.CONT.CONTINUE('world', '!')

}, scope)

CONT.THEN(function (str1, str2) {

    var CONTINUE = this.CONT.getCONTINUE()

    setTimeout(function () {

        CONTINUE(str1, str2)

    }, 1000)
})

CONT.THEN(function (str1, str2) {
    console.log("Current scope 2: %s", this)

    console.log(str1, str2)
})

CONT.now()

Hey don’t scan, read the sources :)

Instead of manual instantiation of JooseX.CPS you can use global `TRY` helper which just creates a new instance and delegates to its `TRY` method. And of course, the `TRY/THEN` method return the continuation itself, so the calls can be chained. There is also a `DELAY` method and so we comes to the initial “Hello world”:

require('task-joose-nodejs')

TRY(function () {

    console.log('Hello')

    this.CONT.CONTINUE()

}).DELAY(1000).THEN(function () {

    console.log('world!')

}).now()

Nesting

Instead of calling the `CONTINUE` of the inner continuation, you can add a nested continuation – using `TRY` as usual. Keep in mind, that you need to activate it with `now`. When the last task of the nested continuation will call `CONTINUE` the control flow will continue on the outer level:

require('task-joose-nodejs')

TRY(function () {

    console.log('Hello')

    this.CONT.DELAY(500).THEN(function () {

        console.log(', ')

        this.CONT.CONTINUE()

    }).DELAY(500).THEN(function () {

        this.CONT.CONTINUE('world')

    }).now()

}).THEN(function (str) {

    console.log(str)

}).now()

Pretty intuitive, don’t you think? Important note: once you’ve nested the continuation (instead of calling the `CONTINUE`) you must not call its `CONTINUE` method to avoid the split of the control flow. From this point, its the nested continuation responsibility to return the control flow.

There is also a `RETURN` method to skip the other tasks from the current nesting level and return the control flow to the outer.

Exceptions

We talked about the correspondence to the `try/catch/finally` block, and here’s their equivalent in the CPS. Surprisingly, they are named `TRY/CATCH/FINALLY` :) . Simplest case:

require('task-joose-nodejs')

TRY(function () {

    throw "oops"

}).CATCH(function (e) {

    console.log("Exception caught: %s", e)

    this.CONT.CONTINUE()

}).FINALLY(function () {

    console.log("Cleanup happens")

}).now()

More text in bold: The `CATCH/FINALLY` tasks are also “continued”. That is, to return the control flow from them you need to use `this.CONT.CONTINUE()` as in `TRY/THEN`. This will allow you to write asynchronous exceptions handlers.

Nesting and Exceptions

The exceptions will correctly propagate from the nested continuations:

require('task-joose-nodejs')

TRY(function () {

    console.log('Hello')

    this.CONT.DELAY(500).THEN(function () {

        console.log(',')

        throw "world"

    }).DELAY(500).THEN(function () {

        console.log("YOU SHOULD N'T SEE THIS TEXT")

    }).now()

}).CATCH(function (str) {

    console.log(str)

}).now()

they could be re-thrown just fine:

require('task-joose-nodejs')

TRY(function () {

    console.log('Hello')

    this.CONT.DELAY(500).THEN(function () {

        console.log(',')

        throw "f$$k"

    }).CATCH(function (str) {

        throw "polite"

    }).now()

}).CATCH(function (str) {

    console.log(str, "world")

}).now()

And in general, behaves very much like “usual” exceptions.

Show me the parallel

Of course any library for asynchronous control flow should provide the “parallel” mode. In JooseX.CPS you can activate it with `AND`:

require('task-joose-nodejs')

TRY(function () {

    console.log('Activated branch 1')

    var CONTINUE = this.CONT.getCONTINUE()

    setTimeout(function () {

        console.log('world')

        CONTINUE('!')

    }, 1000)

}).AND(function () {

    console.log('Activated branch 2')

    var CONTINUE = this.CONT.getCONTINUE()

    setTimeout(function () {

        console.log('Hello')

        CONTINUE('!')

    }, 500)

}).THEN(function (res1, res2) {

    console.log(res1[0], res2[0])

}).now()

If you’ll run this code, you’ll  see that branches were activated simultaneously, but the 2 final exclamation characters appears only after the branches gets merged. The very 1st task following the `AND` group receives the `arguments` objects passed to `CONTINUE` in the order of branches declaration.

Of course, you can nest a new continuation into any of the branches:

require('task-joose-nodejs')

TRY(function () {

    console.log('Activated branch 1')

    this.CONT.CONTINUE('Branch 1 result')

}).AND(function () {

    console.log('Activated branch 2')

    this.CONT.DELAY(500).THEN(function () {

        console.log('The very long')

        this.CONT.CONTINUE()

    }).DELAY(500).THEN(function () {

        console.log('asynchronous task')

        this.CONT.CONTINUE()

    }).DELAY(500).THEN(function () {

        console.log('from several steps')

        this.CONT.CONTINUE('Branch 2 result')

    }).now()

}).THEN(function (res1, res2) {

    console.log(res1[0], res2[0])

}).now()

Some sugar

JooseX.CPS.Continuation also has lower-cased synonyms for the most used methods:

• THEN – then
• AND – and
• CATCH – except
• FINALLY – ensure

Also, the `this.CONT.THEN( … ).now()` can be written as `this.CONT.andThen( … )`

These synonyms simplifies a code a little bit and makes it more readable, but it still contain a lot of boilerplate. This boilerplate can be greatly reduced when using JooseX.CPS trait, which adds the “continued” methods to your class.

It will be described in the part 2 of the tutorial, stay tuned!

In this post I’d like to present the building tools I’m using for my projects. Hopefully after it there will be more contributions :)

I’m using a plugin for Dist::Zilla. Dist::Zilla comes from the severe perl world, where people with strong spirit writes a code which is 10 years backward-compatible (thanks to chromatic thats gradually changing).

Generally, Dist::Zilla is purposed to assist the CPAN authors, but it turned out, that many concepts are language and distribution platform neutral. For example integration with Git – in any language or any distribution platform you definitely want to tag the new release with version number. Then you probably want  to commit the updates. Another example will be `Changes` file – before release it definitely makes sense to check whether it contains the entries for new version. Etc.

All such routine tasks can be automated with the Dist::Zilla. It can create a barebone distribution, generate an appropriate LICENCE file, test the distribution, release it on `npm` and much more!

Setup

Installation of the perl modules is usually scary for people without perl background. So you are welcome to the #joose channel on freenode with any questions. There are also known problems on MacOS, so Mac owners are even more welcome :) Note, that you need to have ‘git’ installed.

I found it much easier to use the recently appeared Miyagawa’s “cpan-minus” tool. So, to install the cpanminus:

> curl -L http://cpanmin.us | perl - --sudo App::cpanminus

then

> cpanm Dist::Zilla::Plugin::JSAN --sudo

After installation, setup the Dist::Zilla in interactive mode (this is required only once):

> dzil setup
What's your name?...

When asked about the default license, here’s the list of available identificators (at the bottom and strip the leading “Software::License::”). The information you provided will be used during creation of new distributions.

Starting a distribution

To start a new distribution:

> dzil new -P JSAN -p joose Sample-Dist

[DZ] making target dir /home/nickolay/Playground/dzil/Sample-Dist
[DZ] writing files to /home/nickolay/Playground/dzil/Sample-Dist
[Git::Init] Initializing a new git repository in /home/nickolay/Playground/dzil/Sample-Dist
[DZ] dist minted in ./Sample-Dist>

This command should create a barebone distribution like this. It will also create a git repository and perform a famous “initial commit”! :) See this page to know how to customize the template.

And thats it! Distribution will contain a single main module with some documentation. After some hacking, you can easily publish the distribution in `npm` (see below).

The distribution will also contain a rudimentary test suite, which you can run in NodeJS (assuming you have test-run installed) with:

> node t/index.js

or, if you’ve completed the 3.1 and 3.2 steps from this document, in browsers. Put the distribution in some web-directory and point the browser to the URL like:

http://localhost/my_workspace/Sample-Dist/t/index.htnml

dist.ini

The whole building process is managed by the `dist.ini` file in the root of distribution. Its a text file which lists the used plugins and their parameters. For example:

; Auto-increment the version number, based on the tags information from git
[Git::NextVersion]
first_version   = 0.0.1

; include the link to git repo and web page (if "origin" points to github)
[GithubMeta]

For the details on the individual plugin, prepend its name with “Dist::Zilla::Plugin” and perform a CPAN search.

See the documentation for the full details on the JSAN plugin, here I’ll briefly describe some of the typical tasks.

Building the distribution

‘build’ command takes your sources as the input, process them (add/remove/change some files) and generate the output (location can be specified with ‘in’ option):

> dzil build [--in /target/dir]

Various plugins participate in this phase. Typical example will be [License] plugin, which adds a LICENSE file with the text of your license.

Releasing the distribution

‘release’ command first build the distribution, then runs plugins which implements the “release” phase. There is a [JSAN::NPM::Publish] plugin, which can publish your dist in npm:

> dzil release

Use case

For example, I’d like to include the version information in the source files, like:

Class('Some.Class', {
    VERSION    : 0.01

    has : { ... },
    methods : { ... }
})

Then before each release, you need to manually update the version information. That quickly becomes PITA. Thankfully, this task can easily automated, especially that Dist::Zilla already can increment the version, based on the information from git tags. So, if we’ll include a version placeholder in the sources:

Class('Some.Class', {
    /*VERSION,*/

    has : { ... },
    methods : { ... }
})

and the plugin to the “dist.ini”, which know how to find that placeholder and replace it with actual version:

[JSAN::PkgVersion]

then during each build dzil will perform all the routine work.

Conclusion

Setting up a proper quality control for your release process is very important. Currently I’m maintaining  ~30 distributions. Under “maintaining” I don’t mean I have that much github repos. I mean all of them have proper versions, tags, docs, test suites, changelogs etc. All of them are released on `npm`. I just can’t imagine how I would do that without Dist::Zilla.

See also

[JSAN] plugin docs and http://dzil.org/

What

KiokuJS is a persistence layer for Joose. Its a freestyle port of KiokuDB which does the same thing in the Moose world.

KiokuJS provides transparent storing/retrieving for instances of Joose classes. Native data structures (Object/Array/etc) can be stored as well. “Transparent” means that class can be stored without any modifications to its declaration (see also But).

KiokuJS is cross-platform – it runs equally well in browsers and NodeJS. KiokuJS is non-blocking – all interactions with DB are asynchronous.

Why

Remember the last Web2.0 project you worked on. Quite probably it was looking as:

Note, that you need to define the serialization procedure 4(!) times. Even if you use some kind of client-side helpers of modern frameworks, you still need to (manually) map your client-side JavaScript data structures to the storage layer.

Meanwhile, nowadays its obvious, that JavaScript has evolved from client-side animation libraries into full-stack platform. There are no reasons why your applications can’t be fully written in JavaScript and benefit from the homogeneous environment both on client and server.

So the goals of KiokuJS are:

• eliminate all the serialization boilerplate from the application
• use a single model definition both on client and server
• encapsulate all the storage-layer details in high-level interface

Example

// class declaration

Class('Person', {

    has : {
        self    : null,

        name    : null,

        spouse  : null
    },

    methods : {

        initialize : function () {
            this.self = this // circular ref
        }
    }
})

// arbitrary data structure

var Homer = new Person({
    name    : "Homer"
})

var Marge = new Person({
    name    : "Marge"
})

Homer.spouse = Marge
Marge.spouse = Homer

// handler setup

var handle = new KiokuJS.Backend.CouchDB({
    dbURL   : 'http://local:5984/someDB'
})

// storing

var scope = handle.newScope()

scope.store(Homer, Marge).andThen(function (homerID, margeID) {
    ...
})

// retrieving

var scope = handle.newScope()

scope.lookUp(homerID, margeID).andThen(function (Homer2, Marge2) {

    // Homer2.self   === Homer2
    // Homer2.spouse  === Marge2

    // Marge2.self   === Marge2
    // Marge2.spouse  === Homer2
})

How

From the KiokuJS viewpoint, the data structures represents a cyclic, directed graph:Data graph is then serialized into JSON structures like:

{
    "className": "Person",
    "ID": "2D9E24EE-89BA-A361-B0E6-2D8A6C1226F6",
    "isRoot": true,
    "data": {
        "self": {
            "$ref": "2D9E24EE-89BA-A361-B0E6-2D8A6C1226F6"
        },
        "name": "Homer",
        "spouse": {
            "$ref": "0CFB5A77-8B58-BC7F-9B7D-DBDD78A29AC2"
        }
    },
    "$entry": true
}

{
    "className": "Person",
    "ID": "0CFB5A77-8B58-BC7F-9B7D-DBDD78A29AC2",
    "isRoot": true,
    "data": {
        "self": {
            "$ref": "0CFB5A77-8B58-BC7F-9B7D-DBDD78A29AC2"
        },
        "name": "Marge",
        "spouse": {
            "$ref": "2D9E24EE-89BA-A361-B0E6-2D8A6C1226F6"
        }
    },
    "$entry": true
}

There are a number of options, defining the serialization details (implemented as attribute traits). When required, its also possible to provide a custom serialization format for specific class.

Its worth to note, that all asynchronous interfaces in Kioku are implemented using the continuation-passing style extension for Joose, which provides a precise control over the asynchronous methods. The previous storing example can be elaborated like this:

// storing

UI.maskScreen("Please wait")

scope.store(Homer, Marge).then(function (homerID, margeID) {

    alert('Objects stored successfully')

    this.CONTINUE()

}).CATCH(function (e) {

    alert('There were an exception [' + e + '] during storing operation')

    this.CONTINUE()

}).FINALLY(function () {

    UI.unMaskScreen()

}).now()

But

Persistence is mostly transparent, but there are some limitations. Generally any closures-related magic won’t be recognized, simply because there are no according introspection capabilities in the language. For example, if you have a function, tied to a scope with some closures, as the value of the attribute:

var closure = { foo : 'bar' }

homer.magic = function () { return closure.foo }

then KiokuJS will store the function declaration only, not the scope it was defined in. So after retrieving the instance from the backend and running “homer.magic()” there will be an exception like [“closure” is not defined].

Another limitation is related to searching. Searching is constrained to backends capabilities only.

Backends

Kioku is backend-agnostic, but currently mostly target NoSQL (key/value) backends because of their simplicity, native JSON support and HTTP APIs, which allows to directly access the DB from the browsers.  With some additional efforts can be implemented a relational backend, either static (when classes definitions will be generated from the relational schema) or dynamic (when the relational schema will be adjusted to the classes definition).

For the development and testing there is a “Hash” backend. For the production – CouchDB backend.

Test suite of Kioku is organized that way, that makes it possible to run for each backend.

There is also a special backend (actually a role for backend) – KiokuJS.Backend.Batch, intended for usage in browsers. It allows to combine the individual requests for separate documents into batches, saving a lot of HTTP requests.

Writing new backends is easy – they only need to define 4 methods.

Status

KiokuJS is at 0.01 version. This means its “stable enough”, has the extensive test suite, but is mostly intended for early adopters, which are welcome to Joose IRC channel with any questions. Currently there is no documentation.

To install Kioku, run

> npm install kiokujs

You may also want to install CouchDB backend with:

> npm install kiokujs-backend-couchdb

After that this mini demo app should just work from NodeJS, supposing you have correct value for NODE_PATH environment variable (see items 3.x in Joose installation).

Roadmap

• Add more backends – Riak and MongoDB are 1st candidacies.
• Try to implement backends for graph databases (very promising).
• Syncler (draft name) – real-time backend, which use the optimistic replication to synchronize the state of replicas between clients.