Joose blog

Blog about Joose – advanced class system for JavaScript

Syncler – distributed applications for human beings

with 9 comments

1. Intro

Distributed systems are hard by definition. Thats why its important to not mix the complexity of distributivity with the complexity of the data domain and keep them separated.

In this post we’ll present Syncler – a distributed, optimistic replication backend for KiokuJS. Syncler will keep several replicas of arbitrary data synchronized, in real-time.

Syncler is unobtrusive and does not require from developer to understand all the mechanics of replication.  Replica is simply a set of Joose classes and/or native JavaScript data structures. Application uses accessors to mutate the attributes of classes, and Syncler takes care about all replication details. To be informed about updates in own replica, application can subscribe to various events using the observable pattern.

Syncler has been designed after Bayou (described in this paper). Using the classification of optimistic replication systems from this paper, Syncler can be characterized as the system with:

  • operations transfers and semantic conflicts resolution
  • many masters (though a single master is being used for updates stabilization)
  • real-time updates propagation schedule
  • a star topology for updates propagation
  • eventual consistency (with very low replicas divergence period due to real-time updates)

To understand the advantages and drawbacks of these design choices please refer to the mentioned papers and other papers on the same topic. Most of the design choices were inherited from Bayou, however Bayou is more focused on off-line usage and Syncler – on real-time synchronization.

In addition to that, Syncler is cross-platform (runs equally well in browsers and NodeJS) and non-blocking.

In fact, Syncler is not a backend per se, but a special role, which can turn any KiokuJS backend into distributed system. Despite that at the moment there’s only one backend (for CouchDB), writing new ones is trivial (backend for Riak looks very promising).

As the transport, Syncler uses Socket.IO. Due to relative instability of most of the Socket.IO transports, at this moment, Syncler work correctly only with websockets.

2. Optimistic replication

Optimistic replication is a special type of replication, in which all masters apply own updates (and “see” the results) immediately. It can be said that every master is optimistic about that his update won’t cause any conflict.

Naturally, because the updates are applied without coordination among all replicas, the state of replicas may slightly diverge in the course of time, but the final state of all replicas is guaranteed to be the same (so called “eventual” or “weak” consistency).

Optimistic replication creates an illusion of zero network latency for own updates and greatly improves the responsiveness of the UI.

3. Applications

The application areas for Syncler is quite wide – it can be used as the basement of any kind of collaborative environment, where is a need to synchronize the operations,  originating from multiple users. Syncler is currently not bound to some specific data domain area.

3.1 Cooperative editing systems.

Probably the most obvious and well-known application area is so called “group-ware”. Typically its the graphical editor which allows several users to simultaneously edit some graphical scene, consisting from various graphical primitives. As the showcase of Syncler, we’ve developed such drawing application “ShareBoard”  (see section 6).

But naturally, the application doesn’t have to focus on graphic edition and can be used in any data domain.

3.2 Online multi-player games

Another very perspective area is the multi-player games, as games can be thought as some special kind of collaboration. This area is gaining interest along with modern browsers gaining visualization capabilities.

Syncler can greatly reduce the “entry barrier” for the creation of a new game as it handles a lower-level question of game scene synchronization, allowing the developers to focus on the gameplay process itself.

4. Architecture and realization

In this section we’ll shortly cover the architecture of the application, based on Syncler along with typical implementation patterns. Everything is still preliminary and subject for refactoring. We’ll use the code fragments from the ShareBoard system.

4.1 Replica

Replica is simply a set of Joose classes, having a `Syncler.Object` role. Currently replica also need to have a special “topic” class, with `Syncler.Topic` role (this requirement will be removed).

For example the board scene is modeled with the following class:

Class('ShareBoard.Model.Board', {

    does            : Syncler.Topic.UUID,

    has : {
        createdBy       : null,
        createdAt       : Joose.I.Now,

        elementsByUUID  : Syncler.I.Object,

        ...
    },

    methods : {

        add: function (element) {
            this.elementsByUUID.set(element.uuid, element)

            return element
        },

        remove : function (element) {
            this.elementsByUUID.remove(element.uuid)
        },

        each : function (func, scope) {
            return this.elementsByUUID.each(func, scope || this)
        },

        ...
    }
})

As you can see, board is generally a collection of board elements. And a base class for board element looks like:

Class('ShareBoard.Model.Board.Element', {

    does         : [
        'Syncler.Object',
        'KiokuJS.Feature.Class.OwnUUID'
    ],

    has : {
        board           : { required : true },

        x               : { required : true },
        y               : { required : true },

        color           : 'blue',

        ....
    },

    methods : {

        getBubbleTarget : function () {
            return this.board
        },

        remove : function () {
            this.board.remove(this)
        },

        moveTo : function (x, y) {
            this.setX(x)
            this.setY(y)
        }
    }
})

There are several subclasses of `Board.Element`, which add additional presentation logic.

4.2 Updates

To update the replica, application just calls the accessors methods. For example to move the board element (graphical primitive) to another position one can call the `moveTo` method, which looks as simple as:

    moveTo : function (x, y) {
        this.setX(x)
        this.setY(y)
    }

The wrappers for native JS data structures (which are required in the absence of proxies) provides obvious accessors for typical opeartions, for example to add/remove the board element to/from board, one can call `add/remove` method,
which updates the key in the underlaying JS Object.

    add: function (element) {
        this.elementsByUUID.set(element.uuid, element)

        return element
    },

    remove : function (element) {
        this.elementsByUUID.remove(element.uuid)
    }

The updates gets applied immediately and will propagate to other replicas in the background. As you can see, all the replication logic is packed into the meta-layer, the application is free from it.

Syncler assumes that any update can be rolled back (thats one of the design decisions of Bayou), thus all updates captures the “precondition” information which is being used for consistency checks.

4.3 Observable

To know about updates in replica, application should subscribe to various mutation events, which will be emitted by the meta-class. For example, to listen to mutation of any attribute in the underlying `model` instance, the base widget class can do the following:

Class('ShareBoard.View.Board.Element', {

    has : {
        boardWidget     : { is : 'rw', required : true },
        model           : { is : 'rw', required : true },

        ...
    },

    methods : {

        initialize : function () {
            this.model.on(
                '/mutation/apply/Syncler.Mutation.Class.Attribute',
                this.onAttributeMutate,
                this
            )
        },

        onAttributeMutate : function (event, mutation) {
            var attributeName       = mutation.attributeName

            if (attributeName == 'status')
                this.updateStyle()
            else
                this.updatePosition()
        },
        ...
    }
})

Another example – the board widget listen to `onNewInstanceOf` event of the replica itself and add widgets as new model elements appears:

        replica.onNewInstanceOf(
            'ShareBoard.Model.Board.Element',
            this.onNewBoardElement,
            this
        )

        onNewBoardElement : function (event, element, e) {
            var widgetClass = Joose.S.strToClass(element.widgetClass)

            var widget = new widgetClass({
                boardWidget : this,
                model       : element
            })

            this.addWidget(widget)

            widget.render()
        }

One more example – the board widget listen to the `remove` event of the underlying set of elements – and removes the widgets:

board.elementsByUUID.on(
    '/mutation/commit/Syncler.Mutation.Object.Remove',
    this.onBoardElementRemove,
    this
)

4.4 MVC

Its worth to note, that Syncler gently but insistently structures the application in MVC pattern.

Any Syncler application naturally has a Model (which is a replica being synchronized). Then, when implementing the UI (View), its not possible to put any Model logic in View, it must only listen the events from Model. The reason is simple – otherwise, UI will ignore the updates from other collaborators.

So, the distributed nature of the application, when there is an external source of model mutations, already implies the separation between Model and View. The only needed step for pure MVC is making the View as much stateless as possible, and moving the processing logic to Controllers. This step is not as obvious as the 1st one though and requires certain qualification level from the programmer.

5. Status and roadmap

Syncler is still on very early development stage. In fact, even the name of the framework isn’t final yet. However, it has an extensive test suite (including a stress load test) so it can already host simple applications ala ShareBoard.

The roadmap items list is quite long, among the other things are:

  • Write some simple multi-player game to better research the capabilities of current approach.
  • Research the possibilities of using state-transfer algorithms and multi-master storage schemes.
  • Implementing a backend for KiokuJS, based on some graph database. This should be a very welcome addition, as graph databases usually have query languages and hopefully some of them even will provide certain form of referential integrity.
  • Adopting the other Socket.IO transports.

6. The showcase system – ShareBoard

ShareBoard is the proof of concept showcase of Syncler. Its a simple collaborative graphical editor, which allows several users to edit the same “online white-board” scene.

Try it yourself (in web-socket enabled browser, like Chrome): http://shareboard.joose.it

Tip: After creation of the board – open several browser windows with it side-by-side, or drop the link to you friend.

7. Conclusion

We’ve presented Syncler – a distributed system, implemented as the backend for KiokuJS.

The creation of the showcase system “ShareBoard” has proved, that the demonstrated approach is very simple (so far as this word can be applied to distributed computing) and clean from the architectural point of view.

Syncler is still on very early development stage so further research directions were defined.

moveTo : function (x, y) {
this.setX(x)
this.setY(y)
}add: function (element) {
this.elementsByUUID.set(element.uuid, element)return element
},remove : function (element) {
this.elementsByUUID.remove(element.uuid)
}

Written by Nickolay Platonov

April 8th, 2011 at 2:13 pm

JooseX.CPS Tutorial – Part II

with one comment

This is the 2nd part of the JooseX.CPS tutorial. Before reading it, make sure you’ve groked the 1st part. In the 1st part, we were managing the chains of anonymous functions. Such task can quickly become cumbersome, because the functions are, well, anonymous. Lets see how we can bring this concept to the higher level – out goal is to integrate the continuation passing style right into Joose classes.

Phase 1 – Synchronous

As the show case, lets use the following simple task. You have a class – wrapper around the text file, which provides simple convenience methods, like “read”, “write” etc. The wrapper is being used by Manager, which has one method – `processFile`. This method takes the source file name, reads the file and then saves the upper-cased content of the source file to another file, appending `-res` to the filename. Skim the code below:

require('task-joose-nodejs')

var fs = require('fs')

Class('TextFile', {

    has : {
        fileName    : { required : true },

        data        : null
    },

    methods : {

        asUpperCase : function () {
            return this.data.toUpperCase()
        },

        read : function () {
            return this.data = fs.readFileSync(this.fileName, 'utf8')
        },

        write : function (text) {
            this.data = text

            this.save()
        },

        save : function () {
            this.saveAs(this.fileName)
        },

        saveAs : function (fileName) {
            fs.writeFileSync(fileName, this.data)
        }
    }
})

Class('Manager', {

    my : {

        methods : {
            processFile : function (fileName) {

                var sourceFile = new TextFile({ fileName : fileName })

                sourceFile.read()

                var targetFile = new TextFile({
                    fileName    : fileName.replace(/\.(\w\w\w)$/, '-res.$1')
                })

                targetFile.write(sourceFile.asUpperCase())
            }
        }
    }
})

try {
    Manager.processFile('source.txt')
} catch (e) {

    console.log('Exception caught: %s', e)
}

Everything operates synchronously, any exceptions can be caught with `try/catch` blocks and all is fine. Now save the code in some js file and run it in node. You should see this:

Exception caught: Error: ENOENT, No such file or directory 'source.txt'

Of course, there is no source file. Create the ‘source.txt’ in the same directory as the code file and run it again. This time everything should work and the resulting file ‘source-res.txt’ , containing the uppercased content of the source file should appear.

Now, remove the write permission on the result file and run the script again. You’ll see

Exception caught: Error: EACCES, Permission denied 'source-res.txt'

Note, how we were able to caught the both exceptions from the “processFile” method (1st one thrown from “read” and 2nd  – from “saveAs”) without any additional gymnastic.

Ok, lets re-write the code in asynchronous way, keeping the functionality the same.

Phase 2 – Asynchronous

So here’s how it will looks like (note though that we won’t catch “usual” exceptions – only those reported by NodeJS methods):

require('task-joose-nodejs')

var fs = require('fs')

Class('TextFile', {

    has : {
        fileName    : { required : true },

        data        : null
    },

    methods : {

        asUpperCase : function () {
            return this.data.toUpperCase()
        },

        read : function (callback, errback) {
            var me = this

            fs.readFile(this.fileName, 'utf8', function (err, data) {

                if (err) {
                    errback(err)
                    return
                }

                me.data = data

                callback(data)
            })
        },

        write : function (text, callback, errback) {
            this.data = text

            this.save(callback, errback)
        },

        save : function (callback, errback) {
            this.saveAs(this.fileName, callback, errback)
        },

        saveAs : function (fileName, callback, errback) {
            fs.writeFile(fileName, this.data, function (err) {
                if (err)
                    errback(err)
                else
                    callback()
            })
        }
    }
})

Class('Manager', {

    my : {

        methods : {
            processFile : function (fileName, callback, errback) {

                var sourceFile = new TextFile({ fileName : fileName })

                sourceFile.read(function () {

                    var targetFile = new TextFile({
                        fileName    : fileName.replace(/\.(\w\w\w)$/, '-res.$1')
                    })

                    targetFile.write(sourceFile.asUpperCase(), callback, errback)

                }, errback)
            }
        }
    }
})

Manager.processFile('source.txt', function () {}, function (e) {

    console.log('Exception caught: %s', e)
})

I hear you said “Piece of cake?”. How about such requirements then – “write the content to the target file, and only if the operation completed successfully, remove the source file. If there was an error during `processFile` method, append it a log file. And process the array of filenames in no more than 5 parallel threads”. If you say “thats 2 pieces of cake” on this – then my congratulations, you are a callback’s overlord :). Otherwise, read further.

CPS conversion

So lets convert the classes above to the CPS. First thing to note is – we have “synchronous” (usual) and “asynchronous” methods.

Synchronous methods returns the result immediately, with the `return` statement. In the same way they throw the exceptions immediately, with `throw`.

“Asynchronous” methods returns the result, by passing it to the callback, and they throws the exceptions by passing them to errbacks. The call to callback/errback may happen after some arbitrary delay.

Synchronous and asynchronous methods

Lets split the methods into 2 groups, by their synchronicity:

require('task-joose-nodejs')

var fs = require('fs')

Class('TextFile', {
    trait : JooseX.CPS,
    ...

    // synchronous methods goes here
    methods : {

        asUpperCase : function () {
            return this.data.toUpperCase()
        }
    },

    // asynchronous methods goes here
    continued : {

        // right here
        methods : {

            read : function (callback, errback) {
                ...
            },

            write : function (text, callback, errback) {
                ...
            },

            save : function (callback, errback) {
                ...
            },

            saveAs : function (fileName, callback, errback) {
                ...
            }
        }
    }
})

Class('Manager', {

    my : {

        trait   : JooseX.CPS,

        continued   : {

            methods : {

                processFile : function (fileName, callback, errback) {
                    ...
                }
            }
        }
    }
})

The new class builder `continued` is provided by the trait `JooseX.CPS`. It can contain the following properties: “methods/before/after/override” which has the same semantic as usual builders. All asynchronous methods (or method modifiers) goes into that section.

Magic continuation instance

Next thing to note – every asynchronous method has the “callback” and “errback” arguments. Lets clean the definition and move them to the special `this.CONT` symbol instead, which we’ll made available inside of each asynchronous method. Yes, its the continuation instance from the part 1. This magic symbol is something like `this.SUPER` we are all used to, but its not a function – its an object with own methods and properties.

myAsynchronousMethod : function (p1) {

    // this.SUPER calls the implementation of the current method
    // from the superclass
    this.SUPER

    // this.CONT refers to the current continuation instance
    this.CONT
}

Important notes: Inside of each asynchronous method there is a special `this.CONT` symbol available, which refers to the “current” instance of JooseX.CPS.Continuation. When the method starts, the continuation is empty – it doesn’t contain any tasks.

Then, when inside of any asynchronous method, you call any other asynchronous method – the call is converted into the task for current continuation (with its TRY method). For example:

// asynchronous methods section
continued : {
    methods : {

        save : function () {
            // call to another asynchronous method
            this.saveAs(this.fileName).now()
        },

        saveAs : function () {
            //saveAs definition
            ...
        }
    }
}

// the above gets translated to

// asynchronous methods section
continued : {
    methods : {

        save : function () {
            this.CONT.TRY(function () {
                //saveAs definition
                ...
            }, this, [ this.fileName ]).now()
        },

        saveAs : function () {
            ...
        }
    }
}

Note, that call to TRY contains the object, calling the method, as 2nd argument and arguments for method – as 3rd.

Asynchronous methods are prohibited to return any values with `return`. Instead they should return values using the call to `this.CONT.CONTINUE()` method (or throw the exceptions with `this.CONT.THROW()`). Or they can instead launch the nested continuation – in this case it will be responsible for returning values. In the “synchronous” meaning, asynchronous methods will return the continuation instance they are attached to.

Also, as you remember, continuation tasks won’t launch immediately – they needs to be activated with `now`.

Phase 3 – CPS

The full version of the example above will looks like:

require('task-joose-nodejs')

var fs = require('fs')

Class('TextFile', {

    trait           : JooseX.CPS,

    has : {
        fileName    : { required : true },

        data        : null
    },

    methods : {

        asUpperCase : function () {
            return this.data.toUpperCase()
        }

    },

    continued : {
        methods : {

            read : function () {
                var me          = this
                var callback    = this.CONT.getCONTINUE()
                var errback     = this.CONT.getTHROW()

                fs.readFile(this.fileName, 'utf8', function (err, data) {

                    if (err) {
                        errback(err)
                        return
                    }

                    me.data = data

                    callback(data)
                })
            },

            write : function (text) {
                this.data = text

                this.save().now()
            },

            save : function () {
                this.saveAs(this.fileName).now()
            },

            saveAs : function (fileName) {
                var callback    = this.CONT.getCONTINUE()
                var errback     = this.CONT.getTHROW()

                fs.writeFile(fileName, this.data, function (err) {
                    if (err)
                        errback(err)
                    else
                        callback()
                })
            }
        }
    }
})

Class('Manager', {

    my : {
        trait   : JooseX.CPS,

        continued : {

            methods : {
                processFile : function (fileName) {

                    var sourceFile = new TextFile({ fileName : fileName })

                    sourceFile.read().THEN(function () {

                        var targetFile = new TextFile({
                            fileName    : fileName.replace(/\.(\w\w\w)$/, '-res.$1')
                        })

                        targetFile.write(sourceFile.asUpperCase()).now()

                    }).now()
                }
            }
        }
    }
})

Manager.processFile('source.txt').CATCH(function (e) {

    console.log('Exception caught: %s', e)

}).now()

Lets examine what have changed:
1) In the methods which integrates with “raw” asynchronous methods (“read” and “saveAs”) we’ve just removed the callbacks and errbacks from arguments and get them from the current continuation. Its a small winning as the outer interface of the methods becomes a bit simpler.
2) Methods which calls only other CPS methods (“write” and “save”) are now much cleaner – they don’t contain any callbacks at all.
3) The biggest gain is in the “processFile” method (which also uses only CPS methods). It doesn’t contain any callbacks/errbacks now! And all the errors from it get caught correctly (try to launch the code w/o source file and w/o write permissions to result file).
4) We now catch the exceptions from `Manager.processFile` in much more straightforward syntax.

For consolidation, lets see how the `processFile` method will look like on the JooseX.CPS.Continuation level:

processFile : function (fileName) {

    var sourceFile = new TextFile({ fileName : fileName })

    this.CONT.TRY(function () {
        /* `read` method definition here */,
    }, sourceFile, [])

    this.CONT.THEN(function () {

        var targetFile = new TextFile({
            fileName    : fileName.replace(/\.(\w\w\w)$/, '-res.$1')
        })

        this.CONT.TRY(function () {
            /* `write` definition here */
        }, targetFile, [ sourceFile.asUpperCase() ])

        this.CONT.now()
    })

    this.CONT.now()
}

Some features

The fact that methods don’t launch immediately provides couple of interesting features.

First of all, when you call several “continued” methods in a raw and then activate the continuation – the methods will be executed sequentially. Compare asynchronous and synchronous versions:

asyncMethod : function () {
    this.anotherAsyncMethod1()
    this.anotherAsyncMethod2()
    this.anotherAsyncMethod3()
    this.anotherAsyncMethod4().now()
}

syncMethod : function () {
    this.anotherSyncMethod1()
    this.anotherSyncMethod2()
    this.anotherSyncMethod3()
    this.anotherSyncMethod4()
}

Such pattern works when methods do not depend on the returning values from the earlier methods.

Another feature is that the call to asynchronous method can be used as some sort of “promise” (though JooseX.CPS was not designed to provide this functionality). You can pass the current continuation to some other method which will add some generic step to it. Using JooseX.CPS this way wasn’t intended but its possible.

Some sugar

A JooseX.CPS trait will add a JooseX.CPS.ControlFlow role to your class, which allow you to omit the `this.CONT` when calling various methods of current continuation. So you can call ‘this.CONTINUE()` instead of `this.CONT.CONTINUE()`, `this.getCONTINUE()` etc.

Conclusion

Hopefully this post illustrates the ideas from this one. Note how the meta-layer absorbed all the low-level details and provided a much cleaner abstractions.

JooseX.CPS significantly lowers the overhead of creation and managing the asynchronous interface of your classes. No excuses to write blocking code anymore :)

P.S.

Curious how the “2 pieces of cake” example will looks like in CPS? Here it is (or here download from here) . For brevity we avoided synchronization problems when writing to log file and just output to console. Try to run the code, already having one result file, w/o write permission.

require('task-joose-nodejs')

var fs = require('fs')

Class('TextFile', {

    trait           : JooseX.CPS,

    has : {
        fileName    : { required : true },

        data        : null
    },

    methods : {

        asUpperCase : function () {
            return this.data.toUpperCase()
        }

    },

    continued : {
        methods : {

            read : function () {
                var me          = this
                var callback    = this.CONT.getCONTINUE()
                var errback     = this.CONT.getTHROW()

                fs.readFile(this.fileName, 'utf8', function (err, data) {

                    if (err) {
                        errback(err)
                        return
                    }

                    me.data = data

                    callback(data)
                })
            },

            write : function (text) {
                this.data = text

                this.save().now()
            },

            save : function () {
                this.saveAs(this.fileName).now()
            },

            saveAs : function (fileName) {
                var callback    = this.CONT.getCONTINUE()
                var errback     = this.CONT.getTHROW()

                fs.writeFile(fileName, this.data, function (err) {
                    if (err)
                        errback(err)
                    else
                        callback()
                })
            },

            remove : function () {
                var callback    = this.CONT.getCONTINUE()
                var errback     = this.CONT.getTHROW()

                fs.unlink(this.fileName, function (err) {
                    if (err)
                        errback(err)
                    else
                        callback()
                })
            }
        }
    }
})

Class('Manager', {

    my : {
        trait   : JooseX.CPS,

        continued : {

            methods : {

                log : function (text) {
                    console.log(text)

                    this.CONTINUE()
                },

                processArray : function (fileNames) {

                    Joose.A.each(fileNames, function (fileName) {

                        this.AND(function () {

                            this.processFile(fileName).except(function (e) {

                                this.log("Error processing " + fileName + ": " + e).now()

                            }).now()
                        })

                    }, this)

                    this.ANDMAX(5).now()
                },

                processFile : function (fileName) {

                    var sourceFile = new TextFile({ fileName : fileName })

                    sourceFile.read().andThen(function () {

                        var targetFile = new TextFile({
                            fileName    : fileName.replace(/\.(\w\w\w)$/, '-res.$1')
                        })

                        targetFile.write(sourceFile.asUpperCase()).andThen(function () {

                            sourceFile.remove().now()
                        })
                    })
                }
            }
            // eof methods
        }
        // eof continued
    }
})

var fileNames = [
    'source1.txt',
    'source2.txt',
    'source3.txt',
    'source4.txt',
    'source5.txt',
    'source6.txt',
    'source7.txt'
]

Manager.processArray(fileNames, "log.txt").now()

Written by Nickolay Platonov

February 22nd, 2011 at 9:13 am

Posted in Joose

Tagged with ,

JooseX.CPS Tutorial – Part I

with 2 comments

So, after the recent crack of our server, the post about the JooseX.CPS was lost. It was poorly written anyway, so may be its even better that the server was hacked – because I’m writing a new post. JooseX.CPS is quite important extension, and its being extensively used in many other Joose-based projects, like KiokuDB, Syncler and Symbie. The new post is written in form of tutorial and contains a runnable code. At the end of this post you should grok the continuation passing style :)

Setup

To follow this tutorial you will need:

The latter can be installed with `npm install task-joose-nodejs`.

Part I – Meet The Continuation

The tutorial will be split into 2 parts. In this part we’ll examine the low-level JooseX.CPS.Continuation class and its features. The second part will describe a JooseX.CPS trait itself, which provides a CPS sugar layer on top of JooseX.CPS.Continuation for you Joose classes.

Just try it, aka “Hello world”

Before we’ll dig in, lets see how the “hello world” will looks like in CPS world:

require('task-joose-nodejs')

TRY(function () {

    console.log('Hello')

    this.CONT.CONTINUE()

}).DELAY(1000).THEN(function () {

    console.log('world!')

}).now()

Try to launch this in your Node. You should see the following with 1s delay between the words:

nickolay@desktop:~/Documents/JooseBlog/CPS-Tutorial$ node hello_world.js
Hello
world!

Compare with

console.log('Hello')
console.log('world!')

You see the idea? With some additional boilerplate between them, the synchronous the statements work asynchronously, thats it. And as we’ll move forward in this tutorial, the amount of boilerplate will reduce, really.

Continuation anatomy

In the first approximation, continuation is an array of tasks, also containing the “catch” and “finally” functions:

CONT = {

    tasks           : [
        {
            func    : function doThis() { ... },
            scope   : {},
            args    : [ ... ]
        },
        {
            func    : function doThat() { ... },
            ...
        },
        ...
    ],

    catchFunc       : function () { ... },
    finallyFunc     : function () { ... },
}

The “task” is simply a function (mandatory), plus a scope for its execution and an array of arguments (both optional). Such continuation roughly corresponds to the following “synchronous” code:

try {

    doThis()
    doThat()

} catch (e) {

    cathcFunc(e)

} finally {

    finallyFunc()
}

Once more – continuation is a single “try/catch” block, containing an array of tasks. You can add tasks to the continuation with the `TRY(func, scope, args)` method. Adding tasks will not launch them immediately – you need to “activate” the continuation manually with its method `now`.

After activation, continuation will launch the first task. It will embed a new, fresh continuation instance into the scope of task, as the `this.CONT`. To pass the control flow from the 1st task to the next one, you need to call the `CONTINUE` method of the embedded continuation. Any parameters, passed to `CONTINUE` will be available as the arguments of the next task. Lets re-write the “Hello world” example, “by hands”:

require('task-joose-nodejs')

var CONT    = new JooseX.CPS.Continuation()

var scope   = { toString : function () { return 'My scope' } }

CONT.TRY(function () {
    console.log("Current scope 1: %s", this)

    console.log('Hello')

    // a new, fresh instance of JooseX.CPS.Continuation has been embedded:
    // this.CONT != CONT
    this.CONT.CONTINUE('world', '!')

}, scope)

CONT.TRY(function (str1, str2) {
    console.log("Current scope 2: %s", this)

    console.log(str1, str2)
})

CONT.now()

Some sugar: there is `THEN` synonym for the `TRY` method, which sounds more naturally for the 2nd and further tasks. Note, that if you have not provided a scope for the task, it will be taken from the previous task.

The call to `CONTINUE` doesn’t have to be done synchronously, you can delay it. To do that, capture it to the closure with the `getCONTINUE` method. Lets add the 1s pause between the outputs, “manually” (w/o the `DELAY()`):

require('task-joose-nodejs')

var CONT    = new JooseX.CPS.Continuation()

var scope   = { toString : function () { return 'My scope' } }

CONT.TRY(function () {
    console.log("Current scope 1: %s", this)

    console.log('Hello')

    // this.CONT != CONT - its a new, fresh instance of JooseX.CPS.Continuation
    // passing the parameters further
    this.CONT.CONTINUE('world', '!')

}, scope)

CONT.THEN(function (str1, str2) {

    var CONTINUE = this.CONT.getCONTINUE()

    setTimeout(function () {

        CONTINUE(str1, str2)

    }, 1000)
})

CONT.THEN(function (str1, str2) {
    console.log("Current scope 2: %s", this)

    console.log(str1, str2)
})

CONT.now()

Hey don’t scan, read the sources :)

Instead of manual instantiation of JooseX.CPS you can use global `TRY` helper which just creates a new instance and delegates to its `TRY` method. And of course, the `TRY/THEN` method return the continuation itself, so the calls can be chained. There is also a `DELAY` method and so we comes to the initial “Hello world”:

require('task-joose-nodejs')

TRY(function () {

    console.log('Hello')

    this.CONT.CONTINUE()

}).DELAY(1000).THEN(function () {

    console.log('world!')

}).now()

Nesting

Instead of calling the `CONTINUE` of the inner continuation, you can add a nested continuation – using `TRY` as usual. Keep in mind, that you need to activate it with `now`. When the last task of the nested continuation will call `CONTINUE` the control flow will continue on the outer level:

require('task-joose-nodejs')

TRY(function () {

    console.log('Hello')

    this.CONT.DELAY(500).THEN(function () {

        console.log(', ')

        this.CONT.CONTINUE()

    }).DELAY(500).THEN(function () {

        this.CONT.CONTINUE('world')

    }).now()

}).THEN(function (str) {

    console.log(str)

}).now()

Pretty intuitive, don’t you think? Important note: once you’ve nested the continuation (instead of calling the `CONTINUE`) you must not call its `CONTINUE` method to avoid the split of the control flow. From this point, its the nested continuation responsibility to return the control flow.

There is also a `RETURN` method to skip the other tasks from the current nesting level and return the control flow to the outer.

Exceptions

We talked about the correspondence to the `try/catch/finally` block, and here’s their equivalent in the CPS. Surprisingly, they are named `TRY/CATCH/FINALLY` :) . Simplest case:

require('task-joose-nodejs')

TRY(function () {

    throw "oops"

}).CATCH(function (e) {

    console.log("Exception caught: %s", e)

    this.CONT.CONTINUE()

}).FINALLY(function () {

    console.log("Cleanup happens")

}).now()

More text in bold: The `CATCH/FINALLY` tasks are also “continued”. That is, to return the control flow from them you need to use `this.CONT.CONTINUE()` as in `TRY/THEN`. This will allow you to write asynchronous exceptions handlers.

Nesting and Exceptions

The exceptions will correctly propagate from the nested continuations:

require('task-joose-nodejs')

TRY(function () {

    console.log('Hello')

    this.CONT.DELAY(500).THEN(function () {

        console.log(',')

        throw "world"

    }).DELAY(500).THEN(function () {

        console.log("YOU SHOULD N'T SEE THIS TEXT")

    }).now()

}).CATCH(function (str) {

    console.log(str)

}).now()

they could be re-thrown just fine:

require('task-joose-nodejs')

TRY(function () {

    console.log('Hello')

    this.CONT.DELAY(500).THEN(function () {

        console.log(',')

        throw "f$$k"

    }).CATCH(function (str) {

        throw "polite"

    }).now()

}).CATCH(function (str) {

    console.log(str, "world")

}).now()

And in general, behaves very much like “usual” exceptions.

Show me the parallel

Of course any library for asynchronous control flow should provide the “parallel” mode. In JooseX.CPS you can activate it with `AND`:

require('task-joose-nodejs')

TRY(function () {

    console.log('Activated branch 1')

    var CONTINUE = this.CONT.getCONTINUE()

    setTimeout(function () {

        console.log('world')

        CONTINUE('!')

    }, 1000)

}).AND(function () {

    console.log('Activated branch 2')

    var CONTINUE = this.CONT.getCONTINUE()

    setTimeout(function () {

        console.log('Hello')

        CONTINUE('!')

    }, 500)

}).THEN(function (res1, res2) {

    console.log(res1[0], res2[0])

}).now()

If you’ll run this code, you’ll  see that branches were activated simultaneously, but the 2 final exclamation characters appears only after the branches gets merged. The very 1st task following the `AND` group receives the `arguments` objects passed to `CONTINUE` in the order of branches declaration.

Of course, you can nest a new continuation into any of the branches:

require('task-joose-nodejs')

TRY(function () {

    console.log('Activated branch 1')

    this.CONT.CONTINUE('Branch 1 result')

}).AND(function () {

    console.log('Activated branch 2')

    this.CONT.DELAY(500).THEN(function () {

        console.log('The very long')

        this.CONT.CONTINUE()

    }).DELAY(500).THEN(function () {

        console.log('asynchronous task')

        this.CONT.CONTINUE()

    }).DELAY(500).THEN(function () {

        console.log('from several steps')

        this.CONT.CONTINUE('Branch 2 result')

    }).now()

}).THEN(function (res1, res2) {

    console.log(res1[0], res2[0])

}).now()

Some sugar

JooseX.CPS.Continuation also has lower-cased synonyms for the most used methods:

  • THEN – then
  • AND – and
  • CATCH – except
  • FINALLY – ensure

Also, the `this.CONT.THEN( … ).now()` can be written as `this.CONT.andThen( … )`

These synonyms simplifies a code a little bit and makes it more readable, but it still contain a lot of boilerplate. This boilerplate can be greatly reduced when using JooseX.CPS trait, which adds the “continued” methods to your class.

It will be described in the part 2 of the tutorial, stay tuned!

Written by Nickolay Platonov

February 14th, 2011 at 8:17 am

Posted in Joose,JooseX

Tagged with ,

Dist::Zilla – the distributions king!

without comments

In this post I’d like to present the building tools I’m using for my projects. Hopefully after it there will be more contributions :)

I’m using a plugin for Dist::Zilla. Dist::Zilla comes from the severe perl world, where people with strong spirit writes a code which is 10 years backward-compatible (thanks to chromatic thats gradually changing).

Generally, Dist::Zilla is purposed to assist the CPAN authors, but it turned out, that many concepts are language and distribution platform neutral. For example integration with Git – in any language or any distribution platform you definitely want to tag the new release with version number. Then you probably want  to commit the updates. Another example will be `Changes` file – before release it definitely makes sense to check whether it contains the entries for new version. Etc.

All such routine tasks can be automated with the Dist::Zilla. It can create a barebone distribution, generate an appropriate LICENCE file, test the distribution, release it on `npm` and much more!

Setup

Installation of the perl modules is usually scary for people without perl background. So you are welcome to the #joose channel on freenode with any questions. There are also known problems on MacOS, so Mac owners are even more welcome :) Note, that you need to have ‘git’ installed.

I found it much easier to use the recently appeared Miyagawa’s “cpan-minus” tool. So, to install the cpanminus:

> curl -L http://cpanmin.us | perl - --sudo App::cpanminus

then

> cpanm Dist::Zilla::Plugin::JSAN --sudo

After installation, setup the Dist::Zilla in interactive mode (this is required only once):

> dzil setup
What's your name?...

When asked about the default license, here’s the list of available identificators (at the bottom and strip the leading “Software::License::”). The information you provided will be used during creation of new distributions.

Starting a distribution

To start a new distribution:

> dzil new -P JSAN -p joose Sample-Dist

[DZ] making target dir /home/nickolay/Playground/dzil/Sample-Dist
[DZ] writing files to /home/nickolay/Playground/dzil/Sample-Dist
[Git::Init] Initializing a new git repository in /home/nickolay/Playground/dzil/Sample-Dist
[DZ] dist minted in ./Sample-Dist>

This command should create a barebone distribution like this. It will also create a git repository and perform a famous “initial commit”! :) See this page to know how to customize the template.

And thats it! Distribution will contain a single main module with some documentation. After some hacking, you can easily publish the distribution in `npm` (see below).

The distribution will also contain a rudimentary test suite, which you can run in NodeJS (assuming you have test-run installed) with:

> node t/index.js

or, if you’ve completed the 3.1 and 3.2 steps from this document, in browsers. Put the distribution in some web-directory and point the browser to the URL like:

http://localhost/my_workspace/Sample-Dist/t/index.htnml

dist.ini

The whole building process is managed by the `dist.ini` file in the root of distribution. Its a text file which lists the used plugins and their parameters. For example:

; Auto-increment the version number, based on the tags information from git
[Git::NextVersion]
first_version   = 0.0.1

; include the link to git repo and web page (if "origin" points to github)
[GithubMeta]

For the details on the individual plugin, prepend its name with “Dist::Zilla::Plugin” and perform a CPAN search.

See the documentation for the full details on the JSAN plugin, here I’ll briefly describe some of the typical tasks.

Building the distribution

‘build’ command takes your sources as the input, process them (add/remove/change some files) and generate the output (location can be specified with ‘in’ option):

> dzil build [--in /target/dir]

Various plugins participate in this phase. Typical example will be [License] plugin, which adds a LICENSE file with the text of your license.

Releasing the distribution

‘release’ command first build the distribution, then runs plugins which implements the “release” phase. There is a [JSAN::NPM::Publish] plugin, which can publish your dist in npm:

> dzil release

Use case

For example, I’d like to include the version information in the source files, like:

Class('Some.Class', {
    VERSION    : 0.01

    has : { ... },
    methods : { ... }
})

Then before each release, you need to manually update the version information. That quickly becomes PITA. Thankfully, this task can easily automated, especially that Dist::Zilla already can increment the version, based on the information from git tags. So, if we’ll include a version placeholder in the sources:

Class('Some.Class', {
    /*VERSION,*/

    has : { ... },
    methods : { ... }
})

and the plugin to the “dist.ini”, which know how to find that placeholder and replace it with actual version:

[JSAN::PkgVersion]

then during each build dzil will perform all the routine work.

Conclusion

Setting up a proper quality control for your release process is very important. Currently I’m maintaining  ~30 distributions. Under “maintaining” I don’t mean I have that much github repos. I mean all of them have proper versions, tags, docs, test suites, changelogs etc. All of them are released on `npm`. I just can’t imagine how I would do that without Dist::Zilla.

See also

[JSAN] plugin docs and http://dzil.org/

Written by Nickolay Platonov

January 13th, 2011 at 5:09 pm

Posted in Joose

Migration to `npm`

without comments

As you may know, for some time, Joose was being distributed both via OpenJSAN and NPM repositories. However many people were experiencing various issues, when installing and using the JSAN shell. Thats why Joose itself and various Joose extensions have migrated to distribution solely via “npm”. See the http://nodul.es/modules/joose for the list of modules, currently depending on Joose.

Documentation is now hosted on the github: http://bit.ly/joose_manual

If you are planning to use Joose cross-platform (share code between browsers and NodeJS) then you may want to complete the 3.1, 3.2 items from the http://joose.github.com/Joose/doc/html/Joose/Manual/Installation.html

These additional steps are required, since the NPM is based on the synchronous commonjs modules. It creates redirect files which contains the following synchronous assignment:

module.exports = require(from)

I’m not sure, whether its possible to redesign the NPM and whole commonjs codebase to use the asynchronous loading (its still a SSJS, not CommonJS), but synchronous loading is not acceptable for client-side.

The additional setup is completely optional however.

Written by Nickolay Platonov

January 13th, 2011 at 5:09 pm

Posted in Joose

Tagged with ,

Joose 3 internals – Part II. Meta-model

without comments

This post will be very technical and contain a gory details about the implementation of the meta-model in Joose. It will be useful only for extension authors, if you are not planning to write one – nothing interesting ahead.

Well, you were warned.

Everything is a Property.

Property is the basement of the meta-model. It is an abstract name/value pair.

Everything in Joose is a property, i.e. attribute is a property, method is a property, role’s requirement is a property as well.

As it’s been mentioned in the previous post, property is implemented as a class from the 1st meta-level: Joose.Managed.Property (it has a Joose.Proto.Class as the meta-class).

Property can be shared among several classes (via composition relationships) so its totally self-contained and don’t contain any links to its “current” container, only to container it was defined in.

Property’s life stages.

By itself, property is just a description. To receive a concrete implementation (one can say “materialize” property) it should be applied to target. It can be un-applied as well (remember the mutability feature). In general, property has the following life stages: `preApply/apply` and `unApply/postUnApply`.  More on this below.

Everything is a PropertySet.

Joose.Managed.PropertySet inherits from Joose.Managed.Property. It is an abstract, unordered collection of properties, indexed by name:

On the set of property sets (sorry for tautology) we’ll define the following operations:

  • A.cleanClone – create a “clean” clone of A (without any properties)
  • A.clone – create a clone of A, including properties (only properties container will be cloned, not properties itself)
  • A.alias(what) – add aliases for properties, listed in what
  • A.exclude(what) – delete properties, listed in what
  • A.flattenTo(B) – for each property in A,
    • check if the property with the same name in B exists
      • if there is one, and its a special conflict marker property or the property of A itself, then just skip it
      • if there is one and its different from the above – then replace it in B with the conflict marker
      • otherwise, copy that property to B
  • A.composeTo(B) – for each property a in A,
    • check if B has own property with the same name
    • if not – then copy that property to B
  • A.composeFrom(B, C, …)
    • create a clean clone of A
    • flatten each argument into the clean clone (possible aliasing & excluding some properties first)
    • compose the result to A

Listed operations will implement the traits spec. All subclasses of PropertySet will keep the semantic of these operations.

PropertySet’s life stages.

PropertySet simply propagate the `preApply/apply/unApply/postUnApply` stages to its elements. Note, that current implementation is still written in the imperative spirit and rewriting it in more functional way will possible allow us to remove the `unApply/postUnApply` stages at all.

Mutable PropertySet.

Joose.Managed.PropertySet.Mutable inherits from Joose.Managed.PropertySet. Its a bit more specialized property set. It track the depended sets (derivatives) and propagate the changes through them. Mutable set B said to be the derivative of A, if it has been composed from A (possibly along with other sets).

Mutable PropertySet can be in 2 states: opened (this.opened > 0) & closed (this.opened == 0). Mutation is only allowed when property set is opened.

During “open” operation, when switching from closed state to opened the set is de-composed – cleaned from all properties which weren’t defined in this set. Before that all derivatives are also opened. Note, that the set can be opened several times – subsequent “open”s will be no-ops. Each “close” operation should have matching “open”. Initially the set is opened (this.opened == 1)

During “close” operation, when switching from opened state to closed, the set is re-composed using the data from “composedFrom” attribute and inherited “composeFrom” method.

Higher-order PropertySet.

Joose.Managed.PropertySet.Composition inherits from Joose.Managed.PropertySet.Mutable. Its an abstract higher-order mutable property set – a property set, which has other property sets as the elements:

Its no longer unordered and defines the order of properties processing (`processOrder`), also defines the reverse processing order, which is important for unapplying things.

Composition’s behavior.

Composition translates the higher-level operations to individual properties. That is, when composition A is asked to be flattened to composition B for example, it flat it’s individual properties:

Other operations are defined in the same way. Note, that composition is a mutable property set.

Stem

Joose.Managed.Stem inherits from Joose.Managed.PropertySet.Composition. Its a concrete implementation of composition:

Stem contains the following properties (in the processing order): [ 'attributes', 'methods', 'requirements', 'methodsModifiers' ]. Each of those properties is a property set of individual properties – collection of attributes, methods, etc.

Stem is the first property, which is aware of the real JavaScript class it belongs to (targetMeta attribute).

During re-composition stem initiates the `preApply/apply` actions of the whole properties subtree.  In the same way, during de-composition, it initiates the `unApply/postUnApply` stages.

Note, that during `preApply` stage the stem is still opened, and can mutate. During `apply` stage, the class definition is “final” and should only be somehow “materialized” in the prototype of the class, but not changed. The class in which the property should “materialize” itself will be passed as the 1st argument for “apply” method.

Conclusion

This post contains the details about the meta-model implementation in Joose3.  In the next post of this series we’ll briefly describe the anatomy of class from the 2nd meta-level – Joose.Managed.Class.

Written by Nickolay Platonov

January 13th, 2011 at 5:08 pm

Posted in Joose

Tagged with ,

PromoteJS – Community matter

without comments

Recently I’ve been attending the JSConf.eu (many thanks to all organizers for the great event), where Chris Williams (aka @voodootikigod) delivered the very inspiring speech about the JavaScript community.

JavaScript is maturing as a language, as a platform and community should follow.

If everyone who use JavaScript will contribute something back to the community, the world will be much better place, at least for the JavaScript programmers. That don’t has to the code per se, if you’ll just answer someone’s question in IRC or update a wiki page or will help to improve the visibility of the documentation in the search results:

JS Operator Precendence

your karma will improve as well :)

Promote JS!

Written by Nickolay Platonov

January 13th, 2011 at 5:08 pm

Posted in Joose

Introducing KiokuJS

with 3 comments

What

KiokuJS is a persistence layer for Joose. Its a freestyle port of KiokuDB which does the same thing in the Moose world.

KiokuJS provides transparent storing/retrieving for instances of Joose classes. Native data structures (Object/Array/etc) can be stored as well. “Transparent” means that class can be stored without any modifications to its declaration (see also But).

KiokuJS is cross-platform – it runs equally well in browsers and NodeJS. KiokuJS is non-blocking – all interactions with DB are asynchronous.

Why

Remember the last Web2.0 project you worked on. Quite probably it was looking as:

Note, that you need to define the serialization procedure 4(!) times. Even if you use some kind of client-side helpers of modern frameworks, you still need to (manually) map your client-side JavaScript data structures to the storage layer.

Meanwhile, nowadays its obvious, that JavaScript has evolved from client-side animation libraries into full-stack platform. There are no reasons why your applications can’t be fully written in JavaScript and benefit from the homogeneous environment both on client and server.

So the goals of KiokuJS are:

  • eliminate all the serialization boilerplate from the application
  • use a single model definition both on client and server
  • encapsulate all the storage-layer details in high-level interface

Example

// class declaration

Class('Person', {

    has : {
        self    : null,

        name    : null,

        spouse  : null
    },

    methods : {

        initialize : function () {
            this.self = this // circular ref
        }
    }
})

// arbitrary data structure

var Homer = new Person({
    name    : "Homer"
})

var Marge = new Person({
    name    : "Marge"
})

Homer.spouse = Marge
Marge.spouse = Homer

// handler setup

var handle = new KiokuJS.Backend.CouchDB({
    dbURL   : 'http://local:5984/someDB'
})

// storing

var scope = handle.newScope()

scope.store(Homer, Marge).andThen(function (homerID, margeID) {
    ...
})

// retrieving

var scope = handle.newScope()

scope.lookUp(homerID, margeID).andThen(function (Homer2, Marge2) {

    // Homer2.self   === Homer2
    // Homer2.spouse  === Marge2

    // Marge2.self   === Marge2
    // Marge2.spouse  === Homer2
})

How

From the KiokuJS viewpoint, the data structures represents a cyclic, directed graph:Data graph is then serialized into JSON structures like:

{
    "className": "Person",
    "ID": "2D9E24EE-89BA-A361-B0E6-2D8A6C1226F6",
    "isRoot": true,
    "data": {
        "self": {
            "$ref": "2D9E24EE-89BA-A361-B0E6-2D8A6C1226F6"
        },
        "name": "Homer",
        "spouse": {
            "$ref": "0CFB5A77-8B58-BC7F-9B7D-DBDD78A29AC2"
        }
    },
    "$entry": true
}

{
    "className": "Person",
    "ID": "0CFB5A77-8B58-BC7F-9B7D-DBDD78A29AC2",
    "isRoot": true,
    "data": {
        "self": {
            "$ref": "0CFB5A77-8B58-BC7F-9B7D-DBDD78A29AC2"
        },
        "name": "Marge",
        "spouse": {
            "$ref": "2D9E24EE-89BA-A361-B0E6-2D8A6C1226F6"
        }
    },
    "$entry": true
}

There are a number of options, defining the serialization details (implemented as attribute traits). When required, its also possible to provide a custom serialization format for specific class.

Its worth to note, that all asynchronous interfaces in Kioku are implemented using the continuation-passing style extension for Joose, which provides a precise control over the asynchronous methods. The previous storing example can be elaborated like this:

// storing

UI.maskScreen("Please wait")

scope.store(Homer, Marge).then(function (homerID, margeID) {

    alert('Objects stored successfully')

    this.CONTINUE()

}).CATCH(function (e) {

    alert('There were an exception [' + e + '] during storing operation')

    this.CONTINUE()

}).FINALLY(function () {

    UI.unMaskScreen()

}).now()

But

Persistence is mostly transparent, but there are some limitations. Generally any closures-related magic won’t be recognized, simply because there are no according introspection capabilities in the language. For example, if you have a function, tied to a scope with some closures, as the value of the attribute:

var closure = { foo : 'bar' }

homer.magic = function () { return closure.foo }

then KiokuJS will store the function declaration only, not the scope it was defined in. So after retrieving the instance from the backend and running “homer.magic()” there will be an exception like ["closure" is not defined].

Another limitation is related to searching. Searching is constrained to backends capabilities only.

Backends

Kioku is backend-agnostic, but currently mostly target NoSQL (key/value) backends because of their simplicity, native JSON support and HTTP APIs, which allows to directly access the DB from the browsers.  With some additional efforts can be implemented a relational backend, either static (when classes definitions will be generated from the relational schema) or dynamic (when the relational schema will be adjusted to the classes definition).

For the development and testing there is a “Hash” backend. For the production – CouchDB backend.

Test suite of Kioku is organized that way, that makes it possible to run for each backend.

There is also a special backend (actually a role for backend) – KiokuJS.Backend.Batch, intended for usage in browsers. It allows to combine the individual requests for separate documents into batches, saving a lot of HTTP requests.

Writing new backends is easy – they only need to define 4 methods.

Status

KiokuJS is at 0.01 version. This means its “stable enough”, has the extensive test suite, but is mostly intended for early adopters, which are welcome to Joose IRC channel with any questions. Currently there is no documentation.

To install Kioku, run

> npm install kiokujs

You may also want to install CouchDB backend with:

> npm install kiokujs-backend-couchdb

After that this mini demo app should just work from NodeJS, supposing you have correct value for NODE_PATH environment variable (see items 3.x in Joose installation).

Roadmap

  • Add more backends – Riak and MongoDB are 1st candidacies.
  • Try to implement backends for graph databases (very promising).
  • Syncler (draft name) – real-time backend, which use the optimistic replication to synchronize the state of replicas between clients.

Written by Nickolay Platonov

January 13th, 2011 at 5:07 pm

Posted in Joose

Tagged with

Joose 3 internals – Part I. Meta-layers

with 3 comments

This post starts a series, describing the meta-model of the Joose3. It mostly purposed for people brave (or crazy) enough to hack on the internals, though casual users may find it also useful for deeper understanding of the underlaying processes in Joose.

Classes and meta-classes.

Ok, lets start with the definition what is the class. Its the abstract template or pattern, describing some behavior (for example: “Dog”). This template can be “instantiated” (usually arbitrary number of times) – that means the abstract definition receive a concrete implementation in some object (for example: “Pluto”). Such object will be called “an instance” of the class:

On the figure above and the following ones, the filled circles denotes classes, unfilled circles – instances.

Then, what is the meta-class? Its the class, whose instances, in turn, represents classes:

The meta-class defines the higher-level behavior of the class. For example it may define, that class can be only instantiated once. Or, that class may have special attributes with validation. Or, special methods with type-checking, etc.

Ok. You’ve probably already spotted the smell of recursion in the meta-class definition (its the class, whose instances are classes). Naturally, the chicken & egg problem arise – what is the meta-class of the meta-class itself? And what is the meta-class of that meta-meta-class?

Joose.Proto.Class – 1st meta-layer

The chicken & egg problem is being solved in Joose 3 by the 1st meta-layer, which is represented by the Joose.Proto.Class meta-class. Joose.Proto.Class is the class, which is the meta-class for itself! This makes its meta-instance perfectly circular – it refer to itself.

Classes with Joose.Proto.Class meta are very close to “raw” JavaScript and other simple class systems. Virtually everything from the class definition goes directly to the class prototype (except the “isa” builder). Inheritance is done with the “classical” prototype chain hack (see Joose.O.getMutableCopy in Joose.js)

If you really need “raw” JavaScript code (for animation library for example), you can host your class at this meta-level like this:

Class('Raw.Code', {
    meta    : Joose.Proto.Class,

    attr1   : 'init',
    attr2   : null,

    method  : function () { return 'result' }
})

var i = new Raw.Code()

i.method() == 'result' // true

At this meta-level there are no roles, traits, lazy attributes, method modifiers or any other magic, just JS. The other parts of the meta-model are written as classes from this meta-level. The next post will cover this topic in detail, in the meantime we’ll continue the overview of the other meta-layers.

Joose.Managed.Class – 2nd meta-layer

The next meta-layer is presented with Joose.Managed.Class. Joose.Managed.Class is a subclass of Joose.Proto.Class, and has it as a meta-class in the same time!


Important feature is that its safe to inherit from class, which has meta from the the previous layer:
This feature makes the bootstrapping possible.

Classes with Joose.Managed.Class as meta-class are already very capable. They may have method modifiers, roles or traits applied. At this level also appears the Joose.Managed.Role – a special meta-class, which, among other things, prevents class from instantiation.

However, we need one more bootstrapping step. The reason is that, by itself, Joose.Managed.Class has a Joose.Proto.Class as meta. Thus its behavior is very limited, you can’t apply role to Joose.Managed.Class for example. We definitely want our meta-model to have a more capable central meta-class. Another reason is the attributes. We need to be able to apply traits to the attributes. So the default meta-class for the attribute should be at least at “managed” level. But this level just been introduced – will be more elegant to delegate the change of the default attribute’s class to next layer.

Joose.Meta.Class – 3rd meta-layer

This is very simple meta-layer, required mostly for bootstraping purposes: Joose.Meta.Class.


There is also a corresponding Joose.Meta.Role meta-class.

Joose.Meta.Class is the central meta-class of the meta-model. If you need to modify the behavior of all classes, you can apply roles to it (comes in use for JooseX.Meta.Lazy for example). See also next section for explanations hows that possible.

Mutability

All classes in Joose, on all meta-levels are mutable. “Mutable” means that class/role definition can be changed at any time (add/remove attribute or method for example) and the change will be propagated to instances and other depended classes/roles.

Mutability is based on the fact, that during inheritance, prototypes of Joose classes forms the usual prototype chain. Thats why no special actions required when a class changes – the language itself will handle the propagation to sub-classes. A simple example:

var a = {

    method1 : function () { return 'a1' },
    method2 : function () { return 'a2' }
}

var b = Joose.O.getMutableCopy(a) // establish a prototype chain

b.method2 = function () { return 'b2' } // override a method

b.method1() == 'a1' // true
b.method2() == 'b2' // true

// mutation

delete a.method1
delete b.method2

b.method1 = function () { return 'b1' }
a.method3 = function () { return 'a3' } // add a new method to the base object

// eof mutation

b.method1() == 'b1' // true
b.method2() == 'a2' // true
b.method3() == 'a3' // true

Slightly more complex logic is required for roles mutations, but Joose never update the whole graph of depended classes, the most of work is delegated to JavaScript and  number of “manually” updated classes is as minimal as possible.

Conclusion

This post contains a high-level overview of the Joose meta-model. In the next post we’ll focus on the details of 2nd meta-level, which implements the traits specification.

Written by Nickolay Platonov

January 13th, 2011 at 9:02 am

Posted in Joose

Tagged with ,

Why Joose? Part #2 (phylosophical)

with 3 comments

If the the previous (practical) answer on the question “Why Joose?” doesn’t sounds convincing for you, here is the philosophical (aka meta-physical) variant. Yes, we like the “meta” word :)

Abstraction layers

Lets start with the following – why we all are programming anyway? Obviously to solve some real-world tasks. And we solve them, by translating the behavior of real-world systems into machine code. Such translation can’t be direct yet, as the computers are presently “dumb” and it has to be performed in the layered fashion.

That is, the first, outermost abstraction layer is the “user story”, written in the natural human language by the end-users.

The further level will probably be a technical specification, written by software analyst, or (in the agile methodology) a set of tests, representing the data-domain knowledge in the programmer’s head.

Yet another level will be the actual program, most probably written in some high-level language like JavaScript, Perl or Java (we treat everything except the assembler as high-level language here).

There will be also a byte-code layer, etc, the chain ends at the actual bits dancing on the silicon chips.

Complexity

Ok, now lets take a look from another (quite abstract) side. That real world task can be characterized by its “complexity”. This term is somewhat close to the “entropy”, may be its even a synonym.

We can say that each abstraction layer, we’ve used during translation, absorbs some part of that complexity. Like the sponge.

To solve the task, it’s whole complexity should be absorbed. You can’t leave some of it un-absorbed, as that will just means that some aspects of the system weren’t addressed.

So, the whole point is that, if you’ll be used “poorly absorbing” layers as tools, you’ll have to absorb the remaining complexity somewhere else, either in the code or in your head :)

Examples

Imagine you need to analyze some text file and extract repeating patterns from it, using C/C++ string manipulation capabilities. Compare with the same task for Perl/JavaScript style regular expressions.

Imagine you need to write the image recognition program in assembler.

Imagine you need to refactor the complex system and you don’t have the full and complete test suite (outer abstraction layer). Compare with the same task, when you have it. This is an example of poorly absorbed complexity.

Compare the implementations of quicksort algorithm in C and Haskell: http://haskell.org/haskellwiki/Introduction This is an example of layers with different absorbing capabilities.

Joose

So what’s special about the Joose and complexity? Its that Joose has meta-layer, which allows you to modify the behavior of your code (not the system being modeled).

And, modifying the behavior of the code, you can absorb any boilerplate you usually had to write. Moreover, you don’t just reduce the number of lines, you free your mind from that boilerplate. For example, consider this common pattern:

getSomething : function () {
    if (typeof this.something == 'undefined') {
        this.something == //something calculation
    }
    return this.something
}

How you’ll read this code? “If we have that `something` undefined, then we calculate it, and store, we always return `something`” Lots of words isn’t it? And the nature of `something` is still unclear..

In Joose land, you can just say that `something` is a lazy attribute:

has : {
    something : {
        is      : 'rw',
        lazy    : function () {
            return //something calculation
        }
    }
}

As you can see, the complexity of that pattern was absorbed by Joose and don’t pollute the code and programmer’s mind.

Conclusion

Enough of vacant advocacy. In the next post we’ll demonstrate, what all these words means in practice, and how Joose makes your code simpler and more robust.

Stay tuned!

P.S.

Oh, and we forgot to answer on one very important meta-physical question – “Does Joose works with NodeJS?” It does: http://samuraijack.github.com/Task-Joose-NodeJS/

:D

Written by Nickolay Platonov

January 13th, 2011 at 8:19 am

Posted in Joose

Tagged with ,