Joose blog

Blog about Joose – advanced class system for JavaScript

Archive for the ‘Joose’ tag

Syncler – distributed applications for human beings

with 9 comments

1. Intro

Distributed systems are hard by definition. Thats why its important to not mix the complexity of distributivity with the complexity of the data domain and keep them separated.

In this post we’ll present Syncler – a distributed, optimistic replication backend for KiokuJS. Syncler will keep several replicas of arbitrary data synchronized, in real-time.

Syncler is unobtrusive and does not require from developer to understand all the mechanics of replication.  Replica is simply a set of Joose classes and/or native JavaScript data structures. Application uses accessors to mutate the attributes of classes, and Syncler takes care about all replication details. To be informed about updates in own replica, application can subscribe to various events using the observable pattern.

Syncler has been designed after Bayou (described in this paper). Using the classification of optimistic replication systems from this paper, Syncler can be characterized as the system with:

  • operations transfers and semantic conflicts resolution
  • many masters (though a single master is being used for updates stabilization)
  • real-time updates propagation schedule
  • a star topology for updates propagation
  • eventual consistency (with very low replicas divergence period due to real-time updates)

To understand the advantages and drawbacks of these design choices please refer to the mentioned papers and other papers on the same topic. Most of the design choices were inherited from Bayou, however Bayou is more focused on off-line usage and Syncler – on real-time synchronization.

In addition to that, Syncler is cross-platform (runs equally well in browsers and NodeJS) and non-blocking.

In fact, Syncler is not a backend per se, but a special role, which can turn any KiokuJS backend into distributed system. Despite that at the moment there’s only one backend (for CouchDB), writing new ones is trivial (backend for Riak looks very promising).

As the transport, Syncler uses Socket.IO. Due to relative instability of most of the Socket.IO transports, at this moment, Syncler work correctly only with websockets.

2. Optimistic replication

Optimistic replication is a special type of replication, in which all masters apply own updates (and “see” the results) immediately. It can be said that every master is optimistic about that his update won’t cause any conflict.

Naturally, because the updates are applied without coordination among all replicas, the state of replicas may slightly diverge in the course of time, but the final state of all replicas is guaranteed to be the same (so called “eventual” or “weak” consistency).

Optimistic replication creates an illusion of zero network latency for own updates and greatly improves the responsiveness of the UI.

3. Applications

The application areas for Syncler is quite wide – it can be used as the basement of any kind of collaborative environment, where is a need to synchronize the operations,  originating from multiple users. Syncler is currently not bound to some specific data domain area.

3.1 Cooperative editing systems.

Probably the most obvious and well-known application area is so called “group-ware”. Typically its the graphical editor which allows several users to simultaneously edit some graphical scene, consisting from various graphical primitives. As the showcase of Syncler, we’ve developed such drawing application “ShareBoard”  (see section 6).

But naturally, the application doesn’t have to focus on graphic edition and can be used in any data domain.

3.2 Online multi-player games

Another very perspective area is the multi-player games, as games can be thought as some special kind of collaboration. This area is gaining interest along with modern browsers gaining visualization capabilities.

Syncler can greatly reduce the “entry barrier” for the creation of a new game as it handles a lower-level question of game scene synchronization, allowing the developers to focus on the gameplay process itself.

4. Architecture and realization

In this section we’ll shortly cover the architecture of the application, based on Syncler along with typical implementation patterns. Everything is still preliminary and subject for refactoring. We’ll use the code fragments from the ShareBoard system.

4.1 Replica

Replica is simply a set of Joose classes, having a `Syncler.Object` role. Currently replica also need to have a special “topic” class, with `Syncler.Topic` role (this requirement will be removed).

For example the board scene is modeled with the following class:

Class('ShareBoard.Model.Board', {

    does            : Syncler.Topic.UUID,

    has : {
        createdBy       : null,
        createdAt       : Joose.I.Now,

        elementsByUUID  : Syncler.I.Object,

        ...
    },

    methods : {

        add: function (element) {
            this.elementsByUUID.set(element.uuid, element)

            return element
        },

        remove : function (element) {
            this.elementsByUUID.remove(element.uuid)
        },

        each : function (func, scope) {
            return this.elementsByUUID.each(func, scope || this)
        },

        ...
    }
})

As you can see, board is generally a collection of board elements. And a base class for board element looks like:

Class('ShareBoard.Model.Board.Element', {

    does         : [
        'Syncler.Object',
        'KiokuJS.Feature.Class.OwnUUID'
    ],

    has : {
        board           : { required : true },

        x               : { required : true },
        y               : { required : true },

        color           : 'blue',

        ....
    },

    methods : {

        getBubbleTarget : function () {
            return this.board
        },

        remove : function () {
            this.board.remove(this)
        },

        moveTo : function (x, y) {
            this.setX(x)
            this.setY(y)
        }
    }
})

There are several subclasses of `Board.Element`, which add additional presentation logic.

4.2 Updates

To update the replica, application just calls the accessors methods. For example to move the board element (graphical primitive) to another position one can call the `moveTo` method, which looks as simple as:

    moveTo : function (x, y) {
        this.setX(x)
        this.setY(y)
    }

The wrappers for native JS data structures (which are required in the absence of proxies) provides obvious accessors for typical opeartions, for example to add/remove the board element to/from board, one can call `add/remove` method,
which updates the key in the underlaying JS Object.

    add: function (element) {
        this.elementsByUUID.set(element.uuid, element)

        return element
    },

    remove : function (element) {
        this.elementsByUUID.remove(element.uuid)
    }

The updates gets applied immediately and will propagate to other replicas in the background. As you can see, all the replication logic is packed into the meta-layer, the application is free from it.

Syncler assumes that any update can be rolled back (thats one of the design decisions of Bayou), thus all updates captures the “precondition” information which is being used for consistency checks.

4.3 Observable

To know about updates in replica, application should subscribe to various mutation events, which will be emitted by the meta-class. For example, to listen to mutation of any attribute in the underlying `model` instance, the base widget class can do the following:

Class('ShareBoard.View.Board.Element', {

    has : {
        boardWidget     : { is : 'rw', required : true },
        model           : { is : 'rw', required : true },

        ...
    },

    methods : {

        initialize : function () {
            this.model.on(
                '/mutation/apply/Syncler.Mutation.Class.Attribute',
                this.onAttributeMutate,
                this
            )
        },

        onAttributeMutate : function (event, mutation) {
            var attributeName       = mutation.attributeName

            if (attributeName == 'status')
                this.updateStyle()
            else
                this.updatePosition()
        },
        ...
    }
})

Another example – the board widget listen to `onNewInstanceOf` event of the replica itself and add widgets as new model elements appears:

        replica.onNewInstanceOf(
            'ShareBoard.Model.Board.Element',
            this.onNewBoardElement,
            this
        )

        onNewBoardElement : function (event, element, e) {
            var widgetClass = Joose.S.strToClass(element.widgetClass)

            var widget = new widgetClass({
                boardWidget : this,
                model       : element
            })

            this.addWidget(widget)

            widget.render()
        }

One more example – the board widget listen to the `remove` event of the underlying set of elements – and removes the widgets:

board.elementsByUUID.on(
    '/mutation/commit/Syncler.Mutation.Object.Remove',
    this.onBoardElementRemove,
    this
)

4.4 MVC

Its worth to note, that Syncler gently but insistently structures the application in MVC pattern.

Any Syncler application naturally has a Model (which is a replica being synchronized). Then, when implementing the UI (View), its not possible to put any Model logic in View, it must only listen the events from Model. The reason is simple – otherwise, UI will ignore the updates from other collaborators.

So, the distributed nature of the application, when there is an external source of model mutations, already implies the separation between Model and View. The only needed step for pure MVC is making the View as much stateless as possible, and moving the processing logic to Controllers. This step is not as obvious as the 1st one though and requires certain qualification level from the programmer.

5. Status and roadmap

Syncler is still on very early development stage. In fact, even the name of the framework isn’t final yet. However, it has an extensive test suite (including a stress load test) so it can already host simple applications ala ShareBoard.

The roadmap items list is quite long, among the other things are:

  • Write some simple multi-player game to better research the capabilities of current approach.
  • Research the possibilities of using state-transfer algorithms and multi-master storage schemes.
  • Implementing a backend for KiokuJS, based on some graph database. This should be a very welcome addition, as graph databases usually have query languages and hopefully some of them even will provide certain form of referential integrity.
  • Adopting the other Socket.IO transports.

6. The showcase system – ShareBoard

ShareBoard is the proof of concept showcase of Syncler. Its a simple collaborative graphical editor, which allows several users to edit the same “online white-board” scene.

Try it yourself (in web-socket enabled browser, like Chrome): http://shareboard.joose.it

Tip: After creation of the board – open several browser windows with it side-by-side, or drop the link to you friend.

7. Conclusion

We’ve presented Syncler – a distributed system, implemented as the backend for KiokuJS.

The creation of the showcase system “ShareBoard” has proved, that the demonstrated approach is very simple (so far as this word can be applied to distributed computing) and clean from the architectural point of view.

Syncler is still on very early development stage so further research directions were defined.

moveTo : function (x, y) {
this.setX(x)
this.setY(y)
}add: function (element) {
this.elementsByUUID.set(element.uuid, element)return element
},remove : function (element) {
this.elementsByUUID.remove(element.uuid)
}

Written by Nickolay Platonov

April 8th, 2011 at 2:13 pm

Why Joose? Part #2 (phylosophical)

with 3 comments

If the the previous (practical) answer on the question “Why Joose?” doesn’t sounds convincing for you, here is the philosophical (aka meta-physical) variant. Yes, we like the “meta” word :)

Abstraction layers

Lets start with the following – why we all are programming anyway? Obviously to solve some real-world tasks. And we solve them, by translating the behavior of real-world systems into machine code. Such translation can’t be direct yet, as the computers are presently “dumb” and it has to be performed in the layered fashion.

That is, the first, outermost abstraction layer is the “user story”, written in the natural human language by the end-users.

The further level will probably be a technical specification, written by software analyst, or (in the agile methodology) a set of tests, representing the data-domain knowledge in the programmer’s head.

Yet another level will be the actual program, most probably written in some high-level language like JavaScript, Perl or Java (we treat everything except the assembler as high-level language here).

There will be also a byte-code layer, etc, the chain ends at the actual bits dancing on the silicon chips.

Complexity

Ok, now lets take a look from another (quite abstract) side. That real world task can be characterized by its “complexity”. This term is somewhat close to the “entropy”, may be its even a synonym.

We can say that each abstraction layer, we’ve used during translation, absorbs some part of that complexity. Like the sponge.

To solve the task, it’s whole complexity should be absorbed. You can’t leave some of it un-absorbed, as that will just means that some aspects of the system weren’t addressed.

So, the whole point is that, if you’ll be used “poorly absorbing” layers as tools, you’ll have to absorb the remaining complexity somewhere else, either in the code or in your head :)

Examples

Imagine you need to analyze some text file and extract repeating patterns from it, using C/C++ string manipulation capabilities. Compare with the same task for Perl/JavaScript style regular expressions.

Imagine you need to write the image recognition program in assembler.

Imagine you need to refactor the complex system and you don’t have the full and complete test suite (outer abstraction layer). Compare with the same task, when you have it. This is an example of poorly absorbed complexity.

Compare the implementations of quicksort algorithm in C and Haskell: http://haskell.org/haskellwiki/Introduction This is an example of layers with different absorbing capabilities.

Joose

So what’s special about the Joose and complexity? Its that Joose has meta-layer, which allows you to modify the behavior of your code (not the system being modeled).

And, modifying the behavior of the code, you can absorb any boilerplate you usually had to write. Moreover, you don’t just reduce the number of lines, you free your mind from that boilerplate. For example, consider this common pattern:

getSomething : function () {
    if (typeof this.something == 'undefined') {
        this.something == //something calculation
    }
    return this.something
}

How you’ll read this code? “If we have that `something` undefined, then we calculate it, and store, we always return `something`” Lots of words isn’t it? And the nature of `something` is still unclear..

In Joose land, you can just say that `something` is a lazy attribute:

has : {
    something : {
        is      : 'rw',
        lazy    : function () {
            return //something calculation
        }
    }
}

As you can see, the complexity of that pattern was absorbed by Joose and don’t pollute the code and programmer’s mind.

Conclusion

Enough of vacant advocacy. In the next post we’ll demonstrate, what all these words means in practice, and how Joose makes your code simpler and more robust.

Stay tuned!

P.S.

Oh, and we forgot to answer on one very important meta-physical question – “Does Joose works with NodeJS?” It does: http://samuraijack.github.com/Task-Joose-NodeJS/

:D

Written by Nickolay Platonov

January 13th, 2011 at 8:19 am

Posted in Joose

Tagged with ,

Joose.Why

without comments

Class(‘Joose.Why’, {

If someone would came at our IRC channel and would’ve asked me “Why Joose?” we could have the following hypothetical conversation:

Visitor: So why should I care about Joose? I already code in JavaScript – beautiful, functional language, very expressive, I write 500 lines of jQuery/ExtJS/NodeJS code in a day and my boss is happy – why should I bother?

Me: Lets slow down a bit, and clarify the things – JavaScript is not a functional language. Its a dynamic, imperative language, which has a first-class function. But, for example Perl also has the first class function, closures, etc, almost for 2 decades already and no one thinks Perl is a functional language. You are coding in the imperative language – should have no illusions about that.

V: So what?

M: So, the imperative programs tend to quickly turn into mess (“spaghetti code”). Its just the nature of imperative languages.

V: Not my code!

M: Of course, your’s not. But probably on your next job, you’ll have to maintain the codebase, written by someone else, you know what I mean..

V: Yea, other guys just can’t do the things right..

M: And to limit the mess in the imperative world, clever guys invented to separate the functions into groups, and limit their’s side-effects with a single object. They called those groups – classes and the whole thing – Object-Oriented-Programming.

V: Bo-o-ring..

M: Hey, you can do the same thing in JS, and its fun, take a look:

Class('Person', {

    methods : {
        eat : function (food) {
            console.log('yummy')

            return 'yummy'
        }
    }
})

Class('Person.Tidy', {
    isa : Person,

    before : {
        eat : function (food) {
            this.washHands()
        }
    },

    after : {
        eat : function (food) {
            this.brushTeeth()
        }
    },

    methods : {

        washHands : function (food) {
            console.log('washing hands')
        },

        brushTeeth : function (food) {
            console.log('brushing teeth')
        },

        eat : function (food) {
            this.SUPER(food)
        }
    }
})

V: Yet another language, compiling down into JS? No thanks.

M: Nah, its pure JS, just a call to `Class` function, with class name and some JSON, describing the class.

V: Hm..

M: This style is called “Declarative programming”.

V: So I can declare that program should “just work” and it will? :D

M: Almost :) Joose provides declarative constructs for the most common programming patterns, like Roles, attributes delegation, types, Singletons, etc, you can also create your own constructs.

V: My own? Like what?

M: Like if you are writing the Router class, you may make it has not only the attributes and methods, but also “routes”. You decide what the semantic of those “routes” will be and how they will behave. Or for example, you can add additional behavior to attributes (validation, binding, laziness, whatever) or methods (overloading, non-blocking chaining, whatever),

V: I see.. Hey, and does Joose works with NodeJS?

M: Sure: http://samuraijack.github.com/Task-Joose-NodeJS/

V: Ok, cool. Any links to start with?

M: aha: http://bit.ly/joose_manual

})

Written by Nickolay Platonov

January 12th, 2011 at 4:01 pm

Posted in Joose

Tagged with ,