Joose blog

Blog about Joose – advanced class system for JavaScript

Archive for the ‘Meta-layer’ tag

Syncler – distributed applications for human beings

with 9 comments

1. Intro

Distributed systems are hard by definition. Thats why its important to not mix the complexity of distributivity with the complexity of the data domain and keep them separated.

In this post we’ll present Syncler – a distributed, optimistic replication backend for KiokuJS. Syncler will keep several replicas of arbitrary data synchronized, in real-time.

Syncler is unobtrusive and does not require from developer to understand all the mechanics of replication.  Replica is simply a set of Joose classes and/or native JavaScript data structures. Application uses accessors to mutate the attributes of classes, and Syncler takes care about all replication details. To be informed about updates in own replica, application can subscribe to various events using the observable pattern.

Syncler has been designed after Bayou (described in this paper). Using the classification of optimistic replication systems from this paper, Syncler can be characterized as the system with:

  • operations transfers and semantic conflicts resolution
  • many masters (though a single master is being used for updates stabilization)
  • real-time updates propagation schedule
  • a star topology for updates propagation
  • eventual consistency (with very low replicas divergence period due to real-time updates)

To understand the advantages and drawbacks of these design choices please refer to the mentioned papers and other papers on the same topic. Most of the design choices were inherited from Bayou, however Bayou is more focused on off-line usage and Syncler – on real-time synchronization.

In addition to that, Syncler is cross-platform (runs equally well in browsers and NodeJS) and non-blocking.

In fact, Syncler is not a backend per se, but a special role, which can turn any KiokuJS backend into distributed system. Despite that at the moment there’s only one backend (for CouchDB), writing new ones is trivial (backend for Riak looks very promising).

As the transport, Syncler uses Socket.IO. Due to relative instability of most of the Socket.IO transports, at this moment, Syncler work correctly only with websockets.

2. Optimistic replication

Optimistic replication is a special type of replication, in which all masters apply own updates (and “see” the results) immediately. It can be said that every master is optimistic about that his update won’t cause any conflict.

Naturally, because the updates are applied without coordination among all replicas, the state of replicas may slightly diverge in the course of time, but the final state of all replicas is guaranteed to be the same (so called “eventual” or “weak” consistency).

Optimistic replication creates an illusion of zero network latency for own updates and greatly improves the responsiveness of the UI.

3. Applications

The application areas for Syncler is quite wide – it can be used as the basement of any kind of collaborative environment, where is a need to synchronize the operations,  originating from multiple users. Syncler is currently not bound to some specific data domain area.

3.1 Cooperative editing systems.

Probably the most obvious and well-known application area is so called “group-ware”. Typically its the graphical editor which allows several users to simultaneously edit some graphical scene, consisting from various graphical primitives. As the showcase of Syncler, we’ve developed such drawing application “ShareBoard”  (see section 6).

But naturally, the application doesn’t have to focus on graphic edition and can be used in any data domain.

3.2 Online multi-player games

Another very perspective area is the multi-player games, as games can be thought as some special kind of collaboration. This area is gaining interest along with modern browsers gaining visualization capabilities.

Syncler can greatly reduce the “entry barrier” for the creation of a new game as it handles a lower-level question of game scene synchronization, allowing the developers to focus on the gameplay process itself.

4. Architecture and realization

In this section we’ll shortly cover the architecture of the application, based on Syncler along with typical implementation patterns. Everything is still preliminary and subject for refactoring. We’ll use the code fragments from the ShareBoard system.

4.1 Replica

Replica is simply a set of Joose classes, having a `Syncler.Object` role. Currently replica also need to have a special “topic” class, with `Syncler.Topic` role (this requirement will be removed).

For example the board scene is modeled with the following class:

Class('ShareBoard.Model.Board', {

    does            : Syncler.Topic.UUID,

    has : {
        createdBy       : null,
        createdAt       : Joose.I.Now,

        elementsByUUID  : Syncler.I.Object,

        ...
    },

    methods : {

        add: function (element) {
            this.elementsByUUID.set(element.uuid, element)

            return element
        },

        remove : function (element) {
            this.elementsByUUID.remove(element.uuid)
        },

        each : function (func, scope) {
            return this.elementsByUUID.each(func, scope || this)
        },

        ...
    }
})

As you can see, board is generally a collection of board elements. And a base class for board element looks like:

Class('ShareBoard.Model.Board.Element', {

    does         : [
        'Syncler.Object',
        'KiokuJS.Feature.Class.OwnUUID'
    ],

    has : {
        board           : { required : true },

        x               : { required : true },
        y               : { required : true },

        color           : 'blue',

        ....
    },

    methods : {

        getBubbleTarget : function () {
            return this.board
        },

        remove : function () {
            this.board.remove(this)
        },

        moveTo : function (x, y) {
            this.setX(x)
            this.setY(y)
        }
    }
})

There are several subclasses of `Board.Element`, which add additional presentation logic.

4.2 Updates

To update the replica, application just calls the accessors methods. For example to move the board element (graphical primitive) to another position one can call the `moveTo` method, which looks as simple as:

    moveTo : function (x, y) {
        this.setX(x)
        this.setY(y)
    }

The wrappers for native JS data structures (which are required in the absence of proxies) provides obvious accessors for typical opeartions, for example to add/remove the board element to/from board, one can call `add/remove` method,
which updates the key in the underlaying JS Object.

    add: function (element) {
        this.elementsByUUID.set(element.uuid, element)

        return element
    },

    remove : function (element) {
        this.elementsByUUID.remove(element.uuid)
    }

The updates gets applied immediately and will propagate to other replicas in the background. As you can see, all the replication logic is packed into the meta-layer, the application is free from it.

Syncler assumes that any update can be rolled back (thats one of the design decisions of Bayou), thus all updates captures the “precondition” information which is being used for consistency checks.

4.3 Observable

To know about updates in replica, application should subscribe to various mutation events, which will be emitted by the meta-class. For example, to listen to mutation of any attribute in the underlying `model` instance, the base widget class can do the following:

Class('ShareBoard.View.Board.Element', {

    has : {
        boardWidget     : { is : 'rw', required : true },
        model           : { is : 'rw', required : true },

        ...
    },

    methods : {

        initialize : function () {
            this.model.on(
                '/mutation/apply/Syncler.Mutation.Class.Attribute',
                this.onAttributeMutate,
                this
            )
        },

        onAttributeMutate : function (event, mutation) {
            var attributeName       = mutation.attributeName

            if (attributeName == 'status')
                this.updateStyle()
            else
                this.updatePosition()
        },
        ...
    }
})

Another example – the board widget listen to `onNewInstanceOf` event of the replica itself and add widgets as new model elements appears:

        replica.onNewInstanceOf(
            'ShareBoard.Model.Board.Element',
            this.onNewBoardElement,
            this
        )

        onNewBoardElement : function (event, element, e) {
            var widgetClass = Joose.S.strToClass(element.widgetClass)

            var widget = new widgetClass({
                boardWidget : this,
                model       : element
            })

            this.addWidget(widget)

            widget.render()
        }

One more example – the board widget listen to the `remove` event of the underlying set of elements – and removes the widgets:

board.elementsByUUID.on(
    '/mutation/commit/Syncler.Mutation.Object.Remove',
    this.onBoardElementRemove,
    this
)

4.4 MVC

Its worth to note, that Syncler gently but insistently structures the application in MVC pattern.

Any Syncler application naturally has a Model (which is a replica being synchronized). Then, when implementing the UI (View), its not possible to put any Model logic in View, it must only listen the events from Model. The reason is simple – otherwise, UI will ignore the updates from other collaborators.

So, the distributed nature of the application, when there is an external source of model mutations, already implies the separation between Model and View. The only needed step for pure MVC is making the View as much stateless as possible, and moving the processing logic to Controllers. This step is not as obvious as the 1st one though and requires certain qualification level from the programmer.

5. Status and roadmap

Syncler is still on very early development stage. In fact, even the name of the framework isn’t final yet. However, it has an extensive test suite (including a stress load test) so it can already host simple applications ala ShareBoard.

The roadmap items list is quite long, among the other things are:

  • Write some simple multi-player game to better research the capabilities of current approach.
  • Research the possibilities of using state-transfer algorithms and multi-master storage schemes.
  • Implementing a backend for KiokuJS, based on some graph database. This should be a very welcome addition, as graph databases usually have query languages and hopefully some of them even will provide certain form of referential integrity.
  • Adopting the other Socket.IO transports.

6. The showcase system – ShareBoard

ShareBoard is the proof of concept showcase of Syncler. Its a simple collaborative graphical editor, which allows several users to edit the same “online white-board” scene.

Try it yourself (in web-socket enabled browser, like Chrome): http://shareboard.joose.it

Tip: After creation of the board – open several browser windows with it side-by-side, or drop the link to you friend.

7. Conclusion

We’ve presented Syncler – a distributed system, implemented as the backend for KiokuJS.

The creation of the showcase system “ShareBoard” has proved, that the demonstrated approach is very simple (so far as this word can be applied to distributed computing) and clean from the architectural point of view.

Syncler is still on very early development stage so further research directions were defined.

moveTo : function (x, y) {
this.setX(x)
this.setY(y)
}add: function (element) {
this.elementsByUUID.set(element.uuid, element)return element
},remove : function (element) {
this.elementsByUUID.remove(element.uuid)
}

Written by Nickolay Platonov

April 8th, 2011 at 2:13 pm

Joose 3 internals – Part II. Meta-model

without comments

This post will be very technical and contain a gory details about the implementation of the meta-model in Joose. It will be useful only for extension authors, if you are not planning to write one – nothing interesting ahead.

Well, you were warned.

Everything is a Property.

Property is the basement of the meta-model. It is an abstract name/value pair.

Everything in Joose is a property, i.e. attribute is a property, method is a property, role’s requirement is a property as well.

As it’s been mentioned in the previous post, property is implemented as a class from the 1st meta-level: Joose.Managed.Property (it has a Joose.Proto.Class as the meta-class).

Property can be shared among several classes (via composition relationships) so its totally self-contained and don’t contain any links to its “current” container, only to container it was defined in.

Property’s life stages.

By itself, property is just a description. To receive a concrete implementation (one can say “materialize” property) it should be applied to target. It can be un-applied as well (remember the mutability feature). In general, property has the following life stages: `preApply/apply` and `unApply/postUnApply`.  More on this below.

Everything is a PropertySet.

Joose.Managed.PropertySet inherits from Joose.Managed.Property. It is an abstract, unordered collection of properties, indexed by name:

On the set of property sets (sorry for tautology) we’ll define the following operations:

  • A.cleanClone – create a “clean” clone of A (without any properties)
  • A.clone – create a clone of A, including properties (only properties container will be cloned, not properties itself)
  • A.alias(what) – add aliases for properties, listed in what
  • A.exclude(what) – delete properties, listed in what
  • A.flattenTo(B) – for each property in A,
    • check if the property with the same name in B exists
      • if there is one, and its a special conflict marker property or the property of A itself, then just skip it
      • if there is one and its different from the above – then replace it in B with the conflict marker
      • otherwise, copy that property to B
  • A.composeTo(B) – for each property a in A,
    • check if B has own property with the same name
    • if not – then copy that property to B
  • A.composeFrom(B, C, …)
    • create a clean clone of A
    • flatten each argument into the clean clone (possible aliasing & excluding some properties first)
    • compose the result to A

Listed operations will implement the traits spec. All subclasses of PropertySet will keep the semantic of these operations.

PropertySet’s life stages.

PropertySet simply propagate the `preApply/apply/unApply/postUnApply` stages to its elements. Note, that current implementation is still written in the imperative spirit and rewriting it in more functional way will possible allow us to remove the `unApply/postUnApply` stages at all.

Mutable PropertySet.

Joose.Managed.PropertySet.Mutable inherits from Joose.Managed.PropertySet. Its a bit more specialized property set. It track the depended sets (derivatives) and propagate the changes through them. Mutable set B said to be the derivative of A, if it has been composed from A (possibly along with other sets).

Mutable PropertySet can be in 2 states: opened (this.opened > 0) & closed (this.opened == 0). Mutation is only allowed when property set is opened.

During “open” operation, when switching from closed state to opened the set is de-composed – cleaned from all properties which weren’t defined in this set. Before that all derivatives are also opened. Note, that the set can be opened several times – subsequent “open”s will be no-ops. Each “close” operation should have matching “open”. Initially the set is opened (this.opened == 1)

During “close” operation, when switching from opened state to closed, the set is re-composed using the data from “composedFrom” attribute and inherited “composeFrom” method.

Higher-order PropertySet.

Joose.Managed.PropertySet.Composition inherits from Joose.Managed.PropertySet.Mutable. Its an abstract higher-order mutable property set – a property set, which has other property sets as the elements:

Its no longer unordered and defines the order of properties processing (`processOrder`), also defines the reverse processing order, which is important for unapplying things.

Composition’s behavior.

Composition translates the higher-level operations to individual properties. That is, when composition A is asked to be flattened to composition B for example, it flat it’s individual properties:

Other operations are defined in the same way. Note, that composition is a mutable property set.

Stem

Joose.Managed.Stem inherits from Joose.Managed.PropertySet.Composition. Its a concrete implementation of composition:

Stem contains the following properties (in the processing order): [ 'attributes', 'methods', 'requirements', 'methodsModifiers' ]. Each of those properties is a property set of individual properties – collection of attributes, methods, etc.

Stem is the first property, which is aware of the real JavaScript class it belongs to (targetMeta attribute).

During re-composition stem initiates the `preApply/apply` actions of the whole properties subtree.  In the same way, during de-composition, it initiates the `unApply/postUnApply` stages.

Note, that during `preApply` stage the stem is still opened, and can mutate. During `apply` stage, the class definition is “final” and should only be somehow “materialized” in the prototype of the class, but not changed. The class in which the property should “materialize” itself will be passed as the 1st argument for “apply” method.

Conclusion

This post contains the details about the meta-model implementation in Joose3.  In the next post of this series we’ll briefly describe the anatomy of class from the 2nd meta-level – Joose.Managed.Class.

Written by Nickolay Platonov

January 13th, 2011 at 5:08 pm

Posted in Joose

Tagged with ,

Joose 3 internals – Part I. Meta-layers

with 3 comments

This post starts a series, describing the meta-model of the Joose3. It mostly purposed for people brave (or crazy) enough to hack on the internals, though casual users may find it also useful for deeper understanding of the underlaying processes in Joose.

Classes and meta-classes.

Ok, lets start with the definition what is the class. Its the abstract template or pattern, describing some behavior (for example: “Dog”). This template can be “instantiated” (usually arbitrary number of times) – that means the abstract definition receive a concrete implementation in some object (for example: “Pluto”). Such object will be called “an instance” of the class:

On the figure above and the following ones, the filled circles denotes classes, unfilled circles – instances.

Then, what is the meta-class? Its the class, whose instances, in turn, represents classes:

The meta-class defines the higher-level behavior of the class. For example it may define, that class can be only instantiated once. Or, that class may have special attributes with validation. Or, special methods with type-checking, etc.

Ok. You’ve probably already spotted the smell of recursion in the meta-class definition (its the class, whose instances are classes). Naturally, the chicken & egg problem arise – what is the meta-class of the meta-class itself? And what is the meta-class of that meta-meta-class?

Joose.Proto.Class – 1st meta-layer

The chicken & egg problem is being solved in Joose 3 by the 1st meta-layer, which is represented by the Joose.Proto.Class meta-class. Joose.Proto.Class is the class, which is the meta-class for itself! This makes its meta-instance perfectly circular – it refer to itself.

Classes with Joose.Proto.Class meta are very close to “raw” JavaScript and other simple class systems. Virtually everything from the class definition goes directly to the class prototype (except the “isa” builder). Inheritance is done with the “classical” prototype chain hack (see Joose.O.getMutableCopy in Joose.js)

If you really need “raw” JavaScript code (for animation library for example), you can host your class at this meta-level like this:

Class('Raw.Code', {
    meta    : Joose.Proto.Class,

    attr1   : 'init',
    attr2   : null,

    method  : function () { return 'result' }
})

var i = new Raw.Code()

i.method() == 'result' // true

At this meta-level there are no roles, traits, lazy attributes, method modifiers or any other magic, just JS. The other parts of the meta-model are written as classes from this meta-level. The next post will cover this topic in detail, in the meantime we’ll continue the overview of the other meta-layers.

Joose.Managed.Class – 2nd meta-layer

The next meta-layer is presented with Joose.Managed.Class. Joose.Managed.Class is a subclass of Joose.Proto.Class, and has it as a meta-class in the same time!


Important feature is that its safe to inherit from class, which has meta from the the previous layer:
This feature makes the bootstrapping possible.

Classes with Joose.Managed.Class as meta-class are already very capable. They may have method modifiers, roles or traits applied. At this level also appears the Joose.Managed.Role – a special meta-class, which, among other things, prevents class from instantiation.

However, we need one more bootstrapping step. The reason is that, by itself, Joose.Managed.Class has a Joose.Proto.Class as meta. Thus its behavior is very limited, you can’t apply role to Joose.Managed.Class for example. We definitely want our meta-model to have a more capable central meta-class. Another reason is the attributes. We need to be able to apply traits to the attributes. So the default meta-class for the attribute should be at least at “managed” level. But this level just been introduced – will be more elegant to delegate the change of the default attribute’s class to next layer.

Joose.Meta.Class – 3rd meta-layer

This is very simple meta-layer, required mostly for bootstraping purposes: Joose.Meta.Class.


There is also a corresponding Joose.Meta.Role meta-class.

Joose.Meta.Class is the central meta-class of the meta-model. If you need to modify the behavior of all classes, you can apply roles to it (comes in use for JooseX.Meta.Lazy for example). See also next section for explanations hows that possible.

Mutability

All classes in Joose, on all meta-levels are mutable. “Mutable” means that class/role definition can be changed at any time (add/remove attribute or method for example) and the change will be propagated to instances and other depended classes/roles.

Mutability is based on the fact, that during inheritance, prototypes of Joose classes forms the usual prototype chain. Thats why no special actions required when a class changes – the language itself will handle the propagation to sub-classes. A simple example:

var a = {

    method1 : function () { return 'a1' },
    method2 : function () { return 'a2' }
}

var b = Joose.O.getMutableCopy(a) // establish a prototype chain

b.method2 = function () { return 'b2' } // override a method

b.method1() == 'a1' // true
b.method2() == 'b2' // true

// mutation

delete a.method1
delete b.method2

b.method1 = function () { return 'b1' }
a.method3 = function () { return 'a3' } // add a new method to the base object

// eof mutation

b.method1() == 'b1' // true
b.method2() == 'a2' // true
b.method3() == 'a3' // true

Slightly more complex logic is required for roles mutations, but Joose never update the whole graph of depended classes, the most of work is delegated to JavaScript and  number of “manually” updated classes is as minimal as possible.

Conclusion

This post contains a high-level overview of the Joose meta-model. In the next post we’ll focus on the details of 2nd meta-level, which implements the traits specification.

Written by Nickolay Platonov

January 13th, 2011 at 9:02 am

Posted in Joose

Tagged with ,