HIGH SPEED DATA STORE FOR NETWORK/TELECOM ELEMENTS
naidu.trk@gmail.com
WE ARE ACTIVELY LOOKING FOR PARTICIPANTS.  WE NEED SKILLSET IN AREAS LIKE DESIGN, ARCHITECTURE,
PROGRAMMING ( C++),  LANGUAGE DESIGN , CODE GENERATION.

Introduction:

Data store aims to provide a high speed database for storing the operational state of a system. System can be any networking device a router , switch etc. Data store uses the "network database model" which helps to model the schema close to the object model of the system. Data store introduces new abstractions to program with and a schema language closely modelled on "object definition language".

System state of a Network element:
System state is the current state of a network/telecommunication element. System state of any networking device can be expressed in terms of objects and the relationships among them. System state is of read-often and write-rare nature. Software can access this data to do lookups and on do operations on a per packet basis. Data store allows efficient and storage and high speed retrieval of this system state, it also allows composing relationships dynamically among different objects in the system at run time.
if we take the case of a networking router/switch the data  store can be used to store objects like ports, subinterfaces, channels , addresses , routes, next hops etc. In all the succeeding sections we will take the case of a network router to explain the concepts of datastore.
  
Problem with existing representations of system state:
Most of the systems store the state inside kernel. In this model the state is replicated across multiple daemons for their operation. The main problem with this approach is muliple copies of same data floating which results in high wastage of memory. Keeping multiple copies of same data also leads to coherency problem. This coherency problem can be solved by sending notifications to all the daemons to make sure all the daemons are seeing the same picture. Every time the state is change it needs to be updated in all the daemons which includes unnecessary IPC and system call context switches in system.

Some advanced systems use a high speed sql database. The main problem with this approach is the impedance mismatch between the programming language and the sql query language. Daemon writers need to understand the sql schema to access the data base. Data is accessed through complex sql queries which imposes certain latency on the database operations limiting the through put of the device. The application software is plagued with sql query statements all over.

Data store Abstractions:
Data store introduces new abstractions to program with, software developers who intend to  use the data store need to understand these abstractions to write effective code.
       Class:
       Class is similar in meaning to the one used in object oriented programming languages.
       Relationship:
       A relation ship is the relation between 2 classes. Relationships are hooks through which objects connect with each other.
       Method:
       A method is a member function defined in the class and can be called after instantiating the objects of the class with an object reference.
       Transaction:
       A transaction is an atomic operation which involves changes on multiple objects. Transaction ensures either all the changes on all the objects are applied or none of them are applied.   
       Multiple operations on the data store can be batched in to a transaction and issued on data store either all of them are applied or none.

Facilities offered by Data store:

Architecture of data store:
TBD.

Data store object:
Any data item which is stored in the data store is a data object. Objects which are of global interest should only be stored in data store. Object types should be known at compile time for the data store to recognize the object. API's are provided for creation / deletion / get / set operations on the objects. Every time a new object type is added to the data store we need to add its definition in a class defintion file and compile the object plugin again.

Operations on the data store objects:
        Add Operation:
        An add operation results in addition of an object to the data store. This operation sends class based notifications to interested parties. Class based notifications will be discussed in
        later sections. An add operation
involves generation of one or more interprocess communication messages. Occasionally an add operation might spawn a costly transaction depending on
        the object properties. Transactions are discussed in later sections.


        Delete Operation:
        A Delete operation results in deletion of an object from the data store. This operation sends class based notification to interested parties. Delete operation involves one or more inter
        process communication
messages. Occasionally a delete operation might spawn a costly transaction depending on the object properties.  Transactions are discussed in later sections.

       Read operation:
       A read operation is used by the daemon to read a value of a particular object. A read operation does not involve any interprocess communication of any kind. A reference to the object is
       obtained  first and then the reference is used to read the fields of the object. This is in the order of direct memory reference. 
          

       Write operation:
       A write operation results in modification of one of the attributes of an object stored in data store. Write operation might send object based notifications to interested parties. It might also
       spawn a transaction
depending on the properties of the object.

Relationships and composibility:
An object in isolation is of little use. Objects in a system are realted to other objects by a relationship. Depending on the complexity of the system an object can participate with other objects in more than one relationship. Data store provides facilities to compose relationships between objects. Data store allows the programmer to define more powerful relationships among the objects as it uses the programming language constructs with out defining any new languages. There are both static and dynamic aspects of a relationship between 2 objects , these are described below.

    Static aspects of a relationship:
    Relation ships are known at design time and are incorporated in the information model. A class defines all the possible relation ship hooks that an object can participate in at the time of
    schema definition.
    Dynamic aspects of a relationship:
    Relation ship hooks are instantiated and become valid when an object participates in a relation ship with other objects at run time.
    Relationship methods:

    Objects can be composed in to relationships with other objects in data store. Every relationship will support 2 methods "compose" and "decompose". A compose operation will typically add
    the participant object in to a collection and a decompose operation will remove the object from the collection.

Examples of few relationships in the system:
An channel can reside on a port and therefore we can say port and channel are related to each other by "residence" relation. port is hosting the channel and channel is residing on the port, therefore these 2 objects are bound together in system by a named relationship called "residence_relation".

                                                                                                               residence relation


If we take the case of an aggregated trunk in the system , an aggreagated bundle like ae0  has other ethernet ports  in the system as its member links. therefore we can define a relation between member links and the bundle as "bundle_relation".


                                                                                Bundle relation


The partition layout of the channelized ports can also be modelled in terms of relationships like below, a relation called "partition_relation" can be defined to relate the partition and base interface like below.
 
                                                    Partition relation


Addressing of objects in data store:
Addressing is an important aspect of the system and very much impacts the performance of the data access. Every object in the datastore needs to be identified on its own and not in relation or hierarchy its participating in. This imposes some challenges on the datastore. For now every object is identified with a plain string.  Data store needs the type information along with the address string to quickly fetch the objects.

Inter object transactions:
Some times it's required to manipulate multiple objects in one shot. Applications needs facilities to compose multiple operations on multiple/same object in to one transaction. If the transaction succeeds all the changes are succesfully applied else none of the objects gets modified. Data store provides facilities to compose multiple operations in to one transaction. Transactions are pretty costly in nature and should be used sparingly.

Distributed commit:
A commit protocol is required if there are multiple parties interested in validating the change on the object. They can reside on the same node as well as different nodes in the system.  Depending on the object properties a commit protocol is started to seek the approval of the change on the object.  If all the parties approve the change on the object then only the object is modified in the data  store.  A  2 phase commit protocol is used to implement this facility. Daemons/Applications should provide callbacks to the datastore library to be executed in different phases of the commit protocol.

An illustration of the two phase commit protocol is given below, this is just for the illustration purpose and not the actual protocol used.

                                                                             
Notifications:
A  data store needs to notify interested applications when some thing changes in it. Change might be at object level or at a class level or at a relationship level. Notifications should be sparingly used as they can bring down the system to a grinding halt on  high loads.

Class level notification:
A class level notification goes to interested parties when an object of a particular class is created/deleted in the data store. Class level notifications carry a duplicate of object deleted/added as a payload along with them.
Object level notification:
An object level notification goes to interested parties when an object is modified. A duplicate of the object modified is sent in the payload along with the notification.
Relationship based notification:
          Relationship based notifications are sent to applications when an object changes its relationship with other objects.

Constraints and validation:
Constraints are checked before the operations on the datastore.  constraints can execute in the context of the datastore daemon or in the context of the applications. Constraints executed
in the data store context can be inlined with the class definitions in the schema file. Constraints executing on the application side are called during the commit phase. Data store operations
will not proceed on any constraint failure.

Designing for concurrency:
Data store is designed to scale on multi core processors. Multiple operations on the data store can proceed in parallel if the objects being accessed are in disjoint relationship collections.
Data store will implement locking at different granularities to parallelize the access.

Persistence:
Some times objects stored in the data store might need persistency to avoid resync or relearning the state from other nodes or for different reaosons. Applications can turn on persistency on the objects if they want persistency. A nearly consistent persistence model is used rather than absolute consistency. Absolute consistency is costly in nature, as every object being modified should be written to file system and then only can be committed in data store. we implement near consistent persistency which works decent.

Schema definition language:
A schema language based loosely on object definition language is used to specify the schema of the data store. Schema langauge will provide primitives to define classes, relationships and methods. Language also provides support for direct inclusion of C/C++ code and header files in the schema file.  Schema file is compiled by the schema compiler and will generate C++ data type definitions and assosciated API calls to be used by the applications. Applications can include the generated header files and use the data types emitted by the schema compiler.

                             schema file  ---------------> .h, .cc files  ------------------> Application/Datastore plugins
                                                     odl compiler                              gcc compiler

  More detailed explanation and a real life example from atm configuration can be found  here

Open issues:
Few things are not clear at this stage
i.   Application programming interface to the data store.
ii.  Effort involved in defining a new relationships.
iii. What commit protocol can be used for transaction and its performance impact.