I’m happy to report that questions from the previous post were solved. We have found a good solution to detect loss of synchronization, to identify missed data items and of course to synchronize data.
I’ll not mention here all technical details but in few words we need to send periodic heartbeats between all DS in the network. Heartbeat is a request containing no actual data and heartbeat response is a very small data block (which has linear dependence on the total number of overall DS in the network). That response gives us possibility to detect missing data blocks. That’s good.
Also I would like to present new entity in our XdbE core architecture which is called Relay Server (RS). RS improves overall data distribution within XdbE Network. In other words RS is a DS but without storage functionality. RS is not a mandatory entity in the network, DS network can function without it but in large network RS(s) is(are) helping to balance load between all servers by taking away data distribution to itself (themselves).
How it works? Let’s look at the data distribution process in the DS network without RS. Once DS receives an Event from an external network it should start event distribution process that will finally deliver it to all other servers. Orig DS forwards the event to a couple (or more) of DS and they continue distribution in the same manner. Every DS forwards event to couple (or more) DS. For example if we have a network with 100 DS and each DS forwards event to couple of DS then distribution of event takes 6 steps and 50 DS are involved in forwarding. It means that 50 DS have triple network load (receive, send, send). Also each DS should store this event to its DB (thus increasing CPU load).
During high load period RS helps to balance loading of servers by taking away up to 50% of network load. DS can pass event forwarding to RS and RS will forward it to as many DS as it can depending on its current load. If RS is idle then it could forward event personally to each DS in the network. If it is not idle then it could forward event to several DS and at the same time forward to next RS.
One more time. RS is not a mandatory entity for the XdbE Network but RS usage adds flexibility in load-balancing mechanism.
I was talking about data and data distribution in my previous posts but I said nothing about the way how data can be transferred between servers within the XdbE network. I mean the protocol.
In XdbE data is meant to be transferred in two distinct environments: 1) Inside DS Network itself, 2) Between external network and DS Network.
Definitely we should use the standard protocols for both cases.
Here I should say that we need two layers: communication layer to deliver requests/responses through Internet and data interexchange format to serialize data.
The transport layer we are going to use is HTTP. This is a standard request/response protocol which allows to enclose any data. We will not use HTTPS because it requires a certificate for every participating server which will unnessesary complicate the whole system. Instead of HTTPS we will use a kind of encryption over HTTP traffic. Using HTTP protocol we can use standard webserver for catching requests and calling applications and we don’t need to invent and maintain something other than that. Of couse usage of HTTP is obviously a single and best option to connect with external networks as it is the most standard and the most widspread of all protocols considered hence it is less problematic in terms of routing and ultimately speed of data delivery.
Which data exchange format to choose seems to be more complicated question. At first we should remember that the format in question must be lightweight for traffic saving and easy processing for CPU resources saving.
Considering this problem we examined the following potential options:
- XML (XML RPC)
- JSON (JSON RPC)
- Custom proprietory data exchange format
Albait XML (XML RPC) is widespread enough it ss less suited for the task as it increases the size of data package thus increasing the overall network traffic as well as less efficien in terms of CPU resources required for parsing.
Developing Yet Another Data Exchange Format was considered only if a suitable format could not be found amongst existing tools. Fourtunately, there was no need to recall this option.
We selected JSON RPC for data exchange because it is more lightweight, widely supported in different programming languages and can represent any non-recurrent data structure “as is”. RPC over JSON standard exists and might proove to be extremely useful at later stages.
As for the way DS will release user data to clients (external networks or other clients requesting data) I should say that we are going to make different options available here: JSON, XML, Open Social.
We just had a discussion about data circulation within the XdbE Network and here I will take down some of the most interesting and impirtant points.
When I talk about data I mean a piece of information about user. When something is changed in user details or user relations in any external network the new data come into XDBE network by means of Event call. Event call is a data message describing changes in users list or user details: user new status, new relations, etc. Event can be received by any DS in the network (because all DS has equal privileges). Then this Event is distributed across the DS network to provide a complete data redundancy. Later user data can be requested by Data Client (DC) from DS network by means of sending a request to any DS server.
The questions here:
1) How to distribute data in the DS network with highest possible speed and minimal service traffic?
2) How to guarantee that all at the end DS have the same set of data considering that any DS at any moment can become out of service due to network failure hardware problems or any other similar reason?
3) Who should be responsible for data distribution process: the originating side (DS which have received an Event from external network) or the recipient of data (every DS in the network)?
I’ll try to demonstrate several possible approaches considering also all their pros and cons. Let’s call DS who received an Event as Orig DS, other DS in the network are A, B, C. Suppose DS C is down and out of network.
Imagine that Orig DS distributes Event to every DS in the network and is solely responsible for this. Once Orig DS has received an Event it sends the Event to DS A, DS B and DS C. DS C is out of network (offline) and the Event can’t be delivered to it. Orig DS continues with attempts to send data to DS C but suppose at some point while doing this it too goes offline (for whatever reason). It means that to preserve data consistency some process from ether A or B must continue distribution of the Event to C. It also means that both A and B should catch the moment when Orig DS became unavailable and understand that either A or B must continue its work. A is continuing and assume that A goes down as well (in large network anything could happen). Then B must catch this moment and should continue distribution. B finds out the missing data on C and continues. This seemsw to be a rather silly way of doing it because in case of large DS network where several DS are offline the network will be busy doing just one job – sending data to offline DS and syncing this activity between all online DS in the network.
The right way would be a delegation of self-syncing responsibility to each DS. Every DS in the network is responsible to be in synced state. In that case when Event is received by Orig DS it distributes it to several online DS immediately. Then these DS distribute the Event to next several online DS but if some DS is offline nobody will constantly try to deliver the Event to it. When offline DS C returns to online state it checks for missing data and syncs. This is a way how our network should work.
After making this decision we are faced with a number of new questions:
1) How to identify missing Events on offline DS C?
2) How to identify right order of missing Events?
3) How DS can catch the out of network state?
There are no clear answers yet.
At the core of XdbE system is a set of servers which store data (social data). We call them Data Servers (DS). We debate about two possible directions: a) full data set on each DS (full redundancy) and b) spreading the data across the network with some sort of redundancy of course. The decision is fundamental and it will defiine all aspects of the system. We are examining in details all pros and cons for each approach.
If we choose to spread the data across the network then we need a kind of Index Server (IS) in the network or a set of IS to store the mapping between users and DS on which the users details are stored. Of course we can go without IS in the case but only if we have a constant number of DS in our network. In this case we just define a mathematical function like F(user_id) -> DS_id which maps user id to id of specific DS. But the case with limited number of DS in the network is not acceptable and it means that we must use a kind of indexation and store the index somewhere in some way.
Ok. Now it’s a time to refresh our major goals and objectives:
- System must store basic data about users like user id, age, gender and location. System will not store any private or multimedia data for users.
- It is better if data are stored in pear-to-pear network where there are no privileged servers.
If we look at our first objective and calculate the required storage space then it becomes obvious that the data for billion users can be easily stored even on home PC.
When we consider both our first and second goals it becomes apparent that even if we decide to use IS for indexation then all servers in IS will store full set of index data to create no-privileges in the network. Storage and synchronization of full set of index data by each IS provides us with no advantages in comparison with full redundancy of data stored on DS.
We have decided to choose option to create pear-to-pear network with full redundancy where data about all users are stored on every DS in the network. In other words – all participating DSs at any moment of time should have the same set od data.
I’m starting to work on XdbE project. Now I will participate in developing core architecture and everything else. We start from the blank paper where we write major goals to achieve. Next month I will work on it. From the very first view we are going to develop simple application to collect and distribute data. But I know from my experience that details are everything and every detail should become a clear-cut algorithm invented, coded and tested. The system is going to work under huge load created by incoming queries for data read, add and update.
Posted in Updates
|
Tagged programming, team
|
We welcome Sergey Antonyuk AKA Ansy to our team. For quite some time we’ve been looking for a qualified programmer capable of leading the production side of the project. Now we are lucky to have Sergey on board. With his experience as Lead Programmer in various projects, extensive expertise in several programming languages and overall structured and systemic approach to problems he is definetly able to realise not only the most immediate objectives but is capable of leading XdbE programming in the long term.
Posted in Updates
|
Tagged team, XdbE
|
Today we decided to end the formation phase of the project and start formally working on XdbE full scale. Wish us luck in this undertaking!
Posted in Updates
|
Tagged XdbE
|
