Spouting Shite

Eventual consistency Vs Linearizability

Last week I spent a few days with an IT company in Montreal, Canada. We are looking at partnering with this company to share some intellectual property, basically, we want to use some of their applications and modules in our emailcloud service. Unfortunately we came to a road block due to our platform design choices. At this time I can’t name the technology partner as we are still under NDA….for the sake of this blog post let’s call them CoolCo.

CoolCo is a technology company and have been in business for 14 years providing excellent IT security products. These products were designed to be installed on servers based in the client’s offices or in some cases in racks at the ISP. They provide services to 3,000 clients and last year they turned over $17m. Their products are truly enterprise grade and are designed from the bottom up to ensure that they are scalable, secure and that changes are propagated across a cluster of servers incredibly quickly…usually in less than a second. To do this properly they use an atomic structure to their administration system and it works very well.

Our network is rather different and we need to be able handle extreme scalability on a level many times above that of CoolCo. While CoolCo can usually predict the usage of their products (and thus can recommend hardware for their client) we don’t have such a luxury. Our network is incredibly hard to provision hardware for …. we use a bank of 50 Linux servers to handle our base load and then contract in additional servers ( from Amazon and Flexiscale) as we need them over the day. This is perfect for us as we can increase our capacity as needs arise. We can’t use an atomic structure as this will not allow extreme scalability and in its place we use a system of ‘eventual consistency‘.

The eventual consistency model states that, when no updates occur for a long period of time, eventually all updates will propagate through the system and all the replicas will be consistent.

So what does this mean? Well, when we decide to make a change to some of our system settings they take a few minutes to propagate through our network rather than happening instantaneously. So, it takes a minute to add an IP address to the black list rather than making the change immediately. To do this we use a proprietary system called Arachne to push (and allow pull) of our settings. This works very well because when we add new servers to our network (event if they only stay online for a few minutes) they can pull their settings and start processing data. If we kill the server then it is removed by Arachne from the scan banks and completely forgotten meaning that we don’t have orphans left on the network.

So, who cares? Well…we do unfortunately. It seems that because of the two different design methodologies that sharing IP is not as easy as I had expected. Basically, we will need to retool the modules to work on our network and then repeat the process every time CoolCo releases a new version of their modules….causing delays in the implementation of new features……oh the bother!