Scalable platform, from cloud out to the Edge

The vast majority of computer systems today are distributed.  Sharing data in a such a system is problematic, because, as proved by the CAP Theorem, it cannot be at the same time available and strongly consistent.  For decades, research and industry alike had been searching for a solution to this quandary.
In 2009–2011, Marc Shapiro and Nuno Preguiça developed a novel approach, called Conflict-Free Replicated Data Types (CRDTs), whose properties mathematically ensure both availability and convergence.  CRDTs were quickly taken up by both research and industry, for instance being used in the Riak, Redis and CosmosDB databases. The recent research of our team members has focused on making CRDTs practical in highly-available applications, on guaranteeing application-level correctness properties (invariants), and on building an open-source CRDT ecosystem (the AntidoteDB geo-distributed database and associated tools) following a Just-Right Consistency approach.

The Concordant start-up plan represents a new, ambitious step in the development of highly-available, highly-scalable distributed systems, enabling sharing between massive numbers of edge and mobile devices, and extending CRDTs and Just-Right Consistency seamlessly across the whole cloud-to-far-edge spectrum.  It leverages the experience and contributions of Shapiro et al. and builds upon the open-source artefacts they developed.

Despite improved network infrastructure, users still experience expensive roaming charges, high latencies, and extended periods of disconnection; furthermore a user or group may decide to “branch” and work independently for a while. This poses important challenges for the development of mobile applications, as users expect their mobile applications to work at any time and location. According to the offline-first approach, mobile applications should be designed to mask disconnection, providing users with a uniform experience independent of connectivity. This constitutes an improvement over the more traditional approach of many BaaS platforms, where mobile applications depend heavily on the cloud for their operation.

We go further, and advocate an edge-first approach: design mobile applications for very fast response, by having data stored locally on the mobile device and by collaborating directly between nearby clients, at all times (whether online or offline). Consider for instance supporting the field agents of a utility company: the edge-first design enables its users to share information, even when in locations with poor network connectivity, such as basements. Another example: In a mobile game application, fast interaction among nearby users is key to improved interactivity. This approach would also support disaster teams operating their own ad-hoc infrastructure when the public telco system is out of service.

Although highly desirable, the edge-first approach would be very hard for application developers themselves. They would have to replicate the right data on each user device, and propagate updates between replicas. They would have to ensure that propagation does not overwrite any local change, but instead merges all changes. They would have to implement direct peer-to-peer sharing connections between nearby devices. They would have to worry about keeping replicas consistent, and about concurrent updates violating business rules undetectably. Instead, they can outsource these difficult issues to the Concordant platform.

Concordant thus simplifies the development of powerful mobile applications. The Concordant software stack comprises three specialized (but interoperable and consistent) tiers:

In cloud infrastructures, providing the cloud-based backend service for long-term data storage and processing, and long-haul data access.
In Point of Presence (PoP) devices at the edge of the network, such as portable servers or edge servers. C-PoP provides nearby data access, storage and computation services for remote devices (mobile or connected) at the cloud-edge boundary.
In end-user devices. C-Client provides storage and sharing capabilities to the application and the edge; it interfaces with the C-Cloud and/or C-PoP; and it supports peer-to-peer interaction with other C-Clients.
These components will have the same API and semantics, allowing application components to be placed where most appropriate, and even to migrate from one tier to another.
The design of the Concordant platform, with its three components and unified API, allows Concordant to run on infrastructures that support cloud connected terminals, up to infrastructures that extend out to the edge, with Point of Presence servers (PoPs) supporting remote devices and peer-to-peer interaction among client devices. This allows Concordant to support existing mobile application scenarios, and it will offer new possibilities for application providers (such as for collaborative classrooms and 5G operators). To this end, the Concordant platform will provide a compelling set of features:
  • Data on the go: the data you need is always with you. Run your application wherever you are, even offline. Never wait due to slow or flaky network. All your updates are reliably propagated to the cloud asynchronously, as connectivity allows.
  • Support for direct interaction: when working together with your colleagues or playing a game, information flows directly in a peer-to-peer fashion, without depending on the cloud. Local storage and sharing can be boosted by PoP devices running the Concordant stack.
  • Top-notch consistency: when different users write to the same data, updates are not lost or overwritten as in current systems; with Concordant, data is automatically merged as appropriate for the application, at any device running the Concordant platform. Concordant can enforce consistency that goes well beyond convergence, including transactional updates that appear in the proper ("causal") order and enforcement of application-level business rules.
  • The right level of security: not all applications have the same security requirements. Concordant supports different levels of security, from fine-granular access control management up to end-to-end encryption, where data is only ciphered/deciphered in the end-users’ devices and the platform only manages ciphered data.
  • Uniform APIs, semantics and guarantees across the cloud, PoP and edge spectrum: application code is located where it is most convenient or relevant, e.g. on the terminal device for fast response, on the PoP to balance latency and bandwidth, or in the cloud for high bandwidth and coverage.