It's All About the Database

OLTP 2.0

I developed the following concept as a follow up to my last post covering open source software Linux, PostgreSQL, GO (the programming language) and gRPC. Also mentioned here, is proof-of-concept code that I recently built and is now on GitHub.

CONCEPT

Using best of breed, open-source components, design a framework for building an open source, peer-to-peer OLTP “network of databases”.

A “Database Everywhere” APPROACH

Each peer has its' own database --part of a data-event-driven, permissioned network of databases for near real time use.

FRAMEWORK

Treat each database transaction as an asynchronous message.

Achieve reasonable and eventual data consistency.

Factor distances and speeds between peers into the design.

Use regional hubs to balance performance, security, and monitoring.

Keep data sets small.

Borrow from the blockchain, make transactions immutable, AKA 'inserts only'.

Make it easy to understand, use, maintain, and customize.

USE CASES

Anything that needs near real time high-performance, data management like trading systems, B2B E-Commerce, payments, blockchains, data vaults, etc.

PROOF OF CONCEPT CODE

I developed the following proof-of-concept code on GitHub. It demonstrates using a database transaction as an asynchronous message. It is a simple approach using a PostgreSQL database, its' asynchronous messaging features, and some GO programming code. The message is sent over a gRPC connection to a destination database.

REFERENCES

Developing the OLTP 2.0 concept, I reviewed a lot of data management and programming methods, systems, and techniques. Some old, some new. The reference links below reflect the ones I reviewed--especially those listed under these categories:

• Asynchronous Messaging
• Block Chain
• Cloud (PaaS) Databases
• GO(GOLANG)
• gRPC
• Open Source Databases
• PostgreSQL

Interestingly, some approaches seem backwards. Instead of "faster, better and cheaper" we are getting "slower, more complicated and expensive". It is hard to think we really need that--but more about that it in a future post.

LOOKING FORWARD

OLTP 2.0 is a concept worth developing. It is a practical, creative use of “best of breed” methods and technologies. The demo code on GitHub is a seed that hopefully can continue to grow.

To help that, I joined Github Sponsors. It is a framework for supporting open-source developers and bringing sponsorship to the community. To date, my company kDS, LLC has funded this research. I have spent a great amount of time and resources to get to this point and am confident it is an important direction that will make a difference. If your organization supports open-source initiatives or is looking for an opportunity to get involved, please consider a level of sponsorship. It is a win-win for organizations that need to research and develop their data technology future.