A recent study showed that almost 40% of the AI, analytics and data management investments by an organization go towards aggregating data from multiple internal and external applications, reconciling for inconsistencies and creating a clean source of information. Not to mention the IT and operational customer service spending to manage process breaks due to inconsistent data.
Given the importance of AI in today’s data driven digital economy, this inhibits agility and slows down insights driven innovation. In addition, it can lead to significant overhead in meeting compliance needs and regulatory reporting.
In this blog post, we will first review the traditional approach to data management, and then outline a new approach using smart contracts, which can be a useful addition to an organization’s toolkit to address this problem. Finally, we will introduce how this has been achieved in practice using our enterprise orchestration solution Gen.Flow which uses DAML smart contracts.
How do current data management approaches work?
Typically, an enterprise business process is executed by multiple applications that work with each other to take actions and move the workflow forward. Each of these applications may have their own application databases, and would pass data back and forth to keep in sync with each other. For example, a credit cards process may consist of a new card origination system to create a new card, followed by a fraud detection application, an e-statements application, and a customer service application. These applications may be a combination of home-grown or externally sourced technologies, and they may be added at different times for different geographies.
This complexity and business need driven evolution of the applications landscape does not allow a complex enterprise to maintain a single monolithic system. As a result, multiple applications exist, often with their own separate databases, and are synced with each other through enterprise grade mechanisms. Common approaches are middleware enabled hub and spoke architectures, various BPM systems that connect applications, API exchanges, and plain vanilla batch file transfers.
As the business process is executed (as in the above scenario), there is often a need to consolidate the data and satisfy various analytics needs. For example, the marketing organization may need insights regarding targeting and segmentation analytics to optimize upcoming campaigns.
In order to serve these various reporting and analytics needs, the traditional approach has been to aggregate the data from various data sources, clean and reconcile inconsistencies, and then load it into a central enterprise data warehouse. The extent of this central data store may vary by enterprise. Many enterprises have multiple such warehouses (e.g. typically for marketing and compliance), while other enterprises create specific data marts off a central warehouse.
In this world of post-facto aggregation, it is natural that significant effort is expended in staging, reconciling and aggregating data. The bigger the organization, the more data sources and application silos exist, and the overall management infrastructure becomes even more complex. Challenges arise in business innovation due to the complexity of integrating additional services.
Finally, there is the issue of access control and compliance. Not everyone must get the same level of access to information, and every step in the business process must be reported and audited for compliance and regulatory purposes. For example, anti-money laundering and SOX compliance are common use cases.
So, what can be done to solve this problem? How can these multiple application data stores be bridged to produce a clean version of data without having to constantly aggregate and reconcile?
How Smart Contracts Can Enhance Traditional Data Management
Smart contracts bring two fundamental changes to the way we look at our business information and process.
- The first is that any business action causes a change to underlying data in a deterministic manner. That means we can now think of key entities in terms of “digital assets” on which actions are taken. For example, a credit card issued to a customer is a digital asset, on which a fraud flag is raised, a payment is made, a campaign is run etc.
- Second, the entire business process of an enterprise now takes tangible form instead of being buried in multiple flow charts and documents. As any enterprise technology practitioner knows, documentation is out of date as soon as it is produced. With smart contracts, your business process is codified easily into functioning software which is then maintained as often as you make a change.
As a result, a smart contract-based applications landscape enjoys a single version of truth in the form of the contracts store. This source of data does not need to be reconciled or aggregated, but is automatically created as business processes execute over time.
In addition, data no longer just works as a static record driven by applications, it can itself trigger events and drive processes forward when it is changed, regardless of what is set up on the application layers. For example, a fraud alert being raised on a credit card, can trigger an action on the payments receivables system to alter when a payment is due. New views or data marts can be created off the central smart contracts store.
So, we transform the challenge of post-facto aggregation of data, to a problem of post-facto distribution of data to those who need the data.
The Smart Contracts Approach in Practice: Introducing Gen.Flow
At this point it is important to note that while smart contracts and Distributed Ledger Technologies (DLT) are complex to architect and implement, there is a key difference between the two.
- Smart contracts (e.g. DAML smart contracts) are independent of blockchains, and are ready to be adopted by any institution looking to harmonize processes and data, regardless of their technical landscape. Smart contracts do not need a blockchain to run. Instead they can utilize traditional data stores. Gen.Flow makes this possible by using DAML, a smart contracts language and runtime engine that can run on both blockchains and databases.
- Blockchains and distributed ledgers, on the other hand, are indeed a medium-term bet. Most organizations do not have incentives to adopt DLT and blockchain for internal use cases. Some of the most common disincentives are the complexity of consensus protocols, infrastructure management overhead of nodes and security keys, performance bottlenecks and potentially high cost of operations.
On the other hand, consortia and shared-services providers can definitely benefit from modern mutualized ledgers because multiple organizations are involved and require a high-trust threshold.
Based on this new approach using smart contracts, we have created Gen.Flow, a smart contract-driven orchestration layer. This approach solves the data silo problem, reduces the complexity of adoption and accelerates the time-to-market for enterprises. It is the culmination of our efforts to help solve some of the key challenges that organizations have shared with us. By using DAML, Gen.Flow is also capable of integrating seamlessly with an external DLT network.
Some interesting points to note:
- Gen.Flow’s smart contracts-based approach works along with your BPM or EAI based architectures. The BPM or EAI layer invokes the Gen.Flow API exposed by smart contracts.
- In addition, a unified API interface allows for organizations using Robotic Process Automation to integrate robots with smart contracts to unlock next generation intelligent automations and industrial savings. For example, a robot may need to create an audit trail, or access the status of a fraud flag before closing a case. By integrating RPA with smart contracts, the robots can create a ready-made audit and compliance trail.
- Gen.Flow allows organizations to abstract technical complexities of blockchains and smart contracts so they can focus on developing process-driven applications
Gen.Flow leverages smart contracts as a process-driven layer of applications, more abstracted from technology and organizational constraints and more in synch with data itself. Smart contract applications closely emulate and help centralize business rules.
The potential benefits of smart contract applications, particularly for the automation of digital processes, are becoming clearer and more appealing to organizations. They allow organizations to leverage functionalities such as non-repudiation and atomic transactions, process-driven development, seamless multi-party integration and higher abstraction from technical constraints. This consists of approaching automation from a business and process-driven perspective.
In order to augment automation capabilities, many organizations are pairing smart contracts-enabling technologies with Artificial Intelligence and Machine Learning applications. The general idea is using smart contracts to orchestrate processes and transactions closer to both business rules and core data, and harmonizing the underlying data, which can be on multiple databases and, at least, one golden source of truth. Robots can more seamlessly connect to core data and orchestration workflows. Artificial Intelligence and Machine Learning applications can leverage on smart contracts and harmonized data to further extend analytics capabilities – we call this full-stack automation. The smart contracts approach is also a powerful addition to the enterprise data science toolkit, by accelerating the adoption of digital and insights driven business transformation and unlocking operational benefits which can enhance the return on analytics investments.