Using a Smart Data Fabric Architecture to Unlock Value from Distributed Data Assets

A $5-billion global fintech leader handling $7TN of fixed income and equities securities trades per day, needed a better way to manage and distribute growing volumes of data.

Its customers, which include 150 brokerage firms and just about every bank on Wall Street, were facing a growing volume of data coming in from every angle – from traditional sources as well as from new ones, including IoT devices and social media.

There was always the option of moving the data to a new place, but that would have meant using a traditional architecture that in the long run could not support a new generation of requirements, such as the need for fast, on-demand access to real-time data.

“You might be tempted just to make a copy of the data and get rid of legacy problems,” explains the firm's Vice President of Development. “A better solution is to build an aggregation layer, an API-enabled orchestration layer. We wanted a single source of reference rather than moving data to one place.”

That layer is part of what is known as a smart data fabric architecture, which has been embraced by the firm to support the weight of current and future data loads. It represents a new and increasingly popular approach to data management and access.

As an InterSystems solution partner, the firm is using InterSystems IRIS® data platform, which provides many of the capabilities to build a data fabric in one product.

The object of the journey is to give the firm users an integrated, trusted, and real-time view of business data, essentially solving data management issues for global enterprises, all of whom need to better leverage their data.

To remain competitive, they need to make more informed decisions, provide better customer service, and personalize their product offerings.

A data fabric architecture does not replace, but rather complements existing systems. An enterprise data fabric is an architectural approach that speeds and simplifies secure access to data assets across a business. It accesses, transforms, and harmonizes data from multiple sources, on demand, to make it usable and actionable for a wide variety of business applications.

Smart data fabrics take this approach one step further by embedding a wide range of analytics capabilities, including data exploration, business rules, business intelligence, natural language processing, and machine learning, making it faster and easier for organizations to gain new insights and power intelligent predictive and prescriptive services and applications.

The firm’s phased approach to building its smart data fabric began with implementation of its data aggregation layer based on InterSystems IRIS. The layer allows for real-time, intra-day, batch, and streaming ingestion. It integrates not only with the firm’s internal systems, but also with those of its customers, including third-party data stores. It also provides data validation and transformation.

The new smart data fabric architecture supports several initiatives, including a next-generation wealth management application, where the aggregation layer must consolidate data from many different account settlement engines. Users need a real-time, consistent view of a client or an account’s assets and obligations using standard APIs.

“They want speed,” says the firm’s Vice President of Development. “They want one-second responses.”

Testing revealed that the new architecture met requirements for the use case, including not only speed, but also the ability to scale to five times current volume, handle two million transactions daily, and store seven years of data. In addition, the client found that InterSystems IRIS provided a 900% improvement in performance using only 30% of the infrastructure, compared with an alternative approach.

While implementing a smart data fabric does not require pulling the plug on any existing hardware or systems, the firm's Vice President of Development does plan on using it “to modernize our footprint” across the organization.

“We will move into building out APIs on the access layer, so that we can decouple the legacy applications and enable new applications using the API layers, and then eventually decompose some of our legacy applications,” he says.

The smart data fabric has been called the future of data management by top industry analysts, because it responds to what the client and many other firms realize they need to do to stay competitive — aggregate and harmonize their data across business lines; create a gateway to the data for complex analytics; consolidate operational functions; and cut infrastructure costs.

Not only will the fabric leverage their data to its full potential, but it will also support collaboration and innovation, and provide the agility to respond to changing market conditions.