Executive Summary

BloorHistorically, software developers wanting to create engaging applications for their customers have had a fairly straightforward set of requirements:a flexible and productive development environment and a high performance database that could be embedded, fired up, and forgotten about. That is, a database that was reliable and didn’t require tuning.

However, the days of delivering traditional applications that focus on automating relevant business processes are, if not gone, then certainly going. This is not to say that this automation is not valuable. It is simply that just about all the conventional processes that can be automated have been automated. Software providers therefore need to be thinking about how they can deliver what are known as “breakthrough applications”.

What characterizes a breakthrough application? Perhaps, most typically, it changes the way that the user does his work. In other words, it doesn’t just automate processes; it innovates and alters them, often radically. Indeed, breakthrough applications can have such a dramatic effect that they can change the way an entire organization functions. In our view it generally encompasses three major considerations:

  1. Mass personalization: a breakthrough application needs to deliver whatever information the user requires, in whatever format is most appropriate and on whatever platform the user requires. It is no longer sufficient to simply make an application available from a PC. In today’s world, web and mobile access are equally essential.
  2. Access to all the data: breakthrough applications need to be able to access any type of data, regardless of whether it is conventional structured data, content (text, images and so forth) or information gathered from devices such as smart meters. In the past, applications have tended to be purely transactional or wholly devoted to content, for example. In effect, applications have been limited to isolated silos of data rather than spanning all relevant information regardless of its format.
  3. Driving informed action: breakthrough applications drive intelligent action. As we have noted, breakthrough applications go beyond mere process automation and thus there will always be a human element associated with such applications: decisions to be made, actions to be taken, and so on.

The purpose of this paper is to discuss the concept of breakthrough applications in more detail. Further, we will consider the implications for the underlying technology elements (development environment and database) that will underpin these applications. The historic requirements for these elements of a solution, as outlined above, remain valid but there are additional functions that will be needed to support breakthrough applications. Finally, we will consider how one software technology vendor in particular, InterSystems, meets those requirements for supporting breakthrough applications.

Mass personalization

Mass personalization is, of course, a contradiction in terms. So, it will be as well to define what we mean before discussing the matter further. By “mass” we mean the ability to allow application providers to cater to the varying and exact standards of the public (masses) and by “personalization” we mean the ability to tailor the application to the specific desires of each user.

Specifically, users today expect applications to provide the information that they want, when they want it, and how they want it. Users are demanding that applications and information are available on their choice of device rather than the device type being mandated by the application provider. Users also have exacting requirements about how the data is presented to them: that is, in a customized manner tailored exactly to their individual preferences.

Given the rapid growth we have seen in user interfaces, this has a significant impact on the developer of advanced applications. They must not only be able to support today’s devices and interfaces but also have the agility to rapidly and simply introduce support for new devices and interfaces as they emerge, all while providing a consistent user experience. Additionally, advanced applications must have the scalability to provide the individualized customization demanded by users.

In order to achieve this flexibility with respect to the end user there is one key requirement—the logic underpinning the user interface should be isolated from the code that provides the functionality of the application, which should be isolated from the data. This independence means that any one of these elements can be amended or added to without affecting the other parts of the application.

We should note that this requirement for a three-tier model (interface logic, code and data) is by no means new. However, the introduction of new ways of interacting with applications (for example, using a touch screen instead of a keyboard or mouse) means that this need is more acute than it used to be, especially with respect to the separation of user interface considerations from other aspects of development.

Access to all the data

If mass personalization is about extending what was already a requirement, the ability to access all types of data represents a more fundamental shift.

There are fundamentally three types of data to consider: structured, unstructured, and event data, all of which may need to be included within, or accessed by, a breakthrough application.

  1. Structured data: the most common example of this is transactional data. Structure is imposed upon the data by the way in which we capture, store, and process that data—in discrete format within objects or tables, and with a well-defined schema. It’s relatively easy to perform searches and queries on this type of data, and the semantics of doing such searches/ queries are well defined and understood.
  2. Unstructured data: the most common example is text, though this also extends to video, audio, and other rich datatypes. A significant number of applications store portions of their data as unstructured text that is typically implemented as plain text, in rich text format (RTF), as XML, or as a BLOB (Binary Large Object). Video and audio are typically stored as BLOBs. This category of data is generally a few orders of magnitude larger (in size and quantity) than structured data. The most notable point about unstructured data is that it is typically very hard to extract meaning from this content. Searching and querying through this data typically involves a lot of wasted cycles (raw searching and scanning) and, what’s worse, it’s often impossible to make conceptual meaning out of the data.
  3. Event data: there are an increasing number of environments that require the processing of event data, by which we mean the capture of that data; its analysis, perhaps in conjunction with structured and/or unstructured data; and then the process of acting on that information, which is often, though not always, automated. Typical examples of event data include stock ticks, clickstream data, sensor data, log data, and so on. Sensor data, for example, is expanding rapidly as technology takes us into a “smarter” future. There is an on-going explosion in the number of peripheral devices which automatically generate regular/frequent sensor data, and then feed this into the main application database. For example, hospitals have been equipping monitoring systems (such as blood pressure monitors) with the ability to automatically feed the readings and results for a given patient directly into that patient’s chart. There are also similar monitors for heart rate, blood oxygen levels, and so forth. Nor is this limited to healthcare. Smart meters of all types are becoming commonplace—for example, power and utility companies are starting to electronically track power consumption in our homes in order to adequately predict surges in usage so that they can ensure adequate supply. The requirements for storing and managing this sort of data are different from both structured and unstructured data. For example, it is often a change in condition that you are interested in with respect to sensor data. It is not always necessary to store all the “steady-as-you-go” information, so you need some sort of filtering mechanism prior to storage. Further, it is often the case that you need to know the order in which events occurred in order to analyze them properly. This means that event data may need to be stored with timestamps. More generally, the sheer frequency of the generation of this data not only causes an explosion in data acquisition and retention requirements but also creates a need for various searching, sorting, and querying capabilities to make it meaningful within the context of the relevant application.

Traditionally, relational databases have been used to store structured data, content management systems have been employed to support unstructured text data, and event stores (most of which are either flat file systems or column-based databases) have been deployed to hold event data. To some extent that picture is changing, especially with the introduction of NoSQL (‘not only SQL’) databases supporting the analysis of unstructured data. However, it doesn’t change the fundamental issue that you need three separate mechanisms for storing and managing these different sorts of data. This may be fine within a federated data warehousing environment where all you want to do is analytics, but if you want to deliver breakthrough applications to your users you can’t afford to do it using a platform based on three different data stores. You need a single environment that can handle all of these different data types.

Even if you have a single platform capable of handling all these different types of data, it is still likely that you will want to use data from elsewhere. For example, you might want to enrich the information you have with geospatial or location-based data, or you might want to access sources such as Dun & Bradstreet or Experian. For this reason, you will also need a level of integration with third-party systems for accessing and gathering information, whether this is from the Internet or other corporate data sources.

Driving informed action

Breakthrough applications must be capable of driving intelligent action. This can be achieved by an on-going business cycle that involves some or all of the following steps:

  • First, the application should be able to capture data in an appropriate and reliable format whether derived from people, systems, devices, and so on.
  • Next, this captured data needs to be shared throughout the enterprise, freeing it from ‘silos’ and making all the data available to users.
  • Third, the data needs to be digested by the application and turned into something meaningful and useful. This means that the application must be able to understand all of the data, regardless of the type of data—structured, unstructured, or event-based.
  • This leads to the fourth step: the application must enable users to act upon the insight gleaned from the data.

Real-life “Breakthrough Application” examples

Before going on to take a brief look at InterSystems’ technology and how it supports the requirements we have discussed, it will be useful to give some examples of breakthrough applications that existing partners of InterSystems have built.


Related Content

The Optalert Vehicle System is used by many of the world’s largest mining and road transport companies. It is the only real-time safety system in the world that detects the early onset of driver drowsiness by accurately measuring a person’s level of alertness, quantifying to both driver and company their fatigue risk profile. Vehicle operators wear specialized Optalert driver glasses that measure the velocity of the wearer’s eyelid 500 times per second using tiny infrared emitters and receivers built into the frame of the glasses. As may be imagined, this generates huge amounts of data, which is stored in and managed by the InterSystems Caché database. The system uses InterSystems DeepSee, not only for generating real-time alerts (of drowsiness) but also for company reporting, trend analysis, and to detect opportunities for improvement. The Optalert solution is available either via a software-as-a-service business model or as directly licensable software.


Related Content

BTC Business Technology Consulting AG used InterSystems’ integrated technology to develop a breakthrough application for managing the ‘smart energy grid’ in Germany. Smart meters installed in homes and businesses feed a steady stream of data to the Advanced Meter Management system, where the data is analyzed and the results used to efficiently allocate energy resources in real time, so that less is wasted. In accordance with European Union market guidelines, 80% of all households in Germany should be equipped with smart meters by 2020. This represents some 32 million households and an enormous amount of event data that must be captured, analyzed, and acted upon.


Related Content

Press Computer Systems (PCS) is a UK-based supplier of what it describes as “clever solutions for anything but ‘me-too’ publishers”. In particular, it provides an integrated multimedia and digital content management system, known as PCS Knowledge, to the news publishing industry. PCS Knowledge, which has been built using InterSystems’ technology, empowers journalists, editors and production teams to publish news in multiple formats simultaneously because it decouples content from the formats used to convey that content. PCS Knowledge consolidates pagination, wire service management, picture desk, and digital archive management into a single system, and provides intelligent workflow for easier, faster, and richer multimedia publication. In addition, PCS Knowledge includes the capability to analyze, associate, and store related unstructured content including text, sound, video, images and so forth. This allows the application to automatically retrieve and display related content to journalists as they write, and to route articles to appropriate editors based on the content of those articles.

The three preceding user examples illustrate not just the concept of breakthrough applications, but also how InterSystems’ technology enables these applications. Therefore, we will take a closer look at the company’s core products, which work together as an integrated whole.

InterSystems Caché

Caché is, on the one hand, a database (strictly speaking, a database management system) and, on the other, a rapid application development (and deployment) environment. As a database, the product uses a multidimensional storage engine. This means that if you want to use it for relational tables then you can do so, because a table is just a two-dimensional array. Alternatively, the engine supports an object-oriented approach that allows you to store business objects (customers or patients for example), as complete entities without having to disperse this information across multiple tables, while avoiding all the complications of referential integrity that you would require when using a relational approach. Caché also supports direct access to the data structures (called globals) within its multidimensional storage engine, providing very high performance to applications that need it.

The Caché database is well-suited to storing XML, as well as unstructured data of various types. Moreover, InterSystems iKnow technology provides the ability to discover concepts and relationships within unstructured data that is stored within Caché. Its distinct advantage over conventional qualitative analysis techniques is that iKnow does not require a predefined dictionary or ontology.

Caché is highly scalable, has advanced synchronization and replication capabilities, and has one of the most elegant implementations of high availability we have seen in any database product.

As a development environment, Caché supports SQL as well as a wide range of popular programming languages (including C, C++, Java, .NET, Python, and so on). In addition, Caché largely automates the creation and consumption of web services, so it can fit nicely into a service-oriented architecture (SOA).

There are also Caché-specific development capabilities that take advantage of tight integration with the database to provide superior performance. These include:

  • Caché ObjectScript, which allows database access via both object technology and through direct manipulation of globals.
  • Caché eXTreme for Java, which provides direct access to globals from Java, so that Java developers can use their preferred IDE to build very high volume, very low latency applications suitable for handling event data.
  • InterSystems Zen technology, which is an extension of AJAX (Asynchronous JavaScript and XML) that can be used for developing rich web-based applications.

Note that you are not limited to any particular choice; it is common, for example, for Caché developers to mix and match the different access methods supported, depending on requirements.

What capabilities of Caché make it particularly useful for developing breakthrough applications? There are several:

  1. Caché provides the database performance, scalability and reliability needed to support breakthrough applications. Its flexible multidimensional data structures can store both structured and unstructured data. The option for direct database access can provide the high performance needed to handle event data.
  2. Breakthrough applications almost always have web-based user interfaces, and/or rely on web services to enable them to run on multiple platforms, including smart phones and tablet computers. Caché includes capabilities for rapidly creating web services and rich, interactive, web interfaces.
  3. The vast majority of data collected and stored by applications is unstructured. With the introduction of InterSystems iKnow technology, Caché enables breakthrough applications to access, analyze, and use unstructured data.

InterSystems Ensemble

InterSystems Ensemble is a single, architecturally consistent, integration platform (integration server, data server and portal development software) that uses the Caché database for persistence. It enables rapid integration and development of new connectable applications. You can also use Ensemble when existing software needs to be extended or enhanced, as the product includes workflow, browser-based user interfaces, dashboards, and rules-based business processes that can be implemented without rewriting any existing code. It also leverages existing software investments through composite applications, and enables establishment of an enterprise service bus (ESB) or SOA infrastructure.

With regards to the development of breakthrough applications, Ensemble provides two important capabilities:

  1. Ensemble allows developers to connect across all applications—and thus access all data—across the enterprise.
  2. Ensemble allows developers to create the mechanisms through which users can take informed action.

InterSystems DeepSee

InterSystems DeepSee is an active analytics engine that works with both Caché and Ensemble. It provides a development environment for embedding real-time business intelligence capabilities into operational applications without the need for a data warehouse or any other form of extraneous data store. That is, you take live data from your transactional system and perform relevant analytics on it directly, without needing to store it somewhere else first (though you can pull in historic data from a warehouse if you want to) thus allowing developers to build BI functionality directly into transactional applications.

The combination of DeepSee and InterSystems iKnow technology enables active analytics using both structured and unstructured data, and thus has the potential to expand the world’s understanding of what breakthrough applications can do.


In this paper, we have discussed the requirements for breakthrough applications. Of course not every such application requires every element of these requirements but, as we have stated, mass personalization, access to all the data, and driving informed action are all key requirements.

InterSystems does not claim that a single product can provide all you need to build all your breakthrough applications. What it does claim is that Caché is capable of acting as the underpinning for all of the requirements you might have for breakthrough applications. This can be fleshed out through the deployment of Ensemble or DeepSee, and the use of iKnow technology, either singly or in combination, in order to meet your breakthrough application needs. We do not know of any other vendor that offers to do all this, not to mention supporting high availability, disaster recovery, and cloud-based (SaaS) solutions, on the basis of a single integrated technology.

Further Information

Further information about this subject is available from