Integration Technology Primer

SECTION #1: INTEGRATION – THE BUSINESS CASE

What is Integration?
Integration of enterprise applications and data to simplify and automate business processes has become an important focus for many CIOs. It refers to both the tasks of integrating data and applications, as well as to software products that provide integration frameworks and associated tools. Integration enables the sharing of data and business functions across applications.

Integration has become popular because most traditional enterprise applications were custom built to address a specific business need. As enterprises have grown, and the need to share information across departments and business areas becomes more critical, companies are turning to integration to provide a method for interconnecting these distributed and often proprietary systems.

Keep in mind that integration projects come in a wide variety of shapes and scopes. Most projects can be classified as belonging to one of the following four main categories of integration initiatives (defined in section #2):

Data Coordination/Data Consistency
Business Process Management/Orchestration
Business Activity Monitoring
Composite Application Development

Why would I want to do it?
All types of companies today seek to become more agile, responsive, and to maximize their existing IT investments. In order to do so, they need to unlock information and functionality in individual applications and turn them into a shared, enterprise-wide resource. Integration solutions unlock existing information assets and share them across multiple applications and business processes, providing the framework for real-time enterprises and real-time business networks. Integration solutions also enable many companies to create a “single view” of all their enterprise data and an infrastructure for ensuring that applications can exchange and update business-critical data no matter where it resides.

Another way to look at it is that integration technologies enable an organization’s IT infrastructure and applications to rapidly respond to business changes by providing a dynamic way to streamline, integrate, and manage previously independent business processes.

In addition, integration solutions allow you to take stand-alone applications and turn them into resources that can be used by other applications or business processes as part of a composite application. (applications that are composed of both new code, processes, or user interfaces along with calls to existing and legacy applications and services). By enabling the creation of composite applications, integration solutions allow businesses to rapidly implement new applications while increasing an organization’s return on previous investments by leveraging existing application functionality in new ways.

What are the benefits of integration solutions?
There are several benefits of using integration solutions. One is cost savings, courtesy of streamlined business processes and increased efficiency. Integration approaches also help contain costs by allowing organizations to continue to use the data and functionality embodied in their existing applications instead of “ripping and replacing” legacy systems. But integration projects can also have forward-looking benefits. Organizations can gain an instant, real-time view of all their data and operations, which can lead to better decision-making. And integration products provide the flexibility to quickly adapt business processes to accommodate growth and meet new business challenges as they arise.

What are the specific business drivers for today’s integration projects?
There are four major reasons why organizations are implementing integration projects:

New systems haven’t replaced legacy systems.
New enterprise applications (such as SAP, Seibel, etc.) that have been deployed typically do not replace existing legacy applications, but rather they complement or extend them, often generating a need for integration or connectivity between them.
The need to consolidate and globalize.
Mergers and acquisitions have left many organizations with a broad array of mission critical systems that simply can’t be abandoned or replaced. Instead, integrating them and putting systems in place to exchange information among an organization’s various applications becomes a much more realistic solution.
The search for increased productivity.
The economic situation has forced many organizations to search for ways to increase productivity and reduce costs. By using integration platforms to automate existing business processes and applications, organizations can streamline their business while increasing productivity and reducing costs.
Raised expectations from Web experiences.
Regardless of your industry, customers now expect organizations to have a consolidated view of their accounts, transactions, and any other related information – regardless of where that data actually is. In order to stay competitive, organizations need to provide better availability of data and applications to both internal users (such as customer service representatives or salespeople) and to external users (such as customers or partners).

What are integration solutions used for?
Most traditional enterprise applications were built to address a specific business need. As enterprises have grown, and the need to share information across departments, divisions, and globally throughout the enterprise has become critical, companies are turning to integration solutions to provide a method for interconnecting these distributed and often proprietary silos of information systems.

Integration became particularly popular after organizations began deploying packaged enterprise applications like SAP, Seibel, PeopleSoft, JD Edwards, and others. Once companies began deploying these new packaged applications, they soon realized that they needed ways to dynamically exchange information between them and between their existing legacy applications. The need for data consistency among applications was soon followed by the requirement to automate and streamline business processes that crossed between these different systems in order to improve efficiency.

Integration software products evolved to support this new business driver adding support for human interaction and workflow as well as business activity monitoring (the process of monitoring applications and business processes in real time, rather than loading information into a data warehouse for future analysis). This evolutionary integration path allowed organizations to implement entirely new application functionality that builds upon their existing systems, through the development of composite applications.

SECTION #2: THE MANY FACES OF INTEGRATION

What are the four basic types of integration projects?
It might be better to say that there are four basic categories of integration initiatives (further defined below):

Data Coordination
Business Process Orchestration
Business Activity Monitoring
Composite Application Development

Many integration projects will involve more than one of these categories – a fact that should be considered when choosing an integration platform.

What is Data Coordination and what are its benefits?
Data coordination moves transactional data between applications in order to provide consistency.

This core technology set, often referred to as a message broker or integration server, is used as the foundation for every category of integration project described above. Data coordination is enabled by technology that connects applications and data stores to a common middleware platform (or bus), routes the message-based information between these different systems based on content, rules or subscription and dynamically transforms the data so that it is in the native format of the target system(s).

Data coordination transforms application-specific information into a shared enterprise resource reducing the costs and clerical errors associated with entering and maintaining consistent information across disparate systems. The result is improved data quality and increased productivity by ensuring consistent data access across all applications.

What is Business Process Orchestration and what are its benefits?
Business Process Orchestration (also called Business Process Management, or BPM) is the combination of the process re-engineering concepts of the 80’s and the workflow technologies of the 90’s.

Business Process Management (BPM) is focused on modeling and orchestrating the workflow between individual functions in different applications in order to automate and streamline previously independent business processes. It can be thought of as the “integration logic” of a cross application workflow.

In many cases, BPM is primarily focused on “long running” business processes that may take hours, days, months or longer to complete, as well as business processes that span multiple applications and include interaction with users (often called workflow). For example, the process of shipping a computer to the customer of a mail-order computer company is a long running business process that starts with the request-to-ship at the computer company’s warehouse and ends when the return period has ended, which might be 30 or 60 days. The process itself might span both the back office warehouse application as well as the customer support and order entry applications.

Managing long running processes requires that state of the processes be maintained through the entire duration—be that hours, days, weeks, or months. BPM tools typically include modeling, automation and management components as well. The modeling components allow business analyst to define, view, or manage individual processes without having to understand the underlying technical details of how the process is actually implemented.

Business Process Management enables organizations to automate and streamline their manual processes and fuse previously autonomous operational and management systems. The result is a more agile business and efficient business better able to respond to changes in the marketplace.

What is Business Activity Monitoring and what are its benefits?
The goal of Business Activity Monitoring (BAM) is to provide management with immediate awareness of changing business events across the enterprise, so that appropriate and timely decisions can be made. Real-time, event-driven business activity monitoring solutions provide real-time alerts via graphical dashboards and other notification mechanisms, enabling management to immediately react and intervene when key performance indicators change. BAM solutions also complement BPM solutions by providing real-time process monitoring capabilities that enable BPM systems to dynamically and immediately react to changes in the business environment.

By using real-time information to remove delays in managing and executing an enterprises critical business processes, BAM reduces costs and enables faster execution of business processes.

What is Composite Application Development and what are its benefits?
A composite application is an application or new application functionality that combines the data and functionality of an enterprise’s existing applications and data stores, adding new business process logic, custom code and user-facing front ends. Composite applications leverage individual functions or business processes within back-end operational systems and use them as components to build new applications or application functionality. Composite Application Development uses a service-oriented development approach, whereby existing functions and business processes, available as services, are combined in support of new application functionality.

Composite applications enable organizations to gain greater re-use from existing applications and IT investments, facilitating quicker time-to-market for new solutions that rapidly address changing business dynamics and challenges.

Does my integration platform really need to be capable of supporting all four categories of integration?
Yes. Even if you don’t think you need all four integration capabilities right now, the chances are you will in the future. New business opportunities breed new integration projects, or what looks like a small project normally expands dramatically in scope once the initial business benefits are realized. It is therefore usually better to choose a complete integration platform than to try and assemble and manage all the necessary capabilities piecemeal. This technology assembly approach dramatically increases the complexity and costs when integration projects evolve and new technologies are required to implement new integration strategies.

But the “complete” integration platforms – aren’t they difficult to use?
They can be. That’s because some complete integration platforms are really nothing more than “suites” that represent assemblies of formerly independent products. A comprehensive integration platform with a consistent architecture will deliver ease of use – a prime consideration when choosing an integration platform. The easier your platform is to use, the faster your integration projects will be completed. Section #3: “Understanding Integration Platforms” gives more detail about the capabilities that a complete integration platform should provide.

SECTION #3: UNDERSTANDING INTEGRATION PLATFORMS

What are the basic functions of an integration platform?
There are seven capabilities that a comprehensive integration platform should provide. They are:

Connection
Abstraction
Coordination
Storage
Orchestration
Development
Management

Let’s take them one at a time.
What is Connection?
Connection provides a channel of communication between the integration platform and each of the applications that are being integrated. Most integration platforms achieve connection by using adapters. Adapters are pre-written pieces of code that work by speaking the native protocols of the target resources and connecting them to the integration platform or message bus. The use of adapters greatly speeds integration and can significantly reduce costs associated with writing code for each connection.

When choosing an integration platform look at the number and types of adapters it includes. Can the platform connect to your existing applications? Is there a mechanism for building custom adapters should you need to? How easy is it to build and use adapters?

What is Abstraction?
Abstraction is the process of representing the data and functionality of your existing applications in some consistent form. This alleviates the need to have to understand and work with the diverse and different protocols, data, and programming models used by each of the applications you are integrating. Some common approaches to abstraction are:

Using J2EE.
This approach can work well for applications and business logic, but it isn’t as good at dealing with data.
Using XML and Web services.
This approach works well for loosely coupling applications that are already Web service enabled. Legacy and custom applications may need to be retrofitted with Web service“wrappers”.
Using “canonical form”.
Canonical form basically means a standardized representation of the data or functions. Putting something into canonical form would mean that each set of data and application functionality would be translated into a standard format or representation. This approach can be very flexible and powerful, if the canonical form can be exposed or projected in various standard-based object and data formats concurrently e.g. Java, .NET, Web Services, Relational, C++, etc.

When choosing an integration platform, consider the flexibility of its abstraction. Is it easy to represent both data and functionality? Can it represent your legacy applications without requiring that they be rewritten? Do you have both J2EE and .NET-centric applications within your organization? What happens if your Java-based organization acquires a company where all the applications or development are based on .NET?

What is Coordination?
In an integrated system, information is passed between disparate applications in the form of messages. Coordination is the capability to manage the flow of messages in your integrated system. Most integration platforms include message brokers that are responsible for guaranteed delivery, data transformation, and intelligent routing of messages.

When choosing an integration platform, the performance and reliability of the messaging engine is of prime importance, as these will directly affect the scalability and robustness of your integrated system. But also consider the messaging engine’s capabilities: Does it support synchronous, asynchronous, and publish-and-subscribe routing? Does it support content-based routing? What data transformation tools are available and how easy are they to use? Is it integrated with a message warehouse for persisting all messages?

What about Storage?
Integration generates data (metadata, messages, state information of long-running processes, indexes on federated databases, etc.) and every integration platform needs a mechanism for storing, retrieving, and analyzing data. Unfortunately, many integration platforms include only rudimentary capabilities for storage. In those cases, in order to gain robust, scalable storage, you must purchase and integrate an external database, thus adding cost and complexity to your integration project.

When choosing an integration platform look for one that has a built-in shared metadata repository for all the integration touchpoints that will be generated and used by the integration platform. Also look for one that has a built-in message warehouse, and pay particular attention to its capabilities for analyzing the message data in support of reporting and activity monitoring. How well does the platform’s storage mechanism support service-oriented development of composite applications? Does it support distributed environments? Will it impede the overall performance and scalability of the integration platform under heavy load?

What is Orchestration?
Orchestration is another name for modeling and automating business processes. It can be as detailed as writing the code that tells the messaging engine what to do and when to do it. However, orchestration normally refers to business process modeling, whereby business analysts define and model graphical process flows without thinking about the underlying coding complexities involved in automating them. The orchestration engine is able to “read” these process models and execute them. Most integration platforms come with built-in graphical tools for process modeling. A few of them enable standard-based XML-based code to be generated from the process diagrams, thus allowing interoperability with third-party modeling tools.

When choosing an integration platform, consider how much flexibility it gives you with regard to orchestration. Does it provide a powerful graphical modeling tool? Does it support interoperability with third-party BPM products? Does the integration platform support direct coding of processes, and what scripting language does it use? How easy is it to orchestrate complex business processes? Does it support human workflow interaction? Does it have a rules-based engine that provides analysts with a quick and easy ways to define rules that change the process flow?

Why is Development important in integration projects?
In the context of integration, development normally refers to building composite applications and portals, often the ultimate deliverable in an integration project. Composite applications make use of all of the capabilities of the integration platform we have described so far, adding new business logic and user-facing front ends to the abstracted services and business processes previously discussed.

When choosing an integration platform, look for the same capabilities that you would want in any rapid development environment. Is the environment easy to use? Does the development environment support business process modeling? Can you build all the components of your integration solution using the same development environment (e.g. adapters, message maps, transformations, business activity monitoring solutions, etc.)? Does it support the development technologies you already use? Does it work with your favorite third-party development tools?

And Management?
Integrated systems are amongst the most difficult to manage because of their loosely coupled nature. It is therefore vital that your integration platform provides good end-to-end management capabilities, both during development, and during operation of your integrated system. Ideally, your integration platform will keep a record of every single message that passes through the system in a message warehouse. Analysis of the message warehouse allows for quick debugging and identification of problems. Other important management features include: configuration management, queue and process monitoring, detailed event logs and usage histograms.

When choosing an integration platform, consider the completeness and ease-of-use of the management tools. Does the platform save enough diagnostic data? Can that data be analyzed in real time? Are the management tools extensible? Does the platform support the use of standard based (e.g. SNMP) third-party management tools? Can the messages that flow through all business processes be traced?

InterSystems Corporation
World Headquarters
One Memorial Drive
Cambridge, MA 02142-1356
Tel: +1.617.621.0600
Fax: +1.617.494.1631
www.InterSystems.com