Integration Technologies - etude de marche

Synopsis

The typical organisation is now extremely complex, with dozens of different applications, databases, and processes all trying to interoperate at once. Although the ideal scenario would be that all these different pieces would come together and act as a single, well-oiled machine, in reality the picture is very different; the pieces of the puzzle are more likely to interfere with each other, causing system failures and crashes, than they are to mesh together. This is a serious business problem, and various approaches have tried to solve it; some of the most recent involve the use of advanced technologies and business models, such as Service-Oriented Architectures and the Enterprise Service Bus.

Deployment of integration is a serious corporate responsibility, and this Report will be of interest to various C-level executives, as well as providing much-needed support to the management teams with the responsibility of making the strategy work. Topics include definitions of the advances that have been made in the integration space, including assessment of how organisations can plan for greater returns on their investment.

Key Findings
The introduction of Service Oriented Architectures (SOAs) has created a focus of integration that takes process rather than information as a primary driver.
A SOA is a conceptual architectural model and as such should be seen as a way of helping to create a more responsive and proactive organisation, rather than a technology solution.
Just as the Enterprise Service Bus (ESB) model has subsumed some of the functionality of other integration related technologies, so it will itself become an element of larger integration technology solution.
The reliance placed on host systems within an organisation should not preclude the bringing of these technology assets into the larger framework.
The first aspect of creating an integration strategy has to be discovering the total assets that exist within the organisational environment.
Simply because there are new architectural models available, this does and should not preclude the use of existing technologies.
The most effective integration strategy will recognise that different integration technologies all have a role to play.
Security in a highly distributed component-based infrastructure requires a different security strategy, but this should not be considered a barrier to implementing such an infrastructure.
Although process is now becoming the primary driver for many organisations the role of information and effective information models must not be ignored.
Many of the technologies introduced during the past few years are coming together into a more cohesive whole.

Management Summary

ntroduction

Integration has always been high on the list of concerns for IT professionals, and over the years there have been

a number of models and technologies that tried to address these concerns; to ease the pain of integration. The

introduction of the Service Oriented Architecture (SOA) model, along with associated technologies such as the

emergence of the Enterprise Service Bus (ESB) and Business Process Management (BPM) solutions has both

helped create the possibility of more open infrastructures, but also muddied the waters.

The problems that now exist are, in the main, due to the coming together of several technologies, and the

uncertainty as to where precisely each technology might fit in a larger integration scenario. One clear

example of this concerns the aforementioned SOA and the older technology of Web services.

To many, there is little (if any) distinction between the two. While there is a great deal of synergy between

SOA and Web services, one is not dependent upon the other. Integration has always (or should always) have

addressed the issue of why integration is necessary rather than how to carry out integration; at least as the

starting point.

The �why� of integration is concerned with the ability for organisations to operate in a more open and flexible

manner. The creation of a total organisational infrastructure; of which the technical infrastructure is only

one element (albeit an important one). Past integration solutions have more closely addressed the issues of

data and information rather than that of process. Whilst the former two are still clearly important, any

integration strategy should not focus upon them to the exclusion of the latter.

Integration has now become a much wider concern than in the past. However, opening up the integration

requirements to a larger audience has created the possibility of making a reactive and proactive organisation

a real possibility.

Another positive aspect of the newer models of integration (such as SOA) is that it is not a �rip and replace�

model; it is firmly entrenched in the re-use of existing assets. SOA is a conceptual architecture, as opposed

to an installable technology, and as such fits well with more traditional methods of integration; all of which

have relevance in creating the required organisational environment.

Business Issues

As mentioned previously, the key to integration is the question �why integrate?� If this question is not asked,

and not answered in a structured manner, then any integration is bound to fail to a greater or lesser degree.

There are several business drivers that make integration increasingly important. Globalisation means that

competition could come from anywhere. Integration, both internal and along the supply chain, could make

the difference between staying in business and losing out. For example, in many vertical markets, supply

chains today tend to be global, with design carried out in the west, but production undertaken in the east

� and there can be strong competition at both ends of this chain. There has to be integration, not just of

data and information, but also of process to make a global organisation most effective.

Legislation is also a strong driver for integration, particularly where there is a need to consolidate

information from across many different applications in a very short timescale. This type of integration is

often more data and information related rather than application related.

Every IT professional knows that integration is not a facile undertaking, and it is not just technology that

puts barriers in the way of successful integration. There are also business aspects that hinder integration.

At the root of this is likely to be a lack of understanding of exactly what applications really are in use, and

what function they perform. This may seem a surprising statement to make, but often even when there is

a good understanding of what applications are in use, it is not always known exactly what function they

really perform, as the application itself may have moved on significantly from any documentation that may

exist. One clear example of this is when the issue of host systems enter into the picture. In many

organisations they have formed the backbone of the transactional systems, and the thought of attempting

to integrate them into a larger framework always raises concerns regarding future operability.

Butler Group believes there is an issue here that organisations really need to come to terms with. The older

host systems appear to have some mythical presence; with complete reliance on the veracity of information

within those systems and produced by those systems. These systems have been modified over a number of

years, and there is little guarantee that the faith placed in them is now well founded. There is a simple but

effective answer to those who would leave host systems alone with the phrase �if it ain�t broke don�t fix it�.

That answer is �the chances are it is broken, and it is only when it is brought into the mainstream of an

organisational operational environment will the extent of its failings become apparent.

Mentioned previously, the importance of integration from a process viewpoint is a clear driver. With many

organisations responding positively to questions about the importance to them of streamlining business

processes it is perhaps still surprising that relatively few have jumped wholesale into integration projects.

An IntegrationWeek survey in the first quarter of 2004 indicated that 90% of financial services executives

surveyed considered optimising business processes a key target for 2004. Yet only 49% of them said that

application integration projects were on their to do lists for the same year. A disparity that is hard to explain.

However, it might be put down to a fear of cost; something that could be overcome with a clearer

understanding of modern techniques, such as the implementation of a SOA infrastructure.

The idea of a Service Oriented Architecture (SOA) is gaining ground, and is reaching the heights of hyperbole

at the moment. What this means from a business perspective is that each application needs to be

understood from the point of view of what services it provides to the organisation. Applications then need

to be �deconstructed� into their component services � each of which performs a specific business function.

Examples of business services are address lookup, customer credit check, and so on. Within an overall SOA

framework, such services can then be re-assembled in such as way as to be much more flexible, futureproofing

the organisation. Whilst the evolution of Web services has undoubtedly supported the concept of

SOA, it has been around for a number of years, but has only recently become a high profile idea.

Integration Technologies www.butlergroup.com

10 Section 1: Management Summary June 2005

Creating an integration strategy based on process as the primary driver (whether or not one wants to call it

SOA) will allow organisations to better understand the benefits of spending money on integration. For those

more forward looking organisations, this cost element becomes subservient to the fact that the organisation

will be future-proofed both in terms of technology advancements and organisation change.

Clearly, whatever approach is taken there are prerequisites that need to be taken into account. Firstly, the

emphasis has to be on re-use. The cost and upheaval of a �rip-and-replace� mentality has no place in today�s

organisations. Secondly, it is imperative to have take a strategic view of integration. From a business

perspective this will have to include an understanding of the relationships that are held with external

partners, and how they will affect both the internal and external strategy. Lastly, integration has always to

be considered as an ongoing task. To this end a framework for integration has to be built. Even without the

SOA model, there is a strong driver to couple systems as loosely as possible and this is an obtainable end

if the framework approach is taken.

Technology Issues

There are multiple technology issues concerned with integration, and the introduction of newer processbased

models should not lead to ignoring some of the more traditional approaches. There is a place within

an integration strategy for multiple models to exist side-by-side.

However, as the technology that is highest on the radar at the moment the joint issues of SOAs and ESBs

need to be addressed; not least because understanding of these will radically affect the total strategy.

A SOA is an architecture model designed to achieve loose coupling between process-based components.

The key here is in the word �Service�. In order to understand a SOA, the definition of a service has to be

created, and in order to do that the concept of a process has to be codified.

A process is a series of explicit events that are joined together by a set of rules in a logical fashion in order

to either fulfil an expectation (provide a mortgage for a customer, as one example) or to change the state of

something (modify information in a structured fashion held within a repository). Each of these explicit

events carry out a specific task or function. In the case of a mortgage application, one task would be to

retrieve financial information about the prospective client. Another related task would be to take that

information and from it apply in-house rules to create a credit-rating; the result of which would then

determine the next task or action to be taken.

Loose coupling, which is the underpinning basis of a SOA, leads to concerns; not least of which is that of

security. A security model based on tight integration is easier to implement than one based on loosely

coupled components. However, there are methods and technologies available to bring security into this

loosely coupled and highly distributed world.

Closely allied to SOAs are the concept of the ESB. This is an extension to the bus messaging model, and has

subsumed some of the technology that existed as a node on the old bus model into a piece of core functionality.

One prime example of this is the use of an integration server that in the past would exist as such a node and

would use adapters to carry out required transformations. This function can now exist as part of the ESB.

This leads to the issue of precisely what an ESB is. Different vendors would put forward different ideas.

There is a basic split between those that promote ESB as an installable product, and those that treat it as

an architectural model. There are arguments for both ways of thinking, but at the end of the day, if one

considers the functionality of an ESB and applies that to an integration strategy then the arguments become

more of an irrelevance.

The functionality of an ESB (whether productised or as an architectural style) has a set of core

characteristics, which defines it. These characteristics include:

Messaging.

Transformation services.

Content-based routing.

Basic connectivity to defined standards (Web services, J2EE Connector Architecture, Java Message

Service, etc.).

Support for highly distributed environments.

Although there are other functional elements that can be implemented within an ESB, the preceding would be considered a minimum �must have�.

The ESB, or at least the idea of an ESB, is so central to many people�s understanding of a SOA, we have devoted the Technology Evaluation and Comparison Section of this Report over to that topic. Similarly, we have extended this to look at vendors who also have an ESB product or who talk about an ESB style within the Vendor Profile Section. Although the ESB is not the be-all and end-all of integration, as this Report demonstrates, it is a factor whose inclusion needs to be examined closely.

Bringing the ESB (product or model) into the total infrastructure has an effect on host systems, and this needs to be clearly understood. The ESB exists as part of a highly distributed system model, which is the complete antithesis to the basic understanding of the role of host systems.

The more traditional role of middleware and associated technologies cannot be ignored. Although there may be those who talk about service or process-based architectures to the exclusion of everything else, a more pragmatic approach is to realise that these architectures will, at least for some time, have to co-exist with other already implemented models.

The synergy between emerging technologies and existing ones has been mentioned previously, and nowhere is this more important than when one looks at the role XML has to play. Once promoted as the answer to all the integration problems, it has now found its correct place within the grand scheme of things. This is an important technology, but, as with many other technologies, it is only one part of the complex jigsaw.

Much mention has been made of an integration strategy and this needs expanding to provide context. The route chosen for integration will be highly dependent upon both the current implemented infrastructure and the purpose for the integration. Purpose being the keyword; integration should never be carried out for integration�s sake. If one is attempting to integrate thousands of applications in a large-scale integration project, it is likely that a different architecture will be selected to that chosen for a small-scale integration project that is attempting to link three or four diverse systems together. Selecting the right topology can make a difference in terms of integration performance, management of unforeseen events, and maintenance costs into the future. Making the decision about which is best for a particular environment will depend on many aspects, such as the number of systems to be connected, the volumes of data to be shared, and whether integration has to be real-time or is time-independent.

Again, it is important to understand the total integration scenarios available. It might be considered forward thinking to only consider service-based integration, but there is still a need for some of the more traditional technologies, such as point-to-point integration, data integration, or the creation of a hub-and-spoke model. Although we have stated that integration has to be considered from the point of view of process, this does not deny the requirements for other integration necessities. As the vast majority of organisations can be seen as information based, then the creation of an integrated information model has also to come under serious consideration.

The total integrated architecture built will incorporate elements of multiple technologies, each one working within a specific area. This is the major change in integration that has happened in the past few years. No longer are organisations faced with implementing a single model, which to a large extent forced a lowest common denominator architecture onto the organisation.

Market Issues

The integration market is now relatively mature, with some well-established players competing for frequently large budgets. However, there has been much consolidation over recent years, and there is still a lot of pressure on vendors. The first few years of this century have proved very difficult for a number of the well-known players, and only the last year has shown some slight improvement. For many of the newer entrants, there have been some good customer wins, but profits are still in the future.

The ESB market has allowed some of the more niche vendors to gain a presence, but how sustainable this will be in the long term is debatable. Given the duplicity of view as to the precise nature of an ESB, it is likely that vendors in this space who do not extend into the larger integration market will fall by the wayside.

The market size for integration today is some US$5 billion, according to figures from Datamonitor.

Enterprise integration software spend is likely to show modest growth during 2005 of 4.23%, and will then rise over the next three years at a compound average growth of 4.9%. North America is still the most significant market segment geographically. Within Europe, Germany and the UK are the biggest spenders on integration technology.

Datamonitor has also investigated revenues in the various different vertical sectors, and an interesting picture is revealed when examining data in this way. Contrasting 2003 with 2008, we see that the highest spending area, financial services, which is certainly one in which many integration vendors place a significant emphasis, actually sees falling revenue projections. The largest likely growth market is manufacturing, and the reason for this is likely to be related to the large spending hike in that sector for Y2K-related issues. This led to a number of years when that sector did not spend significantly on technology, and it is gradually starting to see the benefits of integration and thus expenditure may rise.

As with all markets, there will be an �ebb and flow� of investment in integration. To a large extent this measurement will depend upon the definition one gives to integration. Just as the Business Process Management (BPM) space is being attacked by pure-play integration vendors, so will BPM vendors start to encroach on the �integration market�, especially as organisations move towards the process-centric approach to infrastructure implementation.

Butler Group Integration Technologies Market Lifecycle Ratings

Butler Group�s vendor ranking and assessment model groups suppliers into Outperform, Perform, and Under-perform categories, and shows the predicted progress through the three major market phases of Early Adopter, Market Adoption, and Market Maturity.

Report Structure

Report Structure

This Report is intended to provide readers with an informative guide to integration. Butler Group recognises that not every factor will be relevant to all situations, and that some organisations will already have implemented integration technologies. Therefore, the Report has been divided into segments relating to the different topics that relate to integration strategy, to make it easy for readers to locate the sections that relate to their particular issues. These sections are:

Section 2 � The Integration Imperative

This section discusses why integration is needed, takes a high level look at integration types and requirements, business issues that hinder integration, and covers making the case for integration.

Section 3 � Technology

The importance of Service Oriented Architectures (SOAs) and the Enterprise Service Bus (ESB) model is discussed in some depth. Other important considerations under discussion are how testing and security will operate in the highly distributed loosely coupled environments. More traditional approaches to integration are not ignored and the role of middleware is also addressed, with special emphasis placed on the Java 2 Enterprise Edition Connector Architecture standard.

Section 4 � Architectures and Models

Integration strategies are considered with the introduction of how multiple integration types can co-exist in the total environment. Although much of this Report looks at the importance of process in establishing an integration strategy, there is also the requirement to build effective information models.

Section 5 � Market Analysis

The Market Analysis section examines what drives the integration market, the evolution of that market over recent years discussing the rise of the ESB concept, and then concludes with projections for the future of the market.

Section 6 � Futures

This section looks at integration moving forward with special emphasis on how Business Process Management (BPM), SOAs and composite applications fit together to help create a more responsive architecture.

Section 7 � Tables

This section first presents Butler Group�s Integration Technologies Feature Matrix for ESBs, which permits the selected ESB offerings to be reviewed side-by-side in terms of the features and capabilities of those solutions.

Section 8 - Vendor Comparisons

Here we include comparisons of the products that have been reviewed as Technology Audits for this report.

Section 9 � Technology Audits

This section provides in-depth Technology Audits for the vendors that have been reviewed for this Report.

Section 10 � Vendor Profiles

This section contains a short product profile, supplier summary and contact information for vendors that have potentially useful or important technology offerings in the integration market, but do not necessarily provide Enterprise Service Bus technologies. The product profiles are provided as an indication of what is obtainable in the market and should not be used as a comprehensive review of all the available offerings.

Section 11 � Glossary

This is a glossary of terms used in the Report.