Challenges in ESB Implementations

ESB as an architecture style is one of the most sought-after vision-state in current IT landscape for medium to large enterprises. why not? we now have complex environment that has evolved over time and we invent news ways to try and make it simple. Never know whether the implementation has simplified or complicated the landscape from future point of view. The notion of ESB has elusive implementations as people have different views on how to implement it and show the benefits to the business.

This prevailing chaos is acting as catalyst to product vendors to sell their products by inducing un-justified fear psychosis. Many financial enterprises have succumb to this play and have implicitly declared their technology roadmap to the hands of vendors.

How do we understand ESB role within enterprises? How does ESB differ from SOA? Can SOA still exists without ESB? What are success factors for ESB adoption? Many books and articles are written and available on these topics. Many different versions also create confusion. To put it in simple terms ESB products are valued based on the capabilities and abilities it brings to simplify the enterprise IT design & architecture. Often large enterprises have committed and invested heavily in buying vendor products.
There are many cases of adoption that have either failed to give the benefits and even have created problems in implementations. Some of the causes and experiences that I have seen are listed here, hoping it would benefit some one around.

• Undefined Usage Patterns : Many enteprises buy vendor products with vendor documents that define product patterns for adoption and starts believing that the product patterns are the golden goose to chase. It is often short-sighted view showing no vision in adoption within the enterprise. With this gap in where & how to apply ESB within enteprises, Designers and Architects make un-informed decision for the sake of project timelines and other pressures. Many places where ESB are used where they are wrongly applied and some place they are left un-used loosing the opportunity for adoption. These decision aggravate and becomes monster, then some-one recognizes the problem but does not find the business support to fund the adoption or change. Hence, the ESB vision & strategy for the enterprise need to be documented and shared to teams to give clear design & usage patterns. In the absence of it, I see lengthier debates, prolonged decisions, and political pressures building in the enterprise, causing costlier adoption and scraping of initiatives.
• Noisy Design : There are different opinions about what kind of activities to be implemented in ESB design. Many vendor based ESB's provide higher level lanugage & constructs to create mediation flows. Design of mediation flows are usually corrupted to include business logic and un-justifiable dependencies. This makes the design Noisy and brittle.
• Adhoc decision : In many occasion project teams take adhoc decisions to meet timelines and also to accomodate team pressures. Avoid tactical decisions or keep the scope for re-factoring at later iterations if you are following agile methods.
• Product rigidity : Products are built on proprietary frameworks structured to meet common scenario. Any customization on it is like walking blind. Also there are heavy components as they come with built in functionality when you install. Performance demands are high on ESB's, where most proprietary products have challenges. Applying multithreaded processing is difficult to apply due to its inherent rigidity.  ESB's need to be lightweight and pluggable to meet highly distributed nature of its usage. Integrating ESB's in Continuous integration environment is also not well supported by vendor products, and the basic feature for unit test features are also difficult. This is the gap in which open source implementation score more and gives fleixibility in usage.  ESB's need to be lightweight and pluggable to meet highly distributed nature of its usage.
• Distributed Design challenges: As the ESB inherently is distributed in nature, one needs to be clear in how this distribution is applied and the techniques used to ensure reliability, perfromance and maintainability. ESB's are built on messaging backbone, the usage of messaging mechanism, to distribute & consolidate capabilities needs to come out from the strategic thinking. Often there are lack of understanding in teams on how this mechanism helps in distribution, posses big challenge. Associated with distributed challenge is end-to-end visibility of processing. Many problems are faced by teams who use message queuing systems, they complain issue of visibility and randomly adhoc behavior. I have seen in enterprises, support teams still tamper with messages in queues, deleting them in worst cases. Clearly these challenges need clear strategy, with more discipline and right processes supported by custom utilities will help the teams manage the challenges.
• ESB Governance : This has often been the most neglected area in ESB adoption. The challenge will be on how to bring all the key systems on to the ESB in a phased and incremental manner. This needs real implementation strategy. On the one hand there will be many projects wanting to use ESB, the other end there will be limited team capacity to build, there will be trade-offs to do. There are some cases of design corrections to do through re-factorings, support new interfaces etc. Enterprise IT teams are struggling to keep the momentum to meet business pressure. One option is to form an ESB centre-of-excellence that consist of Integration Analyst, Integration developers and designers, and Platform architects. Along with them the project manager coordinates activities with project teams and plans for implementation.
• Skill & Competency: This area often becomes the weak-link in the process. It is common saying in the enterprise that the platform skills and analysis skills fall short, and they become the core reasons for project delays, low quality deliverable, and higher spending. So it is important to identify and build compentency, retain skills, and encourage re-factoring. It also depends on which platforms you use as an ESB. Appserver, WMB, Opensource Mule ESB etc. All have there own programming models, deployment options and management tools. Hence the skills varies on platforms. Important roles that are a worth having are:
• Integration Analyst - The person who analyses details of the integrations, identifies data elements, transformations and rules, completeness of integration flows, dependency analyses. Important deliverable for this role will be Integration Spec.
• Integration Developer - The person is expert in platform tools and techniques, experienced in integration design patterns and have hands-on for implementations. He should be experienced in Webservices, XML, EAI patterns, messaging systems etc. He plays role of designer and developer, responsible for quality of deliverables and meeting timelines.
• Platform Architect - The person is experienced as System and Integration architect, responsible for overall design decisions meeting NFR requirements, ensure de-coupling and mediation are performed at right levels and Services are granular meeting enterprise standards. He is one who identifies design risks, and defined design requirements. The deliverable will be Integration design spec, articulating design decisions, key dependency interfaces, translation mechanism. He defines the integration topology, data & interface standards to follow and platform dependent mechanism to follow.

  ESB platform are many, it is often confusing to select the better one as the capabilites are seemingly similar. Open source ESB products like Mule ESB, WSO2,and frameworks like Spring Integrations, Apache Camel, are quite matured for adoption. Due to complexity in IT environments, platform competencies and Design challenges, there are many project delivery delays experienced. Being aware with above mentioned challenges and practices will help in adoption process.