Challenges in ESB Implementations

ESB as an architecture style is one of the most sought-after vision-state in current IT landscape for medium to large enterprises. why not? we now have complex environment that has evolved over time and we invent news ways to try and make it simple. Never know whether the implementation has simplified or complicated the landscape from future point of view. The notion of ESB has elusive implementations as people have different views on how to implement it and show the benefits to the business.

This prevailing chaos is acting as catalyst to product vendors to sell their products by inducing un-justified fear psychosis. Many financial enterprises have succumb to this play and have implicitly declared their technology roadmap to the hands of vendors.

How do we understand ESB role within enterprises? How does ESB differ from SOA? Can SOA still exists without ESB? What are success factors for ESB adoption? Many books and articles are written and available on these topics. Many different versions also create confusion. To put it in simple terms ESB products are valued based on the capabilities and abilities it brings to simplify the enterprise IT design & architecture. Often large enterprises have committed and invested heavily in buying vendor products.
There are many cases of adoption that have either failed to give the benefits and even have created problems in implementations. Some of the causes and experiences that I have seen are listed here, hoping it would benefit some one around.

• Undefined Usage Patterns : Many enteprises buy vendor products with vendor documents that define product patterns for adoption and starts believing that the product patterns are the golden goose to chase. It is often short-sighted view showing no vision in adoption within the enterprise. With this gap in where & how to apply ESB within enteprises, Designers and Architects make un-informed decision for the sake of project timelines and other pressures. Many places where ESB are used where they are wrongly applied and some place they are left un-used loosing the opportunity for adoption. These decision aggravate and becomes monster, then some-one recognizes the problem but does not find the business support to fund the adoption or change. Hence, the ESB vision & strategy for the enterprise need to be documented and shared to teams to give clear design & usage patterns. In the absence of it, I see lengthier debates, prolonged decisions, and political pressures building in the enterprise, causing costlier adoption and scraping of initiatives.
• Noisy Design : There are different opinions about what kind of activities to be implemented in ESB design. Many vendor based ESB's provide higher level lanugage & constructs to create mediation flows. Design of mediation flows are usually corrupted to include business logic and un-justifiable dependencies. This makes the design Noisy and brittle.
• Adhoc decision : In many occasion project teams take adhoc decisions to meet timelines and also to accomodate team pressures. Avoid tactical decisions or keep the scope for re-factoring at later iterations if you are following agile methods.
• Product rigidity : Products are built on proprietary frameworks structured to meet common scenario. Any customization on it is like walking blind. Also there are heavy components as they come with built in functionality when you install. Performance demands are high on ESB's, where most proprietary products have challenges. Applying multithreaded processing is difficult to apply due to its inherent rigidity.  ESB's need to be lightweight and pluggable to meet highly distributed nature of its usage. Integrating ESB's in Continuous integration environment is also not well supported by vendor products, and the basic feature for unit test features are also difficult. This is the gap in which open source implementation score more and gives fleixibility in usage.  ESB's need to be lightweight and pluggable to meet highly distributed nature of its usage.
• Distributed Design challenges: As the ESB inherently is distributed in nature, one needs to be clear in how this distribution is applied and the techniques used to ensure reliability, perfromance and maintainability. ESB's are built on messaging backbone, the usage of messaging mechanism, to distribute & consolidate capabilities needs to come out from the strategic thinking. Often there are lack of understanding in teams on how this mechanism helps in distribution, posses big challenge. Associated with distributed challenge is end-to-end visibility of processing. Many problems are faced by teams who use message queuing systems, they complain issue of visibility and randomly adhoc behavior. I have seen in enterprises, support teams still tamper with messages in queues, deleting them in worst cases. Clearly these challenges need clear strategy, with more discipline and right processes supported by custom utilities will help the teams manage the challenges.
• ESB Governance : This has often been the most neglected area in ESB adoption. The challenge will be on how to bring all the key systems on to the ESB in a phased and incremental manner. This needs real implementation strategy. On the one hand there will be many projects wanting to use ESB, the other end there will be limited team capacity to build, there will be trade-offs to do. There are some cases of design corrections to do through re-factorings, support new interfaces etc. Enterprise IT teams are struggling to keep the momentum to meet business pressure. One option is to form an ESB centre-of-excellence that consist of Integration Analyst, Integration developers and designers, and Platform architects. Along with them the project manager coordinates activities with project teams and plans for implementation.
• Skill & Competency: This area often becomes the weak-link in the process. It is common saying in the enterprise that the platform skills and analysis skills fall short, and they become the core reasons for project delays, low quality deliverable, and higher spending. So it is important to identify and build compentency, retain skills, and encourage re-factoring. It also depends on which platforms you use as an ESB. Appserver, WMB, Opensource Mule ESB etc. All have there own programming models, deployment options and management tools. Hence the skills varies on platforms. Important roles that are a worth having are:
• Integration Analyst - The person who analyses details of the integrations, identifies data elements, transformations and rules, completeness of integration flows, dependency analyses. Important deliverable for this role will be Integration Spec.
• Integration Developer - The person is expert in platform tools and techniques, experienced in integration design patterns and have hands-on for implementations. He should be experienced in Webservices, XML, EAI patterns, messaging systems etc. He plays role of designer and developer, responsible for quality of deliverables and meeting timelines.
• Platform Architect - The person is experienced as System and Integration architect, responsible for overall design decisions meeting NFR requirements, ensure de-coupling and mediation are performed at right levels and Services are granular meeting enterprise standards. He is one who identifies design risks, and defined design requirements. The deliverable will be Integration design spec, articulating design decisions, key dependency interfaces, translation mechanism. He defines the integration topology, data & interface standards to follow and platform dependent mechanism to follow.

  ESB platform are many, it is often confusing to select the better one as the capabilites are seemingly similar. Open source ESB products like Mule ESB, WSO2,and frameworks like Spring Integrations, Apache Camel, are quite matured for adoption. Due to complexity in IT environments, platform competencies and Design challenges, there are many project delivery delays experienced. Being aware with above mentioned challenges and practices will help in adoption process.

 

Integration challenge : Handling Duplicate messages

Challenge

Enterprise integrations often demand distributed message processing. Responsibilities are distributed among systems that handle messages at different processing stages. Integration often involved message queueing technologies that provide quality of services and decouples systems. As with all distributed technologies there are eight fallacies to it that need to taken care off. This could produce duplicate messages, or even requestor by mistake could send two messages with same information.  Take an example of order processing system that accepts orders from different channel applications in xml format. The order request is made up of orderItems, customer information, coupon discount reference. The system need to ensure the orders are not using the same coupon discount reference more than once. There could be various reasons why duplicate orders could be send to system. It could be network failure, system faults, customer entering it twice etc. What strategies do we have to handle this challenge?

Solution option

There are different strategies depending on the way messages are processed. Whether we have intgeration platform that provides support to handle duplicate message through configurations or one has simple applications servers running in clusters and handling messages in parallel.

The IT landscape we are in also makes a difference, do we have control over the producers of messages and can we impose restrictions on how they produce messages. Its good practice not to design with specific client as consumer of the services or capability. Idempotent consumer is an Integration pattern defined and used in many scenarios. The solution has three parts to it

1. How to identify unique messages?
2. Maintaining history of message uniqueness
3. Transactional integrity of consumer.

There need to be a way to recognize the message identity from business sematic point of view. The business context to which the message belongs need to have clear identifier that says this is unique to message per client or across clients. It could be single field/element or combination of fields that may define the uniqueness.

Obviously the next step is to think about how to compare the current message with past messages to determine duplicity. Storing history  information could be in different mechanism. One could store uniqueness identifier in in-memory cache for limited time to improve performance. Other way could be to store identifier in database table with constraints of uniquness on those fields. This would naturally throw exception while insertion. There could be separate table with unique column that stores hashed value. The hash value is derived from the combination of business semantic message identifier. In the case of parallel consumers running in cluster, there could be issue of parallel processing of messages that are duplicate and if you have long transactions with steps like webservice calls that may not participate in global transactions, you may end up in rollback of second transaction but with the damage due to processing the webservice.

Hence, it does make sense to use a independent endpoint in the processing flow  that could filter duplicate messages and allows unique messages forward.

 

Can the broker infrastructure take this responsibility? It could maintain a realtime cache of messages currently processed and will have to compare current message from the cached ones to identify duplicate. Let the application simply check the application database for duplicates in normal query. Some modern messaging systems like HornetQ do give support.

 

Depending on the architecture style one has there are different ways to handle the duplicates. Having and ESB style will help to isolate this concern outside the application and within ESB infratructure. If one is in clustered application server environment like websphere, one could use MDBs as endpoints and queues to write Idempotent message consumer that filters duplicate messages.

 

Anti-pattern - Using ESB or Integration Broker as Application platform

SOA style of architecture is being adopted gradually by the industry. Organizations have started to view their business capabilities as set of well managed Services that are modular and reusable in different business processes. Business is slowly realizing the value of SOA with respect to Reuse, Business-IT alignment, and Agility.

Product vendors have responded swiftly by introducing SOA enabling products that help expedite the adoption process. There are many ESB & Broker products introduced that are expected to introduce loose coupling by placing intermediary that handles integration concerns between business applications.

Some of the commercial products are:
 1. IBM WebSphere Message Broker, Websphere ESB, WebSphere MQ.
 2. TIBCO Businessworks,
 3. Webmethods Broker, Integration Server,
 4. Oracle Service Bus
5. Sonic ESB, etc

Apart from the commercial products there are open source products and lightweight frameworks that can do the role of ESB's for e.g Mule, Service mix, etc.

Often, I have seen many organizations using the ESB or Integration platforms as Application platforms. This means they use them to build business applications that produces and consumes inter-application messages. This is to me an anti-pattern where Integration platforms are loaded with business logic, complex persistence flows and business decisions. It is a kind of bad-smell in the design of the system and usage. I see following dis-advantages in using this model.

1. The purpose of ESB's is to be an integration middleware, de-coupling producer and consumer applications in many ways.
2. ESB's help induce seperation of concern between Integration-logic and Business-logic. All integration logic is isolated in Integration environment, so that if it changes the change is limited to Integration platform and not applications.
3. ESB's are appropriate for connectivity, translations, enrichment's and  validations in the context message processing. When we start putting applications and business logic inside ESB, then it starts smelling and add more complexity to manage.
4. Most commercial vendors provide message flow framework to write mediators. Sequencing of activities within mediators creates a message processing flow. The flow is created to link the source application to the target application. Adding business logic, entity persistence, rules and decisions, controlling application logic in flows will defeat the purpose of seperation of concerns. Hence it increases rigidity and complexity of applications.
5. ESB's are not made to write applications, where applications need complex persistence mechanism, caching support, data-structure support and other application frameworks.
6. Business applications are best written in good object oriented languages, leveraging application frameworks, containers, management tools etc. For e.g. building order processing engines on Broker or ESB is not good idea as it is defeating ESB's purpose, and not leveraging application frameworks.
7. Applications are best written in OO platforms or frameworks like the Java, dotNet, Ruby on Rails etc.. and deployed on Application Servers like Websphere Application Server, JBoss, Tomcat, etc. The advantages of using OO is huge when written in good abstracted manner, resulting in well defined, modular, maintainable, reusable & component based applications.

I have often seen organizations that use Broker platforms to write applications that processes messages but also host business logic, business entity and persistence of business entities. I find this as  anti-pattern. Probable reasons they take this decision are:

1. Immaturity of decision making,
2. Organizations do not spend efforts in evaluation of architectural and platform decisions.
3. Not enough understanding and appreciation of Object oriented platforms.
4. Ivory tower architects making decisions without practical experience on platforms.
5. Not enough experience on Agile, Object oriented principles and processes.
6. Not enough clarity on usage of architectural styles and their fitment.

I would recommend reading following articles to understand fundamentals of ESB style of architecture.

Mark Richards on ESB
Fundamentals of ESB

I do welcome comments on this topic, please write your thoughts..