Integration challenge : Handling Duplicate messages

Challenge

Enterprise integrations often demand distributed message processing. Responsibilities are distributed among systems that handle messages at different processing stages. Integration often involved message queueing technologies that provide quality of services and decouples systems. As with all distributed technologies there are eight fallacies to it that need to taken care off. This could produce duplicate messages, or even requestor by mistake could send two messages with same information.  Take an example of order processing system that accepts orders from different channel applications in xml format. The order request is made up of orderItems, customer information, coupon discount reference. The system need to ensure the orders are not using the same coupon discount reference more than once. There could be various reasons why duplicate orders could be send to system. It could be network failure, system faults, customer entering it twice etc. What strategies do we have to handle this challenge?

Solution option

There are different strategies depending on the way messages are processed. Whether we have intgeration platform that provides support to handle duplicate message through configurations or one has simple applications servers running in clusters and handling messages in parallel.

The IT landscape we are in also makes a difference, do we have control over the producers of messages and can we impose restrictions on how they produce messages. Its good practice not to design with specific client as consumer of the services or capability. Idempotent consumer is an Integration pattern defined and used in many scenarios. The solution has three parts to it

1. How to identify unique messages?
2. Maintaining history of message uniqueness
3. Transactional integrity of consumer.

There need to be a way to recognize the message identity from business sematic point of view. The business context to which the message belongs need to have clear identifier that says this is unique to message per client or across clients. It could be single field/element or combination of fields that may define the uniqueness.

Obviously the next step is to think about how to compare the current message with past messages to determine duplicity. Storing history  information could be in different mechanism. One could store uniqueness identifier in in-memory cache for limited time to improve performance. Other way could be to store identifier in database table with constraints of uniquness on those fields. This would naturally throw exception while insertion. There could be separate table with unique column that stores hashed value. The hash value is derived from the combination of business semantic message identifier. In the case of parallel consumers running in cluster, there could be issue of parallel processing of messages that are duplicate and if you have long transactions with steps like webservice calls that may not participate in global transactions, you may end up in rollback of second transaction but with the damage due to processing the webservice.

Hence, it does make sense to use a independent endpoint in the processing flow  that could filter duplicate messages and allows unique messages forward.

 

Can the broker infrastructure take this responsibility? It could maintain a realtime cache of messages currently processed and will have to compare current message from the cached ones to identify duplicate. Let the application simply check the application database for duplicates in normal query. Some modern messaging systems like HornetQ do give support.

 

Depending on the architecture style one has there are different ways to handle the duplicates. Having and ESB style will help to isolate this concern outside the application and within ESB infratructure. If one is in clustered application server environment like websphere, one could use MDBs as endpoints and queues to write Idempotent message consumer that filters duplicate messages.

 

Global Delivery and life-span of IT Applications

Over the years the concept of applications has changed, especially with the introduction of new models; SOA, SaaS and ESB. Application is a general term used to represent collection of modules, bundled and packaged as a deployment unit. So, there are applications for different domains like the Integration, Web Portals, Messaging, Batch, ETL, etc. Here we use the term more for business applications. 

What is more challenging is that most applications bleed with problems as they grow with more funtionality. Customers demand to add functionality in short timeframe, scale the application to meet volume growth, expect to move with multi-tenancy models, the demand for change is ever-constant. This becomes the key puzzle to solve for delivery teams. Apart from these, the pressure of meeting business timelines, the unclear requirements, environment readiness, skill deficiencies, and most importantly the leadership challenge is pre-dominant in the offshore-onshore defined global delivery model.

There are many theories on how to approach these challenges, with agile methodologies, global delivery models, improved project management, the focus has been towards using different project management techniques. Over the years these techniques have been adopted without much success towards satisfaction of customer. What could be the reasons for failures? Do we really openly talk of failures? How can we approach differently? Does metrics based delivery resolve the issues? Does only agile resolve the issues? I have my strong views on this topic and will be sharing them here..

I believe the life of application is directly propotional to its design integrity. The most important design principle is Modularity. Think of it as Lego block puzzle. Joining the lego blocks in controlled way will create better structures. Be it creating Service in SOA or building web applications or any enterprise applications, if the design is based on modular blocks and their dependencies are controlled efficiently, the application becomes maintainable, understandable and flexible. This gives long life to application.

What is then stopping us to build such applications? I see many IT vendors take greenfield projects offshore working in global delivery model and even more systems for maintenance, but many cases end up with fragile systems that eventually bleed customers time and money. At later stage customers start terming these applications as legacy and start looking for replacement products and loose faith in custom developed applications. So what does it take to improve from this current state? Root cause to this problem lies in in-discipline to design. More important is to have consistent mechanism to ensure design integrity is maintained. How does IT vendors ensure this? Does use of bigger IT team, bigger test teams, bigger project management teams help resolve this? No way!

It is important to realize that design and coding skills are more important than ever before. IT vendors need to invest in maintaining right skilled people to ensure quality is delivered. It is skills in design and development that will ensure change to current system is applied meaningfully and maintains design integrity. People with design skills know how to apply design principles, know how implementation is modularized, how extensibility is supported and how application design integrity is maintained. This is why Object oriented principles, Service oriented principles and Agile methods are more relevant today. Applying these principles will ensure evolution of modular applications over longer period of time, benefiting customer in terms of change, efficiency and quality.
IT vendors need to focus on quality along with efficiency to provide customer satisfaction in a global delivery model. Key to this is people with right skills and that is the real challenge to the IT vendors, especially as they grow big their differentiation factors are diminishing. It is time to take a step back for IT vendors to re-focus on core capabilities and make it real competitive advantage in the crowded outsourcing milieu.