Enterprise Service Buses Hit the Road

Just noticed via Ramesh that InfoWeek has a comparison of different ESB products in its latest issue.

Bern under water

Have been through Bern on the bike this afternoon: The damage on the Matte quartier is huge. From the safety of the high level bridge I watched in awe with other "disaster tourists". Draft wood blocked off one of the normal water passage ways, so the flood took another route: straight along the Matte main road! All 1.100 inhabitants of the Matte area have been evacuated from their houses -- some by force.

The water is still 1-2 metres high and there's a very strong current (20km/h?). I saw one house where water came in through the front and came out gushing through the rear. It looked like this house had been built bang in the middle of a torrential river! Lots of cars totally submerged with only the antenna sticking out of the water – some of them badly damaged and with the roof torn off (wonder how this happened?).

Bern Matte Film 1
Bern Matte Film 2

Java Business Integration (JBI)

For ipt I've given a talk on the topic of JBI recently. The rationale behind JBI is the current lack of standardization on the concept of ESB. Services developed for Sonic cannot be migrated to Iona or BEA. Different vendor’s ESB buses cannot seamlessly be integrated. This seriously hampers the adoption rate of ESBs in their current incarnation. The following statements are maybe a bit cynical, but let’s say I’m playing devil’s advocate:

ESBs are proprietary:
While most ESBs are standards based in their low-level implementation (XML, XSLT, BPEL, etc), they are proprietary at the level where custom services and integration components (such as BPEL engine, XSLT engine, BRE) live. Custom services developed on the basis of one ESB product cannot be migrated to another product without modification. Sonic's service API is incompatible with Iona's or BEA's, for example. Integration components, such as process engine, XSLT engine, BRE are tied to the vendor; no choice is possible. This may be perceived as risking vendor lock-in: who wants to build their enterprise infrastructure on a technology from a single vendor?

Different ESBs are incompatible on the transport level:

Most current ESB implementations are built on JMS. JMS guarantees reliable and guaranteed delivery of messages between components, an important basis for critical business processes. Since the JMS specification does not define a wire protocol, different ESBs are incompatible on the transport level.

On the other hand, it is unrealistic to assume that just a single ESB product will be available or can be deployed enterprise-wide. In most situations there will be variations in the installed software base, possibly as a result of mergers. What happens if external business partners want to join the service network but have invested on a different, incompatible bus? In such situations, interoperability can be established only via message bridges that link together the different ESBs. This can hardly provide the illusion of a seamless ESB service network.

Is JBI the solution?
JBI is out to change all that. JBI doesn't require a J2EE stack. This means that third-party vendors of integration components can plug their wares into the JBI framework even without having a full J2EE stack. Customers can therefore build "their" ESB server using a best-of-breed approach, e.g., take the XSLT engine, a BPEL engine, BRE engine of their choice. These might all come from different vendors who focus on their core strengths. Since components are portable between different JBI implementations, this approach avoids vendor lock-in. Different transport implementations can also be plugged in. This makes incompatible wire protocols much less of an issue.

JBI Overview

Service Engines (SE) from different vendors (e.g., an XSLT engine, BPEL engine, container for custom ESB services) plug into the normalized message router (NMR). The NMR mediates all communication between SEs. If the destination SE happens to be on a foreign host, remote communication is initiated via an appropriate protocol binding component (BC). The BC might provide JMS transport, WS-RM transport, even CORBA or EDIFACT. The meta-container maintains a routing table for this purpose. Proxying of JBI services to remote consumers is transparent at the SE-level.

Disadvantages (or why didn’t IBM and BEA join the initiative?)

Do we still need it or are WS-frameworks sufficient? They don’t see it as a sufficient step forward and believe that the area covered by JBI can equally well or better be addressed with upcoming WS-* Standards.

It is still early days to predict whether JBI will catch on or end up as yet another container architecture. The inceptors’ vision is to see JBI play an important role in enterprise integration as J2EE did/does today for application development. They also envisage the emergence of an ecosystem of pluggable SEs from different (niche-) vendors.

A few articles on JBI:

• Mark's post on JBI.
• Ron Ten-Hove Interview 1 (SearchWS)
• Ron Ten-Hove Interview 2 (Artima)
• Mark Hapner Interview on JBI (SearchWS)
• New JBI Standard could change Java (SearchWS)
• Toward Integration (IEEE Internet Computing)

Biking the Himalayas: Lhasa - Kathmandu

In October 2002 I did a Bike Trip Lhasa - Kathmandu. This has been a dream for a long time and it was the experience of a lifetime --- well worth all the physical exhaustion! Recently many people have been asking questions on my experiences and recommendations for repeating such an adventure. I've collected a few of the answers in the hope that they're useful to others.

- How hard was the trip (how good does one's physical condition need to be)?
It's hard but absolutely feasible if you are trained and physically fit. Factors to take into account are the altitude, wind, dust in the air (be prepared to wear a dust mask for severeal sections), sun impact, lack of humidity. All these can be compensated for, but you feel what you've been missing on the 4000-m descent back to Nepal!

- Did group members experience severe signs of altitude sickness?
No, just minor

-Over a longer period?
The response is very individual. For some people the symptoms come back with every 500-m climb, for others it's just one/two days upon arrival in Lhasa. I've heard that sometimes people need to fly back again, but it's very rare. I've complemented my diet with iron and vitamin tablets and found this useful.

- What month did you do the trip and how were weather conditions?
October - fine weather - cold at nights (-15 degree Celsius on everest base camp), comfortable temperatures during the day. The support team were fantastic - on the coldest night they even brought us hot-water-bottles into the tent and every morning we were awoken with freshly brewed tea delivered straight to the tent.

- Would you recommend Makalu trekking for this trip?
Absolutely! They were fantastic and I'm still in touch with some of their staff. They are very competent and efficient.

- If you have any knowledge of other organisations offering this tour, how would you compare Makalu to them?
DAV Summit Club is offering a similar trip but I don't know the details.

- Would you travel with Makalu again for similar trips?
yes

- How would you rate the accommodation and the food?
Superb - given the circumstances.

- How would you rate the logistical support?
Excellent - we had one truck and one jeep with us over the whole distance. Food and all supplies were brought in from Nepal in advance to our arrival in lhasa.

- Did you have many problems with your bike? Could the guides help you to fix your problems? What spare parts should we bring?
Bike problems were minor: a few flat tyres and loose bottom bracket. I carried standard replacement parts (inner tubes, tyres, brake and gear cables, plastic click-on pedals for SPD, lube, oil, spokes) and tools (pump, all allen keys for my bike, spoke key, chain tool, etc.) with me. In Tibet it's impossible to get spares! If something breaks en route the only chance is to find a good/creative mechanic/welder in one of the few towns.

- Did you find your bike guide useful or is it also possible to do it without bike guide (only land cruiser support)?
By chance we joined a group led by a bikealpin guide. This was useful, especially since we had fantastic mueslis and sausage flown in from Germany as well as the more bulky tools (e.g., sprocket remover, crank puller) and extra spare parts available.

The SOA Triangle

Recently I’ve worked on a SOA Readiness Assessment for a client from a data-centric perspective. Some deficiencies I observed there are probably representative of other enterprises with a heterogeneous IT system landscape that has grown organically and was integrated as need arose in an ad-hoc and piece-meal fashion:

Limited Flexibility

• mainly batch base file integration between a fragmented system landscape
• difficult to implement new services and answer to customer needs in an adequate time

Data Duplication and Inconsistencies

• Data assets are stored in a wide array of disparate systems and sources ranging from mainframes to relational databases;
• Data is replicated between individual systems that each use disparate databases and directories. Data replication takes place with time delay. Consequently there’s a window of time where conflicting changes to replicated database records can happen. This makes it a challenge to maintain an aequate level of data quality. Manual intervention is frequently necessary.
• Every disparate system has its own master dataset. There‘s no clear concept of what information is relevant where at what point in time. Data is fragmented, no unified view of enterprise data exists; in some instances this becomes obvious to the customers (e.g., multiple call centers, letters going to two different addresses).
  

Unreliable Data Exchange

• Risk of losing in-flow information during a crash (ad-hoc and often batch-file-transfer based processes).

To address these issues I came up with a graphical representation that I want to call the "SOA triangle" (not to be confused with the Bermuda triangle -- although you can get lost in both):

---

Data Access Services (DAS) : A critical foundation and abstraction layer for the access of enterprise information. They isolate both applications/service and underlying datastores/directories from change by acting as a layer of abstraction.

Data Ownership

• Role- and area-based access concepts can be implemented based on a multidimensional CRUD matrix
• Fine grained access control based e.g., on process-level authentication tokens
• Tighter management/operational controls

Encapsulation

• Provide a data model that is specific to the operational unit or a process
• Shield applications from the complexities, location, access protocols and consistency requirements of underlying enterprise data source
• Useful to percolate updates simultaneouosly to multiple backend datastores (see performance/caching later). This improves the consistency/integrity/quality of enterprise data.
• Basis for a step-wise enterprise architecture transformation: Applications/services and underlying datastores are not directly dependent on each other. Therefore, applications are isolated from structural changes/merges/etc of underlying datastores. At the same time, applications can be modified to use the new DAS-interfaces without modifying the databases.
  

Avoid impedance mismatch

• The DAS can offer an multiple process- or OU-specific interfaces to the same backend datastore with high level, business-event oriented semantics.
• This is more realistic than a unified company-wide data model on which everybody must agree (or face a winner-loser situation with some OUs being forced to use a model that doens‘t fulfil their expectations)

---

Messaging/ESB : Serves as the SOA Communication Backbone/Infrastructure

Abstract Communication Layer with Virtualised Service Endpoints

–

• Avoids "SOA Spaghetti" : Services are connected declaratively. Crosscutting concerns, such as delivery guarantees, security, messaging patterns should not be in the responsibility of services.
  
• Reliable Information Exchange between Disparate Systems: Once-and-only-once semantics guarantee that messages are not lost during machine or network failures.
• Asynchronous Communication: Failure of a component in a business process delays the process rather than failing it.
• Improved overall system reliability. This is important, because in a complex IT landscape like Cablecom‘s the chances of individual components failing increases exponentially. Store-and-forward means a process can continue once a failed component comes back online.
• Different message exchange patterns (point-to-point, publish-subscribe) possible

Straight-Through Processing (STP)

–

• Improved agility through harnessing the real-time data that are flowing around the business
• Ability to quickly acquire, analyze, and act on both opportunities and issues within the SOA implementation.

---

Caching : A caching layer can decrease the frequency of interaction with backend data stores.

Performance is of concern in a SOA

• DAS require frequent interaction with backend datastores
• In addition, services ideally are statless; i.e., services keep their state in persistent storage rather than in-memory (see next slide)

•

This issue can be addressed by software caching:

• Products like Times10 offer in-memory replicated reliable caches with persistence guarantees comparable to database.
• Physical database access is therefore reduced (since most requests can be served from the cache)
• greatly improving data access performance.

Extra Special Bitter

ESB

- Naturally leads to a document-oriented processing model

- document-oriented vs. RPC: Not separate methods are called, but instead a document is passed around asynchronously between various steps that make up a business process

- Integration on the level of business events (STP) rather than nightly batch file transfers

- Are data services more RPC-oriented, whereas higher-level business services are document-oriented?

- Virtualisation of service endpoints; therefore a more flexible adaptation of business processes. In addition, the connection complexity is reduced to attaching a service to the bus and then defining the message flow declaratively using high-level graphical tools (see screenshot).The ESB mediates all communication between services – services never communicate directly (although such optimisations can be used internally, as e.g., in SonicESB).

- ESB is a distributed system in contrast to the hub-and-spoke architecture of typical EAI-products. This offers advantages in terms of reliability and scalability.

- Because of ist distributed nature, ESB deployment can be Step-by-step.

- Disadvantages: unrealistic that a single ESB product can be rolled out across the enterprise. It’s likely that there will be multiple ESB products installed (e.g., mergers, fragmented IT strategies, organic development)

Message-Oriented Middleware (MOM)

- guaranteed delivery semantics (e.g., exactly-once, at-most-once, best-effort)

- different messaging domains: point-to-point, publish-subscribe

Possible process steps within the ESB

- intelligent message routing based on message contents (content-based routing)

- transformation (XSLT) between different technical formats

- enrichment of data with additional information

- archival of entry data

- access control verification

- logging

- throughput monitoring

- delegation to existing applications or web services

Paragliding "Schnupperkurs"

A message especially for my mate Jim: Eventually I've done it -- a whole day of paragliding, on my own. Last Saturday I went together with Stefan to Flugschule Diemtigtal where we completed a one-day "schnupperkurs". I came directly from Spicherweid cottage where I stayed the night before with Thomas and his family.

I’d done two tandem flights in the past, but this is the first time that I was on a glider just by myself. The experience was both scary and exciting. Flying 5-10 metres above ground in the flying school was not something I had expected from a beginner’s day! I managed to look stupid with one botched start (landing on my tummy!) and two crash landings.

Our teacher Andy (who’s a test pilot for Thun-based paraglider manufacturer Advance) was excellent and never got out of the calm (despite me constantly ignoring his radio instructions J) --- thanks a lot for a great day!

Altogether it was great fun and hopefully I can repeat it sometime again. On the other hand, I've decided against taking up a full course, because it's too time-consuming for me right now (waiting for the right weather/wind conditions, etc ...), a bit expensive and I'm not convinced about safety. Let's see, maybe I come to another conclusion next year :-)

New MTB

After riding hardtail bikes for the best part of the last 15 years and fervently disapproving of fullies I've eventually changed my mind. I now believe that the technology is good enough to avoid any bobbing going uphill and the only disadvantage of full suspension is the extra weight and less reliability.
I've been convinced that rear suspension is a great advantage downhill and can even be beneficial on tough uphill sections with very rough surface --- you can pedal more evenly and therefore safe energy. And the rear shock is so much fun that you actually start looking for obstacles! I've gone for the Specialized Epic Marathon 2004 with Fox Terralogic Fork and Brain IQ rear suspension.