Thursday, March 29, 2012

JMS MaxMessage Size - persistent messages

Large JMS Messages in WebLogic

 

Applies to

WLS 10.3.5 / OSB 11g - plain vanilla setup, 64bit, 3GB Heap

Requirement

Send large messages (250MB) across OSB using JMS Transport

Background

OSB supports streaming mode for HTTP and File-based protocols - which we tested successfully with multi-GB files. However, things are not so straightforward with JMS proxies which essentially boils down to the WLS JMS subsystem.

Test Setup

The test is based on a JMS client that creates a persistent message with a variable payload size (1-250 MB). It sends this message to a JMS queue on a remote WLS server and receives it synchronously from the same queue. We measure the roundtrip time on the client as well as server load (heap, CPU) on the WLS server.

 

First Run

 

 

At first it looks good - you'd get a near linear increase in response time in relation to the  message size. But then - at 10MB size - you will encounter the first roadblock:

 

weblogic.socket.MaxMessageSizeExceededException: Incoming message of size: ’10000080′ bytes exceeds the configured maximum of: ’10000000′ bytes for protocol : ‘t3′.

 

Increase MaxMessageSize

 

For most protocols, including T3, WLS limits the size of a network call to 10MB by default (see Performance and Tuning Guide). The 10MB limit is a soft limit meant as a protection against DoS-attacks. It is worth noting, that a server is more prone to DoS attacks when increasing this limit. Anyway, let's increase it to see how large a message we can process.

 

  • Go to Managed server--> Protocols --> General and set the Maximum Message Size to something large.
  • To set the maximum message size on a client, use the following command line property: -Dweblogic.MaxMessageSize
  • Verify the Queues maximum accepted message size:

Queue-> Configuration-> Threshholds and Quotas-> Maximum Message Size

 

 

 

Setting Maximum Message Size for Network Protocols

 

 

Increase Server Heap Size

When running the test again you'll notice that the server-side heap is really busy on those large messages. 

 

 

 

Out heap size of max 3GB seems to be a limiting factor. So we increase it to 8GB (note: you need a 64-bit JVM for anything larger than 3GB):

 

USER_MEM_ARGS="-Xms3096m -Xmx8192m -XX:PermSize=128m -XX:MaxPermSize=512m -XX:NewRatio=3 -XX:SurvivorRate=128"

 

And off we go with our next test.  This time we can send up to around 100MB messages. Roundrip times increase up to 50-80 seconds.  Then we get a CORBA exception:

 

weblogic.jms.common.JMSException: weblogic.messaging.dispatcher.DispatcherException: java.rmi.MarshalException: CORBA COMM_FAILURE 1398079691 Maybe; nested exception is:

org.omg.CORBA.COMM_FAILURE:   vmcid: SUN  minor code: 203 completed: Maybe ...

 

 

Bypass CORBA Stack

At this point we got a hint from Oracle support: "Switch from wlclient.jar to wlthint3client.jar so you bypass all the CORBA stack…". This change guarantees that the T3 protocol is used for JMS communications.

 

We can now send messages up to around 220 MB of size - albeit at extremely long roundtrip times of around 10 minutes. Try anything larger and you get:

 

weblogic.jms.common.JMSException: weblogic.messaging.dispatcher.DispatcherException: weblogic.rjvm.PeerGoneException: No message was received for: '240' seconds

at weblogic.jms.dispatcher.DispatcherAdapter.convertToJMSExceptionAndThrow(DispatcherAdapter.java:116)

at weblogic.jms.dispatcher.DispatcherAdapter.dispatchSyncNoTran(DispatcherAdapter.java:61)

at weblogic.jms.client.JMSSession.receiveMessage(JMSSession.java:894)

at weblogic.jms.client.JMSConsumer.receiveInternal(JMSConsumer.java:647)

at weblogic.jms.client.JMSConsumer.receive(JMSConsumer.java:511)

 

 

 

In-flight Compression

 

WLS JMS has another nice feature: online compression of messages during transport (see

Compressing Messages). Since our test messages contain EDIFACT sample payloads they can be reduced to around 10% of their original size.  This also has a dramatic effect on roundtrip times, as  the chart below shows (up to 100MB). 250 MB messages can now be easily transferred and take less than 10 seconds for roundtrips.

 

 

 

Conclusion

We didn't crack the 200MB limit on physical message sizes in WebLogic JMS. However, we found a solution to our particular requirement. It would be interesting to find out, how to get rid of the PeerGoneException in Test-03. Even though, the roundtrip times for this kind of message size are prohibitively long.

16 comments:

Prithviraj said...

Dear Thomas,
Thank you very much for the informative blog. Your blog has helped me to solve some performance related issues in my weblogic server. And honestly speaking, this information is rarely available on the internet and difficult to find.
Also, I would like to know, what is the way to compress the message communicated between client and weblogic server. I have java client and weblogic 10.3.5 server. I referred the link mentioned by you in in-flight compression section. But it doesnt clarify what exactly I have to configure in my console or on my client side. Can you please provide us the required information on this question.
Thanks a lot!

Prithviraj said...
This comment has been removed by the author.
Thomas Rischbeck said...

Hi Prithviraj
Thanks for your feedback. I'm glad that the blog is useful for you. I've posted a follow-up on how to change the compression threshold setting: http://thomasrischbeck.blogspot.ch/2012/07/weblogic-jms-in-flight-message.html
Best, Thomas.

Chinese Fashion said...

Chinese historic clothes consisted predominantly of robes with pants for men and skirts for ladies below them. Contemporary Chinese apparel is actually a western and historical blend of chic dressing and auspicious symbolism.

salediablo said...

Witch Doctor: Do not point. Teleport over (the phrase with the copy. Kick down a few seconds after the paste, and then you see yourself next three green health bar to stop the control field, you Kailing line speed of light from the battlefield, let them thaw.

WOW GOLD said...

Comparing with other MMORPG, Diablo 3 is one of the most succesfull games. At least, I like it best. It is easy to operate as I do not need to operate on four direction key to control the directions my profession plan to go. I just need to click the mouse to lead him, which is totally different from the World of Warcraft or the Guild Wars 2. Secondly, It is easy to level up and run the maps, which means it is easy to enjoy this game. The only one skill to play this game well is to learn something about the diablo 3 weapons. As the weapons play very important role in this game, which can get you abundant diablo 3 gold and chances to win the war. Owing a best d3 weapon is a thing that worth to be proud of. You won't want to miss it. Have fun!

Anonymous said...

once you are out of combat cycle, then your level drops , so you can hold your rankings. diablo 3 gold These assumptions, but it looks like it is consistent with some of the ideas of the development group. They are currently more likely to rely on distant goal to retain players, rather than the core game mechanics.

Biti said...

Dwayna, as a kind of human six God, for major public has a symbolic meaning, but this does not mean that she will never gw2 gold be angry when she believes there is a need also to those who despise her terrible punishment, admonished the crowd, this story tells us, even if you do not worship the Dwayna goddess, do not offend her. http://www.guildwars2gear.com

aion accounts said...

Very cool entry and I so satisfied that I have found this useful information. I will definitely bookmark your site and visit this website again in the future. Thanks a lot for posting this valuable info for us. King regards!

lawn fertilization toronto said...

I agree with your conclusions and will eagerly look forward to your incoming updates. Just saying thanks will not just be adequate, for the wonderful clarity in your writing.It is imperative that we read blog post very carefully

application security testing said...

I would like to thank you for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well. In fact your creative writing abilities has inspired me to start my own blog now.

Injury lawyer said...

I think I will become a great follower.Just want to say your article is striking. The clarity in your post is simply striking and i can take for granted you are an expert on this subject.

heartburn no more Jeff Martin said...

I want to extend my greatest appreciation with the article you post me really love to read it

wowgold said...

Nice to be here and see your post !

utah web design said...

Thanks for taking the time to discuss that, I really feel strongly about it and love learning more on that topic. If achievable, as you gain competence, would you mind updating your blog with more information? It is highly helpful for me

Dich tieng trung quoc said...

I have to agree with you on this