Large JMS Messages in WebLogic
| Applies to | WLS 10.3.5 / OSB 11g - plain vanilla setup, 64bit, 3GB Heap |
| Requirement | Send large messages (250MB) across OSB using JMS Transport |
| Background | OSB supports streaming mode for HTTP and File-based protocols - which we tested successfully with multi-GB files. However, things are not so straightforward with JMS proxies which essentially boils down to the WLS JMS subsystem. |
| Test Setup | The test is based on a JMS client that creates a persistent message with a variable payload size (1-250 MB). It sends this message to a JMS queue on a remote WLS server and receives it synchronously from the same queue. We measure the roundtrip time on the client as well as server load (heap, CPU) on the WLS server. |
First Run
At first it looks good - you'd get a near linear increase in response time in relation to the message size. But then - at 10MB size - you will encounter the first roadblock:
weblogic.socket.MaxMessageSizeExceededException: Incoming message of size: ’10000080′ bytes exceeds the configured maximum of: ’10000000′ bytes for protocol : ‘t3′.
Increase MaxMessageSize
For most protocols, including T3, WLS limits the size of a network call to 10MB by default (see Performance and Tuning Guide). The 10MB limit is a soft limit meant as a protection against DoS-attacks. It is worth noting, that a server is more prone to DoS attacks when increasing this limit. Anyway, let's increase it to see how large a message we can process.
- Go to Managed server--> Protocols --> General and set the Maximum Message Size to something large.
- To set the maximum message size on a client, use the following command line property: -Dweblogic.MaxMessageSize
- Verify the Queues maximum accepted message size:
Queue-> Configuration-> Threshholds and Quotas-> Maximum Message Size
Setting Maximum Message Size for Network Protocols
Increase Server Heap Size
When running the test again you'll notice that the server-side heap is really busy on those large messages.
Out heap size of max 3GB seems to be a limiting factor. So we increase it to 8GB (note: you need a 64-bit JVM for anything larger than 3GB):
USER_MEM_ARGS="-Xms3096m -Xmx8192m -XX:PermSize=128m -XX:MaxPermSize=512m -XX:NewRatio=3 -XX:SurvivorRate=128"
And off we go with our next test. This time we can send up to around 100MB messages. Roundrip times increase up to 50-80 seconds. Then we get a CORBA exception:
weblogic.jms.common.JMSException: weblogic.messaging.dispatcher.DispatcherException: java.rmi.MarshalException: CORBA COMM_FAILURE 1398079691 Maybe; nested exception is:
org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 203 completed: Maybe ...
Bypass CORBA Stack
At this point we got a hint from Oracle support: "Switch from wlclient.jar to wlthint3client.jar so you bypass all the CORBA stack…". This change guarantees that the T3 protocol is used for JMS communications.
We can now send messages up to around 220 MB of size - albeit at extremely long roundtrip times of around 10 minutes. Try anything larger and you get:
weblogic.jms.common.JMSException: weblogic.messaging.dispatcher.DispatcherException: weblogic.rjvm.PeerGoneException: No message was received for: '240' seconds
at weblogic.jms.dispatcher.DispatcherAdapter.convertToJMSExceptionAndThrow(DispatcherAdapter.java:116)
at weblogic.jms.dispatcher.DispatcherAdapter.dispatchSyncNoTran(DispatcherAdapter.java:61)
at weblogic.jms.client.JMSSession.receiveMessage(JMSSession.java:894)
at weblogic.jms.client.JMSConsumer.receiveInternal(JMSConsumer.java:647)
at weblogic.jms.client.JMSConsumer.receive(JMSConsumer.java:511)
In-flight Compression
WLS JMS has another nice feature: online compression of messages during transport (see
Compressing Messages). Since our test messages contain EDIFACT sample payloads they can be reduced to around 10% of their original size. This also has a dramatic effect on roundtrip times, as the chart below shows (up to 100MB). 250 MB messages can now be easily transferred and take less than 10 seconds for roundtrips.
Conclusion
We didn't crack the 200MB limit on physical message sizes in WebLogic JMS. However, we found a solution to our particular requirement. It would be interesting to find out, how to get rid of the PeerGoneException in Test-03. Even though, the roundtrip times for this kind of message size are prohibitively long.
