JMS Errors caused by very long messages

[Matthias Bauer <matthias.bauer.drs@…>]

Last night one of our Daisy Repo servers threw lots of error messages like this:

[ERROR  ] <2011-01-10 01:29:03,277> (org.outerj.daisy.event.EventDispatcherImpl): Exception in event dispatcher.
java.lang.RuntimeException: Failed to execute JMS action, giving up.
	at org.outerj.daisy.jms.impl.JmsClientImpl$SenderImpl.executeWhenConnectionIsUp(JmsClientImpl.java:565)
	at org.outerj.daisy.jms.impl.JmsClientImpl$SenderImpl.send(JmsClientImpl.java:470)
	at org.outerj.daisy.event.EventDispatcherImpl$EventDispatchThread.run(EventDispatcherImpl.java:134)
Caused by: javax.jms.JMSException: Data truncation: Data too long for column 'MSG' at row 1
	at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:46)
	at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1181)
	at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1551)
	at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:465)
	at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:356)
	at org.outerj.daisy.jms.impl.JmsClientImpl$SenderImpl$1.run(JmsClientImpl.java:472)
	at org.outerj.daisy.jms.impl.JmsClientImpl$SenderImpl.executeWhenConnectionIsUp(JmsClientImpl.java:546)
	... 2 more
...

This effectively stopped execution of JMS events.

What's happening: org.outerj.daisy.event.EventDispatcherImpl?.EventDispatchThread?.run() is querying the daisyrepository.events table and tries to submit these events (one by one) to activeMQ. The error finally occurs in activeMQ and is caused by some messages being larger than 64kByte.
The daisyrepository.events table can hold these messages, because it uses LONGTEXT data type for the message column. But the column MSG in table activemq.activemq_msgs is of type BLOB, whence can hold only 64kByte data.

This limitation has actually been fixed in activeMQ 4.1.1 (see ​https://issues.apache.org/jira/browse/AMQ-1130 and ​https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311210&styleName=Html&version=12315614). But this requires the activeMQ tables to be recreated.

The daisy instance in question has been updated some weeks ago from version 2.3 to 2.4 and to 2.4.1. Daisy 2.3 used activeMQ 4.1.0, using the BLOB data type for MSG. Daisy 2.4 updated activeMQ to version 4.1.2. This basically fixes the bug, but you have to explicitly drop the activeMQ tables.

Solution: Drop activeMQ.activemq_acks, activeMQ.activemq_lock and activeMQ.activemq_msgs upon update to Daisy 2.4.x. ActiveMQ will automatically recreate these tables upon repository startup. (It would be a good idea to include that in the update instructions.) Increasing MySQL's max_allowed_packet might also be neccessary. (I've increased it from 1 to 8 MByte.)

(Side Discussion: Where are the long messages coming from?
The messages in question are DocumentUpdate? and VariantUpdate? messages of documents with a high number of versions. These documents are updated 4 times a day by an external script and have some 2000 versions and a similarly long liveVersionHistory now. The liveVerisonHistory is included in the JMS message leading to message sizes of some 250 kByte.)

[karel]

We recently upgraded to ActiveMQ 5.5.0, so this problem should be gone.
Your remark about the size of liveVersionHistory remains valid, but it's not really a bug.
BTW: you can manipulate the liveVersionHistory (manually or via the API). This should help you keep the size of the ActiveMQ messages down).

[anonymous]

Well, the root cause of the bug is the wrong/old column data type in the ActiveMQ activemq_msgs table. Are you sure that updating ActiveMQ will change anything here? IMHO, you still need to either drop that table or change the column type manually.

[karel]

New installations will use a BLOB type for the msgs table. Existing installations will need to change the activemq schema.