id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc
839	JMS Errors caused by very long messages	Matthias Bauer <matthias.bauer.drs@…>	somebody	"Last night one of our Daisy Repo servers threw lots of error messages like this:

{{{
[ERROR  ] <2011-01-10 01:29:03,277> (org.outerj.daisy.event.EventDispatcherImpl): Exception in event dispatcher.
java.lang.RuntimeException: Failed to execute JMS action, giving up.
	at org.outerj.daisy.jms.impl.JmsClientImpl$SenderImpl.executeWhenConnectionIsUp(JmsClientImpl.java:565)
	at org.outerj.daisy.jms.impl.JmsClientImpl$SenderImpl.send(JmsClientImpl.java:470)
	at org.outerj.daisy.event.EventDispatcherImpl$EventDispatchThread.run(EventDispatcherImpl.java:134)
Caused by: javax.jms.JMSException: Data truncation: Data too long for column 'MSG' at row 1
	at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:46)
	at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1181)
	at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1551)
	at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:465)
	at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:356)
	at org.outerj.daisy.jms.impl.JmsClientImpl$SenderImpl$1.run(JmsClientImpl.java:472)
	at org.outerj.daisy.jms.impl.JmsClientImpl$SenderImpl.executeWhenConnectionIsUp(JmsClientImpl.java:546)
	... 2 more
...
}}}

This effectively stopped execution of JMS events.

What's happening: org.outerj.daisy.event.EventDispatcherImpl.EventDispatchThread.run() is querying the daisyrepository.events table and tries to submit these events (one by one) to activeMQ. The error finally occurs in activeMQ and is caused by some messages being larger than 64kByte. 
The daisyrepository.events table can hold these messages, because it uses LONGTEXT data type for the message column. But the column MSG in table activemq.activemq_msgs is of type BLOB, whence can hold only 64kByte data.

This limitation has actually been fixed in activeMQ 4.1.1 (see https://issues.apache.org/jira/browse/AMQ-1130 and https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311210&styleName=Html&version=12315614). But this requires the activeMQ tables to be recreated.

The daisy instance in question has been updated some weeks ago from version 2.3 to 2.4 and to 2.4.1. Daisy 2.3 used activeMQ 4.1.0, using the BLOB data type for MSG. Daisy 2.4 updated activeMQ to version 4.1.2. This basically fixes the bug, but you have to explicitly drop the activeMQ tables.

Solution: Drop activeMQ.activemq_acks, activeMQ.activemq_lock and activeMQ.activemq_msgs upon update to Daisy 2.4.x. ActiveMQ will automatically recreate these tables upon repository startup. (It would be a good idea to include that in the update instructions.) Increasing MySQL's max_allowed_packet might also be neccessary. (I've increased it from 1 to 8 MByte.)

(Side Discussion: Where are the long messages coming from?
The messages in question are DocumentUpdate and VariantUpdate messages of documents with a high number of versions. These documents are updated 4 times a day by an external script and have some 2000 versions and a similarly long liveVersionHistory now. The liveVerisonHistory is included in the JMS message leading to message sizes of some 250 kByte.)
"	Bug	closed	Major		JMS	2.4	Fixed