Ticket #224 (closed enhancement: fixed)
rowlog MQ optimization in message processing
|Reported by:||bruno||Owned by:||evert|
This is an idea to reduce the number of HBase operations done in the MQ-variant of the rowlog.
Currently, the processor reads a batch of records (default maximum 100), and processes them. As part of the processing of each individual message, a delete is performed on the MQ table after the message is processed. Rather than doing this for each message individually, we could do this per batch of processed messages using a multi-delete. Assuming full batches of 100 records are processed, this would mean a 100-to-1 reduction in the number of HBase operations (though that 1 operation will be somewhat heavier).
There is no danger in delaying these deletes, since when a message would not be removed from the MQ table, it will be processed again, but then it will already be marked as done in (or removed from) the row-local queue.