Ticket #48 (closed task: wontfix)
Add process to clean up deleted records
| Reported by: | evert | Owned by: | evert |
|---|---|---|---|
| Priority: | minor | Milestone: | |
| Component: | Repository | Version: | |
| Keywords: | Cc: | lily-developers@… |
Description
When a record is deleted it is flagged as deleted. Any CRUD operations will behave accordingly.
The reason the record is flagged as deleted instead of deleting the corresponding hbase row is because RowLog? information present on the same row are still needed (after the delete) until all related RowLog? (WAL and MQ) tasks are performed.
To avoid too many stale, deleted, records in HBase a process should at regular intervals scan the records and permanently remove them when they are flagged as deleted and all RowLog? tasks are finished.
Change History
comment:4 Changed 3 years ago by evert
- Milestone ~1.0 deleted
This cleanup conflicts with record re-incarnation.
comment:5 Changed 2 years ago by evert
- Status changed from new to closed
- Resolution set to wontfix
When a record is deleted, it is marked as deleted and all the historical data is cleared. When a new record is created with the same id, it's version numbering will continue from where the previous record was. See also the record re-creation section in http://docs.outerthought.org/lily-docs-current/g1/407-lily.html
So, 'stale' records don't use that much space. And removing them would conflict with the record re-creation semantics.
I'm therefore closing this issue. If there would ever be a need to really purge records this issue can be re-opened.
As for a RowLogProcessor? this will/should run as a separate Kauri module on one selected node.