Thomas Pagel on 07 Oct 2023 19:12:33
It would be great if you can define Create/Update/Delete actions from Event Processing. So depending of a field you can say i.e. "if this is field contains "delete" it's a delete operation, use the ID stored in field ID to identify the record to be deleted in your lakehouse". Same with Updates and Inserts. That would be super helpful to have a table which always contains the latest version of a record. This would be i.e. a great help when you receive events from D365 which can be new records, updates or deletes.
Uri Barash (administrator)
This is already feasible with a combination of materialized views and arg_max on the relevant columns. Please see how it is done for Cosmos DB change feeds here: Get the latest versions of Azure Cosmos DB documents. We will consider adding UX support or wizard, based on popuarity of the idea.
- Comments (1)
RE: Event Processing: Allow CUD actions
THANKS @Uri for the feedback. This is an interesting approach but it would require that you keep the full history of data in the KQL DB which is probably not the best idea, considering large volumes of data and the availability of standard SQL endpoints etc., a warehouse/lakehouse based approach might make more sense. But anyway, a step in the right direction :-)