Data deletion in a lakehouse is a lie if you only run a DELETE statement. In regulated environments, "erasure" isn't a logical filter; it is a physical guarantee that sensitive information no longer exists within your storage layer. Most engineers assume that once a record is removed from the current view of a table, they have met their compliance obligations. They are wrong.
Why this matters: Regulations like GDPR and CCPA require the permanent removal of personal data upon request. In modern table formats like Apache Iceberg, data is versioned; a simple delete command leaves the sensitive information in historical snapshots and metadata files, potentially exposing you to non-compliance penalties even after the "delete" is complete.
The challenge is that Lakehouse architectures are designed for immutability. When you issue a DELETE or MERGE command, the engine creates new data files and marks old ones as "replaced" in the metadata.…