Delta Lake deletion vectors

Run one deterministic workload through two persistent paths and compare immediate Parquet rewrites with DV sidecars that hide row positions.

Apply identical changes to both paths

1

Current snapshot and retained history

Deletion Vectors disabled

Affected Parquet files become inactive; replacement files materialize the change.

copy on write
Current snapshot: active data filesread by queries
Inactive filesnot in current snapshot, accessed via time travel

Deletion Vectors enabled

Source Parquet stays active; a directly linked DV sidecar marks hidden row positions.

merge on read
Current snapshot: data + linked DVcombined by readers
Inactive filesnot in current snapshot, accessed via time travel

Why merge on read costs less to write

Cumulative bytes each path physically writes to apply the same DELETE and UPDATE workload. Merge on read keeps source files intact and writes only tiny deletion-vector sidecars, so it avoids rewriting whole Parquet files.

live physical row updated row version row position hidden by linked DV inactive file, excluded from snapshot

How the DV-enabled reader resolves a pair

1. Read source ParquetPhysical rows remain in the active part-*.parquet file.
2. Apply its linked DVdv-part-*.bin stores the deleted row positions for that source file.
3. Return logical rowsMarked positions are filtered; replacement UPDATE rows remain visible.

What a .bin sidecar stores: a RoaringBitmap of only the deleted row positions (not one bit per row), so its size scales with the number of deletes, not the table size. Each DV is grouped with one source file, names that source, and lists the stored positions.

DV density and maintenance

Highest deleted-row ratio: 0% vs 5% threshold

0%5%10%

Largest deleted-row ratio across the DV-enabled files (marker = Delta's 5% maxDeletedRowsRatio). Cross it and OPTIMIZE auto-purges that file; REORG PURGE forces materialization at any ratio.

OPTIMIZE, REORG PURGE, and VACUUM

OPTIMIZE auto-purges a data file only when more than 5% of its rows are marked by its DV (Delta's default maxDeletedRowsRatio = 0.05). One delete against a file sits at the threshold, so it is retained; a second mark crosses it and the file is rewritten. REORG TABLE ... APPLY (PURGE) force-purges affected files regardless of ratio or for compliance. VACUUM later removes only unreferenced artifacts old enough for retention.

Enable deletion vectors

Deletion vectors are opt-in. Default them on for new tables in your Spark session, or set the table property on an existing table.

# New tables created in this session default to deletion vectors
spark.conf.set(
    "spark.databricks.delta.properties.defaults.enableDeletionVectors",
    "true"
)
-- Enable on an existing table
ALTER TABLE your_table
SET TBLPROPERTIES ('delta.enableDeletionVectors' = 'true');

Available in Fabric Runtime 1.3+ (Delta 3.2+). Once enabled, DELETE, UPDATE, and MERGE write compact deletion-vector sidecars instead of rewriting whole Parquet files.