How Fabric Runtime 2.0 clusters only what needs it — vs. the standard strategy that rewrites every file in a partial Z-Cube on each run. A cube stays partial (and keeps being rewritten) until it crosses MIN_CUBE_SIZE (100 GB default), becoming stable and skipped; OPTIMIZE packs cubes up to TARGET_CUBE_SIZE (150 GB default) before spilling data into the next cube.
Append batches of 4 files (4 GB), then run OPTIMIZE on both tables. On the standard side, the partial Z-Cube is fully rewritten every run — even for a 4 GB append. Fast-forward simulates 20 cycles of a 4 GB append followed by OPTIMIZE — the realistic small-batch pattern where full-rewrite cost compounds. Cubes crossing 100 GB become stable and are skipped; data past the 150 GB target spills into a new cube.
Rewrites every file in partial Z-Cubes; stable cubes (over 100 GB) are skipped, packed to a 150 GB target
Table size
0
Total data rewritten
0
Selects only unclustered, small, or unhealthy files — cube size is irrelevant to cost
Table size
0
Total data rewritten
0
Incremental OPTIMIZE is cheap because it only rewrites files that are new. But a new file is only clustered within itself — nothing makes its value range line up with the files already on disk. Every append lays down another band across ranges the table already covers, and those bands accumulate. The end state is a table where every file is clustered and the layout is still bad: a query filtering on a single day has to open most of the files, because most of them claim to contain that day.
Auto reclustering is the counterweight. New (unclustered) files are always selected for clustering — no thresholds involved. Auto reclustering decides whether to additionally pull in already-clustered files whose ranges overlap, and the two configs below set how bad that overlap has to get before it’s worth paying to fix. Each bar below is one file’s min–max range on the clustering column: append overlapping files, run OPTIMIZE, and watch the thresholds trade write cost against file skipping.
Incremental selection alone minimizes write cost. Auto reclustering is what keeps that saving from being quietly repaid in degraded read performance.
Min files in an overlapping group before reclustering triggers (default 4, must be ≥ 2)
Overlap score a file pair must exceed to count as overlapping (default 0.75, lower = more aggressive)
Max overlap depth
1 file
Skipping effectivenessThe share of avoidable file reads a random point query actually skips: 1 − max(0, expected scans − 1) ÷ (total files − 1), where expected scans = unclustered files (always read — no stats yet) + average clustered cover depth. A perfectly disjoint layout scores 100%.
0%
Incremental clustering is on by default in Fabric Spark Runtime 2.0 (Delta 4.1). You just need a clustered table — OPTIMIZE then reclusters only new or unhealthy files.
-- New clustered table
CREATE TABLE my_table (
order_id BIGINT,
order_date DATE,
region STRING,
amount DECIMAL(10, 2)
)
CLUSTER BY (order_date, region);
-- Or change clustering on an existing table
ALTER TABLE my_table CLUSTER BY (order_date, region);# Incremental clustering and auto reclustering are on by default.
# Disable auto reclustering only (diagnostic):
spark.conf.set(
"spark.microsoft.delta.optimize.clustering.strategy.incremental.autoRecluster",
False
)
# Disable the incremental strategy for the session (diagnostic):
spark.conf.set(
"spark.microsoft.delta.optimize.clustering.strategy.incremental",
False
)Run OPTIMIZE after writes. Runtime 2.0 selects only unclustered, unhealthy, or overlapping files instead of rewriting entire Z-Cubes.