Use blob leases to build a reusable, globally coordinated job lock for data engineering workloads.
data systems, performance, and the details that matter
Learn how Fabric Spark Runtime 2.0 incrementally clusters Delta tables, limits write amplification, and maintains file-skipping quality.
Most Spark developers learn to write data with df.write long before they ever encounter df.writeTo. It is simple, familiar, and everywhere: choose a format, pick a mode, add a few options, and save the result to a table or path. For years, that mental model worked well enough. Spark was often writing files first and tables second.
Schema inference is convenient. In production or benchmarking, it is often a silent performance killer.
I’m guilty. I’ve peddled the #NotebookEverything tagline more than a few times.