Leased a blob? No, this isn’t some new car model with a great leasing offer. Blobs are files in ADLS / OneLake, and leasing is the process of temporarily holding an exclusive lock on one of those files. Does this sound useful in data engineering? Maybe not at first, but in this post I’ll show why understanding the extended features of object storage APIs like blob leases can be invaluable when building data systems that scale.
data systems, performance, and the details that matter
Liquid Clustering was already a better abstraction than static partitioning due to its flexible nature. Fabric Spark Runtime 2.0 fixes the part that had me actively cautioning customers to reconsider blindly adopting it: the cost of maintaining the layout.
Most Spark developers learn to write data with df.write long before they ever encounter df.writeTo. It is simple, familiar, and everywhere: choose a format, pick a mode, add a few options, and save the result to a table or path. For years, that mental model worked well enough. Spark was often writing files first and tables second.
Schema inference is convenient. In production or benchmarking, it is often a silent performance killer.
I’m guilty. I’ve peddled the #NotebookEverything tagline more than a few times.