medium-import

Refreshing Spark API for the Agentic Era

Alexy Khrabrov

Alexy Khrabrov

— 2 min read
Share

Press enter or click to view image in full size

Image 2

As many of you know, I’ve been a Spark pioneer from the very beginning, popularizing Spark at SF Scala, cofounding the very first Spark meetup, and presenting it to developer audiences around the world. When Big Data was as strong a buzzword as Agentic AI today, people were similarly confused, mystified, and experiencing widespread FOMO. Big corporations of the day were flooding the community with Big Data marketing — it was the first influx of big money into OSS, that set the next waves (Blockchain, Deep Learning, GenAI, and now Agents).

My approach to Big Data was very simple: I’ll teach you all you need to know about it in 5 minutes. Then I would whip out a terminal with Scala and do something like:

scala> (1 to 100) map ( + 3) filter ( % 2 == 0) reduce ( + )

val res3: Int = 2650

I’d explain it step by step, like so:

(1 to 100)

— it is a collection. When working with data in Scala, we generally operate on collections, whole data structures like vectors, arrays, or maps. We do not iterate over the elements explicitly, as Python list comprehensions do:

numbers = [ x for x in range(1, 101)]

Martin Odersky keynoted the conference I founded, Big Data Scala By the Bay (Scale/Data/AI By the Bay) with a talk,

Spark — the Ultimate Scala Collections

`map and filter are using a lambda function, substituting _ for the current element. And reduce is rolling up the final collection into a scalar. In a lazy execution, showing it would trigger the computation.

Why did map/filter/reduce take developers by storm? Because it felt natural once you got a hang of the basic principle of the collections. If you know that you want to map a function over a collection, filter it, and reduce — the API gave you what you expected.

Being a true genius, in a way, Matei did not create Spark in a vacuum; he discovered it. Just like Newton discovered Physics. At that moment in time, Philosophiæ Naturalis Principia Mathematica answered the needs of science. In our time, Apache Spark expressed what Big Data API is, following from Scala. This is a tribute to the original Spark API — you think of a function, and it’s there.

Today Spark API is a stable set of methods working on DataFrames, Spark SQL, and collections. We should treat Apache Spark as two things:

  • the API specification
  • a reference JVM implementation

That’s similar to Apache Iceberg. Just as the latter is a specification, Spark, and everything in its OSS ecosystem, could be seen as a decoupled spec and body.

At LakeSail, we’ve reimplemented Spark API in Rust, including Iceberg, Delta, and various catalogs, as Sail. In the upcoming series, we’ll look at the data ecosystem through the fresh lens of Sail.

Read more