Tag See all →

#pyspark

2 stories

Banner: Spark batch inference slowed by loading the model in every UDF batch

Performance & Scaling Bug Report

Spark batch scoring 20x slower than expected: the model was loaded inside every UDF call

Ryan Pham · Apr 15, 2026 · 5 min read

Scoring 80 million rows with a 600 MB model took over 9 hours on a 20-node Spark cluster. The pandas UDF loaded the model from storage on every Arrow batch, thousands of times per executor. An iterator UDF that loads once per task cut the job to 28 minutes.

Banner: a row of evenly sized data partitions with one enormously tall partition overflowing a memory box

Performance & Scaling Bug Report

Spark daily 1TB batch dies on one task: skewed join keys, AQE limits and salting

John Nguyen · Feb 18, 2026 · 11 min read

Our nightly Spark job over 1TB of events finished 1,999 of 2,000 join tasks in minutes, then one task ran for an hour and the executor was killed for exceeding memory. The cause was null and hot join keys that adaptive skew handling did not split. Here is the diagnosis and the fix.