Spark Scala UDFs & Advanced Expressions: Examples & Syntax

The dataset

This module uses the orders DataFrame from Foundations:

ordersid, user_id, product, category, amount, qty, status, ordered_at

13.1

Defining a UDF

A UDF wraps an ordinary Scala function so it can run as a column expression. Define it with udf(...), then apply it like any built-in. Here's a tiering function that bins an order amount into a label.

Signature

import org.apache.spark.sql.functions.udf
val myUdf = udf((x: Double) => /* plain Scala */ )
// then: df.withColumn("out", myUdf($"col"))

Scala

import org.apache.spark.sql.functions.udf

val tier = udf((amt: Double) =>
  if (amt >= 50) "high" else if (amt >= 20) "mid" else "low"
)

orders.select($"product", $"amount", tier($"amount").as("tier"))
  .show(false)

a udf binning amount into low/mid/high — plain Scala logic as a column

product	amount	tier
Wireless Mouse	25.0	mid
Notebook	6.5	low
Mechanical Keyboard	80.0	high
Desk Lamp	32.0	mid
Coffee Mug	12.0	low
USB-C Cable	9.0	low

baked output — Spark-only type, not runnable in the browser engine

A UDF must handle null itself — Spark passes nulls straight through, and an unguarded UDF will throw a NullPointerException mid-job. Type the input precisely (Double, not Any) so Spark can bind the right column type. And register the return type when it isn't obvious. But before you write any of this — read the next two sections; a UDF is usually the wrong tool.

13.2

Registering a UDF for Spark SQL

To call your UDF from a SQL string (or a selectExpr), register it on the session with spark.udf.register. Now it's available by name inside spark.sql("...") and in any SQL context.

Signature

spark.udf.register("tier", (amt: Double) => ... )   // name it for SQL
// then usable in SQL:  SELECT product, tier(amount) FROM orders

Scala

spark.udf.register("tier", (amt: Double) =>
  if (amt >= 50) "high" else if (amt >= 20) "mid" else "low"
)

orders.createOrReplaceTempView("orders")
spark.sql("SELECT product, tier(amount) AS tier FROM orders").show(2, false)

the same UDF, registered and called from a SQL string

product	tier
Wireless Mouse	mid
Notebook	low

baked output — Spark-only type, not runnable in the browser engine

udf(...) gives you a Column-API function; spark.udf.register(...) gives you a SQL-callable one — register when analysts or a SQL-driven pipeline need it. The registration is scoped to the SparkSession, so it disappears when the session ends; re-register at the top of each job.

13.3

When NOT to use a UDF

This is the most important section in the module. A Scala UDF is a black box to Catalyst, Spark's optimizer. It can't see inside, so it can't reorder, push down, or codegen your logic — and it must serialize/deserialize every row across the JVM boundary. On big data that's a large, invisible tax.

The rule: exhaust the built-ins first. Our tier UDF is really just when/otherwise. Nested-array logic is transform/filter/aggregate (higher-order functions). String munging is regexp_replace/split. JSON is from_json/get_json_object. Built-ins stay inside Catalyst and codegen to tight JVM bytecode — often 10× faster than the equivalent UDF.

Scala

import org.apache.spark.sql.functions.when

// the same tiering WITHOUT a UDF — Catalyst can optimize this
orders.select($"product",
  when($"amount" >= 50, "high")
    .when($"amount" >= 20, "mid")
    .otherwise("low").as("tier")
).show(2, false)

Run the equivalent Spark SQL — edit it and press Run

SQL

⌘/Ctrl + Enter

If you genuinely must use custom logic (a real algorithm, a call into a Scala library), prefer a Pandas/vectorized UDF in PySpark (Arrow-batched, far faster than a row-at-a-time Python UDF) or keep the Scala UDF as narrow as possible. Always benchmark: run it with and without and check .explain() — a UDF shows up as an opaque BatchEvalPython/ScalaUDF node the optimizer flows around.

13.4

expr & selectExpr — SQL strings as columns

expr parses a SQL expression string into a Column; selectExpr is select that takes those strings directly. They're a concise middle ground — the full power of Spark SQL expressions without leaving the DataFrame API, and no UDF in sight.

Signature

def expr(expr: String): Column                 // one SQL expression → Column
def selectExpr(exprs: String*): DataFrame       // select with SQL-string projections

Scala

orders.selectExpr(
  "product",
  "amount",
  "qty",
  "amount * qty AS line_total"
).show(false)

Run the equivalent Spark SQL — edit it and press Run

SQL

⌘/Ctrl + Enter

selectExpr shines for computed columns that read naturally as SQL — "amount * qty AS line_total", "CASE WHEN … END", "cast(qty as double)". Anything you can write in a SQL SELECT works, including registered UDFs. Use expr inside a normal select/withColumn when you want one string expression among typed columns: withColumn("lt", expr("amount * qty")).

13.5

Hashing: md5, sha2, hash & xxhash64

Hash functions turn a value into a fixed fingerprint — for pseudonymising PII, building surrogate keys, bucketing, or change detection. md5/sha1/sha2 return hex strings; hash/xxhash64 return integers used internally for partitioning.

Signature

def md5(e: Column): Column        def sha1(e: Column): Column
def sha2(e: Column, numBits: Int): Column   // numBits: 224|256|384|512
def crc32(e: Column): Column      def hash(cols: Column*): Column      def xxhash64(cols: Column*): Column

Scala

import org.apache.spark.sql.functions.{md5, sha2}

orders.select(
  $"product",
  md5($"product").as("md5"),
  sha2($"product", 256).as("sha256")
).show(2, false)

md5 → 32-hex-char digest; sha2(…,256) → 64-hex-char digest (values illustrative)

product	md5	sha256
Wireless Mouse	e4f7c1…(32 hex)	9b2e0a…(64 hex)
Notebook	a1c9d3…(32 hex)	7f4b8e…(64 hex)

baked output — Spark-only type, not runnable in the browser engine

For surrogate keys, hash the natural key columns together — sha2(concat_ws("|", $"a", $"b"), 256) — so the same business key always maps to the same id across runs (unlike monotonically_increasing_id, which changes every run). Use concat_ws with a separator, not bare concat, or ("ab","c") and ("a","bc") collide. hash/xxhash64 are for internal bucketing, not cryptographic use; md5/sha1 are not collision-safe for security either — use sha2(…, 256) when it must be strong.

DevAnalytics · Kettle & Co.now

Line totals please 🧮 — for each order give me product, amount, qty, and amount × qty as line_total. Use selectExpr-style SQL.

🎯 Your mission From orders, return product, amount, qty, and the computed amount * qty as line_total.

SQL

⌘/Ctrl + Enter

In Spark: orders.selectExpr("product","amount","qty","amount * qty AS line_total").
SELECT product, amount, qty, amount * qty AS line_total FROM orders;

Frequently asked questions

How do I create a UDF in Spark with Scala?

Wrap a plain Scala function with udf(...): val tier = udf((amt: Double) => if (amt >= 50) "high" else "low"), then apply it like a built-in — df.withColumn("tier", tier($"amount")). To call it from a SQL string, register it with spark.udf.register("tier", ...).

Why are UDFs slow in Spark?

A UDF is opaque to Catalyst, Spark's optimizer, so it cannot reorder, push down, or code-generate the logic, and every row must be serialized across the JVM boundary. Built-in functions and higher-order functions stay inside Catalyst and codegen to tight bytecode, often running 10× faster. Always try the built-ins first.

What is the difference between expr and selectExpr in Spark?

expr("amount * qty") parses a single SQL expression string into a Column you can use inside select or withColumn. selectExpr("product", "amount * qty AS line_total") is a select that takes several such SQL strings directly. Both let you use SQL expression syntax without dropping to a raw spark.sql query.

How do I create a surrogate key or hash a column in Spark?

Use sha2(concat_ws("|", keyCols…), 256) to produce a stable 256-bit hex fingerprint from the natural key columns — the same business key always maps to the same id across runs. Use concat_ws with a separator to avoid collisions, and avoid monotonically_increasing_id for keys since it changes every run.

UDFs & Advanced Expressions

The dataset

Defining a UDF

Registering a UDF for Spark SQL

When NOT to use a UDF

expr & selectExpr — SQL strings as columns

Hashing: md5, sha2, hash & xxhash64

Frequently asked questions

Ready for the full data stack?

Comments