Spark Scala Arrays & Higher-Order Functions: Examples & Syntax

The dataset

This module uses a small carts DataFrame with an Array column — think of each row as a shopping cart:

cartsid, name, items: Array[String]

8.1

array, size, element_at & array_contains

First, the basics: build an array with array(...), measure it with size, index into it with element_at (1-based!), and test membership with array_contains. Here's the dataset we'll use throughout — three carts, each holding an items array:

Scala

val carts = Seq(
  (1, "Aisha", Seq("mouse", "cable", "mug")),
  (2, "Rohan", Seq("keyboard")),
  (3, "Mia",   Seq("lamp", "mug", "mug"))
).toDF("id", "name", "items")

carts.show(false)

the carts DataFrame — items is an Array[String] column

id	name	items
1	Aisha	[mouse, cable, mug]
2	Rohan	[keyboard]
3	Mia	[lamp, mug, mug]

baked output — Spark-only type, not runnable in the browser engine

Signature

def array(cols: Column*): Column           def size(e: Column): Column      // -1 if null
def element_at(arr: Column, i: Int): Column   // 1-based; negative counts from the end
def array_contains(arr: Column, value: Any): Column   // Boolean

Scala

import org.apache.spark.sql.functions.{size, element_at, array_contains}

carts.select(
  $"name",
  size($"items").as("n"),
  element_at($"items", 1).as("first"),
  array_contains($"items", "mug").as("has_mug")
).show(false)

size / element_at / array_contains over items

name	n	first	has_mug
Aisha	3	mouse	true
Rohan	1	keyboard	false
Mia	3	lamp	true

baked output — Spark-only type, not runnable in the browser engine

element_at is 1-based, unlike everything else in programming — element_at(arr, 1) is the first element, and element_at(arr, -1) is the last. Indexing out of bounds throws by default (set spark.sql.ansi.enabled=false to get null instead). size returns -1 for a null array but 0 for an empty one — check for both when it matters.

8.2

array_distinct, array_remove, sort_array & reverse

Cleaning and ordering arrays: array_distinct drops duplicates, array_remove strips a specific value, sort_array / array_sort order the elements, and reverse flips them.

Signature

def array_distinct(e: Column): Column       def array_remove(arr: Column, value: Any): Column
def sort_array(e: Column, asc: Boolean = true): Column   // nulls sort first when asc
def array_sort(e: Column): Column           def reverse(e: Column): Column

Scala

import org.apache.spark.sql.functions.{array_distinct, sort_array}

carts.select(
  $"name",
  array_distinct($"items").as("distinct_items"),
  sort_array($"items").as("sorted")
).show(false)

array_distinct removes Mia's duplicate mug; sort_array orders alphabetically

name	distinct_items	sorted
Aisha	[mouse, cable, mug]	[cable, mouse, mug]
Rohan	[keyboard]	[keyboard]
Mia	[lamp, mug]	[lamp, mug, mug]

baked output — Spark-only type, not runnable in the browser engine

sort_array(arr, false) sorts descending. The difference between sort_array and array_sort: sort_array takes an ascending/descending flag and puts nulls first; array_sort sorts ascending with nulls last and (in newer Spark) accepts a custom comparator lambda. Neither mutates the source column — like all Spark functions they return a new column.

8.3

array_union, array_intersect & array_except

Treat arrays as sets. array_union merges two arrays (dropping duplicates), array_intersect keeps only common elements, and array_except keeps what's in the first but not the second.

Signature

def array_union(a: Column, b: Column): Column       // set union, deduplicated
def array_intersect(a: Column, b: Column): Column   // common elements
def array_except(a: Column, b: Column): Column      // in a but not in b

Scala

import org.apache.spark.sql.functions.{array, lit, array_intersect, array_except}

val promo = array(lit("mug"), lit("cable"))

carts.select(
  $"name",
  array_intersect($"items", promo).as("promo_hits"),
  array_except($"items", promo).as("non_promo")
).show(false)

which cart items are in the promo set {mug, cable}, and which aren't

name	promo_hits	non_promo
Aisha	[cable, mug]	[mouse]
Rohan	[]	[keyboard]
Mia	[mug]	[lamp]

baked output — Spark-only type, not runnable in the browser engine

All three deduplicate their output — the result is set-like even if the inputs had repeats (note Mia's two mugs collapse to one). An empty intersection returns an empty array [], not null, so size(array_intersect(...)) > 0 is a clean "do these overlap?" test.

8.4

explode ★ — one row per element

explode is the function you came here for. It takes an array column and produces one output row per element, copying the other columns down — turning a nested array back into a flat, joinable, groupable table. It's the bridge between nested and relational shapes.

Signature

def explode(e: Column): Column           // array → rows; drops empty/null arrays
def explode_outer(e: Column): Column     // keeps a null row for empty/null arrays
def posexplode(e: Column): Column        // adds the element's position (0-based)
def posexplode_outer(e: Column): Column

5 ways to use this

Explode an array column (basic)
explode inside select vs withColumn
explode vs explode_outer (keep empty rows)
posexplode — get the index too
explode then aggregate (count per element)

1. Explode an array column (basic)

Scala

import org.apache.spark.sql.functions.explode

carts.select(
  $"name",
  explode($"items").as("item")
).show(false)

each array element becomes its own row; name is copied down

name	item
Aisha	mouse
Aisha	cable
Aisha	mug
Rohan	keyboard
Mia	lamp
Mia	mug
Mia	mug

baked output — Spark-only type, not runnable in the browser engine

2. explode inside select vs withColumn

You can explode in a select (above) or with withColumn — same result. withColumn keeps all existing columns and adds the exploded one, which is handy when you have many columns to preserve:

Scala

// keeps id, name, items AND adds item
carts.withColumn("item", explode($"items"))
  .select("id", "name", "item")
  .show(3, false)

explode via withColumn — the original items array is still on the row too

id	name	item
1	Aisha	mouse
1	Aisha	cable
1	Aisha	mug

baked output — Spark-only type, not runnable in the browser engine

3. explode vs explode_outer

Plain explode drops rows whose array is empty or null — a silent data-loss trap. explode_outer keeps one row with a null element instead. Imagine a fourth cart with no items:

Scala

import org.apache.spark.sql.functions.explode_outer

// Leo has an empty cart: (4, "Leo", Seq())
withEmpty.select($"name", explode_outer($"items").as("item")).show(false)

explode_outer keeps Leo (empty cart) with a null item; plain explode would drop him entirely

name	item
Aisha	mouse
Rohan	keyboard
Mia	lamp
Leo	null

baked output — Spark-only type, not runnable in the browser engine

4. posexplode — get the index too

Scala

import org.apache.spark.sql.functions.posexplode

carts.select(
  $"name",
  posexplode($"items")   // yields two columns: pos, col
).show(false)

posexplode adds a 0-based position column (default names: pos, col)

name	pos	col
Aisha	0	mouse
Aisha	1	cable
Aisha	2	mug
Rohan	0	keyboard
Mia	0	lamp
Mia	1	mug
Mia	2	mug

baked output — Spark-only type, not runnable in the browser engine

5. explode then aggregate

The most common pipeline: explode to flatten, then groupBy the element and count — "how many carts contain each item?"

Scala

carts.select(explode($"items").as("item"))
  .groupBy($"item")
  .count()
  .orderBy($"count".desc)
  .show(false)

explode → groupBy → count: mug appears 3 times across all carts

item	count
mug	3
mouse	1
cable	1
keyboard	1
lamp	1

baked output — Spark-only type, not runnable in the browser engine

Two gotchas. (1) You can only explode one array per select — exploding two arrays in the same select is a cross-product and usually a bug; chain two separate explodes or use arrays_zip first. (2) explode multiplies your row count, so it can massively inflate data before a shuffle — filter the array (array_contains, higher-order filter) before exploding when you can.

8.5

slice, flatten, sequence & arrays_zip

Reshaping helpers. slice takes a sub-range; flatten collapses an array-of-arrays one level; sequence generates a range; arrays_zip stitches parallel arrays into an array of structs; array_max/array_min/array_position query the contents.

Signature

def slice(arr: Column, start: Int, length: Int): Column   // 1-based start
def flatten(e: Column): Column      def arrays_zip(cols: Column*): Column
def sequence(start: Column, stop: Column, step: Column): Column
def array_max(e: Column): Column    def array_min(e: Column): Column    def array_position(arr, value): Column

Scala

import org.apache.spark.sql.functions.{slice, array_position}

carts.select(
  $"name",
  slice($"items", 1, 2).as("first_two"),
  array_position($"items", "mug").as("mug_at")   // 1-based, 0 if absent
).show(false)

slice takes the first 2 elements; array_position finds mug (1-based, 0 = not found)

name	first_two	mug_at
Aisha	[mouse, cable]	3
Rohan	[keyboard]	0
Mia	[lamp, mug]	2

baked output — Spark-only type, not runnable in the browser engine

Scala

import org.apache.spark.sql.functions.sequence

Seq(1).toDF("x").select(
  sequence(lit(1), lit(5), lit(1)).as("range")
).show(false)

sequence(1, 5, 1) generates an array — great for exploding into a calendar or number series

range
[1, 2, 3, 4, 5]

baked output — Spark-only type, not runnable in the browser engine

sequence is a hidden gem: sequence(to_date(start), to_date(end), expr("interval 1 day")) generates every date in a range, which you then explode to build a gap-free date dimension without a separate calendar table. slice is 1-based like element_at.

8.6

Higher-order functions: transform, filter, exists, aggregate

The modern way to process array contents without a UDF. Higher-order functions take a lambda and apply it to each element on the JVM natively — far faster than a UDF. transform maps, filter keeps matching elements, exists/forall test a predicate, and aggregate folds the array to a single value.

Signature

transform(arr, x => expr)          // map each element
filter(arr, x => predicate)        // keep matching elements
exists(arr, x => predicate)        forall(arr, x => predicate)   // Boolean
aggregate(arr, start, (acc, x) => expr)   // fold to one value
zip_with(a, b, (x, y) => expr)     // combine two arrays element-wise

Scala

import org.apache.spark.sql.functions.{transform, filter, exists}

carts.select(
  $"name",
  transform($"items", x => upper(x)).as("upper_items"),
  filter($"items", x => x =!= "mug").as("no_mugs"),
  exists($"items", x => x === "mug").as("has_mug")
).show(false)

transform upper-cases each element; filter drops mugs; exists tests membership — all without a UDF

name	upper_items	no_mugs	has_mug
Aisha	[MOUSE, CABLE, MUG]	[mouse, cable]	true
Rohan	[KEYBOARD]	[keyboard]	false
Mia	[LAMP, MUG, MUG]	[lamp]	true

baked output — Spark-only type, not runnable in the browser engine

Reach for these before writing a UDF. A UDF is a black box Spark's optimizer (Catalyst) can't see into, forcing serialization of every row; higher-order functions stay inside the JVM and codegen. Since Spark 3.0 the lambda syntax (x => ...) is native Scala — before that you passed a SQL string to expr("transform(items, x -> upper(x))"), which still works and is worth knowing for selectExpr.

DevAnalytics · Kettle & Co.now

Quick one from the orders table 📦 — the carts arrays live in the app, but here in the SQL engine just prove you've got the aggregation reflex: give me the number of orders per user_id, most first, columns user_id and orders.

🎯 Your mission From orders, return user_id and a count of their orders as orders, highest first. (Arrays run in the app; this mission uses the live SQL tables.)

SQL

⌘/Ctrl + Enter

This is the explode→groupBy→count muscle memory, on flat rows: groupBy("user_id").count().
SELECT user_id, count(*) AS orders FROM orders GROUP BY user_id ORDER BY orders DESC;

Frequently asked questions

What does explode do in Spark?

explode takes an array (or map) column and produces one output row per element, copying the other columns down — flattening nested data into a relational shape you can group and join. Plain explode drops rows whose array is empty or null; use explode_outer to keep them with a null element.

What is the difference between explode and explode_outer in Spark?

They behave identically for non-empty arrays. The difference is at the edges: explode silently drops any row whose array is null or empty, while explode_outer emits one row for it with a null value. Use explode_outer whenever losing those rows would corrupt a count or a left-join.

How do I get the index of an array element when exploding in Spark?

Use posexplode instead of explode. It returns two columns — a 0-based pos and the element col — so you keep track of each element's original position. posexplode_outer is the null-preserving variant.

How do I transform array elements in Spark without a UDF?

Use the higher-order functions transform, filter, exists, forall, and aggregate, which take a lambda and run natively in the JVM. For example transform($"items", x => upper(x)) upper-cases every element. They are much faster than a UDF because Catalyst can optimize and codegen them.

Arrays & Higher-Order Functions

The dataset

array, size, element_at & array_contains

array_distinct, array_remove, sort_array & reverse

array_union, array_intersect & array_except

explode ★ — one row per element

1. Explode an array column (basic)

2. explode inside select vs withColumn

3. explode vs explode_outer

4. posexplode — get the index too

5. explode then aggregate

slice, flatten, sequence & arrays_zip

Higher-order functions: transform, filter, exists, aggregate

Frequently asked questions

Ready for the full data stack?

Comments