Spark Scala Maps: Examples & Syntax

The dataset

This module uses an accounts DataFrame whose attrs column is a Map[String, String]:

accountsid, name, attrs: Map[String,String]

9.1

map, map_from_arrays & map_from_entries

A Map column stores key→value pairs per row. Build one inline with map(k1, v1, k2, v2, …), or from two parallel arrays with map_from_arrays(keys, values). Here's the dataset — each account carries a bag of attributes:

Scala

import org.apache.spark.sql.functions.map

val accounts = Seq(
  (1, "Aisha", Map("plan" -> "pro",  "seats" -> "5")),
  (2, "Rohan", Map("plan" -> "free", "seats" -> "1")),
  (3, "Mia",   Map("plan" -> "pro",  "region" -> "US"))
).toDF("id", "name", "attrs")

accounts.show(false)

the accounts DataFrame — attrs is a Map[String,String] (note Mia has no seats key)

id	name	attrs
1	Aisha	{plan -> pro, seats -> 5}
2	Rohan	{plan -> free, seats -> 1}
3	Mia	{plan -> pro, region -> US}

baked output — Spark-only type, not runnable in the browser engine

Signature

def map(cols: Column*): Column                 // alternating key, value, key, value …
def map_from_arrays(keys: Column, values: Column): Column
def map_from_entries(e: Column): Column         // Array[Struct(key,value)] → Map

Map keys must be non-null and unique — duplicate keys throw (or keep the last, depending on spark.sql.mapKeyDedupPolicy). Values can be null. Unlike a struct, a map's key set can differ from row to row, which is exactly why you'd choose it over a struct for sparse or open-ended attributes.

9.2

element_at, map_keys, map_values & map_entries

Read a value by key with element_at(map, key) (or the shorthand attrs("plan")). Enumerate the structure with map_keys, map_values, and map_entries.

Signature

def element_at(map: Column, key: Any): Column   // value for key, or null if absent
def map_keys(e: Column): Column     def map_values(e: Column): Column
def map_entries(e: Column): Column   // Array[Struct(key, value)]

Scala

import org.apache.spark.sql.functions.{element_at, map_keys}

accounts.select(
  $"name",
  element_at($"attrs", "plan").as("plan"),
  element_at($"attrs", "seats").as("seats"),
  map_keys($"attrs").as("keys")
).show(false)

element_at reads by key — Mia has no seats key, so it returns null (not an error)

name	plan	seats	keys
Aisha	pro	5	[plan, seats]
Rohan	free	1	[plan, seats]
Mia	pro	null	[plan, region]

baked output — Spark-only type, not runnable in the browser engine

Reading a missing key returns null rather than throwing — that's the key difference from array element_at, and it makes maps forgiving for sparse data. The shorthand $"attrs"("plan") / col("attrs")("plan") compiles to the same thing. Use map_keys + array_contains to test whether a key exists before relying on its value.

9.3

explode a map → one row per key/value

The map counterpart of array explode: exploding a map yields two columns, key and value, one row per entry. This is how you turn a wide bag-of-attributes into a tall, filterable, joinable EAV (entity-attribute-value) table.

Signature

explode(mapCol)         // → key, value columns (drops empty maps)
explode_outer(mapCol)   // keeps a null key/value row for empty/null maps
posexplode(mapCol)      // → pos, key, value

Scala

import org.apache.spark.sql.functions.explode

accounts.select(
  $"name",
  explode($"attrs")   // yields key, value
).show(false)

exploding a map gives key + value columns, one row per entry

name	key	value
Aisha	plan	pro
Aisha	seats	5
Rohan	plan	free
Rohan	seats	1
Mia	plan	pro
Mia	region	US

baked output — Spark-only type, not runnable in the browser engine

After exploding to key/value rows you can filter($"key" === "plan"), groupBy($"key") to count attribute coverage, or pivot it back into typed columns. To explode into custom column names, use select($"name", explode($"attrs").as(Seq("attr", "val"))). As with arrays, use explode_outer if empty maps must survive.

9.4

map_concat, map_filter & transform_values

Combine and reshape maps: map_concat merges maps, map_filter keeps entries matching a predicate, and transform_values / transform_keys map over the entries with a lambda — the higher-order functions, map edition.

Signature

def map_concat(cols: Column*): Column          // merge maps (later wins on key clash)
map_filter(map, (k, v) => predicate)           // keep matching entries
transform_values(map, (k, v) => expr)          transform_keys(map, (k, v) => expr)

Scala

import org.apache.spark.sql.functions.{map_filter, transform_values, upper}

accounts.select(
  $"name",
  map_filter($"attrs", (k, v) => k === "plan").as("only_plan"),
  transform_values($"attrs", (k, v) => upper(v)).as("upper_vals")
).show(false)

map_filter keeps only the plan entry; transform_values upper-cases every value

name	only_plan	upper_vals
Aisha	{plan -> pro}	{plan -> PRO, seats -> 5}
Rohan	{plan -> free}	{plan -> FREE, seats -> 1}
Mia	{plan -> pro}	{plan -> PRO, region -> US}

baked output — Spark-only type, not runnable in the browser engine

map_concat is the map "merge" — on a key collision the later map's value wins, which makes it perfect for layering defaults under overrides: map_concat(defaults, userOverrides). Like array higher-order functions, transform_values and map_filter run natively — always prefer them to a UDF over a map column.

Frequently asked questions

What is a Map column in Spark?

A MapType column stores key→value pairs on a single row, where the keys can differ from row to row. It's the right choice for sparse or open-ended attributes (feature stores, config bags, JSON objects with unknown keys) — as opposed to a struct, whose fields are fixed for every row.

How do I get a value from a Map column in Spark?

Use element_at($"attrs", "plan") or the shorthand $"attrs"("plan"). Both return the value for that key, or null if the key is absent — reading a missing key does not throw, which makes maps convenient for sparse data.

How do I explode a Map column in Spark?

Call explode on the map column; it yields two columns — key and value — with one row per entry. Rename them with .as(Seq("key","value")), and use explode_outer if you need rows with empty or null maps to survive.

How do I merge two Map columns in Spark?

Use map_concat(mapA, mapB). When both maps contain the same key, the value from the later argument wins, so map_concat(defaults, overrides) is a clean way to layer user overrides on top of defaults.

Maps

The dataset

map, map_from_arrays & map_from_entries

element_at, map_keys, map_values & map_entries

explode a map → one row per key/value

map_concat, map_filter & transform_values

Frequently asked questions

Ready for the full data stack?

Comments