Spark Scala Structs, JSON & Nested Data: Examples & Syntax

The dataset

This module uses a signups DataFrame — a raw JSON string column (as you'd get from Kafka) plus a parsed struct:

signupsid, raw: String (JSON), user: Struct(name, city, plan)

10.1

struct, named_struct & dot access

A struct bundles several columns into one nested column with named sub-fields. Build it with struct(...) (names inferred) or named_struct("k", v, …), and read a field with dot notation — $"user.name" — or getField.

Scala

import org.apache.spark.sql.functions.struct

val signups = Seq(
  (1, "Aisha", "Pune",  "pro"),
  (2, "Rohan", "Delhi", "free")
).toDF("id", "name", "city", "plan")
 .select($"id", struct($"name", $"city", $"plan").as("user"))

signups.show(false)
signups.printSchema()

struct(name, city, plan) → a single nested `user` column with three sub-fields

id	user
1	{Aisha, Pune, pro}
2	{Rohan, Delhi, free}

baked output — Spark-only type, not runnable in the browser engine

Signature

def struct(cols: Column*): Column                 // field names inferred from columns
def named_struct(nameAndValue: Column*): Column   // explicit "k", v, "k", v …
$"user.name"   col("user.name")   $"user".getField("name")   // read a sub-field

Scala

signups.select(
  $"user.name".as("name"),
  $"user.plan".as("plan")
).show(false)

dot access pulls sub-fields back out to top-level columns

name	plan
Aisha	pro
Rohan	free

baked output — Spark-only type, not runnable in the browser engine

$"user.*" expands all of a struct's fields to top-level columns in one shot — the quickest way to flatten one level. A struct's fields are fixed for every row (unlike a map); that's what lets Spark push down column pruning so reading only user.name from Parquet skips the other fields on disk.

10.2

withField & dropFields

Editing a struct in place: withField adds or replaces a nested field, dropFields removes one — without unpacking and rebuilding the whole struct by hand.

Signature

def withField(fieldName: String, col: Column): Column   // on a struct Column
def dropFields(fieldNames: String*): Column
// e.g. $"user".withField("active", lit(true))

Scala

import org.apache.spark.sql.functions.lit

signups.select(
  $"id",
  $"user".withField("active", lit(true)).as("user")
).show(false)

withField adds a nested `active` field to every struct

id	user
1	{Aisha, Pune, pro, true}
2	{Rohan, Delhi, free, true}

baked output — Spark-only type, not runnable in the browser engine

Before withField (Spark 3.1+) you had to rebuild the entire struct — struct($"user.name", $"user.city", …, lit(true).as("active")) — which was verbose and broke every time the schema changed. withField also creates nested paths: $"user".withField("addr.zip", …) reaches into a sub-struct.

10.3

from_json ★ — parse a JSON string into a struct

Data off Kafka, a log file, or an API usually arrives as a raw JSON string. from_json parses that string into a real struct (or map/array) given a schema, so you can then use typed dot-access on it. This is one of the most-used functions in streaming pipelines.

Signature

def from_json(e: Column, schema: StructType): Column
def from_json(e: Column, schema: String): Column     // DDL string, e.g. "name STRING, plan STRING"
// malformed input → null (PERMISSIVE mode, the default)

Scala

import org.apache.spark.sql.functions.from_json

val raw = Seq(
  (1, """{"name":"Aisha","plan":"pro"}"""),
  (2, """{"name":"Rohan","plan":"free"}""")
).toDF("id", "raw")

val parsed = raw.select(
  $"id",
  from_json($"raw", "name STRING, plan STRING").as("user")
)
parsed.select($"id", $"user.name", $"user.plan").show(false)

from_json turns the raw JSON string into a struct you can dot-access

id	name	plan
1	Aisha	pro
2	Rohan	free

baked output — Spark-only type, not runnable in the browser engine

You must supply the schema — from_json won't infer it per-row (that would be prohibitively expensive at scale). A quick DDL string like "name STRING, plan STRING" is usually enough; for nested shapes build a StructType. A field missing from the JSON becomes null; a completely malformed record becomes a null struct in the default PERMISSIVE mode. Pair with schema_of_json (below) to derive the schema from a sample once, then hard-code it.

10.4

to_json & schema_of_json

The inverse: to_json serializes a struct/map/array column back into a JSON string — for writing to Kafka, an API payload, or a single text column. schema_of_json inspects a sample JSON string and returns its inferred DDL schema, handy for authoring a from_json call.

Signature

def to_json(e: Column): Column                    // struct/map/array → JSON string
def schema_of_json(json: Column): Column          // sample → DDL schema string

Scala

import org.apache.spark.sql.functions.to_json

signups.select(
  $"id",
  to_json($"user").as("json")
).show(false)

to_json serializes the struct back into a compact JSON string

id	json
1	{"name":"Aisha","city":"Pune","plan":"pro"}
2	{"name":"Rohan","city":"Delhi","plan":"free"}

baked output — Spark-only type, not runnable in the browser engine

to_json drops null fields by default (tune with the ignoreNullFields option) and always emits compact JSON. Round-tripping from_json → transform → to_json is the standard shape for a Structured Streaming job that reads JSON off one Kafka topic and writes enriched JSON to another.

10.5

get_json_object ★ & json_tuple — extract without a schema

When you only need one or two fields out of a big JSON blob, defining a whole schema is overkill. get_json_object pulls a single value by JSONPath; json_tuple pulls several top-level keys at once. Both work straight on the string, no schema required.

Signature

def get_json_object(e: Column, path: String): Column   // JSONPath, e.g. "$.user.plan"
def json_tuple(e: Column, fields: String*): Column     // several top-level keys → columns

Scala

import org.apache.spark.sql.functions.get_json_object

val raw = Seq(
  (1, """{"user":{"name":"Aisha","plan":"pro"},"amt":80}""")
).toDF("id", "raw")

raw.select(
  get_json_object($"raw", "$.user.name").as("name"),
  get_json_object($"raw", "$.user.plan").as("plan"),
  get_json_object($"raw", "$.amt").as("amt")
).show(false)

get_json_object with JSONPath reaches into nested keys — no schema needed

name	plan	amt
Aisha	pro	80

baked output — Spark-only type, not runnable in the browser engine

get_json_object always returns a string (cast it yourself), and re-parses the JSON on every call — so pulling five fields means five parses. If you need more than two or three fields, from_json once into a struct is faster and gives you real types. json_tuple($"raw", "name", "plan") is the middle ground: several top-level keys in a single parse (but it can't reach nested paths the way JSONPath can).

10.6

Flattening a nested schema (the recipe)

The practical goal is usually a flat table from deeply nested JSON. The recipe: dot-access or col("a.b.*") to lift struct fields, and explode to unnest arrays — repeat until flat.

Signature

// structs → select the fields (or .* to grab them all)
df.select($"id", $"user.*")
// arrays → explode, then select the exploded struct's fields
df.select($"id", explode($"events").as("e")).select($"id", $"e.*")

Scala

// nested: id, user:{name, city}, events:[{type, ts}, ...]
df
  .select($"id", $"user.*", explode($"events").as("event"))
  .select($"id", $"name", $"city", $"event.type", $"event.ts")
  .show(false)

struct fields lifted with user.*, events array exploded, then event.* lifted — fully flat

id	name	city	type	ts
1	Aisha	Pune	login	2025-06-01
1	Aisha	Pune	purchase	2025-06-02
2	Rohan	Delhi	login	2025-06-03

baked output — Spark-only type, not runnable in the browser engine

For arbitrarily deep schemas, write a small recursive helper that walks df.schema, expanding every StructType with .* and explode-ing every ArrayType until no nested fields remain — a "flatten" utility you'll reuse on every semi-structured source. Beware: each explode multiplies rows, so a record with two arrays of 100 elements each becomes 10,000 rows.

PriyaFounder · Kettle & Co.now

The events land as JSON but the report is flat 🧾 — in this SQL engine just prove the flatten reflex on the orders table: give me each product and its category, one row per order, ordered by id.

🎯 Your mission From orders, return product and category, one row per order, ordered by id. (Struct/JSON flattening runs in the app; this mission uses the live tables.)

SQL

⌘/Ctrl + Enter

The nested version is select($"user.*", explode($"events")); here the rows are already flat.
SELECT product, category FROM orders ORDER BY id;

Frequently asked questions

How do I parse a JSON string column in Spark?

Use from_json(col, schema), passing either a DDL string like "name STRING, plan STRING" or a StructType. It converts the raw JSON string into a real struct you can dot-access. Malformed records become null in the default PERMISSIVE mode, and you must supply the schema — from_json does not infer it per row.

What is the difference between from_json and get_json_object in Spark?

from_json parses the whole JSON once into a typed struct — best when you need several fields. get_json_object(col, "$.path") extracts a single value by JSONPath as a string and re-parses on each call — best when you need only one or two fields and don't want to define a schema.

How do I access a nested field in a Spark struct?

Use dot notation: $"user.name" or col("user.name"), and $"user.*" to expand all sub-fields to top-level columns at once. To add or replace a nested field use $"user".withField("active", lit(true)).

How do I flatten nested JSON in Spark?

Lift struct fields with select($"struct.*") and unnest array columns with explode, repeating until the schema is flat. For deeply nested data, write a recursive helper that walks df.schema and expands every StructType and ArrayType automatically.

Structs, JSON & Nested Data

The dataset

struct, named_struct & dot access

withField & dropFields

from_json ★ — parse a JSON string into a struct

to_json & schema_of_json

get_json_object ★ & json_tuple — extract without a schema

Flattening a nested schema (the recipe)

Frequently asked questions

Ready for the full data stack?

Comments