Spark with Scala — The Functions Handbook

Everything in the DataFrame API starts from a SparkSession. In a notebook (Databricks) or spark-shell it already exists as spark. In an sbt/IntelliJ app you build it yourself:

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder
  .appName("functions-handbook")
  .master("local[*]")   // remove in production; the cluster sets this
  .getOrCreate()

import spark.implicits._   // enables $"col" and .toDF(...)

A DataFrame is a distributed table: named, typed columns and lazily-evaluated rows spread across the cluster. It's an alias for Dataset[Row]. Nothing runs until an action (like show() or collect()) forces it — transformations just build a plan.

The SQL you run in the cells on this page is that same plan, executed in a tiny SQLite engine inside your browser so you can see real output instantly. The Scala on top is the exact DataFrame API you'd write against Spark.

You'll create DataFrames four common ways. First, spark.range(n) — a one-column DataFrame of id from 0 to n-1, perfect for quick experiments:

def range(end: Long): Dataset[Row]
def range(start: Long, end: Long): Dataset[Row]
def range(start: Long, end: Long, step: Long): Dataset[Row]

spark.range(5).show()

Second, from a Scala Seq of tuples with toDF — the fastest way to hand-build test data:

val people = Seq(
  (1, "Aisha", "India",  29, "2025-02-11"),
  (2, "Rohan", "India",  34, "2024-11-03"),
  (3, "Mia",   "USA",    41, "2025-06-20"),
  (4, "Liam",  "UK",     23, "2025-01-08"),
  (5, "Noah",  "Canada", 37, "2024-08-15")
).toDF("id", "name", "country", "age", "signup_date")

people.show()

Third, with an explicit StructType schema when you need exact types (and control over nullability) instead of letting Spark infer them:

import org.apache.spark.sql.types._
import org.apache.spark.sql.Row

val schema = StructType(Seq(
  StructField("id",      IntegerType, nullable = false),
  StructField("name",    StringType,  nullable = true),
  StructField("country", StringType,  nullable = true),
  StructField("age",     IntegerType, nullable = true)
))

val rows = Seq(Row(1, "Aisha", "India", 29), Row(2, "Rohan", "India", 34))
val df = spark.createDataFrame(spark.sparkContext.parallelize(rows), schema)
df.printSchema()

Fourth — how you'll actually load data in production — from files: spark.read.csv / json / parquet. Give it a schema or ask it to infer one:

val orders = spark.read
  .option("header", "true")
  .option("inferSchema", "true")
  .csv("s3://bucket/orders/")

// or, columnar + schema baked in:
val o2 = spark.read.parquet("s3://bucket/orders_parquet/")

Four things you'll reach for constantly: show() to print rows, printSchema() for column types, .schema / .dtypes to inspect programmatically, and .columns for the names.

orders.show(3, truncate = false)
orders.printSchema()
orders.columns          // Array[String]
orders.dtypes           // Array[(String, String)]