Spark Scala Reading & Writing (I/O): Examples & Syntax

The dataset

This module reads and writes the orders DataFrame from Foundations:

ordersid, user_id, product, category, amount, qty, status, ordered_at

14.1

Reading: csv, json & parquet

spark.read returns a DataFrameReader. Pick a format (or use the shortcut methods), set options, then point it at a path. The same fluent shape works for every source.

Signature

spark.read.format("csv").option("header", "true").load("path")
spark.read.csv("path")        spark.read.json("path")        spark.read.parquet("path")
spark.read.option("header", true).option("inferSchema", true).csv("orders.csv")

Scala

val orders = spark.read
  .option("header", "true")
  .option("inferSchema", "true")
  .csv("s3://bucket/orders.csv")

orders.printSchema()
orders.show(3, false)

reading a CSV with header + inferSchema — the resulting DataFrame

id	product	amount	qty
1	Wireless Mouse	25.0	2
2	Notebook	6.5	3
3	Mechanical Keyboard	80.0	1

baked output — Spark-only type, not runnable in the browser engine

A path can be a single file, a directory (Spark reads every file in it), or a glob. Parquet and ORC are self-describing — no header/inferSchema needed, and they carry types and column stats, so prefer them over CSV/JSON for anything downstream. See Parquet vs ORC vs Avro for choosing a columnar format.

14.2

inferSchema vs an explicit schema

inferSchema is convenient but it makes Spark read the data twice (once to sample types, once to load) and can still guess wrong — dates become strings, IDs become doubles. In production, hand Spark an explicit StructType: it's faster and it's a contract that fails loudly when the source drifts.

Signature

import org.apache.spark.sql.types._
val schema = StructType(Seq(
  StructField("id", IntegerType), StructField("amount", DoubleType), StructField("ordered_at", DateType)))
spark.read.schema(schema).csv("path")     // or .schema("id INT, amount DOUBLE, ordered_at DATE")

Scala

import org.apache.spark.sql.types._

val schema = StructType(Seq(
  StructField("id",         IntegerType, nullable = false),
  StructField("product",    StringType),
  StructField("amount",     DoubleType),
  StructField("ordered_at", DateType)
))

val orders = spark.read.schema(schema).option("header", "true").csv("orders.csv")

The DDL-string form — .schema("id INT, amount DOUBLE, ordered_at DATE") — is the quick way to write the same thing. An explicit schema also lets you set mode("FAILFAST") so a malformed row aborts the job instead of silently landing in a _corrupt_record column. For JSON especially, always pin the schema — inference over a large, ragged JSON file is slow and fragile.

14.3

Writing & save modes

df.write returns a DataFrameWriter. The critical choice is the save mode, which decides what happens when the target path already exists.

Signature

df.write.mode("overwrite").parquet("path")
// modes:
"errorifexists" (default) — throw if path exists     "append" — add to existing data
"overwrite" — replace existing data                  "ignore" — silently skip if path exists

Scala

orders.write
  .mode("overwrite")
  .format("parquet")
  .save("s3://bucket/warehouse/orders")

The default is errorifexists — reruns of a job fail on the second attempt, which surprises people. overwrite replaces the whole target (with partitionOverwriteMode=dynamic it replaces only the partitions you're writing). append is how incremental loads grow a table, but naive appends create the small-files problem — coalesce or repartition before writing to control file count (repartition vs coalesce).

14.4

partitionBy, bucketBy & saveAsTable

partitionBy writes data into a directory tree keyed by a column (category=Electronics/, category=Home/) so that later queries filtering on that column skip whole folders — partition pruning. bucketBy pre-hashes rows into a fixed number of files for shuffle-free joins. saveAsTable registers the output in the metastore.

Signature

df.write.partitionBy("category").parquet("path")           // directory per value
df.write.bucketBy(8, "user_id").saveAsTable("orders")       // fixed hashed buckets (managed table)
df.write.mode("overwrite").saveAsTable("db.orders")        // register in the metastore

Scala

orders.write
  .mode("overwrite")
  .partitionBy("category")
  .parquet("s3://bucket/warehouse/orders")

// on disk:
//   warehouse/orders/category=Electronics/part-*.parquet
//   warehouse/orders/category=Home/part-*.parquet
//   warehouse/orders/category=Stationery/part-*.parquet

partitionBy("category") lays the data out as one directory per category value

path	files
category=Electronics/	part-00000.parquet
category=Home/	part-00000.parquet
category=Stationery/	part-00000.parquet

baked output — Spark-only type, not runnable in the browser engine

Partition on a low-cardinality column you frequently filter by (date, region, category) — never on something high-cardinality like user_id, which creates millions of tiny directories and cripples the driver. The partition column is stored in the path, not the files, so it costs no storage. bucketBy only works with saveAsTable (managed tables), not bare save. See the small-files problem for sizing partitions right.

14.5

Choosing a format & further reading

The format you write in dictates read speed, compression, and schema evolution for everything downstream. A quick decision guide:

Scala

// CSV/JSON  → interchange & human eyeballs only; slow, untyped, no pushdown
// Parquet   → the default for analytics: columnar, compressed, predicate pushdown
// ORC       → columnar, great with Hive; strong compression
// Avro      → row-oriented; best for streaming/Kafka & schema evolution
// Delta/Iceberg → Parquet + ACID transactions, time travel, upserts

Default to Parquet for analytical tables — columnar layout means Spark reads only the columns a query touches and skips row groups via min/max stats (predicate pushdown). Read the deep dives: Parquet vs ORC vs Avro, and for transactional tables the Apache Iceberg guide. To go deeper on the DataFrame basics behind all this I/O, see PySpark DataFrame fundamentals.

PriyaFounder · Kettle & Co.now

Before we write it out 💾 — I want the data laid out by category. In this SQL engine, just prove you can bucket it: give me the count of orders per category, columns category and files (pretend each is one file), sorted by category.

🎯 Your mission From orders, return each category and its order count as files, sorted by category — the shape a partitionBy("category") write would produce.

SQL

⌘/Ctrl + Enter

This mirrors write.partitionBy("category") — one group per category.
SELECT category, count(*) AS files FROM orders GROUP BY category ORDER BY category;

Frequently asked questions

How do I read a CSV file in Spark with Scala?

Use spark.read.option("header", "true").option("inferSchema", "true").csv("path"), or supply an explicit schema with .schema(...) for production. The path can be a single file, a directory (all files are read), or a glob. Prefer a fixed schema over inferSchema, which reads the data twice and can guess types wrong.

What are the save modes in Spark write?

Four: errorifexists (the default — throws if the path exists), append (adds to existing data), overwrite (replaces existing data), and ignore (silently does nothing if the path exists). Set them with df.write.mode("overwrite").

What is the difference between partitionBy and bucketBy in Spark?

partitionBy writes a directory tree keyed by a low-cardinality column so queries can skip whole folders (partition pruning). bucketBy hashes rows into a fixed number of files to enable shuffle-free joins and aggregations, and only works with saveAsTable (managed tables), not a bare save.

Which file format should I use in Spark?

Default to Parquet for analytical tables — it is columnar, compressed, and supports predicate pushdown so Spark reads only the columns and row groups a query needs. Use Avro for streaming and schema evolution, ORC in Hive-centric stacks, CSV/JSON only for interchange, and Delta or Iceberg when you need ACID transactions and time travel.

Reading & Writing (I/O)

The dataset

Reading: csv, json & parquet

inferSchema vs an explicit schema

Writing & save modes

partitionBy, bucketBy & saveAsTable

Choosing a format & further reading

Frequently asked questions

Ready for the full data stack?

Comments