Spark Scala Date & Time Functions: Examples & Syntax

The dataset

This module uses the two DataFrames from Foundations — both have date columns:

peopleid, name, country, age, signup_date

ordersid, user_id, product, category, amount, qty, status, ordered_at

5.1

current_date, to_date & to_timestamp

Dates usually arrive as strings. to_date parses a string into a real DateType (and to_timestamp into a TimestampType) so you can do arithmetic and comparisons on it. current_date and current_timestamp give you today/now.

Signature

def current_date(): Column        def current_timestamp(): Column
def to_date(e: Column): Column                     // parses ISO yyyy-MM-dd
def to_date(e: Column, fmt: String): Column        // custom pattern
def to_timestamp(e: Column, fmt: String): Column

Scala

import org.apache.spark.sql.functions.{to_date, current_date}

people.select(
  $"name",
  to_date($"signup_date").as("signup"),
  current_date().as("today")
).show(3, false)

Run the equivalent Spark SQL — edit it and press Run

SQL

⌘/Ctrl + Enter

Pass an explicit format when the string isn't ISO: to_date($"d", "dd/MM/yyyy"). On a bad or mismatched value, Spark returns null rather than erroring — so always check for nulls after parsing untrusted data. A DateType has no time; use to_timestamp when you need the clock.

5.2

year, month, dayofmonth, dayofweek & quarter

Once you have a real date, pull out its components. These are the building blocks of date-based partition columns (year=2025/month=06/) and time-series grouping.

Signature

def year(e: Column): Column     def month(e: Column): Column
def dayofmonth(e: Column): Column   def dayofweek(e: Column): Column   // 1=Sun..7=Sat
def dayofyear(e): Column  def weekofyear(e): Column  def quarter(e): Column
def hour(e): Column  def minute(e): Column  def second(e): Column

Scala

orders.select(
  $"ordered_at",
  year($"ordered_at").as("y"),
  month($"ordered_at").as("m"),
  dayofmonth($"ordered_at").as("d")
).show(4, false)

Run the equivalent Spark SQL — edit it and press Run

SQL

⌘/Ctrl + Enter

Watch the dayofweek convention: Spark numbers it 1 = Sunday … 7 = Saturday. (SQLite's strftime('%w') is 0 = Sunday, so the equivalent adds 1.) When you need Monday-first weeks, compute it explicitly rather than assuming — off-by-one week bugs are silent and nasty.

5.3

date_format — dates → formatted strings

date_format turns a date/timestamp into a string in any layout you specify — for report labels, partition folder names, or file suffixes. It's the inverse of to_date.

Signature

def date_format(dateExpr: Column, format: String): Column   // returns a STRING

2 ways to use this

Year-month label (yyyy-MM)
Full custom layout

1. A `yyyy-MM` month bucket

Scala

orders.select(
  $"product",
  date_format($"ordered_at", "yyyy-MM").as("month")
).show(4, false)

Run the equivalent Spark SQL — edit it and press Run

SQL

⌘/Ctrl + Enter

2. A human-readable layout

Scala

orders.select(
  date_format($"ordered_at", "dd MMM yyyy").as("label")
).show(2, false)

date_format(ordered_at, "dd MMM yyyy")

label
01 Mar 2025
12 May 2025

baked output — Spark-only type, not runnable in the browser engine

Spark's patterns use Java DateTimeFormatter letters: yyyy year, MM month, dd day, HH 24-hour, mm minute, ss second, MMM short month name. Case matters — MM is month, mm is minute, and mixing them up is the classic date_format bug.

5.4

date_add, date_sub, add_months, datediff & months_between

Add and subtract time, and measure the gap between two dates — the core of retention windows, SLA checks, and "orders in the last 30 days" filters.

Signature

def date_add(start: Column, days: Int): Column     def date_sub(start: Column, days: Int): Column
def add_months(start: Column, months: Int): Column
def datediff(end: Column, start: Column): Column    // whole days, end - start
def months_between(end: Column, start: Column): Column   // fractional months

Scala

orders.select(
  $"ordered_at",
  date_add($"ordered_at", 30).as("plus_30d"),
  datediff(current_date(), $"ordered_at").as("age_days")
).show(3, false)

Run the equivalent Spark SQL — edit it and press Run

SQL

⌘/Ctrl + Enter

months_between returns a fractional number of months (useful for tenure/age calculations) — shown baked for exact Spark output:

Scala

orders.select(
  months_between(current_date(), $"ordered_at").as("months_ago")
).show(2, false)

months_between — fractional months, not rounded

months_ago
4.70967742
2.35483871

baked output — Spark-only type, not runnable in the browser engine

Argument order trips people up: datediff(end, start) and months_between(end, start) both take the later date first and return end − start. Reverse them and you get negative numbers. date_add/date_sub take an integer number of days (use a negative in date_add to go back).

5.5

trunc, date_trunc, last_day & next_day

Snap a date to a boundary. trunc rounds a date down to the start of a unit (month, year); date_trunc does the same for timestamps (down to hour, minute…). last_day and next_day jump to month-end and the next given weekday.

Signature

def trunc(date: Column, fmt: String): Column          // 'year' | 'month' | 'week'
def date_trunc(fmt: String, ts: Column): Column       // 'hour' | 'day' | 'month' ...
def last_day(date: Column): Column
def next_day(date: Column, dayOfWeek: String): Column  // e.g. 'Monday'

Scala

orders.select(
  $"ordered_at",
  trunc($"ordered_at", "month").as("month_start"),
  last_day($"ordered_at").as("month_end")
).show(3, false)

Run the equivalent Spark SQL — edit it and press Run

SQL

⌘/Ctrl + Enter

trunc is the clean way to build a monthly grain for grouping — groupBy(trunc($"ts", "month")) beats string-formatting because the result stays a real date you can still sort and compare. Note the argument order flips between the two: trunc(date, fmt) but date_trunc(fmt, ts).

5.6

unix_timestamp, from_unixtime & time zones

Epoch seconds and time zones — where most timestamp bugs live. unix_timestamp converts to epoch seconds; from_unixtime converts back; from_utc_timestamp/to_utc_timestamp shift between UTC and a named zone.

Signature

def unix_timestamp(e: Column): Column       def from_unixtime(e: Column): Column
def from_utc_timestamp(ts: Column, tz: String): Column
def to_utc_timestamp(ts: Column, tz: String): Column
def window(ts: Column, duration: String): Column   // tumbling time windows

Scala

orders.select(
  $"ordered_at",
  unix_timestamp($"ordered_at").as("epoch")
).show(3, false)

Run the equivalent Spark SQL — edit it and press Run

SQL

⌘/Ctrl + Enter

Converting a UTC timestamp into a local zone is a from_utc_timestamp call (baked — the browser engine has no zone database):

Scala

df.select(
  from_utc_timestamp($"ts_utc", "Asia/Kolkata").as("ist")
).show(1, false)

from_utc_timestamp(ts, "Asia/Kolkata") — shifts +5:30

ts_utc	ist
2025-06-20 12:00:00	2025-06-20 17:30:00

baked output — Spark-only type, not runnable in the browser engine

Store timestamps in UTC, convert only for display. The #1 time-zone bug is applying from_utc_timestamp to data that was never UTC. Also set spark.sql.session.timeZone explicitly — relying on the cluster's default makes results non-reproducible across environments. For event-time windowing, window($"ts", "1 hour") buckets rows into tumbling intervals (see the streaming module).

PriyaFounder · Kettle & Co.now

Monthly report time 🗓️ — for every order, give me the product and the month it was placed, formatted like 2025-03. Call the column month.

🎯 Your mission From orders, return product and the order month formatted as YYYY-MM (e.g. 2025-03), in a column called month.

SQL

⌘/Ctrl + Enter

This is date_format($"ordered_at", "yyyy-MM") in Spark — strftime('%Y-%m', ...) in the SQL engine here.
SELECT product, strftime('%Y-%m', ordered_at) AS month FROM orders;

Frequently asked questions

How do I convert a string to a date in Spark?

Use to_date(col) for ISO yyyy-MM-dd strings, or to_date(col, "dd/MM/yyyy") with an explicit pattern for other layouts. For values with a time component, use to_timestamp. Spark returns null (rather than throwing) when a value doesn't match the pattern, so always null-check parsed columns from untrusted data.

What is the difference between date_format and to_date in Spark?

They are inverses. to_date parses a string into a real DateType you can do arithmetic on; date_format takes a date and renders it back into a formatted string for display or partition names. Do all your date math on the parsed DateType, then date_format only at the end for output.

How do I calculate the number of days between two dates in Spark?

Use datediff(end, start), which returns the whole number of days as end − start. The later date goes first — reverse the arguments and you get a negative number. For a fractional gap in months, use months_between(end, start) instead.

Why does Spark date_format return the wrong value?

Almost always a pattern-letter mistake. Spark uses Java DateTimeFormatter letters where MM is month and mm is minute — mixing them is the classic bug. yyyy is year, dd is day, HH is 24-hour. Case is significant, so double-check the pattern string.

Date & Time Functions

The dataset

current_date, to_date & to_timestamp

year, month, dayofmonth, dayofweek & quarter

date_format — dates → formatted strings

1. A `yyyy-MM` month bucket

2. A human-readable layout

date_add, date_sub, add_months, datediff & months_between

trunc, date_trunc, last_day & next_day

unix_timestamp, from_unixtime & time zones

Frequently asked questions

Ready for the full data stack?

Comments

Date & Time Functions

The dataset

current_date, to_date & to_timestamp

year, month, dayofmonth, dayofweek & quarter

date_format — dates → formatted strings

1. A yyyy-MM month bucket

2. A human-readable layout

date_add, date_sub, add_months, datediff & months_between

trunc, date_trunc, last_day & next_day

unix_timestamp, from_unixtime & time zones

Frequently asked questions

Ready for the full data stack?

Comments

1. A `yyyy-MM` month bucket