Filtering & Conditional Logic

filter and where are the exact same method — where is just a SQL-friendly alias. Both keep only the rows for which the condition is true. You can pass a Column expression or a SQL string.

def filter(condition: Column): Dataset[Row]
def filter(conditionExpr: String): Dataset[Row]
def where(condition: Column): Dataset[Row]   // alias of filter
def where(conditionExpr: String): Dataset[Row]

1. A Column expression

The idiomatic form — build the condition with column operators. Keep only people from India:

people.filter($"country" === "India").show()

2. A SQL string

Sometimes a plain SQL string is more readable — especially for people coming from SQL. where makes it read naturally:

people.where("age > 30 AND country = 'India'").show()

3. Chaining = implicit AND

Stacking filter calls is the same as joining them with AND — handy when conditions are built up in stages:

people
  .filter($"country" === "India")
  .filter($"age" > 30)
  .show()

Spark Columns override the operators. Equality is === (triple), inequality is =!= — not == or !=, which compare object references and are a classic beginner bug. Combine conditions with && (and), || (or), and negate with !.

===   equal to           =!=   not equal to
<  <=  >  >=            less / greater (and-or-equal)
&&   and   ||   or   !   not      (wrap each side in parens!)

people.filter(
  ($"country" === "India") && ($"age" =!= 34)
).show()

An OR across countries, negating with ! is also common — everyone not in India or the USA:

people.filter(!(($"country" === "India") || ($"country" === "USA"))).show()

isin matches a column against a list of values (like SQL IN). between matches an inclusive range. Both replace long chains of || / &&.

def isin(list: Any*): Column
def between(lowerBound: Any, upperBound: Any): Column   // inclusive

orders.filter($"category".isin("Electronics", "Home")).show()

between is inclusive on both ends — orders with an amount from 10 to 50:

orders.filter($"amount".between(10, 50)).show()

For string conditions, Spark gives you SQL like (with % and _ wildcards), the convenience methods contains / startsWith / endsWith, and rlike for full regular expressions.

def like(literal: String): Column        // % = any run, _ = one char
def contains(other: Any): Column
def startsWith(literal: String): Column
def endsWith(literal: String): Column
def rlike(literal: String): Column        // Java regex

1. like with wildcards

Products whose name ends in 'e' (%e = anything then an 'e'):

orders.filter($"product".like("%e")).select($"product").show()

2. contains / startsWith / endsWith

These are the readable shortcuts — no wildcards to remember. Products that contain the word 'Cable':

orders.filter($"product".contains("Cable")).select($"product").show()

3. rlike — regular expressions

rlike takes a full Java regex — reach for it when wildcards aren't enough. Match products that start with 'Wireless' or 'USB':

orders.filter($"product".rlike("^(Wireless|USB)"))
  .select($"product").show()

when / otherwise is Spark's CASE WHEN — the workhorse for deriving a column from conditions. Chain multiple whens for multiple branches; end with otherwise for the fallback (skip it and unmatched rows get null). Import it from org.apache.spark.sql.functions.

def when(condition: Column, value: Any): Column
def otherwise(value: Any): Column
// chain: when(c1, v1).when(c2, v2).otherwise(vDefault)

1. Single when + otherwise

import org.apache.spark.sql.functions.when

people.select(
  $"name",
  when($"age" >= 30, "senior").otherwise("junior").as("tier")
).show()

2. Chained whens — multiple branches

Each when is tested in order; the first match wins. Bucket orders by amount:

orders.select(
  $"product", $"amount",
  when($"amount" >= 50, "premium")
    .when($"amount" >= 20, "standard")
    .otherwise("budget").as("band")
).show()

3. No otherwise → null

Drop the otherwise and any row that matches no branch gets null — useful when you only want to label some rows:

orders.select(
  $"product",
  when($"status" === "cancelled", "⚠ refund").as("flag")
).show()

Nulls are unavoidable — they show up from joins, bad data, and optional fields. Test for them with isNull / isNotNull, fill them with coalesce (first non-null wins), or handle them in bulk with the DataFrame.na helpers fill, drop, and replace.

def isNull: Column          def isNotNull: Column
def coalesce(e: Column*): Column   // first non-null argument
df.na.fill(value)   df.na.drop()   df.na.replace(col, map)

To see nulls in action, here's a tiny contacts DataFrame where some phone numbers are missing:

val contacts = Seq(
  ("Aisha", "555-0101"),
  ("Rohan", null),
  ("Mia",   "555-0199"),
  ("Liam",  null)
).toDF("name", "phone")

1. Find the gaps with isNull

contacts.filter($"phone".isNull).show()

2. Fill nulls with coalesce

coalesce returns the first argument that isn't null — perfect for a default value:

import org.apache.spark.sql.functions.{coalesce, lit}

contacts.select(
  $"name",
  coalesce($"phone", lit("no phone on file")).as("phone")
).show()

3. Bulk handling with na.fill / na.drop

The na helpers act on many columns at once. na.fill replaces nulls with a default; na.drop removes any row containing a null:

contacts.na.fill("unknown").show()   // every null string -> "unknown"
contacts.na.drop().show()            // keep only fully-populated rows