Columns & the Building Blocks

Before you can transform a column you have to name one, and Spark gives you six syntaxes that all produce the same Column object. They're interchangeable — teams pick one for consistency. Knowing all six means you can read anyone's code.

col(colName: String): Column
column(colName: String): Column   // identical to col
$"colName": Column                // needs import spark.implicits._
'colName: Column                  // Scala Symbol, same result
df("colName"): Column             // bound to a specific DataFrame
expr("colName + 1"): Column       // parses a SQL expression string

1. col("...") — the portable default

col (and its twin column) takes a column name as a string. It needs no DataFrame reference, so it's the safest choice in reusable helper functions:

import org.apache.spark.sql.functions.col

people.select(col("name"), col("country")).show()

2. $"..." — the idiomatic shorthand

With import spark.implicits._ in scope, $"name" is the community-favourite shorthand for col("name"). It also unlocks operators like $"age" + 1 and $"age" > 30:

people.select($"name", ($"age" + 1).as("age_next_year")).show()

3. df("...") — disambiguate after a join

When two DataFrames both have an id column, col("id") is ambiguous. people("id") binds the reference to a specific DataFrame — the one clean way to resolve it:

people.join(orders, people("id") === orders("user_id"))
  .select(people("name"), orders("product"))
  .show()

4. expr("...") — a SQL string as a Column

expr parses a SQL expression string into a Column — handy when the logic is easier to read as SQL, or built dynamically at runtime:

import org.apache.spark.sql.functions.expr

people.select(
  expr("name"),
  expr("CASE WHEN age >= 30 THEN 'senior' ELSE 'junior' END AS tier")
).show()

select chooses and computes columns — the projection step. It accepts column names, Column expressions, or a mix. selectExpr is a shortcut that takes SQL-expression strings directly:

def select(cols: Column*): DataFrame
def select(col: String, cols: String*): DataFrame
def selectExpr(exprs: String*): DataFrame

// select with Column expressions
people.select($"name", $"age", ($"age" * 12).as("age_in_months")).show()

selectExpr is the same projection expressed as SQL strings — less ceremony when the expressions get gnarly:

people.selectExpr("name", "age", "age * 12 AS age_in_months").show()

withColumn is the workhorse: it returns a new DataFrame with a column added (or replaced, if the name already exists), keeping everything else. This is how you build up derived fields step by step.

def withColumn(colName: String, col: Column): DataFrame
def withColumnRenamed(existing: String, newName: String): DataFrame

1. Add a brand-new derived column

orders.withColumn("total", $"amount" * $"qty")
  .select($"product", $"amount", $"qty", $"total")
  .show()

2. Replace an existing column in place

Reuse the same name and withColumn overwrites it — here we round amount up to whole dollars:

orders.withColumn("amount", ceil($"amount"))
  .select($"product", $"amount")
  .show()

3. Rename a column

orders.withColumnRenamed("user_id", "customer_id")
  .select($"id", $"customer_id", $"product")
  .show()

Three small functions you'll use in almost every job. lit wraps a literal value as a Column (so you can mix constants into expressions). cast changes a column's type. alias / as renames a computed column.

def lit(literal: Any): Column
def col.cast(to: DataType): Column   // or cast("string")
def col.alias(name: String): Column  // as(name) is identical

orders.select(
  $"product",
  lit("USD").as("currency"),                 // a constant column
  $"amount".cast("int").as("amount_whole")   // double -> int
).show()