String Functions

concat glues columns/values together end to end; concat_ws ("with separator") joins them with a delimiter and skips nulls. Both are imported from org.apache.spark.sql.functions.

def concat(exprs: Column*): Column
def concat_ws(sep: String, exprs: Column*): Column   // skips nulls

1. concat — end to end

Build a label like Wireless Mouse (Electronics) from two columns and literals:

import org.apache.spark.sql.functions.{concat, lit}

orders.select(
  concat($"product", lit(" ("), $"category", lit(")")).as("label")
).show(3, false)

2. concat_ws — with a separator

Far cleaner when you have a delimiter — no repeating the separator, and nulls are dropped automatically:

import org.apache.spark.sql.functions.concat_ws

orders.select(
  concat_ws(" - ", $"product", $"category", $"status").as("summary")
).show(3, false)

Casing and length are the everyday cleanup functions. upper/lower change case, initcap title-cases each word, and length returns the character count.

def upper(e: Column): Column     def lower(e: Column): Column
def initcap(e: Column): Column   // Title Case Each Word
def length(e: Column): Column    // number of characters

orders.select(
  $"product",
  upper($"category").as("cat_upper"),
  length($"product").as("name_len")
).show(3, false)

initcap uppercases the first letter of every word and lowercases the rest — great for normalising names typed in inconsistent case:

// initcap normalises 'aiSHA kHAN' -> 'Aisha Khan'
people.select(initcap($"name").as("name")).show(3, false)

trim removes leading and trailing characters (whitespace by default, or a set you choose); ltrim/rtrim do just one side. lpad/rpad pad a string to a fixed width — the go-to for zero-padded IDs and fixed-width exports.

def trim(e: Column): Column           def trim(e: Column, trimStr: String): Column
def ltrim(e: Column): Column          def rtrim(e: Column): Column
def lpad(e: Column, len: Int, pad: String): Column
def rpad(e: Column, len: Int, pad: String): Column

trim with an explicit character set strips those characters from both ends — here removing stray # markers:

// trim whitespace, or a chosen character set
Seq("  hello  ", "##SKU##").toDF("raw").select(
  trim($"raw").as("trimmed"),
  trim($"raw", "#").as("no_hash")
).show(false)

lpad left-pads to a width — turn an id into a fixed 5-digit code:

orders.select(lpad($"id".cast("string"), 5, "0").as("order_code")).show(3, false)

Slice strings with substring (1-indexed pos, then len); grab the ends with left/right; and find a substring's position with instr or locate.

def substring(str: Column, pos: Int, len: Int): Column   // pos is 1-based
def left(str: Column, len: Int): Column     def right(str: Column, len: Int): Column
def instr(str: Column, substring: String): Column        // 1-based, 0 if absent
def locate(substr: String, str: Column, pos: Int): Column // search from pos

orders.select(
  $"product",
  substring($"product", 1, 4).as("first4"),
  instr($"product", "e").as("pos_of_e")
).show(4, false)

split breaks a string into an array<string> on a regex delimiter. It's the front door to array-land: pair it with explode (Arrays module) to turn a delimited field into rows, or index into it with getItem/element_at.

def split(str: Column, pattern: String): Column
def split(str: Column, pattern: String, limit: Int): Column   // cap the pieces

1. Split on a delimiter

The pattern is a regex, so escape regex metacharacters — a literal dot is \\.:

Seq("a,b,c").toDF("csv").select(
  split($"csv", ",").as("parts")
).show(false)

2. Grab one piece with getItem

Arrays are 0-indexed with getItem — pull the domain out of an email:

Seq("mia@shop.com").toDF("email").select(
  split($"email", "@").getItem(1).as("domain")
).show(false)

3. Limit the pieces

A limit caps how many splits happen — the last element keeps the remainder. Useful to split only on the first delimiter:

Seq("2025-06-20").toDF("d").select(
  split($"d", "-", 2).as("parts")   // ["2025", "06-20"]
).show(false)

The two regex workhorses. regexp_replace substitutes every match of a pattern; regexp_extract pulls out a single capture group. Both take Java regex syntax. These are baked below (the in-browser engine has no regex), but the Scala is exactly what you'd run on Spark.

def regexp_replace(str: Column, pattern: String, replacement: String): Column
def regexp_extract(str: Column, pattern: String, groupIdx: Int): Column

1. regexp_replace — scrub

Strip everything that isn't a digit to normalise a phone number:

Seq("(555) 010-1234").toDF("phone").select(
  regexp_replace($"phone", "[^0-9]", "").as("digits")
).show(false)

2. regexp_extract — capture a group

Group 0 is the whole match; group 1 is the first parenthesised group. Extract the numeric part of a SKU:

Seq("SKU-00423-X").toDF("sku").select(
  regexp_extract($"sku", "SKU-([0-9]+)", 1).as("num")
).show(false)

A grab-bag of the remaining everyday helpers. replace swaps a literal substring (no regex); reverse flips a string; translate maps characters one-to-one; format_string is printf for columns; repeat repeats a string N times.

def regexp_replace / replace(str, search, replace): Column   // replace = literal
def reverse(e: Column): Column        def repeat(e: Column, n: Int): Column
def translate(src: Column, from: String, to: String): Column
def format_string(format: String, args: Column*): Column      // printf-style

orders.select(
  $"status",
  replace($"status", "cancelled", "void").as("status2"),
  reverse($"status").as("reversed")
).show(3, false)

format_string builds a formatted string from a printf template — pad an id and stitch in the product:

orders.select(
  format_string("#%03d: %s", $"id", $"product").as("line")
).show(3, false)