Numeric & Math Functions

The rounding family. round rounds half away from zero to a given number of decimals; bround is banker's rounding (half to even); ceil always rounds up and floor always rounds down to a whole number.

def round(e: Column, scale: Int = 0): Column     // half away from zero
def bround(e: Column, scale: Int = 0): Column    // half to even (banker's)
def ceil(e: Column): Column        def floor(e: Column): Column

1. round to N decimals

Round a line total to 2 decimal places — the everyday money case:

import org.apache.spark.sql.functions.round

orders.select(
  $"product",
  round($"amount" * $"qty", 2).as("total")
).show(4, false)

2. ceil / floor

orders.select(
  $"amount",
  ceil($"amount").as("rounded_up"),
  floor($"amount").as("rounded_down")
).show(4, false)

3. bround — half to even

Banker's rounding reduces cumulative bias when you round millions of values — 2.5 rounds to 2, 3.5 rounds to 4 (always toward the even neighbour):

Seq(2.5, 3.5, 4.5).toDF("x").select(
  round($"x").as("round"),
  bround($"x").as("bround")
).show(false)

abs returns the magnitude (drops the sign); signum returns just the sign as -1.0, 0.0, or 1.0 — handy for turning a number into a direction/flag.

def abs(e: Column): Column
def signum(e: Column): Column   // -1.0, 0.0, or 1.0

Seq(-30, 0, 45).toDF("delta").select(
  $"delta",
  abs($"delta").as("magnitude"),
  signum($"delta").as("direction")
).show(false)

Powers and their inverses. pow(base, exp) raises to a power, sqrt/cbrt are the square/cube roots, exp is e^x, and the log family gives you logarithms in any base.

def pow(base: Column, exp: Column): Column    // base ^ exp
def sqrt(e: Column): Column      def cbrt(e: Column): Column
def exp(e: Column): Column       // e^x
def log(e: Column): Column       // natural log (ln)
def log10(e: Column): Column     def log2(e: Column): Column
def log(base: Double, e: Column): Column   // log in a given base

import org.apache.spark.sql.functions.{pow, sqrt, exp}

Seq(16.0, 100.0).toDF("x").select(
  $"x",
  pow($"x", 2).as("squared"),
  sqrt($"x").as("root")
).show(false)

log10 runs live; the natural log (log), log2, and base-N logs behave the same way — here's the shape of the output:

Seq(8.0, 1000.0).toDF("x").select(
  log($"x").as("ln"),
  log2($"x").as("log2"),
  log10($"x").as("log10")
).show(false)

The trig functions work in radians, so convert from degrees with radians first (and back with degrees). sin, cos, tan and their inverses (asin/acos/atan/atan2) are all available.

def sin(e: Column): Column   def cos(e: Column): Column   def tan(e: Column): Column
def radians(e: Column): Column   def degrees(e: Column): Column
def atan2(y: Column, x: Column): Column   // angle of a vector

Seq(0, 30, 90).toDF("deg").select(
  $"deg",
  round(sin(radians($"deg")), 3).as("sin"),
  round(cos(radians($"deg")), 3).as("cos")
).show(false)

greatest and least compare several columns per row (unlike max/min, which aggregate down a column). The rest are occasional specialists: hypot (√(x²+y²)), pmod (always-positive modulo), factorial, and the random generators.

def greatest(exprs: Column*): Column    def least(exprs: Column*): Column
def hypot(a: Column, b: Column): Column    def pmod(dividend: Column, divisor: Column): Column
def factorial(e: Column): Column
def rand(): Column    def randn(): Column    def monotonically_increasing_id(): Column

greatest/least pick the biggest/smallest value across columns on each row — here comparing amount against qty:

import org.apache.spark.sql.functions.{greatest, least}

orders.select(
  $"amount", $"qty",
  greatest($"amount", $"qty").as("bigger"),
  least($"amount", $"qty").as("smaller")
).show(4, false)

format_number turns a number into a display string with thousands separators and a fixed number of decimals — exactly what you want for reports and dashboards (but not for further math, since the result is text).

def format_number(x: Column, d: Int): Column   // returns a STRING

Seq(1234567.891, 42.5).toDF("n").select(
  format_number($"n", 2).as("pretty")
).show(false)