PySpark String Functions: Examples and Common Uses

Formulas & functions · Updated November 22, 2026

PySpark string functions are built-in column operations in pyspark.sql.functions for cleaning, transforming, searching, and parsing text in a Spark DataFrame. Use them with a column name or a Column expression—for example, F.lower(F.trim("name"))—rather than calling ordinary Python string methods on a distributed column.

What PySpark string functions do

A Spark DataFrame column is not a Python string; it represents values distributed across the rows and partitions of a dataset. Functions such as lower, trim, split, and regexp_replace describe work for Spark to perform on those values. The expressions can be evaluated across a large DataFrame without collecting every row into the Python process.

Most commonly used string operations are exposed through pyspark.sql.functions, often imported with the alias F. They return a Column, so they fit into operations such as select, withColumn, and filter. A typical setup looks like this:

from pyspark.sql import functions as F

cleaned = df.withColumn("name_clean", F.trim(F.col("name")))

The same expression can be written with a column name string, as in F.trim("name"). Using F.col("name") can make it clearer that the argument refers to a DataFrame column, especially inside longer expressions.

How column expressions work

String functions are composable: the result of one function can be passed directly into another. For instance, trimming a value before converting it to lowercase is one expression, not two separate trips through the data. To add a derived column, use withColumn; to return only selected expressions, use select. Both approaches leave the original DataFrame unchanged and produce a new DataFrame description.

from pyspark.sql import functions as F

data = [("  Ada Lovelace  ", "ada@example.com"),
        ("Grace Hopper", "grace@example.com"),
        (None, None)]
df = spark.createDataFrame(data, ["name", "email"])

result = df.select(
    F.trim("name").alias("name_clean"),
    F.lower(F.trim("email")).alias("email_normalized")
)

In this example, the first row’s cleaned name is Ada Lovelace, and its normalized email is ada@example.com. The third row remains null in both output columns: functions such as trim and lower generally propagate a null input rather than turning it into an empty string. Naming results with alias is useful because otherwise Spark may assign an expression-based column name.

Common functions for cleaning and formatting

Several functions cover everyday cleanup without regular expressions. Their exact output depends on the input text, so apply them to a representative sample when defining a data-cleaning rule.

  • trim removes leading and trailing spaces by default. ltrim and rtrim target the left and right ends respectively.
  • lower and upper change letter case. initcap capitalizes the first letter of each word, which may not match the preferred styling of names or product labels.
  • length returns a string’s length. A null input produces null; for binary data, length refers to bytes rather than text characters.
  • lpad and rpad pad text to a requested length. They can also shorten an input that is longer than the requested length, so use them only when that behavior is appropriate.
  • translate replaces or removes individual characters according to character mappings. It is not a general substring replacement function.

For example, with a column called code, F.lpad("code", 8, "0") can turn a short identifier such as 42 into 00000042. This is formatting, not a change in the identifier’s underlying meaning. If leading zeroes must be preserved, keep the identifier as a string rather than casting it to an integer.

Searching and extracting text with regular expressions

Regular expressions are useful when a rule involves a pattern rather than a fixed substring. PySpark’s regexp_replace replaces matches, regexp_extract returns a capture group, and the column method rlike tests whether a value matches a pattern. Spark uses Java regular-expression syntax. A Python raw string, written with an r prefix, can make patterns easier to read because Python does not process backslashes in the usual way.

email_parts = df.select(
    F.regexp_extract("email", r"^([^@]+)@([^@]+)$", 1).alias("user"),
    F.regexp_extract("email", r"^([^@]+)@([^@]+)$", 2).alias("domain"),
    F.regexp_replace("email", r"\s+", "").alias("email_no_spaces")
)

For a valid address such as ada@example.com, the capture groups return ada and example.com. The pattern here is a simple split around an at sign, not a complete email-address validator. A common detail to remember is that regexp_extract returns an empty string when the pattern or requested group does not match; it does not necessarily return null. If downstream logic must distinguish no match from a real empty value, build that distinction into the expression.

For a fixed substring, a regular expression may be unnecessary. instr gives the position of a substring and returns zero when it is not found; contains can express a simple inclusion check. substring extracts text using a one-based starting position, unlike Python’s zero-based string slicing. Choose a regex when the pattern is variable, not merely because it can perform the task.

Splitting, joining, and combining strings

split divides a string using a regular-expression delimiter and returns an array. It does not treat the delimiter as a literal in every case: characters with special regex meaning, such as a period, need escaping if they are intended literally. You can access a split result with getItem; array indexing through this method is zero-based.

parts = df.select(
    F.split("email", "@").getItem(0).alias("user"),
    F.split("email", "@").getItem(1).alias("domain")
)

For a simple address, those expressions return the text before and after the at sign. If the input does not contain enough parts, an out-of-range array access returns null. Check the input format before treating a split as guaranteed to produce a particular number of elements.

To combine columns, concat joins values directly, while concat_ws joins values with a separator. For example, F.concat_ws(" ", "first_name", "last_name") forms a full-name field with a space between its parts. A practical difference is null handling: concat returns null if an input is null, whereas concat_ws skips null values. Decide whether skipping a missing component is acceptable before choosing the latter.

Nulls, empty strings, and case handling

Null and empty text are different values. A null means the value is missing or unknown; an empty string is present text containing no characters. Trimming a string made entirely of spaces can produce an empty string, while trimming null leaves it null. This distinction affects filters, counts, joins, and validation, so do not assume that a cleanup expression removes missing values automatically.

To identify either kind of missing text, express both conditions explicitly. For a column named name, a filter can use F.col("name").isNull() | (F.trim(F.col("name")) == ""). To replace nulls with a chosen value, functions such as coalesce or DataFrame na.fill may be appropriate, but replacing nulls changes the data’s meaning and should follow a clear rule.

Lowercasing can help normalize case for matching, but it does not solve every text-comparison issue. Case conversion is not a substitute for a carefully defined policy for accents, locale-specific forms, punctuation, or Unicode normalization. If data comes from multiple languages or systems, test representative values and confirm that the intended matching rules are met.

Choosing built-in functions instead of Python UDFs

For common transformations, prefer Spark’s built-in functions over a Python user-defined function (UDF). Built-ins create Spark expressions that the engine can analyze and optimize as part of the query plan. A Python UDF may require data to cross between the JVM and Python execution environments and can restrict some optimizations. A UDF is still useful when the required logic cannot reasonably be expressed with built-in operations, but it is usually not the first choice for trimming, case conversion, splitting, or regex replacement.

It is also better to avoid collecting a large column to Python just to process it with ordinary string methods. Operations such as df.collect() bring rows to the driver, which can consume substantial memory and does not scale like a distributed column expression. Keep transformations in DataFrame operations and collect only small results when that is genuinely needed.

Common mistakes and practical checks

  • Calling a Python method on a column: df.name.lower() is not the standard way to lowercase a Spark column. Use F.lower(F.col("name")).
  • Confusing literal text with a regex: split and regex functions interpret patterns. Escape regex metacharacters or use a non-regex function when the task is a literal search.
  • Assuming Python indexing rules: Spark’s substring starts at position one, while array access with getItem starts at index zero.
  • Ignoring null behavior: A null may flow through an expression, cause a combined result to be null, or be skipped by concat_ws. Verify behavior with null examples.
  • Applying cleanup without checking the result: Preview a sample with select and inspect edge cases such as leading spaces, empty text, unexpected delimiters, and punctuation before using the rule on a full dataset.

For larger jobs, inspect the resulting schema and a small sample after a transformation. If a pattern is central to a data-quality rule, test both expected matches and near-misses. This catches issues such as a regex that accepts only one format, a split that silently produces null for malformed input, or padding that truncates longer identifiers.

Frequently Asked Questions

What is the PySpark function for converting a string to lowercase?
Use pyspark.sql.functions.lower, commonly imported as F.lower. For a DataFrame column named city, use F.lower(F.col("city")) in select or withColumn.

How do I remove spaces from a PySpark string column?
Use trim to remove leading and trailing spaces. To remove whitespace within the text as well, use a regular-expression replacement such as F.regexp_replace("text", r"\s+", ""). Choose the rule carefully: removing internal spaces changes the value.

Does PySpark split use a regular expression?
Yes. The delimiter passed to split is a regex pattern, not always a literal string. Escape special regex characters when they should match themselves, and check how the function handles malformed or missing input.

How do I replace part of a string in PySpark?
Use regexp_replace for pattern-based replacement. For individual character mappings, consider translate. For fixed-position extraction, substring may be simpler than a regex.

Can I use Python string methods on a PySpark column?
Not as though the column were a local Python string. A DataFrame column is a Spark expression. Use built-in functions such as F.trim, F.upper, and F.regexp_replace so Spark can apply the operation to rows across the cluster.

Related

Try it: the Formula Explainer: paste any formula and see each function and reference in plain English.

More in Formulas & functions

Sources: Function behaviour and menu paths: Microsoft Support documentation for Excel. Method & sources

Spotted an error in this guide? Let us know.