Home Original page

Column (Spark 4.2.0 JavaDoc)

Method Details

  • isin

    A boolean expression that is evaluated to true if the value of this expression is contained by the evaluated values of the arguments.

    Note: Since the type of the elements in the list are inferred only during the run time, the elements will be "up-casted" to the most common type for comparison. For eg: 1) In the case of "Int vs String", the "Int" will be up-casted to "String" and the comparison will look like "String vs String". 2) In the case of "Float vs Double", the "Float" will be up-casted to "Double" and the comparison will look like "Double vs Double"

    Parameters:
    list - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.5.0

  • node

    public org.apache.spark.sql.internal.ColumnNode node()

  • toString

    Overrides:
    toString in class Object

  • equals

    public boolean equals(Object that)

    Overrides:
    equals in class Object

  • hashCode

    public int hashCode()

    Overrides:
    hashCode in class Object

  • as

    Provides a type hint about the expected return value of this column. This information can be used by operations such as select on a Dataset to automatically convert the results into the correct JVM types.

    Parameters:
    evidence$1 - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.6.0

  • apply

    Extracts a value or values from a complex type. The following types of extraction are supported:

    • Given an Array, an integer ordinal can be used to retrieve a single value.
    • Given a Map, a key of the correct type can be used to retrieve an individual value.
    • Given a Struct, a string fieldName can be used to extract that field.
    • Given an Array of Structs, a string fieldName can be used to extract filed of every struct in that array, and return an Array of fields.
    Parameters:
    extraction - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.4.0

  • equalTo

    Equality test. 

    
   // Scala:
   df.filter( df("colA") === df("colB") )

   // Java
   import static org.apache.spark.sql.functions.*;
   df.filter( col("colA").equalTo(col("colB")) );
    Parameters:
    other - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • notEqual

    Inequality test. 

    
   // Scala:
   df.select( df("colA") !== df("colB") )
   df.select( !(df("colA") === df("colB")) )

   // Java:
   import static org.apache.spark.sql.functions.*;
   df.filter( col("colA").notEqual(col("colB")) );
    Parameters:
    other - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • gt

    Greater than. 

    
   // Scala: The following selects people older than 21.
   people.select( people("age") > lit(21) )

   // Java:
   import static org.apache.spark.sql.functions.*;
   people.select( people.col("age").gt(21) );
    Parameters:
    other - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • lt

    Less than. 

    
   // Scala: The following selects people younger than 21.
   people.select( people("age") < 21 )

   // Java:
   people.select( people.col("age").lt(21) );
    Parameters:
    other - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • leq

    Less than or equal to. 

    
   // Scala: The following selects people age 21 or younger than 21.
   people.select( people("age") <= 21 )

   // Java:
   people.select( people.col("age").leq(21) );
    Parameters:
    other - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • geq

    Greater than or equal to an expression. 

    
   // Scala: The following selects people age 21 or older than 21.
   people.select( people("age") >= 21 )

   // Java:
   people.select( people.col("age").geq(21) )
    Parameters:
    other - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • eqNullSafe

    Equality test that is safe for null values.

    Parameters:
    other - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • when

    Evaluates a list of conditions and returns one of multiple possible result expressions. If otherwise is not defined at the end, null is returned for unmatched conditions. 

    
   // Example: encoding gender string column into integer.

   // Scala:
   people.select(when(people("gender") === "male", 0)
     .when(people("gender") === "female", 1)
     .otherwise(2))

   // Java:
   people.select(when(col("gender").equalTo("male"), 0)
     .when(col("gender").equalTo("female"), 1)
     .otherwise(2))
    Parameters:
    condition - (undocumented)
    value - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.4.0

  • otherwise

    Evaluates a list of conditions and returns one of multiple possible result expressions. If otherwise is not defined at the end, null is returned for unmatched conditions. 

    
   // Example: encoding gender string column into integer.

   // Scala:
   people.select(when(people("gender") === "male", 0)
     .when(people("gender") === "female", 1)
     .otherwise(2))

   // Java:
   people.select(when(col("gender").equalTo("male"), 0)
     .when(col("gender").equalTo("female"), 1)
     .otherwise(2))
    Parameters:
    value - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.4.0

  • between

    True if the current column is between the lower bound and upper bound, inclusive.

    Parameters:
    lowerBound - (undocumented)
    upperBound - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.4.0

  • isNaN

    True if the current expression is NaN.

    Returns:
    (undocumented)
    Since:
    1.5.0

  • isNull

    True if the current expression is null.

    Returns:
    (undocumented)
    Since:
    1.3.0

  • isNotNull

    public Column isNotNull()

    True if the current expression is NOT null.

    Returns:
    (undocumented)
    Since:
    1.3.0

  • or

    Boolean OR. 

    
   // Scala: The following selects people that are in school or employed.
   people.filter( people("inSchool") || people("isEmployed") )

   // Java:
   people.filter( people.col("inSchool").or(people.col("isEmployed")) );
    Parameters:
    other - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • and

    Boolean AND. 

    
   // Scala: The following selects people that are in school and employed at the same time.
   people.select( people("inSchool") && people("isEmployed") )

   // Java:
   people.select( people.col("inSchool").and(people.col("isEmployed")) );
    Parameters:
    other - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • plus

    Sum of this expression and another expression. 

    
   // Scala: The following selects the sum of a person's height and weight.
   people.select( people("height") + people("weight") )

   // Java:
   people.select( people.col("height").plus(people.col("weight")) );
    Parameters:
    other - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • minus

    Subtraction. Subtract the other expression from this expression. 

    
   // Scala: The following selects the difference between people's height and their weight.
   people.select( people("height") - people("weight") )

   // Java:
   people.select( people.col("height").minus(people.col("weight")) );
    Parameters:
    other - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • multiply

    Multiplication of this expression and another expression. 

    
   // Scala: The following multiplies a person's height by their weight.
   people.select( people("height") * people("weight") )

   // Java:
   people.select( people.col("height").multiply(people.col("weight")) );
    Parameters:
    other - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • divide

    Division this expression by another expression. 

    
   // Scala: The following divides a person's height by their weight.
   people.select( people("height") / people("weight") )

   // Java:
   people.select( people.col("height").divide(people.col("weight")) );
    Parameters:
    other - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • mod

    Modulo (a.k.a. remainder) expression.

    Parameters:
    other - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • isin

    public Column isin(scala.collection.immutable.Seq<Object> list)

    A boolean expression that is evaluated to true if the value of this expression is contained by the evaluated values of the arguments.

    Note: Since the type of the elements in the list are inferred only during the run time, the elements will be "up-casted" to the most common type for comparison. For eg: 1) In the case of "Int vs String", the "Int" will be up-casted to "String" and the comparison will look like "String vs String". 2) In the case of "Float vs Double", the "Float" will be up-casted to "Double" and the comparison will look like "Double vs Double"

    Parameters:
    list - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.5.0

  • isInCollection

    public Column isInCollection(scala.collection.Iterable<?> values)

    A boolean expression that is evaluated to true if the value of this expression is contained by the provided collection.

    Note: Since the type of the elements in the collection are inferred only during the run time, the elements will be "up-casted" to the most common type for comparison. For eg: 1) In the case of "Int vs String", the "Int" will be up-casted to "String" and the comparison will look like "String vs String". 2) In the case of "Float vs Double", the "Float" will be up-casted to "Double" and the comparison will look like "Double vs Double"

    Parameters:
    values - (undocumented)
    Returns:
    (undocumented)
    Since:
    2.4.0

  • isInCollection

    A boolean expression that is evaluated to true if the value of this expression is contained by the provided collection.

    Note: Since the type of the elements in the collection are inferred only during the run time, the elements will be "up-casted" to the most common type for comparison. For eg: 1) In the case of "Int vs String", the "Int" will be up-casted to "String" and the comparison will look like "String vs String". 2) In the case of "Float vs Double", the "Float" will be up-casted to "Double" and the comparison will look like "Double vs Double"

    Parameters:
    values - (undocumented)
    Returns:
    (undocumented)
    Since:
    2.4.0

  • isin

    A boolean expression that is evaluated to true if the value of this expression is contained by the provided Dataset/DataFrame.

    Parameters:
    ds - (undocumented)
    Returns:
    (undocumented)
    Since:
    4.1.0

  • like

    SQL like expression. Returns a boolean column based on a SQL LIKE match.

    Parameters:
    literal - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • rlike

    SQL RLIKE expression (LIKE with Regex). Returns a boolean column based on a regex match.

    Parameters:
    literal - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • ilike

    SQL ILIKE expression (case insensitive LIKE).

    Parameters:
    literal - (undocumented)
    Returns:
    (undocumented)
    Since:
    3.3.0

  • getItem

    An expression that gets an item at position ordinal out of an array, or gets a value by key key in a MapType.

    Parameters:
    key - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • withField

    An expression that adds/replaces field in StructType by name. 

    
   val df = sql("SELECT named_struct('a', 1, 'b', 2) struct_col")
   df.select($"struct_col".withField("c", lit(3)))
   // result: {"a":1,"b":2,"c":3}

   val df = sql("SELECT named_struct('a', 1, 'b', 2) struct_col")
   df.select($"struct_col".withField("b", lit(3)))
   // result: {"a":1,"b":3}

   val df = sql("SELECT CAST(NULL AS struct<a:int,b:int>) struct_col")
   df.select($"struct_col".withField("c", lit(3)))
   // result: null of type struct<a:int,b:int,c:int>

   val df = sql("SELECT named_struct('a', 1, 'b', 2, 'b', 3) struct_col")
   df.select($"struct_col".withField("b", lit(100)))
   // result: {"a":1,"b":100,"b":100}

   val df = sql("SELECT named_struct('a', named_struct('a', 1, 'b', 2)) struct_col")
   df.select($"struct_col".withField("a.c", lit(3)))
   // result: {"a":{"a":1,"b":2,"c":3}}

   val df = sql("SELECT named_struct('a', named_struct('b', 1), 'a', named_struct('c', 2)) struct_col")
   df.select($"struct_col".withField("a.c", lit(3)))
   // result: org.apache.spark.sql.AnalysisException: Ambiguous reference to fields

    This method supports adding/replacing nested fields directly e.g. 

    
   val df = sql("SELECT named_struct('a', named_struct('a', 1, 'b', 2)) struct_col")
   df.select($"struct_col".withField("a.c", lit(3)).withField("a.d", lit(4)))
   // result: {"a":{"a":1,"b":2,"c":3,"d":4}}

    However, if you are going to add/replace multiple nested fields, it is more optimal to extract out the nested struct before adding/replacing multiple fields e.g. 

    
   val df = sql("SELECT named_struct('a', named_struct('a', 1, 'b', 2)) struct_col")
   df.select($"struct_col".withField("a", $"struct_col.a".withField("c", lit(3)).withField("d", lit(4))))
   // result: {"a":{"a":1,"b":2,"c":3,"d":4}}
    Parameters:
    fieldName - (undocumented)
    col - (undocumented)
    Returns:
    (undocumented)
    Since:
    3.1.0

  • dropFields

    public Column dropFields(scala.collection.immutable.Seq<String> fieldNames)

    An expression that drops fields in StructType by name. This is a no-op if schema doesn't contain field name(s). 

    
   val df = sql("SELECT named_struct('a', 1, 'b', 2) struct_col")
   df.select($"struct_col".dropFields("b"))
   // result: {"a":1}

   val df = sql("SELECT named_struct('a', 1, 'b', 2) struct_col")
   df.select($"struct_col".dropFields("c"))
   // result: {"a":1,"b":2}

   val df = sql("SELECT named_struct('a', 1, 'b', 2, 'c', 3) struct_col")
   df.select($"struct_col".dropFields("b", "c"))
   // result: {"a":1}

   val df = sql("SELECT named_struct('a', 1, 'b', 2) struct_col")
   df.select($"struct_col".dropFields("a", "b"))
   // result: org.apache.spark.sql.AnalysisException: [DATATYPE_MISMATCH.CANNOT_DROP_ALL_FIELDS] Cannot resolve "update_fields(struct_col, dropfield(), dropfield())" due to data type mismatch: Cannot drop all fields in struct.;

   val df = sql("SELECT CAST(NULL AS struct<a:int,b:int>) struct_col")
   df.select($"struct_col".dropFields("b"))
   // result: null of type struct<a:int>

   val df = sql("SELECT named_struct('a', 1, 'b', 2, 'b', 3) struct_col")
   df.select($"struct_col".dropFields("b"))
   // result: {"a":1}

   val df = sql("SELECT named_struct('a', named_struct('a', 1, 'b', 2)) struct_col")
   df.select($"struct_col".dropFields("a.b"))
   // result: {"a":{"a":1}}

   val df = sql("SELECT named_struct('a', named_struct('b', 1), 'a', named_struct('c', 2)) struct_col")
   df.select($"struct_col".dropFields("a.c"))
   // result: org.apache.spark.sql.AnalysisException: Ambiguous reference to fields

    This method supports dropping multiple nested fields directly e.g. 

    
   val df = sql("SELECT named_struct('a', named_struct('a', 1, 'b', 2)) struct_col")
   df.select($"struct_col".dropFields("a.b", "a.c"))
   // result: {"a":{"a":1}}

    However, if you are going to drop multiple nested fields, it is more optimal to extract out the nested struct before dropping multiple fields from it e.g. 

    
   val df = sql("SELECT named_struct('a', named_struct('a', 1, 'b', 2)) struct_col")
   df.select($"struct_col".withField("a", $"struct_col.a".dropFields("b", "c")))
   // result: {"a":{"a":1}}
    Parameters:
    fieldNames - (undocumented)
    Returns:
    (undocumented)
    Since:
    3.1.0

  • getField

    An expression that gets a field by name in a StructType.

    Parameters:
    fieldName - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • substr

    An expression that returns a substring.

    Parameters:
    startPos - expression for the starting position.
    len - expression for the length of the substring.
    Returns:
    (undocumented)
    Since:
    1.3.0

  • substr

    public Column substr(int startPos, int len)

    An expression that returns a substring.

    Parameters:
    startPos - starting position.
    len - length of the substring.
    Returns:
    (undocumented)
    Since:
    1.3.0

  • contains

    Contains the other element. Returns a boolean column based on a string match.

    Parameters:
    other - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • startsWith

    String starts with. Returns a boolean column based on a string match.

    Parameters:
    other - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • startsWith

    String starts with another string literal. Returns a boolean column based on a string match.

    Parameters:
    literal - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • endsWith

    String ends with. Returns a boolean column based on a string match.

    Parameters:
    other - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • endsWith

    String ends with another string literal. Returns a boolean column based on a string match.

    Parameters:
    literal - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • alias

    Gives the column an alias. Same as as. 

    
   // Renames colA to colB in select output.
   df.select($"colA".alias("colB"))
    Parameters:
    alias - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.4.0

  • as

    Gives the column an alias. 

    
   // Renames colA to colB in select output.
   df.select($"colA".as("colB"))

    If the current column has metadata associated with it, this metadata will be propagated to the new column. If this not desired, use the API as(alias: String, metadata: Metadata) with explicit metadata.

    Parameters:
    alias - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • as

    public Column as(scala.collection.immutable.Seq<String> aliases)

    (Scala-specific) Assigns the given aliases to the results of a table generating function. 

    
   // Renames colA to colB in select output.
   df.select(explode($"myMap").as("key" :: "value" :: Nil))
    Parameters:
    aliases - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.4.0

  • as

    Assigns the given aliases to the results of a table generating function. 

    
   // Renames colA to colB in select output.
   df.select(explode($"myMap").as("key" :: "value" :: Nil))
    Parameters:
    aliases - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.4.0

  • as

    public Column as(scala.Symbol alias)

    Gives the column an alias. 

    
   // Renames colA to colB in select output.
   df.select($"colA".as("colB"))

    If the current column has metadata associated with it, this metadata will be propagated to the new column. If this not desired, use the API as(alias: String, metadata: Metadata) with explicit metadata.

    Parameters:
    alias - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • as

    Gives the column an alias with metadata. 

    
   val metadata: Metadata = ...
   df.select($"colA".as("colB", metadata))
    Parameters:
    alias - (undocumented)
    metadata - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • name

    Gives the column a name (alias). 

    
   // Renames colA to colB in select output.
   df.select($"colA".name("colB"))

    If the current column has metadata associated with it, this metadata will be propagated to the new column. If this not desired, use the API as(alias: String, metadata: Metadata) with explicit metadata.

    Parameters:
    alias - (undocumented)
    Returns:
    (undocumented)
    Since:
    2.0.0

  • cast

    Casts the column to a different data type. 

    
   // Casts colA to IntegerType.
   import org.apache.spark.sql.types.IntegerType
   df.select(df("colA").cast(IntegerType))

   // equivalent to
   df.select(df("colA").cast("int"))
    Parameters:
    to - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • cast

    Casts the column to a different data type, using the canonical string representation of the type. The supported types are: string, boolean, byte, short, int, long, float, double, decimal, date, timestamp. 

    
   // Casts colA to integer.
   df.select(df("colA").cast("int"))
    Parameters:
    to - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.3.0

  • try_cast

    Casts the column to a different data type and the result is null on failure. 

    
   // Casts colA to IntegerType.
   import org.apache.spark.sql.types.IntegerType
   df.select(df("colA").try_cast(IntegerType))

   // equivalent to
   df.select(df("colA").try_cast("int"))
    Parameters:
    to - (undocumented)
    Returns:
    (undocumented)
    Since:
    4.0.0

  • try_cast

    Casts the column to a different data type and the result is null on failure. 

    
   // Casts colA to integer.
   df.select(df("colA").try_cast("int"))
    Parameters:
    to - (undocumented)
    Returns:
    (undocumented)
    Since:
    4.0.0

  • desc

    Returns a sort expression based on the descending order of the column. 

    
   // Scala
   df.sort(df("age").desc)

   // Java
   df.sort(df.col("age").desc());
    Returns:
    (undocumented)
    Since:
    1.3.0

  • desc_nulls_first

    public Column desc_nulls_first()

    Returns a sort expression based on the descending order of the column, and null values appear before non-null values. 

    
   // Scala: sort a DataFrame by age column in descending order and null values appearing first.
   df.sort(df("age").desc_nulls_first)

   // Java
   df.sort(df.col("age").desc_nulls_first());
    Returns:
    (undocumented)
    Since:
    2.1.0

  • desc_nulls_last

    public Column desc_nulls_last()

    Returns a sort expression based on the descending order of the column, and null values appear after non-null values. 

    
   // Scala: sort a DataFrame by age column in descending order and null values appearing last.
   df.sort(df("age").desc_nulls_last)

   // Java
   df.sort(df.col("age").desc_nulls_last());
    Returns:
    (undocumented)
    Since:
    2.1.0

  • asc

    Returns a sort expression based on ascending order of the column. 

    
   // Scala: sort a DataFrame by age column in ascending order.
   df.sort(df("age").asc)

   // Java
   df.sort(df.col("age").asc());
    Returns:
    (undocumented)
    Since:
    1.3.0

  • asc_nulls_first

    public Column asc_nulls_first()

    Returns a sort expression based on ascending order of the column, and null values return before non-null values. 

    
   // Scala: sort a DataFrame by age column in ascending order and null values appearing first.
   df.sort(df("age").asc_nulls_first)

   // Java
   df.sort(df.col("age").asc_nulls_first());
    Returns:
    (undocumented)
    Since:
    2.1.0

  • asc_nulls_last

    public Column asc_nulls_last()

    Returns a sort expression based on ascending order of the column, and null values appear after non-null values. 

    
   // Scala: sort a DataFrame by age column in ascending order and null values appearing last.
   df.sort(df("age").asc_nulls_last)

   // Java
   df.sort(df.col("age").asc_nulls_last());
    Returns:
    (undocumented)
    Since:
    2.1.0

  • explain

    public void explain(boolean extended)

    Prints the expression to the console for debugging purposes.

    Parameters:
    extended - (undocumented)
    Since:
    1.3.0

  • bitwiseOR

    Compute bitwise OR of this expression with another expression. 

    
   df.select($"colA".bitwiseOR($"colB"))
    Parameters:
    other - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.4.0

  • bitwiseAND

    Compute bitwise AND of this expression with another expression. 

    
   df.select($"colA".bitwiseAND($"colB"))
    Parameters:
    other - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.4.0

  • bitwiseXOR

    Compute bitwise XOR of this expression with another expression. 

    
   df.select($"colA".bitwiseXOR($"colB"))
    Parameters:
    other - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.4.0

  • over

    Defines a windowing column. 

    
   val w = Window.partitionBy("name").orderBy("id")
   df.select(
     sum("price").over(w.rangeBetween(Window.unboundedPreceding, 2)),
     avg("price").over(w.rowsBetween(Window.currentRow, 4))
   )
    Parameters:
    window - (undocumented)
    Returns:
    (undocumented)
    Since:
    1.4.0

  • over

    Defines an empty analytic clause. In this case the analytic function is applied and presented for all rows in the result set. 

    
   df.select(
     sum("price").over(),
     avg("price").over()
   )
    Returns:
    (undocumented)
    Since:
    2.0.0

  • outer

    Mark this column as an outer column if its expression refers to columns from an outer query. This is used to trigger lazy analysis of Spark Classic DataFrame, so that we can use it to build subquery expressions. Spark Connect DataFrame is always lazily analyzed and does not need to use this function. 

    
   // Spark can't analyze this `df` now as it doesn't know how to resolve `t1.col`.
   val df = spark.table("t2").where($"t2.col" === $"t1.col".outer())

   // Since this `df` is lazily analyzed, you won't see any error until you try to execute it.
   df.collect()  // Fails with UNRESOLVED_COLUMN error.

   // Now Spark can resolve `t1.col` with the outer plan `spark.table("t1")`.
   spark.table("t1").where(df.exists())
    Returns:
    (undocumented)
    Since:
    4.0.0

  • transform

    Concise syntax for chaining custom transformations. 

    
   def addPrefix(c: Column): Column = concat(lit("prefix_"), c)

   df.select($"name".transform(addPrefix))

   // Chaining multiple transformations
   df.select($"name".transform(addPrefix).transform(upper))
    Parameters:
    f - (undocumented)
    Returns:
    (undocumented)
    Since:
    4.1.0