All Implemented Interfaces:
Serializable, scala.collection.immutable.Iterable<StructField>, scala.collection.immutable.Seq<StructField>, scala.collection.immutable.SeqOps<StructField,scala.collection.immutable.Seq,scala.collection.immutable.Seq<StructField>>, scala.collection.Iterable<StructField>, scala.collection.IterableFactoryDefaults<StructField,scala.collection.Iterable>, scala.collection.IterableOnce<StructField>, scala.collection.IterableOnceOps<StructField,scala.collection.Iterable,scala.collection.Iterable>, scala.collection.IterableOps<StructField,scala.collection.Iterable,scala.collection.Iterable>, scala.collection.Seq<StructField>, scala.collection.SeqOps<StructField,scala.collection.Seq,scala.collection.Seq<StructField>>, scala.Equals, scala.Function1<Object,StructField>, scala.PartialFunction<Object,StructField>, scala.Product

A StructType object can be constructed by 


 StructType(fields: Seq[StructField])

For a StructType object, one or multiple StructFields can be extracted by names. If multiple StructFields are extracted, a StructType object will be returned. If a provided name does not have a matching field, it will be ignored. For the case of extracting a single StructField, a null will be returned.

Scala Example: 


 import org.apache.spark.sql._
 import org.apache.spark.sql.types._

 val struct =
   StructType(
     StructField("a", IntegerType, true) ::
     StructField("b", LongType, false) ::
     StructField("c", BooleanType, false) :: Nil)

 // Extract a single StructField.
 val singleField = struct("b")
 // singleField: StructField = StructField(b,LongType,false)

 // If this struct does not have a field called "d", it throws an exception.
 struct("d")
 // java.lang.IllegalArgumentException: d does not exist.
 //   ...

 // Extract multiple StructFields. Field names are provided in a set.
 // A StructType object will be returned.
 val twoFields = struct(Set("b", "c"))
 // twoFields: StructType =
 //   StructType(StructField(b,LongType,false), StructField(c,BooleanType,false))

 // Any names without matching fields will throw an exception.
 // For the case shown below, an exception is thrown due to "d".
 struct(Set("b", "c", "d"))
 // java.lang.IllegalArgumentException: d does not exist.
 //    ...

A Row object is used as a value of the StructType.

Scala Example: 


 import org.apache.spark.sql._
 import org.apache.spark.sql.types._

 val innerStruct =
   StructType(
     StructField("f1", IntegerType, true) ::
     StructField("f2", LongType, false) ::
     StructField("f3", BooleanType, false) :: Nil)

 val struct = StructType(
   StructField("a", innerStruct, true) :: Nil)

 // Create a Row with the schema defined by struct
 val row = Row(Row(1, 2, true))
Since:
1.3.0
See Also:

  • Nested Class Summary

    Nested classes/interfaces inherited from interface scala.Function1

    scala.Function1.UnliftOps<A extends Object,B extends Object>, scala.Function1.UnliftOps$

    Nested classes/interfaces inherited from interface scala.collection.IterableOps

    scala.collection.IterableOps.SizeCompareOps, scala.collection.IterableOps.SizeCompareOps$, scala.collection.IterableOps.WithFilter<A extends Object,CC extends Object>

    Nested classes/interfaces inherited from interface scala.PartialFunction

    scala.PartialFunction.ElementWiseExtractor<A extends Object,B extends Object>, scala.PartialFunction.ElementWiseExtractor$

  • Constructor Summary

    Constructors

    No-arg constructor for kryo.

  • Method Summary

    Creates a new StructType by adding a new nullable field with no metadata where the dataType is specified as a String.

    Creates a new StructType by adding a new field with no metadata where the dataType is specified as a String.

    Creates a new StructType by adding a new field and specifying metadata where the dataType is specified as a String.

    Creates a new StructType by adding a new field and specifying metadata where the dataType is specified as a String.

    Creates a new StructType by adding a new nullable field with no metadata.

    Creates a new StructType by adding a new field with no metadata.

    Creates a new StructType by adding a new field and specifying metadata.

    Creates a new StructType by adding a new field and specifying metadata.

    apply(int fieldIndex)

    apply(scala.collection.immutable.Set<String> names)

    Returns a StructType containing StructFields of the given names, preserving the original order of fields.

    String representation for the type saved in external catalogs.

    int

    The default size of a value of the StructType is the total default sizes of all field types.

    boolean

    int

    Returns the index of a given field.

    Returns all field names in an array.

    fields()

    Returns a StructType that contains missing fields recursively from source to target.

    Creates StructType for a given DDL-formatted string, which is a comma separated list of field definitions, e.g., a INT, b STRING.

    int

    hashCode()

    iterator()

    int

    length()

    names()

    Returns all field names in an array.

    org.apache.spark.sql.catalyst.util.CaseInsensitiveMap<DataType>

    void

    Readable string representation for the type.

    sql()

    toDDL()

    Returns a string containing a schema in DDL format.

    Returns the same data type but set all nullability fields are true (StructField.nullable, ArrayType.containsNull, and MapType.valueContainsNull).

    toString()

    treeString(int maxDepth)

    Methods inherited from interface scala.Function1

    apply, apply$mcDD$sp, apply$mcDF$sp, apply$mcDI$sp, apply$mcDJ$sp, apply$mcFD$sp, apply$mcFF$sp, apply$mcFI$sp, apply$mcFJ$sp, apply$mcID$sp, apply$mcIF$sp, apply$mcII$sp, apply$mcIJ$sp, apply$mcJD$sp, apply$mcJF$sp, apply$mcJI$sp, apply$mcJJ$sp, apply$mcVD$sp, apply$mcVF$sp, apply$mcVI$sp, apply$mcVJ$sp, apply$mcZD$sp, apply$mcZF$sp, apply$mcZI$sp, apply$mcZJ$sp, compose

    Methods inherited from interface scala.collection.Iterable

    className, coll, collectionClassName, lazyZip, seq, toIterable

    Methods inherited from interface scala.collection.IterableFactoryDefaults

    empty, fromSpecific, newSpecificBuilder

    Methods inherited from interface scala.collection.IterableOnce

    knownSize, stepper

    Methods inherited from interface scala.collection.IterableOnceOps

    $colon$bslash, $div$colon, addString, addString, addString, aggregate, collectFirst, copyToArray, copyToArray, copyToArray, copyToBuffer, corresponds, count, exists, find, fold, foldLeft, foldRight, forall, foreach, hasDefiniteSize, max, maxBy, maxByOption, maxOption, min, minBy, minByOption, minOption, mkString, mkString, mkString, nonEmpty, product, reduce, reduceLeft, reduceLeftOption, reduceOption, reduceRight, reduceRightOption, reversed, sum, to, toArray, toBuffer, toIndexedSeq, toIterator, toList, toMap, toSet, toStream, toVector

    Methods inherited from interface scala.collection.IterableOps

    $plus$plus, collect, companion, drop, dropRight, dropWhile, filter, filterNot, flatMap, flatten, groupBy, grouped, groupMap, groupMapReduce, head, headOption, init, inits, isTraversableAgain, last, lastOption, map, partition, partitionMap, repr, scan, scanLeft, scanRight, sizeIs, slice, sliding, sliding, span, splitAt, tail, tails, take, takeRight, takeWhile, tapEach, toTraversable, transpose, unzip, unzip3, view, withFilter, zip, zipAll, zipWithIndex

    Methods inherited from interface scala.PartialFunction

    andThen, andThen, applyOrElse, compose, elementWise, isDefinedAt, lift, orElse, runWith, unapply

    Methods inherited from interface scala.Product

    productArity, productElement, productElementName, productElementNames, productIterator, productPrefix

    Methods inherited from interface scala.collection.immutable.Seq

    iterableFactory, toSeq

    Methods inherited from interface scala.collection.Seq

    canEqual, stringPrefix

    Methods inherited from interface scala.collection.SeqOps

    $colon$plus, $colon$plus$plus, $plus$colon, $plus$plus$colon, appended, appendedAll, combinations, concat, contains, containsSlice, corresponds, diff, distinct, distinctBy, endsWith, findLast, indexOf, indexOf, indexOfSlice, indexOfSlice, indexWhere, indexWhere, indices, intersect, isDefinedAt, isEmpty, lastIndexOf, lastIndexOf$default$2, lastIndexOfSlice, lastIndexOfSlice, lastIndexWhere, lastIndexWhere, lengthCompare, lengthCompare, lengthIs, occCounts, padTo, patch, permutations, prefixLength, prepended, prependedAll, reverse, reverseIterator, reverseMap, sameElements, scala$collection$SeqOps$$toGenericSeq, search, search, segmentLength, segmentLength, size, sizeCompare, sizeCompare, sortBy, sorted, sortWith, startsWith, startsWith$default$2, union, updated, view

  • Constructor Details

    • StructType

    • StructType

      public StructType()

      No-arg constructor for kryo.

  • Method Details

    • fromDDL

      Creates StructType for a given DDL-formatted string, which is a comma separated list of field definitions, e.g., a INT, b STRING.

      Parameters:
      ddl - (undocumented)
      Returns:
      (undocumented)
      Since:
      2.2.0

    • findMissingFields

      Returns a StructType that contains missing fields recursively from source to target. Note that this doesn't support looking into array type and map type recursively.

      Parameters:
      source - (undocumented)
      target - (undocumented)
      resolver - (undocumented)
      Returns:
      (undocumented)

    • fields

    • fieldNames

      public String[] fieldNames()

      Returns all field names in an array.

    • names

      Returns all field names in an array. This is an alias of fieldNames.

      Returns:
      (undocumented)
      Since:
      2.4.0

    • nameToDataType

      public scala.collection.immutable.Map<String,DataType> nameToDataType()

    • nameToDataTypeCaseInsensitive

      public org.apache.spark.sql.catalyst.util.CaseInsensitiveMap<DataType> nameToDataTypeCaseInsensitive()

    • equals

      public boolean equals(Object that)

      Specified by:
      equals in interface scala.Equals
      Specified by:
      equals in interface scala.collection.Seq<StructField>
      Overrides:
      equals in class Object

    • toString

      Specified by:
      toString in interface scala.Function1<Object,StructField>
      Specified by:
      toString in interface scala.collection.Iterable<StructField>
      Specified by:
      toString in interface scala.collection.Seq<StructField>
      Overrides:
      toString in class Object

    • hashCode

      public int hashCode()

      Specified by:
      hashCode in interface scala.collection.Seq<StructField>
      Overrides:
      hashCode in class Object

    • add

      Creates a new StructType by adding a new field. 

      
 val struct = (new StructType)
   .add(StructField("a", IntegerType, true))
   .add(StructField("b", LongType, false))
   .add(StructField("c", StringType, true))
      Parameters:
      field - (undocumented)
      Returns:
      (undocumented)

    • add

      Creates a new StructType by adding a new nullable field with no metadata.

      val struct = (new StructType) .add("a", IntegerType) .add("b", LongType) .add("c", StringType)

      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      Returns:
      (undocumented)

    • add

      Creates a new StructType by adding a new field with no metadata.

      val struct = (new StructType) .add("a", IntegerType, true) .add("b", LongType, false) .add("c", StringType, true)

      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      nullable - (undocumented)
      Returns:
      (undocumented)

    • add

      Creates a new StructType by adding a new field and specifying metadata. 

      
 val struct = (new StructType)
   .add("a", IntegerType, true, Metadata.empty)
   .add("b", LongType, false, Metadata.empty)
   .add("c", StringType, true, Metadata.empty)
      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      nullable - (undocumented)
      metadata - (undocumented)
      Returns:
      (undocumented)

    • add

      Creates a new StructType by adding a new field and specifying metadata. 

      
 val struct = (new StructType)
   .add("a", IntegerType, true, "comment1")
   .add("b", LongType, false, "comment2")
   .add("c", StringType, true, "comment3")
      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      nullable - (undocumented)
      comment - (undocumented)
      Returns:
      (undocumented)

    • add

      Creates a new StructType by adding a new nullable field with no metadata where the dataType is specified as a String. 

      
 val struct = (new StructType)
   .add("a", "int")
   .add("b", "long")
   .add("c", "string")
      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      Returns:
      (undocumented)

    • add

      Creates a new StructType by adding a new field with no metadata where the dataType is specified as a String. 

      
 val struct = (new StructType)
   .add("a", "int", true)
   .add("b", "long", false)
   .add("c", "string", true)
      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      nullable - (undocumented)
      Returns:
      (undocumented)

    • add

      Creates a new StructType by adding a new field and specifying metadata where the dataType is specified as a String. 

      
 val struct = (new StructType)
   .add("a", "int", true, Metadata.empty)
   .add("b", "long", false, Metadata.empty)
   .add("c", "string", true, Metadata.empty)
      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      nullable - (undocumented)
      metadata - (undocumented)
      Returns:
      (undocumented)

    • add

      Creates a new StructType by adding a new field and specifying metadata where the dataType is specified as a String. 

      
 val struct = (new StructType)
   .add("a", "int", true, "comment1")
   .add("b", "long", false, "comment2")
   .add("c", "string", true, "comment3")
      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      nullable - (undocumented)
      comment - (undocumented)
      Returns:
      (undocumented)

    • apply

      Parameters:
      name - (undocumented)
      Returns:
      (undocumented)
      Throws:
      IllegalArgumentException - if a field with the given name does not exist

    • apply

      public StructType apply(scala.collection.immutable.Set<String> names)

      Returns a StructType containing StructFields of the given names, preserving the original order of fields.

      Parameters:
      names - (undocumented)
      Returns:
      (undocumented)
      Throws:
      IllegalArgumentException - if at least one given field name does not exist

    • fieldIndex

      public int fieldIndex(String name)

      Returns the index of a given field.

      Parameters:
      name - (undocumented)
      Returns:
      (undocumented)
      Throws:
      IllegalArgumentException - if a field with the given name does not exist

    • treeString

      public String treeString()

    • treeString

      public String treeString(int maxDepth)

    • printTreeString

      public void printTreeString()

    • apply

      Specified by:
      apply in interface scala.collection.SeqOps<StructField,scala.collection.Seq,scala.collection.Seq<StructField>>

    • length

      public int length()

      Specified by:
      length in interface scala.collection.SeqOps<StructField,scala.collection.Seq,scala.collection.Seq<StructField>>

    • iterator

      public scala.collection.Iterator<StructField> iterator()

      Specified by:
      iterator in interface scala.collection.IterableOnce<StructField>

    • defaultSize

      public int defaultSize()

      The default size of a value of the StructType is the total default sizes of all field types.

      Specified by:
      defaultSize in class DataType
      Returns:
      (undocumented)

    • simpleString

      public String simpleString()

      Description copied from class: DataType

      Readable string representation for the type.

      Overrides:
      simpleString in class DataType

    • catalogString

      public String catalogString()

      Description copied from class: DataType

      String representation for the type saved in external catalogs.

      Overrides:
      catalogString in class DataType

    • sql

      Overrides:
      sql in class DataType

    • toDDL

      Returns a string containing a schema in DDL format. For example, the following value: StructType(Seq(StructField("eventId", IntegerType), StructField("s", StringType))) will be converted to eventId INT, s STRING. The returned DDL schema can be used in a table creation.

      Returns:
      (undocumented)
      Since:
      2.4.0

    • toNullable

      Returns the same data type but set all nullability fields are true (StructField.nullable, ArrayType.containsNull, and MapType.valueContainsNull).

      Returns:
      (undocumented)
      Since:
      4.0.0