Object

org.apache.spark.sql.types.DataType

Direct Known Subclasses:
ArrayType, BinaryType, BooleanType, CalendarIntervalType, DateType, DayTimeIntervalType, GeographyType, GeometryType, MapType, NullType, NumericType, ObjectType, StringType, StructType, TimestampNTZType, TimestampType, TimeType, UserDefinedType, VariantType, YearMonthIntervalType

public abstract class DataType extends Object

The base type of all Spark SQL data types.

Since:
1.3.0

  • Constructor Summary

    Constructors

  • Method Summary

    String representation for the type saved in external catalogs.

    abstract int

    The default size of a value of this data type, used internally for size estimation.

    static boolean

    Compares two types, ignoring nullability of ArrayType, MapType, StructType, and ignoring case sensitivity of field names in StructType.

    static boolean

    Compares two types, ignoring nullability of ArrayType, MapType, StructType.

    static boolean

    Returns true if the two data types share the same "shape", i.e.

    static boolean

    Returns true if the two data types have the same field names in order recursively.

    json()

    The compact JSON representation of this data type.

    Parses data type from a string with schema.

    Readable string representation for the type.

    sql()

    typeName()

    Name of the type used in JSON serialization.

  • Constructor Details

    • DataType

      public DataType()

  • Method Details

    • COLLATIONS_METADATA_KEY

      public static String COLLATIONS_METADATA_KEY()

    • fromDDL

    • parseTypeWithFallback

      Parses data type from a string with schema. It calls parser for schema. If it fails, calls fallbackParser. If the fallback function fails too, combines error message from parser and fallbackParser.

      Parameters:
      schema - The schema string to parse by parser or fallbackParser.
      parser - The function that should be invoke firstly.
      fallbackParser - The function that is called when parser fails.
      Returns:
      The data type parsed from the schema schema.

    • fromJson

    • equalsStructurally

      public static boolean equalsStructurally(DataType from, DataType to, boolean ignoreNullability)

      Returns true if the two data types share the same "shape", i.e. the types are the same, but the field names don't need to be the same.

      Parameters:
      ignoreNullability - whether to ignore nullability when comparing the types
      from - (undocumented)
      to - (undocumented)
      Returns:
      (undocumented)

    • equalsStructurallyByName

      Returns true if the two data types have the same field names in order recursively.

      Parameters:
      from - (undocumented)
      to - (undocumented)
      resolver - (undocumented)
      Returns:
      (undocumented)

    • equalsIgnoreNullability

      public static boolean equalsIgnoreNullability(DataType left, DataType right)

      Compares two types, ignoring nullability of ArrayType, MapType, StructType.

      Parameters:
      left - (undocumented)
      right - (undocumented)
      Returns:
      (undocumented)

    • equalsIgnoreCaseAndNullability

      public static boolean equalsIgnoreCaseAndNullability(DataType from, DataType to)

      Compares two types, ignoring nullability of ArrayType, MapType, StructType, and ignoring case sensitivity of field names in StructType.

      Parameters:
      from - (undocumented)
      to - (undocumented)
      Returns:
      (undocumented)

    • defaultSize

      public abstract int defaultSize()

      The default size of a value of this data type, used internally for size estimation.

      Returns:
      (undocumented)

    • typeName

      Name of the type used in JSON serialization.

    • json

      The compact JSON representation of this data type.

    • prettyJson

      public String prettyJson()

      The pretty (i.e. indented) JSON representation of this data type.

    • simpleString

      public String simpleString()

      Readable string representation for the type.

    • catalogString

      public String catalogString()

      String representation for the type saved in external catalogs.

    • sql