All Implemented Interfaces:

Serializable, JavaRDDLike<T,JavaRDD<T>>

public class JavaRDD<T> extends Object

See Also:

• Serialized Form

• Constructor Summary

  Constructors

  JavaRDD(RDD<T> rdd, scala.reflect.ClassTag<T> classTag)

• Method Summary

  cache()

  Persist this RDD with the default storage level (MEMORY_ONLY).

  scala.reflect.ClassTag<T>

  classTag()

  coalesce(int numPartitions)

  Return a new RDD that is reduced into numPartitions partitions.

  coalesce(int numPartitions, boolean shuffle)

  Return a new RDD that is reduced into numPartitions partitions.

  distinct()

  Return a new RDD containing the distinct elements in this RDD.

  distinct(int numPartitions)

  Return a new RDD containing the distinct elements in this RDD.

  Return a new RDD containing only the elements that satisfy a predicate.

  fromRDD(RDD<T> rdd, scala.reflect.ClassTag<T> evidence$1)

  Get the ResourceProfile specified with this RDD or None if it wasn't specified.

  Return the intersection of this RDD and another one.

  Set this RDD's storage level to persist its values across operations after the first time it is computed.

  randomSplit(double[] weights)

  Randomly splits this RDD with the provided weights.

  randomSplit(double[] weights, long seed)

  Randomly splits this RDD with the provided weights.

  rdd()

  repartition(int numPartitions)

  Return a new RDD that has exactly numPartitions partitions.

  sample(boolean withReplacement, double fraction)

  Return a sampled subset of this RDD with a random seed.

  sample(boolean withReplacement, double fraction, long seed)

  Return a sampled subset of this RDD, with a user-supplied seed.

  Assign a name to this RDD

  sortBy(Function<T,S> f, boolean ascending, int numPartitions)

  Return this RDD sorted by the given key function.

  Return an RDD with the elements from this that are not in other.

  subtract(JavaRDD<T> other, int numPartitions)

  Return an RDD with the elements from this that are not in other.

  Return an RDD with the elements from this that are not in other.

  static <T> RDD<T>

  toString()

  Return the union of this RDD and another one.

  unpersist()

  Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.

  unpersist(boolean blocking)

  Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.

  Specify a ResourceProfile to use when calculating this RDD.

  Methods inherited from interface org.apache.spark.api.java.JavaRDDLike

  aggregate, cartesian, checkpoint, collect, collectAsync, collectPartitions, context, count, countApprox, countApprox, countApproxDistinct, countAsync, countByValue, countByValueApprox, countByValueApprox, first, flatMap, flatMapToDouble, flatMapToPair, fold, foreach, foreachAsync, foreachPartition, foreachPartitionAsync, getCheckpointFile, getNumPartitions, getStorageLevel, glom, groupBy, groupBy, id, isCheckpointed, isEmpty, iterator, keyBy, map, mapPartitions, mapPartitions, mapPartitionsToDouble, mapPartitionsToDouble, mapPartitionsToPair, mapPartitionsToPair, mapPartitionsWithIndex, mapToDouble, mapToPair, max, min, name, partitioner, partitions, pipe, pipe, pipe, pipe, pipe, reduce, saveAsObjectFile, saveAsTextFile, saveAsTextFile, take, takeAsync, takeOrdered, takeOrdered, takeSample, takeSample, toDebugString, toLocalIterator, top, top, treeAggregate, treeAggregate, treeAggregate, treeReduce, treeReduce, zip, zipPartitions, zipWithIndex, zipWithUniqueId

• Constructor Details

  • JavaRDD

    public JavaRDD(RDD<T> rdd, scala.reflect.ClassTag<T> classTag)

• Method Details

  • fromRDD

    public static <T> JavaRDD<T> fromRDD(RDD<T> rdd, scala.reflect.ClassTag<T> evidence$1)

  • toRDD

    public static <T> RDD<T> toRDD(JavaRDD<T> rdd)

  • rdd

    public RDD<T> rdd()

  • classTag

    public scala.reflect.ClassTag<T> classTag()

  • wrapRDD

  • cache

    Persist this RDD with the default storage level (MEMORY_ONLY).

    Returns:

    (undocumented)

  • persist

    Set this RDD's storage level to persist its values across operations after the first time it is computed. This can only be used to assign a new storage level if the RDD does not have a storage level set yet..

    Parameters:

    newLevel - (undocumented)

    Returns:

    (undocumented)

  • withResources

    Specify a ResourceProfile to use when calculating this RDD. This is only supported on certain cluster managers and currently requires dynamic allocation to be enabled. It will result in new executors with the resources specified being acquired to calculate the RDD.

    Parameters:

    rp - (undocumented)

    Returns:

    (undocumented)

  • getResourceProfile

    Get the ResourceProfile specified with this RDD or None if it wasn't specified.

    Returns:

    the user specified ResourceProfile or null if none was specified

  • unpersist

    Mark the RDD as non-persistent, and remove all blocks for it from memory and disk. This method blocks until all blocks are deleted.

    Returns:

    (undocumented)

  • unpersist

    public JavaRDD<T> unpersist(boolean blocking)

    Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.

    Parameters:

    blocking - Whether to block until all blocks are deleted.

    Returns:

    (undocumented)

  • distinct

    Return a new RDD containing the distinct elements in this RDD.

    Returns:

    (undocumented)

  • distinct

    public JavaRDD<T> distinct(int numPartitions)

    Return a new RDD containing the distinct elements in this RDD.

    Parameters:

    numPartitions - (undocumented)

    Returns:

    (undocumented)

  • filter

    Return a new RDD containing only the elements that satisfy a predicate.

    Parameters:

    f - (undocumented)

    Returns:

    (undocumented)

  • coalesce

    public JavaRDD<T> coalesce(int numPartitions)

    Return a new RDD that is reduced into numPartitions partitions.

    Parameters:

    numPartitions - (undocumented)

    Returns:

    (undocumented)

  • coalesce

    public JavaRDD<T> coalesce(int numPartitions, boolean shuffle)

    Return a new RDD that is reduced into numPartitions partitions.

    Parameters:

    numPartitions - (undocumented)

    shuffle - (undocumented)

    Returns:

    (undocumented)

  • repartition

    public JavaRDD<T> repartition(int numPartitions)

    Return a new RDD that has exactly numPartitions partitions.

    Can increase or decrease the level of parallelism in this RDD. Internally, this uses a shuffle to redistribute data.

    If you are decreasing the number of partitions in this RDD, consider using coalesce, which can avoid performing a shuffle.

    Parameters:

    numPartitions - (undocumented)

    Returns:

    (undocumented)

  • sample

    public JavaRDD<T> sample(boolean withReplacement, double fraction)

    Return a sampled subset of this RDD with a random seed.

    Parameters:

    withReplacement - can elements be sampled multiple times (replaced when sampled out)

    fraction - expected size of the sample as a fraction of this RDD's size without replacement: probability that each element is chosen; fraction must be [0, 1] with replacement: expected number of times each element is chosen; fraction must be greater than or equal to 0

    Returns:

    (undocumented)

    Note:

    This is NOT guaranteed to provide exactly the fraction of the count of the given RDD.

  • sample

    public JavaRDD<T> sample(boolean withReplacement, double fraction, long seed)

    Return a sampled subset of this RDD, with a user-supplied seed.

    Parameters:

    withReplacement - can elements be sampled multiple times (replaced when sampled out)

    fraction - expected size of the sample as a fraction of this RDD's size without replacement: probability that each element is chosen; fraction must be [0, 1] with replacement: expected number of times each element is chosen; fraction must be greater than or equal to 0

    seed - seed for the random number generator

    Returns:

    (undocumented)

    Note:

    This is NOT guaranteed to provide exactly the fraction of the count of the given RDD.

  • randomSplit

    public JavaRDD<T>[] randomSplit(double[] weights)

    Randomly splits this RDD with the provided weights.

    Parameters:

    weights - weights for splits, will be normalized if they don't sum to 1

    Returns:

    split RDDs in an array

  • randomSplit

    public JavaRDD<T>[] randomSplit(double[] weights, long seed)

    Randomly splits this RDD with the provided weights.

    Parameters:

    weights - weights for splits, will be normalized if they don't sum to 1

    seed - random seed

    Returns:

    split RDDs in an array

  • union

    Return the union of this RDD and another one. Any identical elements will appear multiple times (use .distinct() to eliminate them).

    Parameters:

    other - (undocumented)

    Returns:

    (undocumented)

  • intersection

    Return the intersection of this RDD and another one. The output will not contain any duplicate elements, even if the input RDDs did.

    Parameters:

    other - (undocumented)

    Returns:

    (undocumented)

    Note:

    This method performs a shuffle internally.

  • subtract

    Return an RDD with the elements from this that are not in other.

    Uses this partitioner/partition size, because even if other is huge, the resulting RDD will be less than or equal to us.

    Parameters:

    other - (undocumented)

    Returns:

    (undocumented)

  • subtract

    Return an RDD with the elements from this that are not in other.

    Parameters:

    other - (undocumented)

    numPartitions - (undocumented)

    Returns:

    (undocumented)

  • subtract

    Return an RDD with the elements from this that are not in other.

    Parameters:

    other - (undocumented)

    p - (undocumented)

    Returns:

    (undocumented)

  • toString

    Overrides:

    toString in class Object

  • setName

    Assign a name to this RDD

  • sortBy

    public <S> JavaRDD<T> sortBy(Function<T,S> f, boolean ascending, int numPartitions)

    Return this RDD sorted by the given key function.

    Parameters:

    f - (undocumented)

    ascending - (undocumented)

    numPartitions - (undocumented)

    Returns:

    (undocumented)