All Implemented Interfaces:

Serializable, org.apache.spark.internal.Logging, Params, HasInputCol, HasNumFeatures, HasOutputCol, DefaultParamsWritable, Identifiable, MLWritable

Maps a sequence of terms to their term frequencies using the hashing trick. Currently we use Austin Appleby's MurmurHash 3 algorithm (MurmurHash3_x86_32) to calculate the hash code value for the term object. Since a simple modulo is used to transform the hash function to a column index, it is advisable to use a power of two as the numFeatures parameter; otherwise the features will not be mapped evenly to the columns.

See Also:

• Serialized Form

• Nested Class Summary

  Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

  org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter

• Constructor Summary

  Constructors

• Method Summary

  binary()

  Binary toggle to control term frequency counts.

  Creates a copy of this instance with the same UID and some extra params.

  boolean

  getBinary()

  int

  int

  Returns the index of the input term.

  inputCol()

  Param for input column name.

  Param for Number of features.

  outputCol()

  Param for output column name.

  read()

  void

  Saves this ML instance to the input path, a shortcut of write.save(path).

  setBinary(boolean value)

  setNumFeatures(int value)

  toString()

  Transforms the input dataset.

  Check transform validity and derive the output schema from the input schema.

  uid()

  An immutable unique ID for the object and its derivatives.

  Methods inherited from interface org.apache.spark.internal.Logging

  initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logBasedOnLevel, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, MDC, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext

  Methods inherited from interface org.apache.spark.ml.param.Params

  clear, copyValues, defaultCopy, defaultParamMap, estimateMatadataSize, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, onParamChange, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn

• Constructor Details

  • HashingTF

    public HashingTF()

  • HashingTF

    public HashingTF(String uid)

• Method Details

  • read

  • load

  • numFeatures

    public final IntParam numFeatures()

    Param for Number of features. Should be greater than 0.

    Specified by:

    numFeatures in interface HasNumFeatures

    Returns:

    (undocumented)

  • outputCol

    Param for output column name.

    Specified by:

    outputCol in interface HasOutputCol

    Returns:

    (undocumented)

  • inputCol

    Description copied from interface: HasInputCol

    Param for input column name.

    Specified by:

    inputCol in interface HasInputCol

    Returns:

    (undocumented)

  • uid

    An immutable unique ID for the object and its derivatives.

    Specified by:

    uid in interface Identifiable

    Returns:

    (undocumented)

  • hashFuncVersion

    public int hashFuncVersion()

  • setInputCol

  • setOutputCol

  • binary

    Binary toggle to control term frequency counts. If true, all non-zero counts are set to 1. This is useful for discrete probabilistic models that model binary events rather than integer counts. (default = false)

    Returns:

    (undocumented)

  • setNumFeatures

    public HashingTF setNumFeatures(int value)

  • getBinary

    public boolean getBinary()

  • setBinary

    public HashingTF setBinary(boolean value)

  • transform

    Transforms the input dataset.

    Specified by:

    transform in class Transformer

    Parameters:

    dataset - (undocumented)

    Returns:

    (undocumented)

  • transformSchema

    Check transform validity and derive the output schema from the input schema.

    We check validity for interactions between parameters during transformSchema and raise an exception if any parameter value is invalid. Parameter value checks which do not depend on other parameters are handled by Param.validate().

    Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.

    Specified by:

    transformSchema in class PipelineStage

    Parameters:

    schema - (undocumented)

    Returns:

    (undocumented)

  • indexOf

    public int indexOf(Object term)

    Returns the index of the input term.

    Parameters:

    term - (undocumented)

    Returns:

    (undocumented)

  • copy

    Description copied from interface: Params

    Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly. See defaultCopy().

    Specified by:

    copy in interface Params

    Specified by:

    copy in class Transformer

    Parameters:

    extra - (undocumented)

    Returns:

    (undocumented)

  • toString

    Specified by:

    toString in interface Identifiable

    Overrides:

    toString in class Object

  • save

    public void save(String path)

    Description copied from interface: MLWritable

    Saves this ML instance to the input path, a shortcut of write.save(path).

    Specified by:

    save in interface MLWritable

    Parameters:

    path - (undocumented)