datafu.pig.hash

Class Hasher

java.lang.Object

org.apache.pig.EvalFunc<T>

datafu.pig.util.ContextualEvalFunc<T>

datafu.pig.util.SimpleEvalFunc<java.lang.String>

datafu.pig.hash.Hasher

Direct Known Subclasses:

HasherRand

public class Hasher
extends SimpleEvalFunc<java.lang.String>

Computes a hash value of a string and outputs it in hex. The default constructor produces a fast 32-bit non-cryptographic-strength hash ('murmur3-32') that is reasonable if all you need is good mixing. Additional constructors are available to choose alternative hash functions:

• 'murmur3-32', [optional seed] or 'murmur3-128', [optional seed]: Returns a murmur3 hash of the given length. Murmur3 is fast, with has exceptionally good statistical properties; it's a good choice if all you need is good mixing of the inputs. It is not cryptographically secure; that is, given an output value from murmur3, there are efficient algorithms to find an input yielding the same output value. Supply the seed as a string that Integer.decode can handle. Examples: datafu.pig.hash.Hasher('murmur3-32', '0x56789abc'); or datafu.pig.hash.Hasher('murmur3-32', '-12345678');.
• 'sip24', [optional seed]: Returns a 64-bit SipHash-2-4 hash. SipHash is competitive in performance with Murmur3, and is simpler and faster than the cryptographic algorithms below. When used with a seed, it can be considered cryptographically secure: given the output from a sip24 instance but not the seed used, we cannot efficiently craft a message yielding the same output from that instance. To supply a seed, pass in a 32-character string representing the seed in hexadecimal. If none is given, k = '00010203…0e0f' is used.
• 'adler32': Returns an Adler-32 checksum (32 hash bits) by delegating to Java's Adler32 Checksum.
• 'crc32': Returns a CRC-32 checksum (32 hash bits) by delegating to Java's CRC32 Checksum.
• 'md5': Returns an MD5 hash (128 hash bits) using Java's MD5 MessageDigest.
• 'sha1': Returns a SHA-1 hash (160 hash bits) using Java's SHA-1 MessageDigest.
• 'sha256': Returns a SHA-256 hash (256 hash bits) using Java's SHA-256 MessageDigest.
• 'sha512': Returns a SHA-512 hash (160 hash bits) using Java's SHA-512 MessageDigest.
• 'good-{integer number of bits}': Returns a general-purpose, non-cryptographic-strength, streaming hash function that produces hash codes of length at least minimumBits. Users without specific' compatibility requirements and who do not persist the hash codes are encouraged to choose this hash function. (Cryptographers, like dieticians and fashionistas, occasionally realize that We've Been Doing it Wrong This Whole Time. Using 'good-*' lets you track What the Experts From (Milan|NIH|IEEE) Say To (Wear|Eat|Hash With) this Fall.) Expect values returned by this hasher to change run-to-run.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.pig.EvalFunc

org.apache.pig.EvalFunc.SchemaType

Field Summary

Fields

Modifier and Type	Field and Description
protected com.google.common.hash.HashFunction	hash_func
protected static java.lang.String	SEEDED_HASH_NAMES

Fields inherited from class org.apache.pig.EvalFunc

log, pigLogger, reporter, returnType

Constructor Summary

Constructors

Constructor and Description
Hasher() Generates hash values according to murmur3-32, a non-cryptographic-strength hash function with good mixing.
Hasher(java.lang.String algorithm) Generates hash values according to the hash function given by algorithm.
Hasher(java.lang.String algorithm, java.lang.String seed) Generates hash values according to the hash function given by algorithm, with initial seed given by the seed.

Method Summary

Modifier and Type	Method and Description
java.lang.String	call(java.lang.String val)
static int	intFromHex(java.lang.String hex_str)
static long	longFromHex(java.lang.String hex_str)
protected void	makeHashFunc(java.lang.String algorithm, java.lang.String seed) Returns the HashFunction named by algorithm, with initial seed given by the seed.

Methods inherited from class datafu.pig.util.SimpleEvalFunc

exec, getReturnType, outputSchema

Methods inherited from class datafu.pig.util.ContextualEvalFunc

getContextProperties, getInstanceName, getInstanceProperties, onReady, setUDFContextSignature

Methods inherited from class org.apache.pig.EvalFunc

allowCompileTimeCalculation, finish, getArgToFuncMapping, getCacheFiles, getInputSchema, getLogger, getPigLogger, getReporter, getSchemaName, getSchemaType, getShipFiles, isAsynchronous, progress, setInputSchema, setPigLogger, setReporter, warn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

hash_func

protected com.google.common.hash.HashFunction hash_func

SEEDED_HASH_NAMES

protected static final java.lang.String SEEDED_HASH_NAMES

See Also:

Constant Field Values

Constructor Detail

Hasher

public Hasher()
       throws java.lang.IllegalArgumentException,
              java.lang.RuntimeException

Generates hash values according to murmur3-32, a non-cryptographic-strength hash function with good mixing.

Throws:

java.lang.IllegalArgumentException - for an internal error

java.lang.RuntimeException - for an internal error

Hasher

public Hasher(java.lang.String algorithm)
       throws java.lang.IllegalArgumentException,
              java.lang.RuntimeException

Generates hash values according to the hash function given by algorithm. See the Hasher class docs for a list of algorithms and guidance on selection.

Parameters:

algorithm - the hash algorithm to use

Throws:

java.lang.IllegalArgumentException - for an invalid algorithm

java.lang.RuntimeException - for an internal error

See Also:

makeHashFunc(String algorithm)

Hasher

public Hasher(java.lang.String algorithm,
              java.lang.String seed)
       throws java.lang.IllegalArgumentException,
              java.lang.RuntimeException

Generates hash values according to the hash function given by algorithm, with initial seed given by the seed. See the Hasher class docs for a list of algorithms and guidance on selection.

Parameters:

algorithm - the hash algorithm to use

seed - the initial seed to use

Throws:

java.lang.IllegalArgumentException - for an invalid algorithm or seed

java.lang.RuntimeException - when the seed cannot be parsed or other internal error

See Also:

makeHashFunc(String algorithm, String seed)

Method Detail

makeHashFunc

protected void makeHashFunc(java.lang.String algorithm,
                            java.lang.String seed)
                     throws java.lang.IllegalArgumentException,
                            java.lang.RuntimeException

Returns the HashFunction named by algorithm, with initial seed given by the seed. See the Hasher class docs for a list of algorithms and guidance on selection. The seed is interpreted as follows:

• With algorithm 'murmur3-32' or 'murmur3-128', supply a 32-bit number as a string of exactly 8 hexadecimal digits.
• With algorithm 'sip24' supply a 128-bit number as a string of exactly 32 hexadecimal digits.

Parameters:

algorithm - the hash algorithm to use

seed - the initial seed to use

Throws:

java.lang.IllegalArgumentException - for an invalid seed given the algorithm

java.lang.RuntimeException - when the seed cannot be parsed

longFromHex

public static long longFromHex(java.lang.String hex_str)

intFromHex

public static int intFromHex(java.lang.String hex_str)

call

public java.lang.String call(java.lang.String val)