datafu.pig.hash.lsh

Class LSHFunc

java.lang.Object

org.apache.pig.EvalFunc<org.apache.pig.data.DataBag>

datafu.pig.hash.lsh.LSHFunc

Direct Known Subclasses:

CosineDistanceHash, L1PStableHash, L2PStableHash

public abstract class LSHFunc
extends org.apache.pig.EvalFunc<org.apache.pig.data.DataBag>

The base UDF for locality sensitive hashing.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.pig.EvalFunc

org.apache.pig.EvalFunc.SchemaType

Field Summary

Fields

Modifier and Type	Field and Description
protected LSHFamily	lsh
protected java.lang.Long	seed

Fields inherited from class org.apache.pig.EvalFunc

log, pigLogger, reporter, returnType

Constructor Summary

Constructors

Constructor and Description
LSHFunc(java.lang.String sSeed)

Method Summary

Methods

Modifier and Type	Method and Description
protected abstract LSHCreator	createLSHCreator()
org.apache.pig.data.DataBag	exec(org.apache.pig.data.Tuple t) The expectation is that the input is a Tuple of numeric data representing a real vector.
protected abstract int	getDimension()
protected long	getSeed()
org.apache.pig.impl.logicalLayer.schema.Schema	outputSchema(org.apache.pig.impl.logicalLayer.schema.Schema input) The schema returned here is a bag containing pairs.

Methods inherited from class org.apache.pig.EvalFunc

allowCompileTimeCalculation, finish, getArgToFuncMapping, getCacheFiles, getInputSchema, getLogger, getPigLogger, getReporter, getReturnType, getSchemaName, getSchemaType, getShipFiles, isAsynchronous, progress, setInputSchema, setPigLogger, setReporter, setUDFContextSignature, warn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

lsh

protected LSHFamily lsh

seed

protected java.lang.Long seed

Constructor Detail

LSHFunc

public LSHFunc(java.lang.String sSeed)

Method Detail

createLSHCreator

protected abstract LSHCreator createLSHCreator()

getDimension

protected abstract int getDimension()

exec

public org.apache.pig.data.DataBag exec(org.apache.pig.data.Tuple t)
                                 throws java.io.IOException

The expectation is that the input is a Tuple of numeric data representing a real vector.

Specified by:

exec in class org.apache.pig.EvalFunc<org.apache.pig.data.DataBag>

Returns:

DataBag containing tuples which have the lsh_id (the hash ID in the family) and the hash

Throws:

java.io.IOException

getSeed

protected long getSeed()

outputSchema

public org.apache.pig.impl.logicalLayer.schema.Schema outputSchema(org.apache.pig.impl.logicalLayer.schema.Schema input)

The schema returned here is a bag containing pairs. The pairs are the lsh_id (or the ID of the hash) and the hash value.

Overrides:

outputSchema in class org.apache.pig.EvalFunc<org.apache.pig.data.DataBag>