datafu.pig.stats

Class HyperLogLogPlusPlus

java.lang.Object

org.apache.pig.EvalFunc<T>

org.apache.pig.AccumulatorEvalFunc<T>

org.apache.pig.AlgebraicEvalFunc<java.lang.Long>

datafu.pig.stats.HyperLogLogPlusPlus

All Implemented Interfaces:

org.apache.pig.Accumulator<java.lang.Long>, org.apache.pig.Algebraic

public class HyperLogLogPlusPlus
extends org.apache.pig.AlgebraicEvalFunc<java.lang.Long>

A UDF that applies the HyperLogLog++ cardinality estimation algorithm.

This uses the implementation of HyperLogLog++ from stream-lib. The HyperLogLog++ algorithm is an enhanced version of HyperLogLog as described in here.

This is a streaming implementation, and therefore the input data does not need to be sorted.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HyperLogLogPlusPlus.Final
static class	HyperLogLogPlusPlus.Initial
static class	HyperLogLogPlusPlus.Intermediate

Nested classes/interfaces inherited from class org.apache.pig.EvalFunc

org.apache.pig.EvalFunc.SchemaType

Field Summary

Fields inherited from class org.apache.pig.EvalFunc

log, pigLogger, reporter, returnType

Constructor Summary

Constructors

Constructor and Description
HyperLogLogPlusPlus() Constructs a HyperLogLog++ estimator.
HyperLogLogPlusPlus(java.lang.String p) Constructs a HyperLogLog++ estimator.

Method Summary

Methods

Modifier and Type	Method and Description
protected static com.clearspring.analytics.stream.cardinality.HyperLogLogPlus	countDisctinct(org.apache.pig.data.Tuple input, int p)
java.lang.String	getFinal()
java.lang.String	getInitial()
java.lang.String	getIntermed()
org.apache.pig.impl.logicalLayer.schema.Schema	outputSchema(org.apache.pig.impl.logicalLayer.schema.Schema input)

Methods inherited from class org.apache.pig.AlgebraicEvalFunc

accumulate, cleanup, getValue

Methods inherited from class org.apache.pig.AccumulatorEvalFunc

exec

Methods inherited from class org.apache.pig.EvalFunc

allowCompileTimeCalculation, finish, getArgToFuncMapping, getCacheFiles, getInputSchema, getLogger, getPigLogger, getReporter, getReturnType, getSchemaName, getSchemaType, getShipFiles, isAsynchronous, progress, setInputSchema, setPigLogger, setReporter, setUDFContextSignature, warn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HyperLogLogPlusPlus

public HyperLogLogPlusPlus()

Constructs a HyperLogLog++ estimator.

HyperLogLogPlusPlus

public HyperLogLogPlusPlus(java.lang.String p)

Constructs a HyperLogLog++ estimator.

Parameters:

p - precision value

Method Detail

outputSchema

public org.apache.pig.impl.logicalLayer.schema.Schema outputSchema(org.apache.pig.impl.logicalLayer.schema.Schema input)

Overrides:

outputSchema in class org.apache.pig.EvalFunc<java.lang.Long>

getFinal

public java.lang.String getFinal()

Specified by:

getFinal in interface org.apache.pig.Algebraic

Specified by:

getFinal in class org.apache.pig.AlgebraicEvalFunc<java.lang.Long>

getInitial

public java.lang.String getInitial()

Specified by:

getInitial in interface org.apache.pig.Algebraic

Specified by:

getInitial in class org.apache.pig.AlgebraicEvalFunc<java.lang.Long>

getIntermed

public java.lang.String getIntermed()

Specified by:

getIntermed in interface org.apache.pig.Algebraic

Specified by:

getIntermed in class org.apache.pig.AlgebraicEvalFunc<java.lang.Long>

countDisctinct

protected static com.clearspring.analytics.stream.cardinality.HyperLogLogPlus countDisctinct(org.apache.pig.data.Tuple input,
                                                                          int p)
                                                                                      throws java.lang.NumberFormatException,
                                                                                             java.io.IOException

Throws:

java.lang.NumberFormatException

java.io.IOException