datafu.pig.bags

Class CountDistinctUpTo

java.lang.Object

org.apache.pig.EvalFunc<T>

org.apache.pig.AccumulatorEvalFunc<java.lang.Integer>

datafu.pig.bags.CountDistinctUpTo

All Implemented Interfaces:

org.apache.pig.Accumulator<java.lang.Integer>, org.apache.pig.Algebraic

public class CountDistinctUpTo
extends org.apache.pig.AccumulatorEvalFunc<java.lang.Integer>
implements org.apache.pig.Algebraic

Generates a count of the number of distinct tuples in a bag, up to a preset limit. If the number of distinct tuples is small, performance is comparable to using Pig's DISTINCT and COUNT in a nested foreach. Use this UDF when your threshold is low, and some records have a distinct count that is much higher. In such cases this UDF will prevent memory problems and perform an order of magnitude faster than using pure Pig. Example:

 DEFINE CountDistinctUpTo10 datafu.pig.bags.CountDistinctUpTo('10');
 DEFINE CountDistinctUpTo3 datafu.pig.bags.CountDistinctUpTo('3');
 
 -- input: 
 -- {(A),(B),(D),(A),(C),(E),(A),(B),(A),(B)}
 input = LOAD 'input' AS (B: bag {T: tuple(alpha:CHARARRAY)});
 
 -- output: 
 -- (5)
 output = FOREACH input GENERATE CountDistinctUpTo10(B); 

 -- output2: 
 -- (3)
 output2 = FOREACH input GENERATE CountDistinctUpTo3(B); 
  
 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CountDistinctUpTo.Final Receives output either from initial results or intermediate Outputs an integer with the number of distinct tuples, up to the maximum desired.
static class	CountDistinctUpTo.Initial Outputs a tuple containing a DataBag containing a single tuple T (the original schema) or an empty bag T -> ({T})
static class	CountDistinctUpTo.Intermediate Receives a bag of bags, each containing a single tuple with the original input schema T Outputs a bag of distinct tuples each with the original schema T: {({T}),({T}),({T})} -> ({T, T, T}) or if the maximum is reached, null: {({T}),({T}),({T}) ..} -> (null)

Nested classes/interfaces inherited from class org.apache.pig.EvalFunc

org.apache.pig.EvalFunc.SchemaType

Field Summary

Fields inherited from class org.apache.pig.EvalFunc

log, pigLogger, reporter, returnType

Constructor Summary

Constructors

Constructor and Description
CountDistinctUpTo(java.lang.String maxAmount)

Method Summary

Methods

Modifier and Type	Method and Description
void	accumulate(org.apache.pig.data.Tuple tuple)
void	cleanup()
java.lang.String	getFinal()
java.lang.String	getInitial()
java.lang.String	getIntermed()
java.lang.Integer	getValue()
org.apache.pig.impl.logicalLayer.schema.Schema	outputSchema(org.apache.pig.impl.logicalLayer.schema.Schema input)

Methods inherited from class org.apache.pig.AccumulatorEvalFunc

exec

Methods inherited from class org.apache.pig.EvalFunc

allowCompileTimeCalculation, finish, getArgToFuncMapping, getCacheFiles, getInputSchema, getLogger, getPigLogger, getReporter, getReturnType, getSchemaName, getSchemaType, getShipFiles, isAsynchronous, progress, setInputSchema, setPigLogger, setReporter, setUDFContextSignature, warn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CountDistinctUpTo

public CountDistinctUpTo(java.lang.String maxAmount)

Method Detail

accumulate

public void accumulate(org.apache.pig.data.Tuple tuple)
                throws java.io.IOException

Specified by:

accumulate in interface org.apache.pig.Accumulator<java.lang.Integer>

Specified by:

accumulate in class org.apache.pig.AccumulatorEvalFunc<java.lang.Integer>

Throws:

java.io.IOException

cleanup

public void cleanup()

Specified by:

cleanup in interface org.apache.pig.Accumulator<java.lang.Integer>

Specified by:

cleanup in class org.apache.pig.AccumulatorEvalFunc<java.lang.Integer>

getValue

public java.lang.Integer getValue()

Specified by:

getValue in interface org.apache.pig.Accumulator<java.lang.Integer>

Specified by:

getValue in class org.apache.pig.AccumulatorEvalFunc<java.lang.Integer>

getInitial

public java.lang.String getInitial()

Specified by:

getInitial in interface org.apache.pig.Algebraic

getIntermed

public java.lang.String getIntermed()

Specified by:

getIntermed in interface org.apache.pig.Algebraic

getFinal

public java.lang.String getFinal()

Specified by:

getFinal in interface org.apache.pig.Algebraic

outputSchema

public org.apache.pig.impl.logicalLayer.schema.Schema outputSchema(org.apache.pig.impl.logicalLayer.schema.Schema input)

Overrides:

outputSchema in class org.apache.pig.EvalFunc<java.lang.Integer>