datafu.pig.sampling

Class WeightedReservoirSample

java.lang.Object

org.apache.pig.EvalFunc<T>

org.apache.pig.AccumulatorEvalFunc<org.apache.pig.data.DataBag>

datafu.pig.sampling.ReservoirSample

datafu.pig.sampling.WeightedReservoirSample

All Implemented Interfaces:

org.apache.pig.Accumulator<org.apache.pig.data.DataBag>, org.apache.pig.Algebraic

@Nondeterministic
public class WeightedReservoirSample
extends ReservoirSample

Performs a weighted random sample using an in-memory reservoir to produce a weighted random sample of a given size based on the A-Res algorithm described in paper.

Species with larger weight have higher probability to be selected in the final sample set.

This UDF inherits from ReservoirSample and it is guaranteed to produce a sample of the given size. Similarly it comes at the cost of scalability. since it uses internal storage with size equaling the desired sample to guarantee the exact sample size.

Its constructor takes 2 arguments:

• The 1st argument specifies the sample size which should be a string of positive integer.
• The 2nd argument specifies the index of the weight field in the input tuple, which should be a string of non-negative integer that is no greater than the input tuple size.

Example:

 define WeightedSample datafu.pig.sampling.WeightedReservoirSample('1','1');
 input = LOAD 'input' AS (v1:chararray, v2:INT);
 input_g = GROUP input ALL;
 sampled = FOREACH input_g GENERATE WeightedSample(input);
 
 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	WeightedReservoirSample.Final
static class	WeightedReservoirSample.Initial
static class	WeightedReservoirSample.Intermediate

Nested classes/interfaces inherited from class org.apache.pig.EvalFunc

org.apache.pig.EvalFunc.SchemaType

Field Summary

Fields inherited from class datafu.pig.sampling.ReservoirSample

numSamples, scoreGen

Fields inherited from class org.apache.pig.EvalFunc

log, pigLogger, reporter, returnType

Constructor Summary

Constructors

Constructor and Description
WeightedReservoirSample(java.lang.String strNumSamples, java.lang.String strWeightIdx)

Method Summary

Methods

Modifier and Type	Method and Description
java.lang.String	getFinal()
java.lang.String	getInitial()
java.lang.String	getIntermed()
protected datafu.pig.sampling.ScoredTuple.ScoreGenerator	getScoreGenerator()
org.apache.pig.impl.logicalLayer.schema.Schema	outputSchema(org.apache.pig.impl.logicalLayer.schema.Schema input)

Methods inherited from class datafu.pig.sampling.ReservoirSample

accumulate, cleanup, exec, getValue

Methods inherited from class org.apache.pig.EvalFunc

allowCompileTimeCalculation, finish, getArgToFuncMapping, getCacheFiles, getInputSchema, getLogger, getPigLogger, getReporter, getReturnType, getSchemaName, getSchemaType, getShipFiles, isAsynchronous, progress, setInputSchema, setPigLogger, setReporter, setUDFContextSignature, warn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WeightedReservoirSample

public WeightedReservoirSample(java.lang.String strNumSamples,
                       java.lang.String strWeightIdx)

Method Detail

getScoreGenerator

protected datafu.pig.sampling.ScoredTuple.ScoreGenerator getScoreGenerator()

Overrides:

getScoreGenerator in class ReservoirSample

outputSchema

public org.apache.pig.impl.logicalLayer.schema.Schema outputSchema(org.apache.pig.impl.logicalLayer.schema.Schema input)

Overrides:

outputSchema in class ReservoirSample

getInitial

public java.lang.String getInitial()

Specified by:

getInitial in interface org.apache.pig.Algebraic

Overrides:

getInitial in class ReservoirSample

getIntermed

public java.lang.String getIntermed()

Specified by:

getIntermed in interface org.apache.pig.Algebraic

Overrides:

getIntermed in class ReservoirSample

getFinal

public java.lang.String getFinal()

Specified by:

getFinal in interface org.apache.pig.Algebraic

Overrides:

getFinal in class ReservoirSample