datafu.pig.sampling

Class SimpleRandomSample

java.lang.Object

org.apache.pig.EvalFunc<T>

org.apache.pig.AccumulatorEvalFunc<T>

org.apache.pig.AlgebraicEvalFunc<org.apache.pig.data.DataBag>

datafu.pig.sampling.SimpleRandomSample

All Implemented Interfaces:

org.apache.pig.Accumulator<org.apache.pig.data.DataBag>, org.apache.pig.Algebraic

public class SimpleRandomSample
extends org.apache.pig.AlgebraicEvalFunc<org.apache.pig.data.DataBag>

Scalable simple random sampling (ScaSRS).

This UDF implements a scalable simple random sampling algorithm described in

 X. Meng, Scalable Simple Random Sampling and Stratified Sampling, ICML 2013.
 

It takes a bag of n items and a sampling probability p as the inputs, and outputs a simple random sample of size exactly ceil(p*n) in a bag, with probability at least 99.99%. For example, the following script generates a simple random sample with sampling probability 0.1:

 DEFINE SRS datafu.pig.sampling.SimpleRandomSample();

 item    = LOAD 'input' AS (x:double);
 sampled = FOREACH (GROUP item ALL) GENERATE FLATTEN(SRS(item, 0.01));
 

Optionally, user can provide a good lower bound of n as the third argument to help reduce the size of intermediate data, for example:

 DEFINE SRS datafu.pig.sampling.SimpleRandomSample();

 item    = LOAD 'input' AS (x:double);
 summary = FOREACH (GROUP item ALL) GENERATE COUNT(item) AS count;
 sampled = FOREACH (GROUP item ALL) GENERATE FLATTEN(SRS(item, 0.01, summary.count));
 

This UDF is very useful for stratified sampling. For example, the following script keeps all positive examples while downsampling negatives with probability 0.1:

 DEFINE SRS datafu.pig.sampling.SimpleRandomSample();

 item    = LOAD 'input' AS (x:double, label:int);
 grouped = FOREACH (GROUP item BY label) GENERATE item, (group == 1 ? 1.0 : 0.1) AS p;
 sampled = FOREACH grouped GENERATE FLATTEN(SRS(item, p));
 

In a Java Hadoop MapReduce job, we can output selected items directly using MultipleOutputs. However, this feature is not available in a Pig UDF. So we still let selected items go through the sort phase. However, as long as the sample size is not huge, this should not be a big problem.

In the first version, the sampling probability is specified in the constructor. This method is deprecated now and will be removed in the next release.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SimpleRandomSample.Final
static class	SimpleRandomSample.Initial
static class	SimpleRandomSample.Intermediate

Nested classes/interfaces inherited from class org.apache.pig.EvalFunc

org.apache.pig.EvalFunc.SchemaType

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	OUTPUT_BAG_NAME_PREFIX Prefix for the output bag name.

Fields inherited from class org.apache.pig.EvalFunc

log, pigLogger, reporter, returnType

Constructor Summary

Constructors

Constructor and Description
SimpleRandomSample()
SimpleRandomSample(java.lang.String samplingProbability) Deprecated. Should specify the sampling probability in the function call.

Method Summary

Modifier and Type	Method and Description
java.lang.String	getFinal()
java.lang.String	getInitial()
java.lang.String	getIntermed()
org.apache.pig.impl.logicalLayer.schema.Schema	outputSchema(org.apache.pig.impl.logicalLayer.schema.Schema input)

Methods inherited from class org.apache.pig.AlgebraicEvalFunc

accumulate, cleanup, getValue

Methods inherited from class org.apache.pig.AccumulatorEvalFunc

exec

Methods inherited from class org.apache.pig.EvalFunc

allowCompileTimeCalculation, finish, getArgToFuncMapping, getCacheFiles, getInputSchema, getLogger, getPigLogger, getReporter, getReturnType, getSchemaName, getSchemaType, getShipFiles, isAsynchronous, progress, setInputSchema, setPigLogger, setReporter, setUDFContextSignature, warn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

OUTPUT_BAG_NAME_PREFIX

public static final java.lang.String OUTPUT_BAG_NAME_PREFIX

Prefix for the output bag name.

See Also:

Constant Field Values

Constructor Detail

SimpleRandomSample

public SimpleRandomSample()

SimpleRandomSample

@Deprecated
public SimpleRandomSample(java.lang.String samplingProbability)

Deprecated. Should specify the sampling probability in the function call.

Constructs this UDF with a sampling probability.

Parameters:

samplingProbability - sampling probability

Method Detail

getInitial

public java.lang.String getInitial()

Specified by:

getInitial in interface org.apache.pig.Algebraic

Specified by:

getInitial in class org.apache.pig.AlgebraicEvalFunc<org.apache.pig.data.DataBag>

getIntermed

public java.lang.String getIntermed()

Specified by:

getIntermed in interface org.apache.pig.Algebraic

Specified by:

getIntermed in class org.apache.pig.AlgebraicEvalFunc<org.apache.pig.data.DataBag>

getFinal

public java.lang.String getFinal()

Specified by:

getFinal in interface org.apache.pig.Algebraic

Specified by:

getFinal in class org.apache.pig.AlgebraicEvalFunc<org.apache.pig.data.DataBag>

outputSchema

public org.apache.pig.impl.logicalLayer.schema.Schema outputSchema(org.apache.pig.impl.logicalLayer.schema.Schema input)

Overrides:

outputSchema in class org.apache.pig.EvalFunc<org.apache.pig.data.DataBag>