datafu.pig.stats.entropy

Class Entropy

java.lang.Object

org.apache.pig.EvalFunc<T>

org.apache.pig.AccumulatorEvalFunc<java.lang.Double>

datafu.pig.stats.entropy.Entropy

All Implemented Interfaces:

org.apache.pig.Accumulator<java.lang.Double>

public class Entropy
extends org.apache.pig.AccumulatorEvalFunc<java.lang.Double>

Calculate entropy H(X) of random variable X following entropy's wiki definition

This UDF's constructor takes 2 arguments.

The 1st argument, the type of entropy estimator algorithm we currently support, includes:

• empirical (empirical entropy estimator)
• chaosh (Chao-Shen entropy estimator)

The default estimation algorithm is empirical.

The 2nd argument, the logarithm base we currently support, includes:

• log (use Euler's number as the logarithm base)
• log2 (use 2 as the logarithm base)
• log10 (use 10 as the logarithm base)

The default logarithm base is log.

Note:

• The input to this UDF must be a sorted bag of raw data tuples of X. An exception will be thrown if the input bag is not sorted
• The returned entropy value is of double type.

How to use:

This UDF calculates entropy from raw data tuples without the need to pre-compute per tuple occurrence frequency.

It could be used in a nested FOREACH after a GROUP BY, in which we sort the inner bag and use the sorted bag as this UDF's input.

Example:

 
 --calculate empirical entropy with Euler's number as the logarithm base
 define Entropy datafu.pig.stats.entropy.Entropy();

 input = LOAD 'input' AS (grp: chararray, val: double);

 -- calculate the input's entropy in each group
 input_group_g = GROUP input BY grp;
 entropy_group = FOREACH input_group_g {
   input_val = input.val;
   input_ordered = ORDER input_val BY $0;
   GENERATE FLATTEN(group) AS group, Entropy(input_ordered) AS entropy; 
 }
 
 

See Also:

CondEntropy, EmpiricalCountEntropy

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.pig.EvalFunc

org.apache.pig.EvalFunc.SchemaType

Field Summary

Fields inherited from class org.apache.pig.EvalFunc

log, pigLogger, reporter, returnType

Constructor Summary

Constructors

Constructor and Description
Entropy()
Entropy(java.lang.String type)
Entropy(java.lang.String type, java.lang.String base)

Method Summary

Modifier and Type	Method and Description
void	accumulate(org.apache.pig.data.Tuple input)
void	cleanup()
java.lang.Double	getValue()
org.apache.pig.impl.logicalLayer.schema.Schema	outputSchema(org.apache.pig.impl.logicalLayer.schema.Schema input)

Methods inherited from class org.apache.pig.AccumulatorEvalFunc

exec

Methods inherited from class org.apache.pig.EvalFunc

allowCompileTimeCalculation, finish, getArgToFuncMapping, getCacheFiles, getInputSchema, getLogger, getPigLogger, getReporter, getReturnType, getSchemaName, getSchemaType, getShipFiles, isAsynchronous, progress, setInputSchema, setPigLogger, setReporter, setUDFContextSignature, warn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Entropy

public Entropy()
        throws org.apache.pig.backend.executionengine.ExecException

Throws:

org.apache.pig.backend.executionengine.ExecException

Entropy

public Entropy(java.lang.String type)
        throws org.apache.pig.backend.executionengine.ExecException

Throws:

org.apache.pig.backend.executionengine.ExecException

Entropy

public Entropy(java.lang.String type,
               java.lang.String base)
        throws org.apache.pig.backend.executionengine.ExecException

Throws:

org.apache.pig.backend.executionengine.ExecException

Method Detail

accumulate

public void accumulate(org.apache.pig.data.Tuple input)
                throws java.io.IOException

Specified by:

accumulate in interface org.apache.pig.Accumulator<java.lang.Double>

Specified by:

accumulate in class org.apache.pig.AccumulatorEvalFunc<java.lang.Double>

Throws:

java.io.IOException

getValue

public java.lang.Double getValue()

Specified by:

getValue in interface org.apache.pig.Accumulator<java.lang.Double>

Specified by:

getValue in class org.apache.pig.AccumulatorEvalFunc<java.lang.Double>

cleanup

public void cleanup()

Specified by:

cleanup in interface org.apache.pig.Accumulator<java.lang.Double>

Specified by:

cleanup in class org.apache.pig.AccumulatorEvalFunc<java.lang.Double>

outputSchema

public org.apache.pig.impl.logicalLayer.schema.Schema outputSchema(org.apache.pig.impl.logicalLayer.schema.Schema input)

Overrides:

outputSchema in class org.apache.pig.EvalFunc<java.lang.Double>