datafu.pig.stats.entropy

Class EmpiricalCountEntropy

java.lang.Object

org.apache.pig.EvalFunc<T>

org.apache.pig.AccumulatorEvalFunc<java.lang.Double>

datafu.pig.stats.entropy.EmpiricalCountEntropy

All Implemented Interfaces:

org.apache.pig.Accumulator<java.lang.Double>, org.apache.pig.Algebraic

public class EmpiricalCountEntropy
extends org.apache.pig.AccumulatorEvalFunc<java.lang.Double>
implements org.apache.pig.Algebraic

Calculate the empirical entropy of random variable X given its occurrence frequencies, following entropy's wiki definition.

This UDF's constructor takes 1 argument: the logarithm base, whose definition is the same as that defined in Entropy

Note:

• Unlike Entropy, which calculates entropy from sorted raw data bag in accumulative mode, this UDF calculates entropy from the data's occurrence frequencies which does not need to be sorted, either in accumulative or algebraic mode.
• Each tuple of the UDF's input bag must only have 1 field, the occurrence frequency of a data instance, and the data type of this field must be int or long. Otherwise, an exception will be thrown.
• Negative frequency number will be silently discarded and a warning message will be logged in the job's log file.
• The returned entropy value is of double type.

How to use:

To use this UDF, customer needs to pre-compute the occurrence frequency of each data instance, often in an outer GROUP BY , and then use this UDF to calculate entropy with those frequency numbers in another outer GROUP BY.

Compared with Entropy, this UDF is more scalable when we need to handle a very large data set, since it could distribute computation onto mappers and take advantage of combiners to reduce intermedidate output from mappers to reducers.

Example:

 

 define Entropy datafu.pig.stats.entropy.EmpiricalCountEntropy();

 input = LOAD 'input' AS (val: double);

 -- calculate the occurrence of each instance
 counts_g = GROUP input BY val;
 counts = FOREACh counts_g GENERATE COUNT(input) AS cnt;

 -- calculate entropy
 input_counts_g = GROUP counts ALL;
 entropy = FOREACH input_counts_g GENERATE Entropy(counts) AS entropy;
 
 

Use case to calculate mutual information using EmpiricalCountEntropy:

 

 define Entropy datafu.pig.stats.entropy.EmpiricalCountEntropy();

 input = LOAD 'input' AS (valX: double, valY: double);

 ------------
 -- calculate mutual information I(X, Y) using entropy
 -- I(X, Y) = H(X) + H(Y) -  H(X, Y)
 ------------

 input_x_y_g = GROUP input BY (valX, valY);
 input_x_y_cnt = FOREACH input_x_y_g GENERATE flatten(group) as (valX, valY), COUNT(input) AS cnt;

 input_x_g = GROUP input_x_y_cnt BY valX;
 input_x_cnt = FOREACH input_x_g GENERATE flatten(group) as valX, SUM(input_x_y_cnt.cnt) AS cnt;

 input_y_g = GROUP input_x_y_cnt BY valY;
 input_y_cnt = FOREACH input_y_g GENERATE flatten(group) as valY, SUM(input_x_y_cnt.cnt) AS cnt;

 input_x_y_entropy_g = GROUP input_x_y_cnt ALL;
 input_x_y_entropy = FOREACH input_x_y_entropy_g {
                         input_x_y_entropy_cnt = input_x_y_cnt.cnt;
                         GENERATE Entropy(input_x_y_entropy_cnt) AS x_y_entropy;
                     }

 input_x_entropy_g = GROUP input_x_cnt ALL;
 input_x_entropy = FOREACH input_x_entropy_g {
                         input_x_entropy_cnt = input_x_cnt.cnt;
                         GENERATE Entropy(input_x_entropy_cnt) AS x_entropy;
                   }

 input_y_entropy_g = GROUP input_y_cnt ALL;
 input_y_entropy = FOREACH input_y_entropy_g {
                         input_y_entropy_cnt = input_y_cnt.cnt;
                         GENERATE Entropy(input_y_entropy_cnt) AS y_entropy;
                   }

 input_mi_cross = CROSS input_x_y_entropy, input_x_entropy, input_y_entropy;
 input_mi = FOREACH input_mi_cross GENERATE (input_x_entropy::x_entropy +
                                             input_y_entropy::y_entropy - 
                                             input_x_y_entropy::x_y_entropy) AS mi;
 
 

See Also:

Entropy

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	EmpiricalCountEntropy.Final
static class	EmpiricalCountEntropy.Initial
static class	EmpiricalCountEntropy.Intermediate

Nested classes/interfaces inherited from class org.apache.pig.EvalFunc

org.apache.pig.EvalFunc.SchemaType

Field Summary

Fields inherited from class org.apache.pig.EvalFunc

log, pigLogger, reporter, returnType

Constructor Summary

Constructors

Constructor and Description
EmpiricalCountEntropy()
EmpiricalCountEntropy(java.lang.String base)

Method Summary

Modifier and Type	Method and Description
void	accumulate(org.apache.pig.data.Tuple input)
void	cleanup()
protected static org.apache.pig.data.Tuple	combine(org.apache.pig.data.DataBag values)
java.lang.String	getFinal()
java.lang.String	getInitial()
java.lang.String	getIntermed()
java.lang.Double	getValue()
org.apache.pig.impl.logicalLayer.schema.Schema	outputSchema(org.apache.pig.impl.logicalLayer.schema.Schema input)

Methods inherited from class org.apache.pig.AccumulatorEvalFunc

exec

Methods inherited from class org.apache.pig.EvalFunc

allowCompileTimeCalculation, finish, getArgToFuncMapping, getCacheFiles, getInputSchema, getLogger, getPigLogger, getReporter, getReturnType, getSchemaName, getSchemaType, getShipFiles, isAsynchronous, progress, setInputSchema, setPigLogger, setReporter, setUDFContextSignature, warn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

EmpiricalCountEntropy

public EmpiricalCountEntropy()
                      throws org.apache.pig.backend.executionengine.ExecException

Throws:

org.apache.pig.backend.executionengine.ExecException

EmpiricalCountEntropy

public EmpiricalCountEntropy(java.lang.String base)
                      throws org.apache.pig.backend.executionengine.ExecException

Throws:

org.apache.pig.backend.executionengine.ExecException

Method Detail

getFinal

public java.lang.String getFinal()

Specified by:

getFinal in interface org.apache.pig.Algebraic

getInitial

public java.lang.String getInitial()

Specified by:

getInitial in interface org.apache.pig.Algebraic

getIntermed

public java.lang.String getIntermed()

Specified by:

getIntermed in interface org.apache.pig.Algebraic

combine

protected static org.apache.pig.data.Tuple combine(org.apache.pig.data.DataBag values)
                                            throws org.apache.pig.backend.executionengine.ExecException

Throws:

org.apache.pig.backend.executionengine.ExecException

accumulate

public void accumulate(org.apache.pig.data.Tuple input)
                throws java.io.IOException

Specified by:

accumulate in interface org.apache.pig.Accumulator<java.lang.Double>

Specified by:

accumulate in class org.apache.pig.AccumulatorEvalFunc<java.lang.Double>

Throws:

java.io.IOException

getValue

public java.lang.Double getValue()

Specified by:

getValue in interface org.apache.pig.Accumulator<java.lang.Double>

Specified by:

getValue in class org.apache.pig.AccumulatorEvalFunc<java.lang.Double>

cleanup

public void cleanup()

Specified by:

cleanup in interface org.apache.pig.Accumulator<java.lang.Double>

Specified by:

cleanup in class org.apache.pig.AccumulatorEvalFunc<java.lang.Double>

outputSchema

public org.apache.pig.impl.logicalLayer.schema.Schema outputSchema(org.apache.pig.impl.logicalLayer.schema.Schema input)

Overrides:

outputSchema in class org.apache.pig.EvalFunc<java.lang.Double>