datafu.pig.stats.entropy

Class CondEntropy

java.lang.Object

org.apache.pig.EvalFunc<T>

org.apache.pig.AccumulatorEvalFunc<java.lang.Double>

datafu.pig.stats.entropy.CondEntropy

All Implemented Interfaces:

org.apache.pig.Accumulator<java.lang.Double>

public class CondEntropy
extends org.apache.pig.AccumulatorEvalFunc<java.lang.Double>

Calculate conditional entropy H(Y|X) of random variables X and Y following conditional entropy's wiki definition, X is the conditional variable and Y is the variable that conditions on X.

Each tuple of the input bag has 2 fields, the 1st field is an object instance of variable X and the 2nd field is an object instance of variable Y. An exception will be thrown if the number of fields is not 2.

This UDF's constructor definition and parameters are the same as that of Entropy

Note:

• The input bag to this UDF must be sorted on X and Y, with X in the first sort order. An exception will be thrown if the input bag is not sorted.
• The returned entropy value is of double type.

How to use:

This UDF calculates conditional entropy given raw data tuples of X and Y without the need to pre-compute per tuple occurrence frequency.

It could be used in a nested FOREACH after a GROUP BY, in which we sort the inner bag and use the sorted bag as this UDF's input.

Example:

 --define empirical conditional entropy with Euler's number as the logarithm base
 define CondEntropy datafu.pig.stats.entropy.CondEntropy();

 input = LOAD 'input' AS (grp: chararray, valX: double, valY: double);

 -- calculate conditional entropy H(Y|X) in each group
 input_group_g = GROUP input BY grp;
 entropy_group = FOREACH input_group_g {
   input_val = input.(valX, valY)
   input_ordered = ORDER input_val BY $0, $1;
   GENERATE FLATTEN(group) AS group, CondEntropy(input_ordered) AS cond_entropy; 
 }
 
 

Use case to calculate mutual information:

 ------------
 -- calculate mutual information I(X, Y) using conditional entropy UDF and entropy UDF
 -- I(X, Y) = H(Y) - H(Y|X)
 ------------

 define CondEntropy datafu.pig.stats.entropy.CondEntropy();
 define Entropy datafu.pig.stats.entropy.Entropy();

 input = LOAD 'input' AS (grp: chararray, valX: double, valY: double);

 -- calculate the I(X,Y) in each group
 input_group_g = GROUP input BY grp;
 mutual_information = FOREACH input_group_g {
      input_val_x_y = input.(valX, valY);
      input_val_x_y_ordered = ORDER input_val_x_y BY $0,$1;
      input_val_y = input.valY;
      input_val_y_ordered = ORDER input_val_y BY $0;
      cond_h_x_y = CondEntropy(input_val_x_y_ordered);
      h_y = Entropy(input_val_y_ordered);
      GENERATE FLATTEN(group), h_y - cond_h_x_y;
 }
 
 

See Also:

Entropy

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.pig.EvalFunc

org.apache.pig.EvalFunc.SchemaType

Field Summary

Fields inherited from class org.apache.pig.EvalFunc

log, pigLogger, reporter, returnType

Constructor Summary

Constructors

Constructor and Description
CondEntropy()
CondEntropy(java.lang.String type)
CondEntropy(java.lang.String type, java.lang.String base)

Method Summary

Methods

Modifier and Type	Method and Description
void	accumulate(org.apache.pig.data.Tuple input)
void	cleanup()
java.lang.Double	getValue()
org.apache.pig.impl.logicalLayer.schema.Schema	outputSchema(org.apache.pig.impl.logicalLayer.schema.Schema input)

Methods inherited from class org.apache.pig.AccumulatorEvalFunc

exec

Methods inherited from class org.apache.pig.EvalFunc

allowCompileTimeCalculation, finish, getArgToFuncMapping, getCacheFiles, getInputSchema, getLogger, getPigLogger, getReporter, getReturnType, getSchemaName, getSchemaType, getShipFiles, isAsynchronous, progress, setInputSchema, setPigLogger, setReporter, setUDFContextSignature, warn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CondEntropy

public CondEntropy()
            throws org.apache.pig.backend.executionengine.ExecException

Throws:

org.apache.pig.backend.executionengine.ExecException

CondEntropy

public CondEntropy(java.lang.String type)
            throws org.apache.pig.backend.executionengine.ExecException

Throws:

org.apache.pig.backend.executionengine.ExecException

CondEntropy

public CondEntropy(java.lang.String type,
           java.lang.String base)
            throws org.apache.pig.backend.executionengine.ExecException

Throws:

org.apache.pig.backend.executionengine.ExecException

Method Detail

accumulate

public void accumulate(org.apache.pig.data.Tuple input)
                throws java.io.IOException

Specified by:

accumulate in interface org.apache.pig.Accumulator<java.lang.Double>

Specified by:

accumulate in class org.apache.pig.AccumulatorEvalFunc<java.lang.Double>

Throws:

java.io.IOException

getValue

public java.lang.Double getValue()

Specified by:

getValue in interface org.apache.pig.Accumulator<java.lang.Double>

Specified by:

getValue in class org.apache.pig.AccumulatorEvalFunc<java.lang.Double>

cleanup

public void cleanup()

Specified by:

cleanup in interface org.apache.pig.Accumulator<java.lang.Double>

Specified by:

cleanup in class org.apache.pig.AccumulatorEvalFunc<java.lang.Double>

outputSchema

public org.apache.pig.impl.logicalLayer.schema.Schema outputSchema(org.apache.pig.impl.logicalLayer.schema.Schema input)

Overrides:

outputSchema in class org.apache.pig.EvalFunc<java.lang.Double>