datafu.pig.text.opennlp

Class TokenizeWhitespace

java.lang.Object

org.apache.pig.EvalFunc<org.apache.pig.data.DataBag>

datafu.pig.text.opennlp.TokenizeWhitespace

public class TokenizeWhitespace
extends org.apache.pig.EvalFunc<org.apache.pig.data.DataBag>

The OpenNLP Tokenizers segment an input character sequence into tokens. This one uses the OpenNLP class WhitespaceTokenizer.

Example:

 
 define TokenizeWhitespace datafu.pig.text.opennlp.TokenizeWhitespace();

 -- input:
 -- ("I believe the Masons have infiltrated the Apache PMC.")
 infoo = LOAD 'input' AS (text:chararray);

 -- output:
 -- ({(I),(believe),(the),(Masons),(have),(infiltrated),(the),(Apache),(PMC),(.)})
 outfoo = FOREACH infoo GENERATE TokenizeWhitespace(text) as tokens;
 
 

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.pig.EvalFunc

org.apache.pig.EvalFunc.SchemaType

Field Summary

Fields inherited from class org.apache.pig.EvalFunc

log, pigLogger, reporter, returnType

Constructor Summary

Constructors

Constructor and Description
TokenizeWhitespace()

Method Summary

Modifier and Type	Method and Description
org.apache.pig.data.DataBag	exec(org.apache.pig.data.Tuple input)
org.apache.pig.impl.logicalLayer.schema.Schema	outputSchema(org.apache.pig.impl.logicalLayer.schema.Schema input)

Methods inherited from class org.apache.pig.EvalFunc

allowCompileTimeCalculation, finish, getArgToFuncMapping, getCacheFiles, getInputSchema, getLogger, getPigLogger, getReporter, getReturnType, getSchemaName, getSchemaType, getShipFiles, isAsynchronous, progress, setInputSchema, setPigLogger, setReporter, setUDFContextSignature, warn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenizeWhitespace

public TokenizeWhitespace()

Method Detail

exec

public org.apache.pig.data.DataBag exec(org.apache.pig.data.Tuple input)
                                 throws java.io.IOException

Specified by:

exec in class org.apache.pig.EvalFunc<org.apache.pig.data.DataBag>

Throws:

java.io.IOException

outputSchema

public org.apache.pig.impl.logicalLayer.schema.Schema outputSchema(org.apache.pig.impl.logicalLayer.schema.Schema input)

Overrides:

outputSchema in class org.apache.pig.EvalFunc<org.apache.pig.data.DataBag>