datafu.hourglass.jobs
Class TimeBasedJob

java.lang.Object
  org.apache.hadoop.conf.Configured
      datafu.hourglass.jobs.AbstractJob
          datafu.hourglass.jobs.TimeBasedJob

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable

Direct Known Subclasses:

AbstractNonIncrementalJob, IncrementalJob

public abstract class TimeBasedJob
extends AbstractJob

Base class for Hadoop jobs that consume time-partitioned data.

This class has the same configuration and methods as AbstractJob. In addition it also recognizes the following properties:

• num.days - Number of consecutive days of input data to consume
• days.ago - Number of days to subtract off the end of the consumption window
• start.date - Start date for window in yyyyMMdd format
• end.date - End date for window in yyyyMMdd format

Methods are available as well for setting these configuration parameters.

Author:

"Matthew Hayes"

Constructor Summary

TimeBasedJob()
Initializes the job.

TimeBasedJob(java.lang.String name, java.util.Properties props)
Initializes the job with a job name and properties.

Method Summary

java.lang.Integer	getDaysAgo() Gets the number of days to subtract off the end of the consumption window.
java.util.Date	getEndDate() Gets the end date.
java.lang.Integer	getNumDays() Gets the number of consecutive days to process.
java.util.Date	getStartDate() Gets the start date.
void	setDaysAgo(java.lang.Integer daysAgo) Sets the number of days to subtract off the end of the consumption window.
void	setEndDate(java.util.Date endDate) Sets the end date.
void	setNumDays(java.lang.Integer numDays) Sets the number of consecutive days to process.
void	setProperties(java.util.Properties props) Sets the configuration properties.
void	setStartDate(java.util.Date startDate) Sets the start date.
protected void	validate() Validation required before running job.

Methods inherited from class datafu.hourglass.jobs.AbstractJob

config, createRandomTempPath, ensurePath, getCountersParentPath, getFileSystem, getInputPaths, getName, getNumReducers, getOutputPath, getProperties, getRetentionCount, getTempPath, initialize, isUseCombiner, randomTempPath, run, setCountersParentPath, setInputPaths, setName, setNumReducers, setOutputPath, setRetentionCount, setTempPath, setUseCombiner

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TimeBasedJob

public TimeBasedJob()

Initializes the job.

TimeBasedJob

public TimeBasedJob(java.lang.String name,
                    java.util.Properties props)

Initializes the job with a job name and properties.

Parameters:

name - Job name

props - Configuration properties

Method Detail

setProperties

public void setProperties(java.util.Properties props)

Description copied from class: AbstractJob

Sets the configuration properties.

Overrides:

setProperties in class AbstractJob

Parameters:

props - Properties

getNumDays

public java.lang.Integer getNumDays()

Gets the number of consecutive days to process.

Returns:

number of days to process

setNumDays

public void setNumDays(java.lang.Integer numDays)

Sets the number of consecutive days to process.

Parameters:

numDays - number of days to process

getDaysAgo

public java.lang.Integer getDaysAgo()

Gets the number of days to subtract off the end of the consumption window.

Returns:

Days ago

setDaysAgo

public void setDaysAgo(java.lang.Integer daysAgo)

Sets the number of days to subtract off the end of the consumption window.

Parameters:

daysAgo - Days ago

getStartDate

public java.util.Date getStartDate()

Gets the start date.

Returns:

Start date

setStartDate

public void setStartDate(java.util.Date startDate)

Sets the start date.

Parameters:

startDate - start date

getEndDate

public java.util.Date getEndDate()

Gets the end date.

Returns:

end date

setEndDate

public void setEndDate(java.util.Date endDate)

Sets the end date.

Parameters:

endDate - end date

validate

protected void validate()

Description copied from class: AbstractJob

Validation required before running job.

Overrides:

validate in class AbstractJob