Apache DataFu™

Getting Started

DataFu Spark Docs

DataFu Pig Docs

DataFu Hourglass Docs

Community

Apache Software Foundation

Getting Started

Download

Apache DataFu is available for download as a source release and as compiled artifacts stored in a Maven repository. Please note that the latest version of datafu-pig and datafu-hourglass is not the same as the latest version of datafu-spark.

Source Releases

The latest source release can be found here:

Previous releases:

Validation

It is important to validate the release using either the PGP signature (.asc file) or hashes (.sha512 file). For more information on verification of Apache releases, see here. The KEYS file can be found here.

Once fetched, the KEYS file can be imported and the .asc file can used to verify the release.

gpg --import KEYS
gpg --verify apache-datafu-sources-2.0.0.tgz.asc apache-datafu-sources-2.0.0.tgz

The sha512sum tool can be used to compute a SHA-512 hash that can be compared against the .sha512 file:

sha512sum apache-datafu-sources-2.0.0.tgz
cat apache-datafu-sources-2.0.0.tgz.sha512

Note that the hashes are only intended to check that the file has been downloaded correctly. They do not provide guarantees on the authenticity of the file. The signature should instead be used for this purpose. For more information see here.

Setup

Make sure you have Gradle installed. Extract the source and bootstrap the gradlew script that's used for building. The gradlew script uses the specific version of Gradle that DataFu is intended to be built with.

tar xvf apache-datafu-sources-2.0.0.tgz
cd apache-datafu-sources-2.0.0
gradle -b bootstrap.gradle

To build the JARs, run:

./gradlew assemble

This will produce JARs in the following directories:

  • datafu-spark/build/libs
  • datafu-pig/build/libs
  • datafu-hourglass/build/libs

Local Maven Install

DataFu artifacts can be installed to your local maven repository like so:

./gradlew install

Assuming your local maven repository is at ~/.m2, you should see the DataFu artifacts under ~/.m2/repository/org/apache/datafu/. You should now be able to declare a dependency on DataFu artifacts as shown below.

Maven

The latest release can be found in Apache's Maven Repository for DataFu:

You can also use a dependency management system to download the DataFu artifacts and all their dependencies. Some examples appear below.

SBT:

libraryDependencies += "org.apache.datafu" %%  "datafu-spark" % "2.0.0" intransitive

Gradle:

compile "org.apache.datafu:datafu-spark:2.0.0"

Maven:

<dependency>
  <groupId>org.apache.datafu</groupId>
  <artifactId>datafu-spark</artifactId>
  <version>2.0.0</version>
</dependency>

Next Steps

See the following guides for next steps:

Apache Feather
Copyright © 2011-2024 The Apache Software Foundation, Licensed under the Apache License, Version 2.0.
Apache DataFu, DataFu, Apache Pig, Apache Hadoop, Hadoop, Apache, and the Apache feather logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and other countries.