Running Jaql

Introduction

We've provided several tools to help you try out Jaql. Below, we show how to run the Jaql shell, run the examples from the documentatation, and run unit tests.

Jaql Shell

The Jaql shell is a very simple way to run Jaql queries. Currently, it can run with a Hadoop MiniCluster or your own cluster. Its pre-requisites are as follows:

• Java 1.5 or greater
• Cygwin if using MS Windows
• If running your own, distributed cluster, Hadoop 0.15.3, Hadoop 0.16.4, and Hadoop 0.17.1

1. Download the tar (for Hadoop 0.16.* clusters)
For other versions of Hadoop, download: Jaql + Hadoop 0.15.3 or Jaql + Hadoop 0.17.1

2. untar jaql-0.3.tgz into DIR

3. Setup the environment:
export JAQL_HOME=[DIR]/jaql-0.3 Note: If you already have a Hadoop cluster installed, when running jaql in local mode, be sure to unset Hadoop related environment variables (e.g., HADOOP_HOME, HADOOP_CONF_DIR, ...).

4. run it: from DIR/jaql-0.3, run ./bin/jaql JaqlShell. You should see the jaql> prompt if successful.

By default, JaqlShell launches a Hadoop MiniCluster. A MiniCluster runs HDFS, MapReduce, and HBase as multiple threads in a single process. If you already have a distributed cluster installed and what to try out Jaql, simply pass in the -cluster flag as follows: ./jaql JaqlShell -cluster.

JaqlShell supports the following options:

-cluster [use an existing cluster, do not launch a MiniCluster]
-jars    [list of comma separated jars to include user defined expressions or data stores]
-dir     [MiniCluster option: the root HDFS directory]
-fmt     [MiniCluster option: format HDFS?]
-n       [MiniCluster options: how many nodes to use]

Examples

All examples from the documentation can be found in DIR/jaql/docs/examples.txt. JaqlShell can be used to evaluate the examples either interactively or as a script. To run interactively, fire up the shell, then cut-and-paste. To run as a script, pipe in the example file to the shell, i.e., ./jaql JaqlShell < docs/examples.txt.

Note:Queries from the documentation that intentionally produce exceptions were excluded from examples.txt.

When extending Jaql, be sure to specify the jar file that includes your extensions to the shell. For example, if your extension code is in myextensions.jar, specify the jar file to the shell as follows: ./jaql JaqlShell -jars myextensions.jar.

Unit Tests

The "tests" directory includes other tests that can be used as additional examples. The tests are included under DIR/jaql/tests. Each test includes a Queries file (i.e., DIR/jaql/tests/coreQueries.txt) and a Gold file (i.e., DIR/jaql/tests/coreGold.txt) that lists expected outputs, including when failures are expected. To run all of the tests, from DIR/jaql, run ant test. The test outputs and logs will be placed in DIR/jaql/build/test/cache and DIR/jaql/build/test, respectively. To run only one of the tests, from DIR/jaql, run ant test -Dtest.include="TestCore" for example. The tests can also be run on a distributed cluster as follows: ant test-cluster.