\n",
"\n",
"# Hadoop MiniCluster\n",
" \n",
" \n",
"\n",
"The Hadoop MiniCluster is a lightweight, single-node Hadoop cluster that is primarily used for testing and development purposes. It provides a way to simulate a distributed Hadoop environment on a single machine, allowing developers to experiment with and test their Hadoop applications without the need for a full-scale, multi-node cluster.\n",
"\n",
"Quote from [the official Hadoop documentation](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CLIMiniCluster.html):\n",
"\n",
"\n",
"> *Using the CLI MiniCluster, users can simply start and stop a single-node Hadoop cluster with a single command, and without the need to set any environment variables or manage configuration files. The CLI MiniCluster starts both a YARN/MapReduce & HDFS clusters.*\n",
"\n",
"\n",
"In this notebook we download the Hadoop core and guide you through the steps required to launch the MiniCluster."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"cellView": "form",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 98
},
"execution": {
"iopub.execute_input": "2024-10-13T09:38:37.111977Z",
"iopub.status.busy": "2024-10-13T09:38:37.111765Z",
"iopub.status.idle": "2024-10-13T09:38:37.123356Z",
"shell.execute_reply": "2024-10-13T09:38:37.122707Z"
},
"id": "8Rl-ti3i2aWM",
"outputId": "e93a8d0e-0eb8-4fee-edb3-04e10a4fd0d4"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
\n",
"\n",
" Note:\n",
"While the MiniCluster is useful for many development and testing scenarios, it's important to note that it does not fully replicate the complexities and challenges of a true multi-node Hadoop cluster. For production-scale testing or performance evaluations, a larger, more representative cluster setup is recommended.\n",
"
\n",
"\n",
" Note:\n",
"While the MiniCluster is useful for many development and testing scenarios, it's important to note that it does not fully replicate the complexities and challenges of a true multi-node Hadoop cluster. For production-scale testing or performance evaluations, a larger, more representative cluster setup is recommended.\n",
"
\n",
"\"\"\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "piPXIeS3KPDd"
},
"source": [
"## About this tutorial\n",
"\n",
"This tutorial dives deep to help you really get the hang of things, explaining every step in Big Data processing, even if it takes a bit of time, so just hang in there and be patient! "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Plpb6qoZ8EVh"
},
"source": [
"# Preamble"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xvhtO--1RWug"
},
"source": [
"## Download Hadoop"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:38:37.156026Z",
"iopub.status.busy": "2024-10-13T09:38:37.155452Z",
"iopub.status.idle": "2024-10-13T09:39:32.751495Z",
"shell.execute_reply": "2024-10-13T09:39:32.750942Z"
},
"id": "qUVKZ2j0vTjS",
"outputId": "6223c729-12a3-4010-b232-4d0fe4de25c3"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"13-Oct-24 09:38:37 AM - INFO: Java is already installed: /usr/lib/jvm/temurin-11-jdk-amd64\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"13-Oct-24 09:38:37 AM - INFO: Downloading hadoop-3.4.0.tar.gz\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"13-Oct-24 09:39:20 AM - INFO: Uncompressing hadoop-3.4.0.tar.gz\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"13-Oct-24 09:39:32 AM - INFO: HADOOP_HOME is /home/runner/work/big_data/big_data/hadoop-3.4.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"13-Oct-24 09:39:32 AM - INFO: PATH is /home/runner/work/big_data/big_data/hadoop-3.4.0/bin:/opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/11.0.24-8/x64/bin:/opt/hostedtoolcache/Python/3.8.18/x64/bin:/opt/hostedtoolcache/Python/3.8.18/x64:/snap/bin:/home/runner/.local/bin:/opt/pipx_bin:/home/runner/.cargo/bin:/home/runner/.config/composer/vendor/bin:/usr/local/.ghcup/bin:/home/runner/.dotnet/tools:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin\n"
]
}
],
"source": [
"import urllib.request\n",
"import os\n",
"import shutil\n",
"import tarfile\n",
"import logging\n",
"import subprocess\n",
"import time\n",
"import sys\n",
"############\n",
"# COSTANTS #\n",
"############\n",
"\n",
"# URL for downloading Hadoop (archive site https://archive.apache.org/dist/)\n",
"HADOOP_URL = \"https://archive.apache.org/dist/hadoop/core/hadoop-3.4.0/hadoop-3.4.0.tar.gz\"\n",
"\n",
"# logging level (should be one of: DEBUG, INFO, WARNING, ERROR, CRITICAL)\n",
"LOGGING_LEVEL = \"INFO\" #@param [\"DEBUG\", \"INFO\", \"WARNING\", \"ERROR\", \"CRITICAL\"]\n",
"\n",
"# setup logging\n",
"for handler in logging.root.handlers[:]:\n",
" logging.root.removeHandler(handler)\n",
"\n",
"logging_level = getattr(logging, LOGGING_LEVEL.upper(), 10)\n",
"\n",
"logging.basicConfig(level=logging_level, \\\n",
" format='%(asctime)s - %(levelname)s: %(message)s', \\\n",
" datefmt='%d-%b-%y %I:%M:%S %p')\n",
"\n",
"logger = logging.getLogger('my_logger')\n",
"\n",
"JAVA_PATH = '/usr/lib/jvm/java-11-openjdk-amd64'\n",
"\n",
"# true if running on Google Colab\n",
"IN_COLAB = 'google.colab' in sys.modules\n",
"if IN_COLAB:\n",
" from google.colab import output\n",
"\n",
"# setup logging\n",
"for handler in logging.root.handlers[:]:\n",
" logging.root.removeHandler(handler)\n",
"\n",
"logging_level = getattr(logging, LOGGING_LEVEL.upper(), 10)\n",
"\n",
"logging.basicConfig(level=logging_level, \\\n",
" format='%(asctime)s - %(levelname)s: %(message)s', \\\n",
" datefmt='%d-%b-%y %I:%M:%S %p')\n",
"\n",
"logger = logging.getLogger('my_logger')\n",
"\n",
"# set variable JAVA_HOME (install Java if necessary)\n",
"def is_java_installed():\n",
" os.environ['JAVA_HOME'] = os.path.realpath(shutil.which(\"java\")).split('/bin')[0]\n",
" return os.environ['JAVA_HOME']\n",
"\n",
"def install_java():\n",
" # Uncomment and modify the desired version\n",
" # java_version= 'openjdk-11-jre-headless'\n",
" # java_version= 'default-jre'\n",
" # java_version= 'openjdk-17-jre-headless'\n",
" # java_version= 'openjdk-18-jre-headless'\n",
" java_version= 'openjdk-19-jre-headless'\n",
"\n",
" print(f\"Java not found. Installing {java_version} ... (this might take a while)\")\n",
" try:\n",
" cmd = f\"apt install -y {java_version}\"\n",
" subprocess_output = subprocess.run(cmd, shell=True, check=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)\n",
" stdout_result = subprocess_output.stdout\n",
" # Process the results as needed\n",
" logger.info(\"Done installing Java {}\".format(java_version))\n",
" os.environ['JAVA_HOME'] = os.path.realpath(shutil.which(\"java\")).split('/bin')[0]\n",
" logger.info(\"JAVA_HOME is {}\".format(os.environ['JAVA_HOME']))\n",
" except subprocess.CalledProcessError as e:\n",
" # Handle the error if the command returns a non-zero exit code\n",
" logger.warn(\"Command failed with return code {}\".format(e.returncode))\n",
" logger.warn(\"stdout: {}\".format(e.stdout))\n",
"\n",
"# Install Java if not available\n",
"if is_java_installed():\n",
" logger.info(\"Java is already installed: {}\".format(os.environ['JAVA_HOME']))\n",
"else:\n",
" logger.info(\"Installing Java\")\n",
" install_java()\n",
"\n",
"\n",
"# download Hadoop\n",
"file_name = os.path.basename(HADOOP_URL)\n",
"if os.path.isfile(file_name):\n",
" logger.info(\"{} already exists, not downloading\".format(file_name))\n",
"else:\n",
" logger.info(\"Downloading {}\".format(file_name))\n",
" urllib.request.urlretrieve(HADOOP_URL, file_name)\n",
"\n",
"# uncompress archive\n",
"dir_name = file_name[:-7]\n",
"if os.path.exists(dir_name):\n",
" logger.info(\"{} is already uncompressed\".format(file_name))\n",
"else:\n",
" logger.info(\"Uncompressing {}\".format(file_name))\n",
" tar = tarfile.open(file_name)\n",
" tar.extractall()\n",
" tar.close()\n",
"\n",
"# environment variables\n",
"os.environ['HADOOP_HOME'] = os.path.join(os.getcwd(), dir_name)\n",
"logger.info(\"HADOOP_HOME is {}\".format(os.environ['HADOOP_HOME']))\n",
"os.environ['PATH'] = ':'.join([os.path.join(os.environ['HADOOP_HOME'], 'bin'), os.environ['PATH']])\n",
"logger.info(\"PATH is {}\".format(os.environ['PATH']))\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "wWzin-_oRuTl"
},
"source": [
"### Check environment variables used by `mapred`\n",
"\n",
"The following steps are not needed but they might be useful to get familiar the `mapred` command.\n",
"\n",
"`mapred minicluster` is the command that we are going to use to start the MiniCluster once a few variables and libraries are taken care of."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:39:32.753542Z",
"iopub.status.busy": "2024-10-13T09:39:32.753349Z",
"iopub.status.idle": "2024-10-13T09:39:33.063692Z",
"shell.execute_reply": "2024-10-13T09:39:33.063050Z"
},
"id": "FmCr_cI05Fvm",
"outputId": "13ab3991-d6d1-40b2-e11f-b903f0703b40"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Usage: mapred [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]\r\n",
" or mapred [OPTIONS] CLASSNAME [CLASSNAME OPTIONS]\r\n",
" where CLASSNAME is a user-provided Java class\r\n",
"\r\n",
" OPTIONS is none or any of:\r\n",
"\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"--config dir Hadoop config directory\r\n",
"--debug turn on shell script debug mode\r\n",
"--help usage information\r\n",
"\r\n",
" SUBCOMMAND is one of:\r\n",
"\r\n",
"\r\n",
" Admin Commands:\r\n",
"\r\n",
"frameworkuploader mapreduce framework upload\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"hsadmin job history server admin interface\r\n",
"\r\n",
" Client Commands:\r\n",
"\r\n",
"classpath prints the class path needed for running mapreduce\r\n",
" subcommands\r\n",
"envvars display computed Hadoop environment variables\r\n",
"job manipulate MapReduce jobs\r\n",
"minicluster CLI MiniCluster\r\n",
"pipes run a Pipes job\r\n",
"queue get information regarding JobQueues\r\n",
"sampler sampler\r\n",
"version print the version\r\n",
"\r\n",
" Daemon Commands:\r\n",
"\r\n",
"historyserver run job history servers as a standalone daemon\r\n",
"\r\n",
"SUBCOMMAND may print help when invoked w/o parameters or with -h.\r\n"
]
}
],
"source": [
"!mapred -h"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:39:33.065960Z",
"iopub.status.busy": "2024-10-13T09:39:33.065731Z",
"iopub.status.idle": "2024-10-13T09:39:33.265831Z",
"shell.execute_reply": "2024-10-13T09:39:33.265184Z"
},
"id": "XSF0r1CJvTjT",
"outputId": "381816ac-2a99-4a5e-c3ee-58ba5322d8ec"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"JAVA_HOME='/usr/lib/jvm/temurin-11-jdk-amd64'\r\n",
"HADOOP_MAPRED_HOME='/home/runner/work/big_data/big_data/hadoop-3.4.0'\r\n",
"MAPRED_DIR='share/hadoop/mapreduce'\r\n",
"MAPRED_LIB_JARS_DIR='share/hadoop/mapreduce/lib'\r\n",
"HADOOP_CONF_DIR='/home/runner/work/big_data/big_data/hadoop-3.4.0/etc/hadoop'\r\n",
"HADOOP_TOOLS_HOME='/home/runner/work/big_data/big_data/hadoop-3.4.0'\r\n",
"HADOOP_TOOLS_DIR='share/hadoop/tools'\r\n",
"HADOOP_TOOLS_LIB_JARS_DIR='share/hadoop/tools/lib'\r\n"
]
}
],
"source": [
"!mapred envvars"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-vNiekW25pbB"
},
"source": [
"## Set `HADOOP_TOOLS_LIB_JARS_DIR`\n",
"\n",
"This variable needs to point to the folder containing Hadoop libraries. As you can see in the output of `mapred envvars`, by default this is set incorrectly to `share/hadoop/tools/lib`."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"execution": {
"iopub.execute_input": "2024-10-13T09:39:33.268129Z",
"iopub.status.busy": "2024-10-13T09:39:33.267908Z",
"iopub.status.idle": "2024-10-13T09:39:33.271578Z",
"shell.execute_reply": "2024-10-13T09:39:33.271019Z"
},
"id": "EKAwTLHdvTjU"
},
"outputs": [],
"source": [
"os.environ['HADOOP_TOOLS_LIB_JARS_DIR'] = os.path.join(os.environ['HADOOP_HOME'], 'share/hadoop/tools/lib/') #IMPORTANT"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zrUFZyHOSBb5"
},
"source": [
"## Install the `mockito` library\n",
"\n",
"To figure out which version of `mockito` is compatible with the current version of Hadoop, check this page for library dependency analysis: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/dependency-analysis.html\n",
"\n",
"We need Mockito 2.28.2."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pQrBSymx6xdT"
},
"source": [
"After a new Hadoop installation, the mockito library is not yet there!"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"execution": {
"iopub.execute_input": "2024-10-13T09:39:33.273487Z",
"iopub.status.busy": "2024-10-13T09:39:33.273289Z",
"iopub.status.idle": "2024-10-13T09:39:33.457952Z",
"shell.execute_reply": "2024-10-13T09:39:33.457269Z"
},
"id": "84AGO_GwvTjU"
},
"outputs": [],
"source": [
"!find hadoop-3.4.0 -name \"mockito*\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ztngaw_S67Hd"
},
"source": [
"Download it from the Maven repository."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:39:33.460281Z",
"iopub.status.busy": "2024-10-13T09:39:33.460067Z",
"iopub.status.idle": "2024-10-13T09:39:33.614059Z",
"shell.execute_reply": "2024-10-13T09:39:33.613421Z"
},
"id": "9W4RaUGbvTjV",
"outputId": "99ccdabf-d9bf-46d3-8caa-96e2d54eb0f7"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"--2024-10-13 09:39:33-- https://repo1.maven.org/maven2/org/mockito/mockito-core/2.28.2/mockito-core-2.28.2.jar\r\n",
"Resolving repo1.maven.org (repo1.maven.org)... 146.75.76.209, 2a04:4e42:83::209\r\n",
"Connecting to repo1.maven.org (repo1.maven.org)|146.75.76.209|:443... connected.\r\n",
"HTTP request sent, awaiting response... 200 OK\r\n",
"Length: 591179 (577K) [application/java-archive]\r\n",
"Saving to: ‘mockito-core-2.28.2.jar’\r\n",
"\r\n",
"\r",
"mockito-core-2.28.2 0%[ ] 0 --.-KB/s \r",
"mockito-core-2.28.2 100%[===================>] 577.32K --.-KB/s in 0.008s \r\n",
"\r\n",
"2024-10-13 09:39:33 (73.2 MB/s) - ‘mockito-core-2.28.2.jar’ saved [591179/591179]\r\n",
"\r\n"
]
}
],
"source": [
"!wget --no-clobber https://repo1.maven.org/maven2/org/mockito/mockito-core/2.28.2/mockito-core-2.28.2.jar"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rlWXc92bSjeE"
},
"source": [
"Install the library in a location where it can be found."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 0
},
"execution": {
"iopub.execute_input": "2024-10-13T09:39:33.616433Z",
"iopub.status.busy": "2024-10-13T09:39:33.615938Z",
"iopub.status.idle": "2024-10-13T09:39:33.621650Z",
"shell.execute_reply": "2024-10-13T09:39:33.621124Z"
},
"id": "jk6CvZZY4ehO",
"outputId": "05d80737-cc4c-4c61-cc8a-1b84f3440671"
},
"outputs": [
{
"data": {
"text/plain": [
"'/home/runner/work/big_data/big_data/hadoop-3.4.0/share/hadoop/mapreduce/mockito-core-2.28.2.jar'"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"shutil.copy('mockito-core-2.28.2.jar', os.path.join(os.environ['HADOOP_HOME'],'share/hadoop/mapreduce/'))"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:39:33.623628Z",
"iopub.status.busy": "2024-10-13T09:39:33.623429Z",
"iopub.status.idle": "2024-10-13T09:39:33.627976Z",
"shell.execute_reply": "2024-10-13T09:39:33.627417Z"
},
"id": "x5KDVOgFvTjU",
"outputId": "b08dae4a-5d9b-4627-c900-d5244c27f6db"
},
"outputs": [
{
"data": {
"text/plain": [
"['hadoop-mapreduce-client-common-3.4.0.jar',\n",
" 'hadoop-mapreduce-client-jobclient-3.4.0-tests.jar',\n",
" 'hadoop-mapreduce-client-hs-plugins-3.4.0.jar',\n",
" 'hadoop-mapreduce-client-shuffle-3.4.0.jar',\n",
" 'hadoop-mapreduce-client-app-3.4.0.jar',\n",
" 'sources',\n",
" 'hadoop-mapreduce-client-jobclient-3.4.0.jar',\n",
" 'hadoop-mapreduce-examples-3.4.0.jar',\n",
" 'hadoop-mapreduce-client-core-3.4.0.jar',\n",
" 'mockito-core-2.28.2.jar',\n",
" 'hadoop-mapreduce-client-uploader-3.4.0.jar',\n",
" 'hadoop-mapreduce-client-hs-3.4.0.jar',\n",
" 'hadoop-mapreduce-client-nativetask-3.4.0.jar',\n",
" 'jdiff']"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"os.listdir(os.path.join(os.environ['HADOOP_HOME'],'share/hadoop/mapreduce/'))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FweWN3enS3HV"
},
"source": [
"## Create folders for the datanodes\n",
"\n",
"These folders are needed for the correct functioning of the MiniCluster."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"execution": {
"iopub.execute_input": "2024-10-13T09:39:33.630040Z",
"iopub.status.busy": "2024-10-13T09:39:33.629663Z",
"iopub.status.idle": "2024-10-13T09:39:33.755900Z",
"shell.execute_reply": "2024-10-13T09:39:33.755203Z"
},
"id": "KZg6NLydvTjU"
},
"outputs": [],
"source": [
"!mkdir -p ./target/test/data/dfs/{name-0-1,name-0-2}"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:39:33.758093Z",
"iopub.status.busy": "2024-10-13T09:39:33.757822Z",
"iopub.status.idle": "2024-10-13T09:39:33.883887Z",
"shell.execute_reply": "2024-10-13T09:39:33.883286Z"
},
"id": "XThvV4kuVYpU",
"outputId": "56e8287a-47e4-4359-c029-1f5fbdad7bd0"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{name-0-1,name-0-2}\r\n"
]
}
],
"source": [
"!ls ./target/test/data/dfs/"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7Odx4SZbOhL4"
},
"source": [
"# Start the MiniCluster\n",
"\n",
"\n",
"To see a full list of options run `mapred minicluster -help`."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:39:33.886271Z",
"iopub.status.busy": "2024-10-13T09:39:33.885831Z",
"iopub.status.idle": "2024-10-13T09:39:34.460051Z",
"shell.execute_reply": "2024-10-13T09:39:34.459359Z"
},
"id": "R7YqK3brVG2J",
"outputId": "7ef5aa31-196e-4d68-8aee-c43a45fd4932"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"usage: ...\r\n",
" -D Options to pass into configuration object\r\n",
" -datanodes How many datanodes to start (default 1)\r\n",
" -format Format the DFS (default false)\r\n",
" -help Prints option help.\r\n",
" -jhsport JobHistoryServer port (default 0--we choose)\r\n",
" -namenode URL of the namenode (default is either the DFS\r\n",
" cluster or a temporary dir)\r\n",
" -nnhttpport NameNode HTTP port (default 0--we choose)\r\n",
" -nnport NameNode port (default 0--we choose)\r\n",
" -nodemanagers How many nodemanagers to start (default 1)\r\n",
" -nodfs Don't start a mini DFS cluster\r\n",
" -nomr Don't start a mini MR cluster\r\n",
" -rmport ResourceManager port (default 0--we choose)\r\n",
" -writeConfig Save configuration to this XML file.\r\n",
" -writeDetails Write basic information to this JSON file.\r\n"
]
}
],
"source": [
"!mapred minicluster -help"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CKX9vXJ9MRlI"
},
"source": [
"If you are not running this notebook for the first time or have edited the `core-site.xml` file you should now empty it to get the default initial configuration.\n",
"\n",
"**Note:** the file `core-site.xml` needs to exist and contain the lines\n",
"\n",
"```\n",
"\n",
"\n",
"\n",
"```\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:39:34.462662Z",
"iopub.status.busy": "2024-10-13T09:39:34.462218Z",
"iopub.status.idle": "2024-10-13T09:39:34.644786Z",
"shell.execute_reply": "2024-10-13T09:39:34.644145Z"
},
"id": "VANlMygeMukl",
"outputId": "c9069a09-fb71-4262-87b4-365664f7de9f"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/home/runner/work/big_data/big_data/hadoop-3.4.0/etc/hadoop/core-site.xml\r\n"
]
}
],
"source": [
"# check if the file is there\n",
"!find $HADOOP_HOME -name \"core-site.xml\""
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:39:34.647356Z",
"iopub.status.busy": "2024-10-13T09:39:34.646709Z",
"iopub.status.idle": "2024-10-13T09:39:34.829642Z",
"shell.execute_reply": "2024-10-13T09:39:34.828875Z"
},
"id": "FyNiiZki_1s1",
"outputId": "e38a9975-38f8-49f9-c347-f7d5327f74e7"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\r\n",
"\r\n",
"\r\n",
"\r\n",
"\r\n",
"\r\n",
"\r\n",
"\r\n"
]
}
],
"source": [
"# view the contents of the file\n",
"!cat $(find $HADOOP_HOME -name \"core-site.xml\")"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"execution": {
"iopub.execute_input": "2024-10-13T09:39:34.832137Z",
"iopub.status.busy": "2024-10-13T09:39:34.831740Z",
"iopub.status.idle": "2024-10-13T09:39:34.835546Z",
"shell.execute_reply": "2024-10-13T09:39:34.834917Z"
},
"id": "w4ihWl6hMz2s"
},
"outputs": [],
"source": [
"with open(os.environ['HADOOP_HOME']+'/etc/hadoop/core-site.xml', 'w') as file:\n",
" file.write(\"\\n\")"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:39:34.837746Z",
"iopub.status.busy": "2024-10-13T09:39:34.837383Z",
"iopub.status.idle": "2024-10-13T09:39:35.019711Z",
"shell.execute_reply": "2024-10-13T09:39:35.019025Z"
},
"id": "C4QcevYKdvf7",
"outputId": "901ed557-b095-488d-85b1-bbeccbfd6cba"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\r\n",
""
]
}
],
"source": [
"!cat $(find $HADOOP_HOME -name \"core-site.xml\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_2vNXuN1AE6q"
},
"source": [
"## `mapred minicluster -format`\n",
"\n",
"Finally, we are all set up to start the MiniCluster.\n",
"\n",
"Make sure to include the `-format` option to initialize and format the filesystem.\n",
"\n",
"Other than that, we are using the defaults for all the other options.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "y7jGgVy3WF4S"
},
"source": [
"Note that this process runs forever thus blocking the notebook. In order to proceed with the rest of the notebook, just interrupt the running cell.\n",
"\n",
"We'll see later how to run the MiniCluster as a subprocess without blocking the notebook's cells execution.\n",
"\n",
"Uncomment the next cell to launch the MiniCluster!"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"execution": {
"iopub.execute_input": "2024-10-13T09:39:35.022256Z",
"iopub.status.busy": "2024-10-13T09:39:35.021846Z",
"iopub.status.idle": "2024-10-13T09:39:35.024986Z",
"shell.execute_reply": "2024-10-13T09:39:35.024331Z"
},
"id": "5c46g83gvTjX"
},
"outputs": [],
"source": [
"#!mapred minicluster -format"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "IlU0eS5_QGbq"
},
"source": [
"If the MiniMRCluster started correctly, you should see a line like this at the bottom:\n",
"\n",
" 2024-01-14 13:53:15,112 INFO mapreduce.MiniHadoopClusterManager: Started MiniMRCluster"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Oc0xM9VNRei5"
},
"source": [
"# Stop the MiniCluster process\n",
"\n",
"To continue to work with this notebook, you need to stop the MiniCluster by terminating the execution of the previous cell."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fJGsLuZ46pXf"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "F9BfFS27vTjX"
},
"source": [
"# Start the MiniCluster as a subprocess\n",
"\n",
"It is convenient to start the MiniCluster as a subprocess in order prevent it from blocking the execution of other notebook cells.\n",
"\n",
"The MiniCluster is a Java process with multiple listening ports.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0X1zImnU5Afb"
},
"source": [
"## Use `lsof` to show listening ports\n",
"\n",
"To check ports that have listening services use `lsof`.\n",
"\n",
"The output should look like this:\n",
"\n",
"```\n",
"COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\n",
"node 7 root 21u IPv6 19666 0t0 TCP *:8080 (LISTEN)\n",
"kernel_manager_ 20 root 3u IPv4 18322 0t0 TCP 172.28.0.12:6000 (LISTEN)\n",
"colab-fileshim. 61 root 3u IPv4 19763 0t0 TCP 127.0.0.1:3453 (LISTEN)\n",
"jupyter-noteboo 79 root 7u IPv4 19989 0t0 TCP 172.28.0.12:9000 (LISTEN)\n",
"python3 428 root 21u IPv4 25506 0t0 TCP 127.0.0.1:38217 (LISTEN)\n",
"python3 467 root 3u IPv4 26425 0t0 TCP 127.0.0.1:43729 (LISTEN)\n",
"python3 467 root 5u IPv4 26426 0t0 TCP 127.0.0.1:60229 (LISTEN)\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:39:35.027530Z",
"iopub.status.busy": "2024-10-13T09:39:35.027169Z",
"iopub.status.idle": "2024-10-13T09:39:35.191131Z",
"shell.execute_reply": "2024-10-13T09:39:35.190350Z"
},
"id": "v_ieJekiBv-2",
"outputId": "25c07f4f-56ba-4431-9c8a-f4cd72a25b7c"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\r\n",
"python 4369 runner 10u IPv4 74469 0t0 TCP 127.0.0.1:37001 (LISTEN)\r\n",
"python 4369 runner 12u IPv4 74471 0t0 TCP 127.0.0.1:51093 (LISTEN)\r\n",
"python 4369 runner 14u IPv4 74473 0t0 TCP 127.0.0.1:43921 (LISTEN)\r\n",
"python 4369 runner 23u IPv4 74477 0t0 TCP 127.0.0.1:49399 (LISTEN)\r\n",
"python 4369 runner 28u IPv4 74481 0t0 TCP 127.0.0.1:36477 (LISTEN)\r\n",
"python 4369 runner 36u IPv4 72829 0t0 TCP 127.0.0.1:44503 (LISTEN)\r\n"
]
}
],
"source": [
"!lsof -n -i -P +c0 -sTCP:LISTEN"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tg9V_qXGZ1fe"
},
"source": [
"Options used in `lsof`:\n",
"\n",
"* `-i` specifies that you want to display only network files, that is open network connections\n",
"* `-n` and `-P` tell `lsof` to show IP addresses (`-n`) and ports (`-P`) in numeric form. This makes `lsof` faster as it saves the time for name lookups.\n",
"* the option `+c0` is used to show a longer substring of the name of the UNIX command associated with the process (https://linux.die.net/man/8/lsof)\n",
"* `-sTCP:LISTEN` filters for TCP connections in state `LISTEN`\n",
"\n",
"See also [lsof and listening ports](https://unix.stackexchange.com/questions/26887/lsof-and-listening-ports) on Stackexchange."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a-OKONXJBwkf"
},
"source": [
"## Start the subprocess\n",
"\n",
"Start the MiniCluster as a subprocess using Python's [`subprocess` library](https://docs.python.org/3/library/subprocess.html).\n",
"\n",
"The files `out.txt` and `err.txt` will contain respectively the standard output and the standard error emitted by the `mapred minicluster` command."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"execution": {
"iopub.execute_input": "2024-10-13T09:39:35.193719Z",
"iopub.status.busy": "2024-10-13T09:39:35.193312Z",
"iopub.status.idle": "2024-10-13T09:39:35.202534Z",
"shell.execute_reply": "2024-10-13T09:39:35.201683Z"
},
"id": "aTBwruBkAuP_"
},
"outputs": [],
"source": [
"import subprocess\n",
"with open('out.txt', \"w\") as stdout_file, open('err.txt', \"w\") as stderr_file:\n",
" process = subprocess.Popen(\n",
" [\"mapred\", \"minicluster\", \"-format\"],\n",
" stdout=stdout_file,\n",
" stderr=stderr_file\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gY484JgGCVUd"
},
"source": [
"Wait for a couple of seconds because the services might not be available immediately."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"execution": {
"iopub.execute_input": "2024-10-13T09:39:35.204996Z",
"iopub.status.busy": "2024-10-13T09:39:35.204740Z",
"iopub.status.idle": "2024-10-13T09:40:05.238381Z",
"shell.execute_reply": "2024-10-13T09:40:05.237727Z"
},
"id": "dk_FGmuOCoBW"
},
"outputs": [],
"source": [
"if not IN_COLAB:\n",
" time.sleep(30)\n",
"else:\n",
" time.sleep(10)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6iBMkb6fCqUU"
},
"source": [
"Now check for listening ports again (you can also refresh the next cell with ctrl-enter)."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:05.241041Z",
"iopub.status.busy": "2024-10-13T09:40:05.240620Z",
"iopub.status.idle": "2024-10-13T09:40:05.418325Z",
"shell.execute_reply": "2024-10-13T09:40:05.417462Z"
},
"id": "Rt9J9bj4A_5m",
"outputId": "a18dc323-201b-4680-c488-897b64a3737e"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\r\n",
"python 4369 runner 10u IPv4 74469 0t0 TCP 127.0.0.1:37001 (LISTEN)\r\n",
"python 4369 runner 12u IPv4 74471 0t0 TCP 127.0.0.1:51093 (LISTEN)\r\n",
"python 4369 runner 14u IPv4 74473 0t0 TCP 127.0.0.1:43921 (LISTEN)\r\n",
"python 4369 runner 23u IPv4 74477 0t0 TCP 127.0.0.1:49399 (LISTEN)\r\n",
"python 4369 runner 28u IPv4 74481 0t0 TCP 127.0.0.1:36477 (LISTEN)\r\n",
"python 4369 runner 36u IPv4 72829 0t0 TCP 127.0.0.1:44503 (LISTEN)\r\n",
"java 4612 runner 341u IPv4 72927 0t0 TCP 127.0.0.1:40811 (LISTEN)\r\n",
"java 4612 runner 354u IPv4 72930 0t0 TCP 127.0.0.1:37841 (LISTEN)\r\n",
"java 4612 runner 364u IPv4 76854 0t0 TCP 127.0.0.1:32815 (LISTEN)\r\n",
"java 4612 runner 367u IPv4 72948 0t0 TCP 127.0.0.1:41281 (LISTEN)\r\n",
"java 4612 runner 420u IPv4 76859 0t0 TCP 127.0.0.1:33233 (LISTEN)\r\n",
"java 4612 runner 421u IPv4 76863 0t0 TCP 127.0.0.1:45237 (LISTEN)\r\n",
"java 4612 runner 444u IPv4 82849 0t0 TCP *:8031 (LISTEN)\r\n",
"java 4612 runner 473u IPv4 79114 0t0 TCP *:10033 (LISTEN)\r\n",
"java 4612 runner 483u IPv4 79117 0t0 TCP *:19888 (LISTEN)\r\n",
"java 4612 runner 485u IPv4 79118 0t0 TCP *:8088 (LISTEN)\r\n",
"java 4612 runner 499u IPv4 85381 0t0 TCP 127.0.0.1:36717 (LISTEN)\r\n",
"java 4612 runner 509u IPv4 82843 0t0 TCP *:8033 (LISTEN)\r\n",
"java 4612 runner 529u IPv4 82853 0t0 TCP *:8030 (LISTEN)\r\n",
"java 4612 runner 539u IPv4 85385 0t0 TCP 127.0.0.1:44123 (LISTEN)\r\n",
"java 4612 runner 549u IPv4 79130 0t0 TCP 127.0.0.1:38125 (LISTEN)\r\n",
"java 4612 runner 559u IPv4 79134 0t0 TCP 127.0.0.1:39879 (LISTEN)\r\n",
"java 4612 runner 569u IPv4 79138 0t0 TCP *:40817 (LISTEN)\r\n",
"java 4612 runner 570u IPv4 79139 0t0 TCP 127.0.0.1:43613 (LISTEN)\r\n"
]
}
],
"source": [
"!lsof -n -i -P +c0 -sTCP:LISTEN"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "crP9O3O9DfUT"
},
"source": [
"You should have gotten something like this (a total of $18$ listening ports associated with the MiniCluster process):\n",
"\n",
"```\n",
"COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\n",
"node 6 root 21u IPv6 17373 0t0 TCP *:8080 (LISTEN)\n",
"kernel_manager_ 20 root 3u IPv4 17180 0t0 TCP 172.28.0.12:6000 (LISTEN)\n",
"colab-fileshim. 58 root 3u IPv4 19499 0t0 TCP 127.0.0.1:3453 (LISTEN)\n",
"jupyter-noteboo 75 root 7u IPv4 19658 0t0 TCP 172.28.0.12:9000 (LISTEN)\n",
"python3 4081 root 21u IPv4 108431 0t0 TCP 127.0.0.1:44519 (LISTEN)\n",
"python3 4108 root 3u IPv4 109755 0t0 TCP 127.0.0.1:46699 (LISTEN)\n",
"python3 4108 root 5u IPv4 109756 0t0 TCP 127.0.0.1:51813 (LISTEN)\n",
"java 17261 root 347u IPv4 390097 0t0 TCP 127.0.0.1:38817 (LISTEN)\n",
"java 17261 root 357u IPv4 391184 0t0 TCP 127.0.0.1:41631 (LISTEN)\n",
"java 17261 root 367u IPv4 390801 0t0 TCP 127.0.0.1:34651 (LISTEN)\n",
"java 17261 root 370u IPv4 390804 0t0 TCP 127.0.0.1:35015 (LISTEN)\n",
"java 17261 root 399u IPv4 390814 0t0 TCP 127.0.0.1:46503 (LISTEN)\n",
"java 17261 root 400u IPv4 390817 0t0 TCP 127.0.0.1:44665 (LISTEN)\n",
"java 17261 root 423u IPv4 401418 0t0 TCP *:8031 (LISTEN)\n",
"java 17261 root 440u IPv4 395796 0t0 TCP *:10033 (LISTEN)\n",
"java 17261 root 450u IPv4 400174 0t0 TCP *:19888 (LISTEN)\n",
"java 17261 root 455u IPv4 396244 0t0 TCP 127.0.0.1:43877 (LISTEN)\n",
"java 17261 root 465u IPv4 400367 0t0 TCP *:8088 (LISTEN)\n",
"java 17261 root 470u IPv4 400420 0t0 TCP *:8033 (LISTEN)\n",
"java 17261 root 490u IPv4 401422 0t0 TCP *:8030 (LISTEN)\n",
"java 17261 root 500u IPv4 400426 0t0 TCP 127.0.0.1:37335 (LISTEN)\n",
"java 17261 root 510u IPv4 400470 0t0 TCP 127.0.0.1:42337 (LISTEN)\n",
"java 17261 root 520u IPv4 401450 0t0 TCP 127.0.0.1:40401 (LISTEN)\n",
"java 17261 root 530u IPv4 401454 0t0 TCP *:42359 (LISTEN)\n",
"java 17261 root 531u IPv4 401457 0t0 TCP 127.0.0.1:38543 (LISTEN)\n",
"```\n",
"\n",
"The `java` process is the one responsible for providing the MiniCluster services by listening on several ports.\n",
"\n",
"There are two known ports for the Web interfaces (see [https://hadoop.apache.org/docs/.../ClusterSetup.html#Web_Interfaces](https://hadoop.apache.org/docs/r3.0.1/hadoop-project-dist/hadoop-common/ClusterSetup.html#Web_Interfaces)):\n",
"* $8088$ the port to the ResourceManager\n",
"* $19888$ the port to the MapReduce JobHistory Server\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rHb_H6-rt2NQ"
},
"source": [
"### Access the Web UI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vUo-K1HmKTIv"
},
"source": [
"Let us check the Web interface at port $8088$."
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:05.421389Z",
"iopub.status.busy": "2024-10-13T09:40:05.420924Z",
"iopub.status.idle": "2024-10-13T09:40:05.725928Z",
"shell.execute_reply": "2024-10-13T09:40:05.725198Z"
},
"id": "GOKETlOVKTRg",
"outputId": "c3c0a621-d346-4841-8c14-9697c1dd1f84"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"--2024-10-13 09:40:05-- http://localhost:8088/\r\n",
"Resolving localhost (localhost)... ::1, 127.0.0.1\r\n",
"Connecting to localhost (localhost)|::1|:8088... failed: Connection refused.\r\n",
"Connecting to localhost (localhost)|127.0.0.1|:8088... connected.\r\n",
"HTTP request sent, awaiting response... "
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"302 Found\r\n",
"Location: http://localhost:8088/node [following]\r\n",
"--2024-10-13 09:40:05-- http://localhost:8088/node\r\n",
"Reusing existing connection to localhost:8088.\r\n",
"HTTP request sent, awaiting response... "
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"200 OK\r\n",
"Length: 6597 (6.4K) [text/html]\r\n",
"Saving to: ‘index.html’\r\n",
"\r\n",
"\r",
"index.html 0%[ ] 0 --.-KB/s \r",
"index.html 100%[===================>] 6.44K --.-KB/s in 0s \r\n",
"\r\n",
"2024-10-13 09:40:05 (694 MB/s) - ‘index.html’ saved [6597/6597]\r\n",
"\r\n"
]
}
],
"source": [
"!wget http://localhost:8088"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jO6uA0_aLbX_"
},
"source": [
"We can serve the ResourceManager UI in the browser through Google Colab."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 52
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:05.728835Z",
"iopub.status.busy": "2024-10-13T09:40:05.728371Z",
"iopub.status.idle": "2024-10-13T09:40:05.732083Z",
"shell.execute_reply": "2024-10-13T09:40:05.731507Z"
},
"id": "4a1mpPpYL25c",
"outputId": "2f72432f-1ab6-41d4-a3ac-39d72dd23e0f"
},
"outputs": [],
"source": [
"if IN_COLAB:\n",
" # serve the Web UI on Colab\n",
" print(\"Click on the link below to open the Resource Manager Web UI 🚀\")\n",
" output.serve_kernel_port_as_window(8088, path='/node')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zoL28RXvRyZV"
},
"source": [
"The port $19888$ is redirected to the same page as port $8088$, so it won't be very useful. I'm not sure if this is due to a missing configuration parameter or if it's a bug."
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:05.734102Z",
"iopub.status.busy": "2024-10-13T09:40:05.733892Z",
"iopub.status.idle": "2024-10-13T09:40:05.873623Z",
"shell.execute_reply": "2024-10-13T09:40:05.872775Z"
},
"id": "Ll29MYBUSAhg",
"outputId": "efcf5335-4eae-40df-b101-603983590c27"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"--2024-10-13 09:40:05-- http://localhost:19888/\r\n",
"Resolving localhost (localhost)... ::1, 127.0.0.1\r\n",
"Connecting to localhost (localhost)|::1|:19888... failed: Connection refused.\r\n",
"Connecting to localhost (localhost)|127.0.0.1|:19888... connected.\r\n",
"HTTP request sent, awaiting response... 302 Found\r\n",
"Location: http://localhost:19888/node [following]\r\n",
"--2024-10-13 09:40:05-- http://localhost:19888/node\r\n",
"Reusing existing connection to localhost:19888.\r\n",
"HTTP request sent, awaiting response... 200 OK\r\n",
"Length: 6597 (6.4K) [text/html]\r\n",
"Saving to: ‘index.html.1’\r\n",
"\r\n",
"\r",
"index.html.1 0%[ ] 0 --.-KB/s \r",
"index.html.1 100%[===================>] 6.44K --.-KB/s in 0s \r\n",
"\r\n",
"2024-10-13 09:40:05 (585 MB/s) - ‘index.html.1’ saved [6597/6597]\r\n",
"\r\n"
]
}
],
"source": [
"!wget http://localhost:19888"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 52
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:05.876300Z",
"iopub.status.busy": "2024-10-13T09:40:05.876059Z",
"iopub.status.idle": "2024-10-13T09:40:05.879814Z",
"shell.execute_reply": "2024-10-13T09:40:05.879135Z"
},
"id": "XwxSD_Ez4mA5",
"outputId": "2ceef797-d3a6-4da5-a55c-4d223815dbe9"
},
"outputs": [],
"source": [
"if IN_COLAB:\n",
" # serve the Web UI on Colab\n",
" print(\"Click on the link below to open the MapReduce JobHistory Server Web UI 🚀\")\n",
" output.serve_kernel_port_as_window(19888, path='/node')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qq0i7cnLuSvm"
},
"source": [
"### Use ngrok to access the Web UI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "z6c_QA-fsJib"
},
"source": [
"In the free tier of Google Colab this functionality might not be available (see https://research.google.com/colaboratory/faq.html#limitations-and-restrictions). As an alternative, you can use [ngrok](https://ngrok.com/) after signing up for a free account.\n",
"\n",
"Check the NGROK box below if you want to use ngrok.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"execution": {
"iopub.execute_input": "2024-10-13T09:40:05.881875Z",
"iopub.status.busy": "2024-10-13T09:40:05.881683Z",
"iopub.status.idle": "2024-10-13T09:40:05.884686Z",
"shell.execute_reply": "2024-10-13T09:40:05.884090Z"
},
"id": "jD7pB1dHGaMB"
},
"outputs": [],
"source": [
"# you should set this to True\n",
"NGROK = False #@param {type:\"boolean\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "24l7XugFGsfQ"
},
"source": [
"We are going to use the Python ngrok client `pyngrok` (see the [Colab example](https://pyngrok.readthedocs.io/en/latest/integrations.html#colab-http-example))."
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:05.887037Z",
"iopub.status.busy": "2024-10-13T09:40:05.886506Z",
"iopub.status.idle": "2024-10-13T09:40:05.890673Z",
"shell.execute_reply": "2024-10-13T09:40:05.890102Z"
},
"id": "Ed-77ALaqlel",
"outputId": "9d6e8bda-bd81-4f79-f324-d10e9a7d9ba1"
},
"outputs": [],
"source": [
"if NGROK:\n",
" !pip install pyngrok\n",
" from pyngrok import ngrok, conf\n",
" import getpass\n",
"\n",
" print(\"Enter your authtoken, which can be copied from https://dashboard.ngrok.com/get-started/your-authtoken\")\n",
" conf.get_default().auth_token = getpass.getpass()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mI2SrsWBrOeI"
},
"source": [
"After entering the ngrok authorization token, you can open a connection."
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:05.892578Z",
"iopub.status.busy": "2024-10-13T09:40:05.892390Z",
"iopub.status.idle": "2024-10-13T09:40:05.895460Z",
"shell.execute_reply": "2024-10-13T09:40:05.894902Z"
},
"id": "xvvmU6ZArDz_",
"outputId": "5e3d79f5-5e51-4861-c9e9-6a4cb921af9d"
},
"outputs": [],
"source": [
"if NGROK:\n",
" # Open a ngrok tunnel to the HTTP server\n",
" public_url = ngrok.connect(19888).public_url"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:05.897576Z",
"iopub.status.busy": "2024-10-13T09:40:05.897211Z",
"iopub.status.idle": "2024-10-13T09:40:05.900299Z",
"shell.execute_reply": "2024-10-13T09:40:05.899727Z"
},
"id": "uzpDDM-xrbHu",
"outputId": "3f228a4c-655c-4dd2-a4f9-61ce2e2f68f8"
},
"outputs": [],
"source": [
"if NGROK:\n",
" print(f'Click on {public_url} to open the MapReduce JobHistory Server Web UI')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WV2BBzEwvv_X"
},
"source": [
"You can safely ignore the warning since we are not disclosing any confidential information and proceed with clicking on the \"Visit site\" button."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PTN1yVvhwYUr"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7UQg9PjpNJRj"
},
"source": [
"## Stop the MiniCluster subprocess\n",
"\n",
"To stop the MiniCluster subprocess use `process.kill()` (remember: `process` is the variable name for the MiniCluster subprocess)."
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"execution": {
"iopub.execute_input": "2024-10-13T09:40:05.902369Z",
"iopub.status.busy": "2024-10-13T09:40:05.902183Z",
"iopub.status.idle": "2024-10-13T09:40:05.907136Z",
"shell.execute_reply": "2024-10-13T09:40:05.906487Z"
},
"id": "9_UGJUPYNWuH"
},
"outputs": [],
"source": [
"process.kill()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "F5nVzcd2NcgZ"
},
"source": [
"The Java process should now be gone."
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:05.909470Z",
"iopub.status.busy": "2024-10-13T09:40:05.909034Z",
"iopub.status.idle": "2024-10-13T09:40:06.086905Z",
"shell.execute_reply": "2024-10-13T09:40:06.086050Z"
},
"id": "PGJ0Bs0ENbDZ",
"outputId": "32bf4e82-d198-43c4-d745-648b5a79d7e3"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\r\n",
"python 4369 runner 10u IPv4 74469 0t0 TCP 127.0.0.1:37001 (LISTEN)\r\n",
"python 4369 runner 12u IPv4 74471 0t0 TCP 127.0.0.1:51093 (LISTEN)\r\n",
"python 4369 runner 14u IPv4 74473 0t0 TCP 127.0.0.1:43921 (LISTEN)\r\n",
"python 4369 runner 23u IPv4 74477 0t0 TCP 127.0.0.1:49399 (LISTEN)\r\n",
"python 4369 runner 28u IPv4 74481 0t0 TCP 127.0.0.1:36477 (LISTEN)\r\n",
"python 4369 runner 36u IPv4 72829 0t0 TCP 127.0.0.1:44503 (LISTEN)\r\n"
]
}
],
"source": [
"!lsof -n -i -P +c0 -sTCP:LISTEN"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Nc5RhrAl8o_F"
},
"source": [
"In case there are still some `java` processes lingering around, kill them with"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {
"execution": {
"iopub.execute_input": "2024-10-13T09:40:06.090071Z",
"iopub.status.busy": "2024-10-13T09:40:06.089602Z",
"iopub.status.idle": "2024-10-13T09:40:06.222694Z",
"shell.execute_reply": "2024-10-13T09:40:06.221758Z"
},
"id": "p62GHDXh1fiU"
},
"outputs": [],
"source": [
"!pkill -f java"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "F9H_aFvWARei"
},
"source": [
"Verify that the Java processes are gone."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Tn6kOUmvA0FS"
},
"source": [
"# Start the MiniCluster with your own ports\n",
"\n",
"This time we will also set the ports for various services:\n",
"\n",
" mapred minicluster -format -jhsport 8900 -nnhttpport 8901 -nnport 8902 -rmport 8903\n",
"\n",
"Ports:\n",
"\n",
"port number | description\n",
"--- | ---\n",
"8900 | JobHistoryServer port\n",
"8901 | NameNode HTTP port\n",
"8902 | NameNode port\n",
"8903 | ResourceManager port\n",
"\n",
"\n",
"**Note:** we chose these port numbers (8900, 8901, 8902, 8903) arbitrarily, you can pick other numbers as long as they do not conflict with ports that are already is use."
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"execution": {
"iopub.execute_input": "2024-10-13T09:40:06.225483Z",
"iopub.status.busy": "2024-10-13T09:40:06.225205Z",
"iopub.status.idle": "2024-10-13T09:40:06.235456Z",
"shell.execute_reply": "2024-10-13T09:40:06.234648Z"
},
"id": "fvmYd5HhvTjX"
},
"outputs": [],
"source": [
"import subprocess\n",
"with open('out.txt', \"w\") as stdout_file, open('err.txt', \"w\") as stderr_file:\n",
" process = subprocess.Popen(\n",
" [\"mapred\", \"minicluster\", \"-format\", \"-jhsport\", \"8900\", \"-nnhttpport\", \"8901\", \"-nnport\", \"8902\", \"-rmport\", \"8903\"],\n",
" stdout=stdout_file,\n",
" stderr=stderr_file\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:06.238267Z",
"iopub.status.busy": "2024-10-13T09:40:06.237729Z",
"iopub.status.idle": "2024-10-13T09:40:36.260918Z",
"shell.execute_reply": "2024-10-13T09:40:36.260240Z"
},
"id": "UzhdeackOYAa",
"outputId": "f147bd51-4fcd-42fd-8070-dd0fdf605c9d"
},
"outputs": [],
"source": [
"if not IN_COLAB:\n",
" time.sleep(30)\n",
"else:\n",
" time.sleep(10)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kmW_0xbWPb8k"
},
"source": [
"List the Java ports. These should be all ports associated with the MIniCluster `java` process.\n",
"\n",
"Note: `grep \"^COMMAND\\|java\"` means \"filter out the lines that begin with the string `COMMAND` or that contain the string `java`\". This is to preserve the header line."
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:36.263298Z",
"iopub.status.busy": "2024-10-13T09:40:36.263085Z",
"iopub.status.idle": "2024-10-13T09:40:36.445427Z",
"shell.execute_reply": "2024-10-13T09:40:36.444668Z"
},
"id": "5w_SHSeSvTjX",
"outputId": "4f7d404c-2269-4dd3-acdc-35ba4775ab25"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\r\n",
"java 5129 runner 341u IPv4 84793 0t0 TCP 127.0.0.1:8901 (LISTEN)\r\n",
"java 5129 runner 354u IPv4 79164 0t0 TCP 127.0.0.1:8902 (LISTEN)\r\n",
"java 5129 runner 364u IPv4 87194 0t0 TCP 127.0.0.1:38697 (LISTEN)\r\n",
"java 5129 runner 367u IPv4 87197 0t0 TCP 127.0.0.1:41527 (LISTEN)\r\n",
"java 5129 runner 420u IPv4 87216 0t0 TCP 127.0.0.1:38035 (LISTEN)\r\n",
"java 5129 runner 421u IPv4 87220 0t0 TCP 127.0.0.1:36527 (LISTEN)\r\n",
"java 5129 runner 444u IPv4 93129 0t0 TCP *:8031 (LISTEN)\r\n",
"java 5129 runner 473u IPv4 88768 0t0 TCP *:10033 (LISTEN)\r\n",
"java 5129 runner 483u IPv4 89164 0t0 TCP *:19888 (LISTEN)\r\n",
"java 5129 runner 485u IPv4 94167 0t0 TCP *:8088 (LISTEN)\r\n",
"java 5129 runner 499u IPv4 88773 0t0 TCP 127.0.0.1:8900 (LISTEN)\r\n",
"java 5129 runner 509u IPv4 94174 0t0 TCP *:8033 (LISTEN)\r\n",
"java 5129 runner 529u IPv4 93134 0t0 TCP *:8030 (LISTEN)\r\n",
"java 5129 runner 539u IPv4 89169 0t0 TCP 127.0.0.1:8903 (LISTEN)\r\n",
"java 5129 runner 549u IPv4 93135 0t0 TCP 127.0.0.1:44085 (LISTEN)\r\n",
"java 5129 runner 559u IPv4 88778 0t0 TCP 127.0.0.1:38203 (LISTEN)\r\n",
"java 5129 runner 569u IPv4 89174 0t0 TCP *:45391 (LISTEN)\r\n",
"java 5129 runner 570u IPv4 93138 0t0 TCP 127.0.0.1:42883 (LISTEN)\r\n"
]
}
],
"source": [
"!lsof -n -i -P +c0 -sTCP:LISTEN | grep \"^COMMAND\\|java\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "oIt1BO5EPRIE"
},
"source": [
"You should now see\n",
"\n",
"```\n",
"COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\n",
"java 30246 root 347u IPv4 671659 0t0 TCP 127.0.0.1:8901 (LISTEN)\n",
"java 30246 root 357u IPv4 672029 0t0 TCP 127.0.0.1:8902 (LISTEN)\n",
"java 30246 root 367u IPv4 674008 0t0 TCP 127.0.0.1:44721 (LISTEN)\n",
"java 30246 root 370u IPv4 672082 0t0 TCP 127.0.0.1:36789 (LISTEN)\n",
"java 30246 root 399u IPv4 674082 0t0 TCP 127.0.0.1:37975 (LISTEN)\n",
"java 30246 root 400u IPv4 674085 0t0 TCP 127.0.0.1:37675 (LISTEN)\n",
"java 30246 root 423u IPv4 681319 0t0 TCP *:8031 (LISTEN)\n",
"java 30246 root 440u IPv4 680955 0t0 TCP *:10033 (LISTEN)\n",
"java 30246 root 450u IPv4 681014 0t0 TCP *:19888 (LISTEN)\n",
"java 30246 root 455u IPv4 681987 0t0 TCP 127.0.0.1:8900 (LISTEN)\n",
"java 30246 root 465u IPv4 682035 0t0 TCP *:8088 (LISTEN)\n",
"java 30246 root 470u IPv4 681313 0t0 TCP *:8033 (LISTEN)\n",
"java 30246 root 490u IPv4 681325 0t0 TCP *:8030 (LISTEN)\n",
"java 30246 root 500u IPv4 682046 0t0 TCP 127.0.0.1:8903 (LISTEN)\n",
"java 30246 root 510u IPv4 681370 0t0 TCP 127.0.0.1:34521 (LISTEN)\n",
"java 30246 root 520u IPv4 682057 0t0 TCP 127.0.0.1:36981 (LISTEN)\n",
"java 30246 root 530u IPv4 681373 0t0 TCP *:46657 (LISTEN)\n",
"java 30246 root 531u IPv4 682061 0t0 TCP 127.0.0.1:39897 (LISTEN)\n",
"```\n",
"\n",
"Our ports $8900$, $8901$, $8902$, and $8903$ are included in the list."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "W19UpXR5NXbu"
},
"source": [
"The log messages are in the file `err.txt`. The last 10 lines should look like this:\n",
"\n",
"```\n",
"2023-12-27 21:36:08,820 INFO server.MiniYARNCluster: All Node Managers connected in MiniYARNCluster\n",
"2023-12-27 21:36:08,820 INFO v2.MiniMRYarnCluster: MiniMRYARN ResourceManager address: localhost:8903\n",
"2023-12-27 21:36:08,821 INFO v2.MiniMRYarnCluster: MiniMRYARN ResourceManager web address: 0.0.0.0:8088\n",
"2023-12-27 21:36:08,821 INFO v2.MiniMRYarnCluster: MiniMRYARN HistoryServer address: localhost:8900\n",
"2023-12-27 21:36:08,822 INFO v2.MiniMRYarnCluster: MiniMRYARN HistoryServer web address: 26769af38ddc:19888\n",
"2023-12-27 21:36:08,823 INFO mapreduce.MiniHadoopClusterManager: Started MiniMRCluster\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:36.447975Z",
"iopub.status.busy": "2024-10-13T09:40:36.447753Z",
"iopub.status.idle": "2024-10-13T09:40:36.575666Z",
"shell.execute_reply": "2024-10-13T09:40:36.574839Z"
},
"id": "B6K9s-A9NdBa",
"outputId": "3f270543-87f5-4468-b61d-afbf06715e63"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2024-10-13 09:40:12,669 INFO nodemanager.NodeStatusUpdaterImpl: Registered with ResourceManager as localhost:44085 with total resource of \r\n",
"2024-10-13 09:40:12,669 INFO capacity.AbstractLeafQueue: LeafQueue: root.default update max app related, maxApplications=10000, maxApplicationsPerUser=10000, Abs Cap:1.0, Cap: 1.0, MaxCap : 1.0\r\n",
"2024-10-13 09:40:12,670 INFO capacity.CapacityScheduler: Added node localhost:44085 clusterResource: \r\n",
"2024-10-13 09:40:12,671 INFO capacity.AbstractLeafQueue: LeafQueue: root.default update max app related, maxApplications=10000, maxApplicationsPerUser=10000, Abs Cap:1.0, Cap: 1.0, MaxCap : 1.0\r\n",
"2024-10-13 09:40:12,677 INFO server.MiniYARNCluster: All Node Managers connected in MiniYARNCluster\r\n",
"2024-10-13 09:40:12,677 INFO v2.MiniMRYarnCluster: MiniMRYARN ResourceManager address: localhost:8903\r\n",
"2024-10-13 09:40:12,678 INFO v2.MiniMRYarnCluster: MiniMRYARN ResourceManager web address: 0.0.0.0:8088\r\n",
"2024-10-13 09:40:12,678 INFO v2.MiniMRYarnCluster: MiniMRYARN HistoryServer address: localhost:8900\r\n",
"2024-10-13 09:40:12,678 INFO v2.MiniMRYarnCluster: MiniMRYARN HistoryServer web address: fv-az837-567:19888\r\n",
"2024-10-13 09:40:12,678 INFO mapreduce.MiniHadoopClusterManager: Started MiniMRCluster\r\n"
]
}
],
"source": [
"!tail err.txt"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "OdzAzlPmcB0h"
},
"source": [
"Check the NameNode's HTTP port."
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:36.578382Z",
"iopub.status.busy": "2024-10-13T09:40:36.578149Z",
"iopub.status.idle": "2024-10-13T09:40:36.764812Z",
"shell.execute_reply": "2024-10-13T09:40:36.764119Z"
},
"id": "M8S8O_4WvTjY",
"outputId": "025b4bc8-6dc4-47a4-cb88-f5814900e236"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"--2024-10-13 09:40:36-- http://localhost:8901/\r\n",
"Resolving localhost (localhost)... ::1, 127.0.0.1\r\n",
"Connecting to localhost (localhost)|::1|:8901... failed: Connection refused.\r\n",
"Connecting to localhost (localhost)|127.0.0.1|:8901... connected.\r\n",
"HTTP request sent, awaiting response... 302 Found\r\n",
"Location: http://localhost:8901/index.html [following]\r\n",
"--2024-10-13 09:40:36-- http://localhost:8901/index.html\r\n",
"Reusing existing connection to localhost:8901.\r\n",
"HTTP request sent, awaiting response... "
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"200 OK\r\n",
"Length: 1079 (1.1K) [text/html]\r\n",
"Saving to: ‘index.html.2’\r\n",
"\r\n",
"\r",
"index.html.2 0%[ ] 0 --.-KB/s \r",
"index.html.2 100%[===================>] 1.05K --.-KB/s in 0s \r\n",
"\r\n",
"2024-10-13 09:40:36 (131 MB/s) - ‘index.html.2’ saved [1079/1079]\r\n",
"\r\n"
]
}
],
"source": [
"!wget http://localhost:8901"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aKA3MLV5cIEX"
},
"source": [
"## Start the NameNode Web UI\n",
"\n",
"Serve the NameNode UI in the browser through Google Colab (path should be set to `/index.html` as in the output of `wget` otherwise the URL won't work)."
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:36.767687Z",
"iopub.status.busy": "2024-10-13T09:40:36.767238Z",
"iopub.status.idle": "2024-10-13T09:40:36.771757Z",
"shell.execute_reply": "2024-10-13T09:40:36.771222Z"
},
"id": "k-LdeVwBvTjY",
"outputId": "8f1a1bff-a2eb-45d8-92eb-fcc08da112be"
},
"outputs": [],
"source": [
"if IN_COLAB and not NGROK:\n",
" # serve the Web UI on Colab\n",
" print(\"Click on the link below to open the NameNode Web UI 🚀\")\n",
" output.serve_kernel_port_as_window(8901, path='/index.html')\n",
"else:\n",
" if NGROK:\n",
" # disconnect previous tunnels (note: you can have max 3 tunnels open!)\n",
" # see: https://pyngrok.readthedocs.io/en/latest/index.html#get-active-tunnels\n",
" tunnels = ngrok.get_tunnels()\n",
" for t in tunnels:\n",
" ngrok.disconnect(t.public_url)\n",
" # Open a ngrok tunnel to the HTTP server on port 8901\n",
" public_url = ngrok.connect(8901).public_url\n",
" print(f'Click on {public_url} to open the NameNode Web UI 🚀')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zya4FAgnTGHz"
},
"source": [
"By clicking on the above link you should see the NameNode's Web UI in your browser:\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GXZifg7YhvBa"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PWLoLFJabAev"
},
"source": [
"# Test the HDFS filesystem"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9jA_tBoV647p"
},
"source": [
"## Create a directory\n",
"\n",
"**Note:** in order to use the MiniCluster's Hadoop filesystem you need to specify the full path prepending `hdfs://localhost:8902/` otherwise `hdfs` will write to the local filesystem."
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {
"execution": {
"iopub.execute_input": "2024-10-13T09:40:36.773911Z",
"iopub.status.busy": "2024-10-13T09:40:36.773710Z",
"iopub.status.idle": "2024-10-13T09:40:38.351604Z",
"shell.execute_reply": "2024-10-13T09:40:38.350759Z"
},
"id": "wCYTzNvlfzRO"
},
"outputs": [],
"source": [
"%%bash\n",
"# create a folder my_dir\n",
"hdfs dfs -mkdir hdfs://localhost:8902/my_dir"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "NiIBxgBi8KQh"
},
"source": [
"List the contents of `my_dir` (should be empty)."
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {
"execution": {
"iopub.execute_input": "2024-10-13T09:40:38.354213Z",
"iopub.status.busy": "2024-10-13T09:40:38.353900Z",
"iopub.status.idle": "2024-10-13T09:40:40.043000Z",
"shell.execute_reply": "2024-10-13T09:40:40.042109Z"
},
"id": "yATUZZ5hbEHh"
},
"outputs": [],
"source": [
"!hdfs dfs -ls hdfs://localhost:8902/my_dir"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5vKshDCX8Sc2"
},
"source": [
"Upload the local folder `sample_data` to `my_dir` on HDFS"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:40.046410Z",
"iopub.status.busy": "2024-10-13T09:40:40.045889Z",
"iopub.status.idle": "2024-10-13T09:40:40.173916Z",
"shell.execute_reply": "2024-10-13T09:40:40.173126Z"
},
"id": "PiVF1RFzeLXd",
"outputId": "77ff6b8e-af93-41a4-9b57-846d5b4a693c"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ls: cannot access 'sample_data': No such file or directory\r\n"
]
}
],
"source": [
"!ls -lh sample_data"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ayePcxZ7TuXm"
},
"source": [
"Check the total size of the local folder `sample_data` using the command `du` (\"du\" stands for \"disk usage\" and the `-h` option stands for \"human\" as it will format file sizes in a “human-readable” fashion, e.g 55M instead of 55508) ."
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:40.176385Z",
"iopub.status.busy": "2024-10-13T09:40:40.176166Z",
"iopub.status.idle": "2024-10-13T09:40:40.302641Z",
"shell.execute_reply": "2024-10-13T09:40:40.302043Z"
},
"id": "Kp119M2bTrzy",
"outputId": "b16cff16-1d95-468a-f93f-1ac802305fe5"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"du: cannot access 'sample_data': No such file or directory\r\n"
]
}
],
"source": [
"!du -h sample_data"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4zer_jX7T2ZD"
},
"source": [
"Upload `sample_data` to HDFS."
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {
"execution": {
"iopub.execute_input": "2024-10-13T09:40:40.304979Z",
"iopub.status.busy": "2024-10-13T09:40:40.304558Z",
"iopub.status.idle": "2024-10-13T09:40:42.054494Z",
"shell.execute_reply": "2024-10-13T09:40:42.053688Z"
},
"id": "WfV62y1yeu6O"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"put: `sample_data': No such file or directory\r\n"
]
}
],
"source": [
"!hdfs dfs -put sample_data hdfs://localhost:8902/my_dir/"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gP1gYl7t8Sh9"
},
"source": [
"Check"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:42.057086Z",
"iopub.status.busy": "2024-10-13T09:40:42.056863Z",
"iopub.status.idle": "2024-10-13T09:40:43.821429Z",
"shell.execute_reply": "2024-10-13T09:40:43.820546Z"
},
"id": "mL8pfXzVUBS_",
"outputId": "83f6dd1d-c0dd-48b2-9eb7-86a1e96e7d4e"
},
"outputs": [],
"source": [
"!hdfs dfs -ls hdfs://localhost:8902/my_dir"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7h7gL8tOlS17"
},
"source": [
"Check the size of `my_dir` on HDFS using the [HDFS equivalent of `du`](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/FileSystemShell.html#du)."
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:43.824080Z",
"iopub.status.busy": "2024-10-13T09:40:43.823853Z",
"iopub.status.idle": "2024-10-13T09:40:45.570715Z",
"shell.execute_reply": "2024-10-13T09:40:45.569825Z"
},
"id": "1zCBrlOEUKpU",
"outputId": "813e3030-27bc-4deb-b058-8e288fabbce6"
},
"outputs": [],
"source": [
"!hdfs dfs -du -h hdfs://localhost:8902/my_dir"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "IQDzG-TOEB9a"
},
"source": [
"Check the contents of the HDFS folder `my_dir`"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:45.573335Z",
"iopub.status.busy": "2024-10-13T09:40:45.573109Z",
"iopub.status.idle": "2024-10-13T09:40:47.289581Z",
"shell.execute_reply": "2024-10-13T09:40:47.288771Z"
},
"id": "U1XOv7NZhY9F",
"outputId": "91925578-7e05-4296-a538-2f2842d1bb7a"
},
"outputs": [],
"source": [
"!hdfs dfs -ls -R -h hdfs://localhost:8902/my_dir"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tuI4QmYnCyvp"
},
"source": [
"You should now see in the Web interface that the \"DFS used\" has increased (you might need to refresh the NameNode UI Web page):"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6LX998zzDDaa"
},
"source": [
"![dfs_used.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAACQ4AAAEECAYAAABj6TuGAAAMa2lDQ1BJQ0MgUHJvZmlsZQAASImVVwdYU8kWnluSkJDQAqFICb0JIr1ICaFFEJAONkISSCgxJgQVOyIquHYRxYquiii6FkAWFbGXRbH3xYLKyrpYUBSVNyEBXfeV753vmzv/PXPmP+XO3HsHAM1erkSSi2oBkCfOl8aFBzNTUtOYpOcAAVRABkygzeXJJKzY2CgAZbD/u7y/Ca2hXHNScP1z/L+KDl8g4wGAjIc4gy/j5UHcDAC+gSeR5gNAVOgtp+ZLFHguxLpSGCDEqxU4S4l3KXCGEjcN2CTEsSG+AoAalcuVZgGgcR/qmQW8LMij8RliFzFfJAZAczjEATwhlw+xIvbheXmTFbgCYjtoL4EYxgO8M77jzPobf8YQP5ebNYSVeQ2IWohIJsnlTv8/S/O/JS9XPujDBjaqUBoRp8gf1vB2zuRIBaZC3CXOiI5R1BriXhFfWXcAUIpQHpGotEeNeTI2rB9gQOzC54ZEQmwMcZg4NzpKpc/IFIVxIIarBZ0myuckQGwA8SKBLDReZbNFOjlO5Quty5SyWSr9Oa50wK/C10N5TiJLxf9GKOCo+DGNQmFCMsQUiK0KREnREGtA7CzLiY9U2YwqFLKjB22k8jhF/FYQxwnE4cFKfqwgUxoWp7IvzZMN5ottEYo40Sp8IF+YEKGsD3aKxx2IH+aCXRGIWYmDPAJZStRgLnxBSKgyd+yFQJwYr+LpleQHxynn4hRJbqzKHrcQ5IYr9BYQu8sK4lVz8aR8uDiV/HimJD82QRknXpjNHR2rjAdfDqIAG4TA3SeHLQNMBtlA1NpV3wXvlCNhgAukIAsIgJNKMzgjeWBEDK/xoBD8CZEAyIbmBQ+MCkAB1H8Z0iqvTiBzYLRgYEYOeAZxHogEufBePjBLPOQtCTyFGtE/vHNh48F4c2FTjP97/aD2m4YFNVEqjXzQI1Nz0JIYSgwhRhDDiPa4ER6A++FR8BoEmyvujfsM5vHNnvCM0EZ4TLhBaCfcmSQqkv4Q5RjQDvnDVLXI+L4WuA3k9MCDcX/IDplxBm4EnHB36IeFB0LPHlDLVsWtqArzB+6/ZfDd01DZkV3IKFmfHES2+3GmhoOGxxCLotbf10cZa8ZQvdlDIz/6Z39XfT7sI3+0xBZhB7Gz2AnsPNaE1QMmdhxrwC5hRxV4aHU9HVhdg97iBuLJgTyif/jjqnwqKilzqXHpdPmsHMsXTMtXbDz2ZMl0qShLmM9kwa+DgMkR85yHM11dXN0AUHxrlK+vt4yBbwjCuPBNV/QOAH9+f39/0zddFNzrhxbA7f/sm872GHxN6ANwrownlxYodbjiQoBvCU240wyBKbAEdjAfV+AJ/EAQCAWjQQxIAKlgIqyyEK5zKZgKZoJ5oASUgeVgDVgPNoNtYBfYCw6AetAEToAz4CK4Am6Ae3D1dICXoBu8B30IgpAQGkJHDBEzxBpxRFwRbyQACUWikDgkFUlHshAxIkdmIvORMmQlsh7ZilQjvyBHkBPIeaQNuYM8QjqRN8gnFEOpqC5qgtqgI1BvlIVGognoBDQLnYIWosXoUrQCrUL3oHXoCfQiegNtR1+iPRjA1DEGZo45Yd4YG4vB0rBMTIrNxkqxcqwKq8Ua4XO+hrVjXdhHnIjTcSbuBFdwBJ6I8/Ap+Gx8Cb4e34XX4afwa/gjvBv/SqARjAmOBF8Ch5BCyCJMJZQQygk7CIcJp+Fe6iC8JxKJDKIt0QvuxVRiNnEGcQlxI3EfsZnYRnxC7CGRSIYkR5I/KYbEJeWTSkjrSHtIx0lXSR2kXjV1NTM1V7UwtTQ1sVqRWrnabrVjalfVnqv1kbXI1mRfcgyZT55OXkbeTm4kXyZ3kPso2hRbij8lgZJNmUepoNRSTlPuU96qq6tbqPuoj1UXqc9Vr1Dfr35O/ZH6R6oO1YHKpo6nyqlLqTupzdQ71Lc0Gs2GFkRLo+XTltKqaSdpD2m9GnQNZw2OBl9jjkalRp3GVY1XmmRNa02W5kTNQs1yzYOalzW7tMhaNlpsLa7WbK1KrSNat7R6tOnaI7VjtPO0l2jv1j6v/UKHpGOjE6rD1ynW2aZzUucJHaNb0tl0Hn0+fTv9NL1Dl6hrq8vRzdYt092r26rbraej566XpDdNr1LvqF47A2PYMDiMXMYyxgHGTcYnfRN9lr5Af7F+rf5V/Q8GwwyCDAQGpQb7DG4YfDJkGoYa5hiuMKw3fGCEGzkYjTWaarTJ6LRR1zDdYX7DeMNKhx0YdtcYNXYwjjOeYbzN+JJxj4mpSbiJxGSdyUmTLlOGaZBptulq02OmnWZ0swAzkdlqs+NmfzD1mCxmLrOCeYrZbW5sHmEuN99q3mreZ2FrkWhRZLHP4oElxdLbMtNytWWLZbeVmdUYq5lWNVZ3rcnW3tZC67XWZ60/2NjaJNsstKm3eWFrYMuxLbStsb1vR7MLtJtiV2V33Z5o722fY7/R/ooD6uDhIHSodLjsiDp6OoocNzq2DScM9xkuHl41/JYT1YnlVOBU4/TImeEc5VzkXO/8aoTViLQRK0acHfHVxcMl12W7y72ROiNHjywa2TjyjauDK8+10vW6G80tzG2OW4Pba3dHd4H7JvfbHnSPMR4LPVo8vnh6eUo9az07vay80r02eN3y1vWO9V7ifc6H4BPsM8enyeejr6dvvu8B37/8nPxy/Hb7vRhlO0owavuoJ/4W/lz/rf7tAcyA9IAtAe2B5oHcwKrAx0GWQfygHUHPWfasbNYe1qtgl2Bp8OHgD2xf9ix2cwgWEh5SGtIaqhOaGLo+9GGYRVhWWE1Yd7hH+Izw5ghCRGTEiohbHBMOj1PN6R7tNXrW6FOR1Mj4yPWRj6McoqRRjWPQMaPHrBpzP9o6WhxdHwNiODGrYh7E2sZOif11LHFs7NjKsc/iRsbNjDsbT4+fFL87/n1CcMKyhHuJdonyxJYkzaTxSdVJH5JDklcmt6eMSJmVcjHVKFWU2pBGSktK25HWMy503JpxHeM9xpeMvznBdsK0CecnGk3MnXh0kuYk7qSD6YT05PTd6Z+5Mdwqbk8GJ2NDRjePzVvLe8kP4q/mdwr8BSsFzzP9M1dmvsjyz1qV1SkMFJYLu0Rs0XrR6+yI7M3ZH3Jicnbm9Ocm5+7LU8tLzzsi1hHniE9NNp08bXKbxFFSImmf4jtlzZRuaaR0hwyRTZA15OvCn/pLcjv5AvmjgoCCyoLeqUlTD07Tniaedmm6w/TF058XhhX+PAOfwZvRMtN85ryZj2axZm2djczOmN0yx3JO8ZyOueFzd82jzMuZ91uRS9HKonfzk+c3FpsUzy1+siB8QU2JRom05NZCv4WbF+GLRItaF7stXrf4aym/9EKZS1l52eclvCUXfhr5U8VP/Uszl7Yu81y2aTlxuXj5zRWBK3at1F5ZuPLJqjGr6lYzV5eufrdm0prz5e7lm9dS1srXtldEVTSss1q3fN3n9cL1NyqDK/dtMN6weMOHjfyNVzcFbardbLK5bPOnLaItt7eGb62rsqkq30bcVrDt2fak7Wd/9v65eofRjrIdX3aKd7bvitt1qtqrunq38e5lNWiNvKZzz/g9V/aG7G2odarduo+xr2w/2C/f/8cv6b/cPBB5oOWg98HaQ9aHNhymHy6tQ+qm13XXC+vbG1Ib2o6MPtLS6Nd4+FfnX3c2mTdVHtU7uuwY5Vjxsf7jhcd7miXNXSeyTjxpmdRy72TKyeunxp5qPR15+tyZsDMnz7LOHj/nf67pvO/5Ixe8L9Rf9LxYd8nj0uHfPH473OrZWnfZ63LDFZ8rjW2j2o5dDbx64lrItTPXOdcv3oi+0XYz8ebtW+Nvtd/m335xJ/fO67sFd/vuzb1PuF/6QOtB+UPjh1W/2/++r92z/eijkEeXHsc/vveE9+TlU9nTzx3Fz2jPyp+bPa9+4fqiqTOs88of4/7oeCl52ddV8qf2nxte2b069FfQX5e6U7o7Xktf979Z8tbw7c537u9aemJ7Hr7Pe9/3obTXsHfXR++PZz8lf3reN/Uz6XPFF/svjV8jv97vz+vvl3Cl3IFfAQw2NDMTgDc7AaClAkCH5zbKOOVZcEAQ5fl1AIH/hJXnxQHxBKAWdorfeHYzAPths5kLuYMAUPzCJwQB1M1tqKlElunmquSiwpMQobe//60JAKRGAL5I+/v7Nvb3f9kOg70DQPMU5RlUIUR4ZtgSoEA3DFK2gh9EeT79Lscfe6CIwB382P8LRN2PY8nL7lUAAACKZVhJZk1NACoAAAAIAAQBGgAFAAAAAQAAAD4BGwAFAAAAAQAAAEYBKAADAAAAAQACAACHaQAEAAAAAQAAAE4AAAAAAAAAkAAAAAEAAACQAAAAAQADkoYABwAAABIAAAB4oAIABAAAAAEAAAkOoAMABAAAAAEAAAEEAAAAAEFTQ0lJAAAAU2NyZWVuc2hvdL9VossAAAAJcEhZcwAAFiUAABYlAUlSJPAAAAHXaVRYdFhNTDpjb20uYWRvYmUueG1wAAAAAAA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4OnhtcHRrPSJYTVAgQ29yZSA2LjAuMCI+CiAgIDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+CiAgICAgIDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiCiAgICAgICAgICAgIHhtbG5zOmV4aWY9Imh0dHA6Ly9ucy5hZG9iZS5jb20vZXhpZi8xLjAvIj4KICAgICAgICAgPGV4aWY6UGl4ZWxZRGltZW5zaW9uPjI2MDwvZXhpZjpQaXhlbFlEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlBpeGVsWERpbWVuc2lvbj4yMzE4PC9leGlmOlBpeGVsWERpbWVuc2lvbj4KICAgICAgICAgPGV4aWY6VXNlckNvbW1lbnQ+U2NyZWVuc2hvdDwvZXhpZjpVc2VyQ29tbWVudD4KICAgICAgPC9yZGY6RGVzY3JpcHRpb24+CiAgIDwvcmRmOlJERj4KPC94OnhtcG1ldGE+CidKQukAAAAcaURPVAAAAAIAAAAAAAAAggAAACgAAACCAAAAggAAUN9677UQAABAAElEQVR4AezdCbxV8/7/8W8DSZEmIa66KEMUwi+Uoswumefpyg0hIVKhmzFDSDIUERmvebqEa/jF5afENaQBl4Qk6QoJ/97f67P+373O2tM5e5+zz9mv7+Nx715r7bXWXuu51unxWF/v9fnW+21FczQEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBAoK4F6BIfK6npzsggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIeAGCQ9wICCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAgiUoQDBoTK86JwyAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIEh7gHEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBAoQwGCQ2V40TllBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQIDnEPIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCBQhgIEh8rwonPKCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggQHOIeQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEECgDAUIDpXhReeUEUAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBAgOMQ9gAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIBAGQoQHCrDi84pI4AAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCBAcIh7AAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQACBMhQgOFSGF51TRgABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEECA4BD3AAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACZShAcKgMLzqnjAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIBAQYNDc+fORRQBBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQSKJPDHP/6xYHsmOFQwSnaEAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggEBxBUo+OLTuuusWV4C9I4AAAggggAACCCCAAAIIIIAAAggggAACaQQ+++wz/w39lGmAWIwAAggggAACCCCAAAIIIFArBex5l+BQrbx8HDQCCCCAAAIIIIAAAggggAACCCCAAAIIVIeAdaQSHKoObX4DAQQQQAABBBBAAAEEEECgugTseZfgUHWJ8zsIIIAAAggggAACCCCAAAIIIIAAAgggUOsErCOV4FCtu3QcMAIIIIAAAggggAACCCCAQAYBe94lOJQBia8QQAABBBBAAAEEEEAAAQQQQAABBBBAoLwFrCOV4FB53wecPQIIIIAAAggggAACCCBQ1wTseZfgUF27spwPAggggAACCCCAAAIIIIAAAggggAACCBRMwDpSCQ4VjJQdIYAAAggggAACCCCAAAIIlICAPe8SHCqBi8EhIIAAAggggAACCCCAAAIIIIAAAggggEBpClhHKsGh0rw+HBUCCCCAAAIIIIAAAggggEDlBOx5l+BQ5fzYCgEEEEAAAQQQQAABBBBAAAEEEEAAAQTKQMA6UgkOlcHF5hQRQAABBBBAAAEEEEAAgTISsOddgkNldNE5VQQQQAABBBBAAAEEEEAAAQQQQAABBBDIT8A6UgkO5efG2ggggAACCCCAAAIIIIAAAqUtYM+7BIdK+zpxdAgggAACCCCAAAIIIIAAAggggAACCCBQgwLWkUpwqAYvAj+NAAIIIIAAAggggAACCCBQcAF73iU4VHBadogAAggggAACCCCAAAIIIIAAAggggAACdUXAOlIJDtWVK8p5IIAAAggggAACCCCAAAIISMCedwkOcT8ggAACCCCAAAIIIIAAAggggAACCCCAAAJpBKwjleBQGiAWI4AAAggggAACCCCAAAII1EoBe94lOFQrLx8HjQACCCCAAAIIIIAAAggggAACCCCAAALVIWAdqQSHqkOb30AAAQQQQAABBBBAAAEEEKguAXveJThUXeL8DgIIIIAAAggggAACCCCAAAIIIIAAAgjUOgHrSCU4VOsuHQeMAAIIIIAAAggggAACCCCQQcCedwkOZUDiKwQQQAABBBBAAAEEEEAAAQQQQAABBBAobwHrSCU4VN73AWePAAIIIIAAAggggAACCNQ1AXveJThU164s54MAAggggAACCCCAAAIIIIAAAggggAACBROwjlSCQwUjZUcIIIAAAggggAACCCCAAAIlIGDPuwSHSuBicAgIIIAAAggggAACCCCAAAIIIIAAAgggUJoC1pFKcKg0rw9HhQACCCCAAAIIIIAAAgggUDkBe94lOFQ5P7ZCAAEEEEAAAQQQQAABBBBAAAEEEEAAgTIQsI5UgkNlcLE5RQQQQAABBBBAAAEEEECgjATseZfgUBlddE4VAQQQQAABBBBAAAEEEEAAAQQQQAABBPITsI5UgkP5ubE2AggggAACCCCAAAIIIIBAaQvY8y7BodK+ThwdAggggAACCCCAAAIIIIAAAggggAACCNSggHWkEhyqwYvATyOAAAIIIIAAAggggAACCBRcwJ53CQ4VnJYdIoAAAggggAACCCCAAAIIIIAAAggggEBdEbCOVIJDdeWKch4IIIAAAggggAACCCCAAAISsOddgkPcDwgggAACCCCAAAIIIIAAAggggAACCCCAQBoB60glOJQGiMUIIIAAAggggAACCCCAAAK1UsCedwkO1crLx0EjgAACCCCAAAIIIIAAAggggAACCCCAQHUIWEcqwaHq0OY3EEAAAQQQQAABBBBAAAEEqkvAnncJDlWXOL+DAAIIIIAAAggggAACCCCAAAIIIIAAArVOwDpSCQ7VukvHASOAAAIIIIAAAggggAACCGQQsOddgkMZkPgKAQQQQAABBBBAAAEEEEAAAQQQQAABBMpbwDpSCQ6V933A2SOAAAIIIIAAAggggAACdU3AnncJDtW1K8v5IIAAAggggAACCCCAAAIIIIAAAggggEDBBKwjleBQwUjZEQIIIIAAAggggAACCCCAQAkI2PNu2QaHvv76a/fWW2+5OXPmuFmzZrn58+e75s2buxYtWrgtttjCdevWzbVt27YELlVuh/DTTz+5F154wc2bN8998803TvPdu3d3O+20k5s9e7Z74oknoh2pk+OAAw6I5plIFpDb22+/HX3ZqlUr16NHj2g+34m6ds/le/5VWf+XX35xY8eOdb/99lu0m0MPPdS1adMmmmcCAQQQQAABBBBAAAEEEEAAAQQQKIaAdaQSHCqGbs3tU/3C7733nu8bfv/9992CBQt8//Caa67pNt10U9e7d++s/cOvvPKKW7x4ccaTWH311V27du3cOuus4xo0aJBxXb5EAAEEEEAAAQQQQAABBKpTwJ53yy44pODB448/7i699NKs3rvvvrsbMGCADxNlXbkGV/j888/dySef7L766quUozjqqKPcSSed5P7xj3+48847L/quc+fObty4cdE8E8kCd999txszZkz0pQJlN954YzSf60RdvOdyPfdCracgXK9evVJ2N378eN+JYws/+ugjt3TpUpt1f/jDH9xqq60WzTOBAAIIIIAAAggggAACCCCAAAIIVEbAOlILHRx655133IwZM3x4RZ9qG220kWvfvr3ba6+93IYbbpjz4aof65lnnnEbbLCBGzZsWMbtJk+e7L799tuM6+jLDh06+PBM1hVzWOHVV191Z555pl/zggsucLvttlvard544w337rvvun/9619OgZ411ljD9/N07NjR9e3b1zVr1izttrl88fPPP7tbbrnF3XnnnVlX33XXXd2QIUNco0aNEtc94ogjnPqkcm1av3///gSIcgVjPQQQQAABBBBAAAEEECiqgD3vllVw6D//+Y8bPny4++c//5kzrqoQKaCw9tpr57xNda944YUX+o6B+O8SHIqL5DdfiOBQXb3n8pOs+tq5BIf69evnO5Xs10aMGOH69Oljs3wigAACCCCAAAIIIIAAAggggAAClRKwjtRCBocefPBBd+WVV2Y8Hr0QqKBJ/fr1M67366+/uv3339+/VKhwzW233ZZx/e233z7j9/blLrvs4kaOHGmzlf5cuHChO+SQQ6IXvvSC49577524v5tvvtlNnDgx8TtbmC14ZOslff773/922n7mzJkVvtZLaPo+3vQS5qhRoxJfUMs3OKR9b7XVVu7yyy93TZo0if8U8wgggAACCCCAAAIIIIBAtQrY825ZBYdGjx7t7r///ryhVUZWb6CsssoqeW9bHRskPewr8KTgkIZzouJQ5a5CIYJDdfWeq5xo5bciOFR5O7ZEAAEEEEAAAQQQQAABBBBAAIGqCVhHaqGCQ/fee6+79tpro4NaddVV3eabb+5WWmklN23atChgoxWGDh3qqw9FKydM3H777e6mm27y32QLDn3zzTdpQzvxXRciOKRQ09lnn+1UcchauuCQzkHnYs1cPvzwQ7do0SJb7D8VjtK55tOWLVvmDj/8cKfq7daOPPJI77veeutFAa1PP/3UPfzww059g9Y22WQTN2HCBJuNPsPg0FVXXeXWWmut6DtN/PDDDz7Q9dRTT7mXX345+u4vf/mLO+aYY6J5JhBAAAEEEEAAAQQQQACBmhCw592yCQ598MEH7vjjj69grXKz+p/GmdbbLyqFqwpD8XbKKaf4N3ziy2t6XmNo77HHHimHEe9Q0Doas9uaxtXOp9SxbVdun1UNDtXVe64m7gMN96ZKYfq0pqHjwjezqDhkMnwigAACCCCAAAIIIIAAAggggEAhBawjtRDBIfU/7rPPPtHhHX300e7EE0+MQiuqXq1KRBp2TE3hmYceeiil2o2GGfvkk0/8EFnPPfece/PNN6P9ZQsOaQgw9aGoqVqz+lfSNf12VYeBj4ek9FtJwSGFeQ488MDoUAYPHuz2228/P6/wkfrZFMzR0GVqlQk1TZo0yY0bN85vr3O75JJL3Lbbbuvnk/5PQZ9zzjkn+ioprBQGh/72t7+lrVqvPi29YPjAAw/4/Wk4urvuuivaNxMIIIAAAggggAACCCCAQE0I2PNuWQSHfvnlF3fCCSdUKEGrh/Jjjz22gr/GF9dbH2HTw+Tf//73xPGn58+f74M5c+fO9Q/sCuZoPHE9AGpc8qRKRXpY1PjXevBVa9CggV9f8wpIaF8//vija9WqlV8ef4jXet99951/Q0bnFjaN0d22bVv/u/pt7efrr7+OVtGY3K1bt47mwwk9pOstno8//thvr/NQ+Vwd7+zZs6NVGzZsmBI+0nY6HmstWrRwa665ps36T40fHgaYtFCdGfXq1fPff/nllylvD+kc1Dmhfb/++us+2NWrVy8Xv2nVWTJr1ix/fHr7SBWitF8d+8orr+z3ne7/dG+oBLHOed68ed5Nb3hpH1UJDhXznlu6dKnvHNJx2x+y7hN1XnXp0iXxHpXRF198ETG0bNnS3wNLlizx95vsdV/oftX49Y0bN47WTZrQ22n6fb2BJTetr2Ow7ZO2CZd9//33/h7T/aDtNVa9hgPUdUs3LKBKSIfBIf196ZjVobZ8+XJ3xhlnpPyNn3XWWW7nnXf294DMwr8BHYvOM6nUtzqiwqY3xXR8NAQQQAABBBBAAAEEEEAAAQQQKE8B638pRHBI4RIFYNQUfvnrX/8a9Y2ZrvryVEX8q6++8ou0frdu3exrpz6PqVOnRvPhRLbg0LPPPuuH6tI2CrGoD6xYTf1tSX2vScEhVXu/4YYb/KGoCtDJJ59c4bDCcJGO20I4FVZMWCBLCyLpa7n37t07Yc3URWPHjo0CPromp512WsoKuQaHtFH8JcN01zDlB5hBAAEEEEAAAQQQQAABBIooYM+78QxGVX6y3or/qP//y4FUZU8rtlVwRq0QD+RJQSAFcW688Ub/G0n/N2zYMPf888+nfKW3QBRWsKZwiErUZhp3Ww+xF110kdt4441tM/+psIMqHYVND7uDBg1KO5a2HqpVNldt+vTpTlWQMrW99trLlzOOD1Wm8brvueeelE01FNSll14avc0UfqlSvP3793enn356tFhBqilTpkTzOjb9jjW9IaRzCZuCR3qLKmwKY9mbSxrb+5FHHom+VuUkBT70No61yy67zPXo0cPPKjylN66ss8XWCT/V0aBKU0nhLQVJdJ3ffvvtcBM/3bdvX99xos4Ba9nuGVtPn8W65/Smk94Gk0tS03U57rjjKlTHCjultN1BBx3kOnXq5MdTj+9L+9D49WGHlP2W7nm9naXx5tM13V+6V5K21za65jqHdE1/FwruhQGiTEOV6V5Tx1G6ttlmm/mOuOuuuy5lFQXs9F3YFIQ65JBDwkXu4osvdgqsqSkkGJbM1jJ5t2nTRpM0BBBAAAEEEEAAAQQQQAABBBCogwLWkVqIfkr1bT366KNe6YILLnC77bZbotjVV18dBWMUolEfl7WqBIdUNUd9ImrqZ9LLjMVoGqJLfSZ68ax58+Z+iDDrZ0sKDoUu8aBUeHzbb799NPu///u/FUJX0ZexCfUv3n///X6p+jplkPRCWWwz/+KbhjdT03k88cQTKavkExxSf7ddx6R9peyYGQQQQAABBBBAAAEEEECgGgTsebcsgkN6oNN//A9bGEAJl9v0xysq7jz99NM26z/33nvvKMikSi0qmTtjxoyUddLNxIcPSwoOKXChh+l0TcOLKahk450PGDAg3ap+ea7BIVV/OfPMMxMDNOl+oDqCQ+oIiL95Y9dNARb5h+OjpztWBVhGjRqV0hGiak/yi4+Pnm4fWp5PcKgY99ytt96aOIxe0jFrjPSwalY8OKTOiWznrvBPGKJRUEvLwvLXSb9ty1RWO+zMUa5QJaDjHSy2fvipvwUNGdi0aVO/uKrBIYX3FAYLmwJl8WpdCqJdccUV4WpOJb+tAlNYzttW0nWJBwPtOz4RQAABBBBAAAEEEEAAAQQQQKD2C1hHaiGCQ6qYrGrjaqqyk65zdsyYMb4attZTH4/6eqwtXrzYKZhjbcGCBVE/ULaKQ6q0oz7PYg+VFQaB9DKXql1b/2xScEjrmIsqDzVr1sxOL/rUPqx/J9+KQ3op0l4eVN/PDjvsEO23KhP5BIfCqkrdu3f3L/RV5bfZFgEEEEAAAQQQQAABBBCoqoA976Z7Nq3M/ku24pAqC91xxx0p5/Tiiy/6AE7Kwjxmwof3XDcLx7lOCg7lsh97E0kPunrgzdT23XdfPw53topDsslUfSnpN6ojOJT0uxYc0ptZms61nXrqqe6www6LVg87aaKFWSbyCQ4V+p7TMGP7779/hSPUdYhXDLKVdN1tqLZ4cMjWyfapII1V01EFLlVoird0x7D11ls7/Z1Yi48Lb8vTfVrwTd9nCg6pTHSmwJ1dt5NOOikl6JfUQaYw2iuvvBId0u677+7OP//8aJ7gUETBBAIIIIAAAggggAACCCCAAAJlI2AdqYUIDr311ltOQ8Cr7bTTTikvuhmoXphT1W69+KaWLegSDsOVLTj05z//2b3//vv+Za+uXbv6vhLNq69SLy1qe1Wqrsq5vvDCC74Kuo5dwRpVTQ9fsksKDmndTE3Dt6mPxvptdIzq38u1aVgy60Mr5BBt2YJDCnh98skn7rXXXkup4K3KT7KmIYAAAggggAACCCCAAAI1KWDPu2URHBoyZIhTUMhaVUvBhuNp2z71louGX9IDnx609QCqSiRhC0MI6YJDKlerykZNmjTxw3ap6krYjjrqKKcAhLWFCxe6ffbZx2b95+OPP+5atGgRLcsUHNID85/+9Kfowdk26tevn9tqq618VRqFYOLBjOoODqnjYq211vLjomu4Nr1dZA/7OmaFQ4YPH+5atmzpOxDkFh6zro/KEderVy9xGDHdExqjvHXr1k5ljjVkWrh/+41cA1aFvud0TVWtJ2w21JY6TjT8l4Z6C1tYCSddcEidGwr46H7UUHwzZ84Md+HC8I69kWYr6JqodLTMvvvuO6dy04899ph97T+t5PXy5ct9KebwmmgFOek+07BxGh4tHHJM99gzzzzjy0ZnCg6pkpH+p7fvFOyxppBdnz59/DXXdU8Kmz344IP+vtI2y5Ytcz179rTN/We8NDbBoRQeZhBAAAEEEEAAAQQQQAABBBAoCwHrSK1KmCYfqHHjxvmh4m2bJ5980q2xxho2W+Ezn+BQGKCpsKNggV5YVD+k+lTyaaoKpP4m9aup70h9dHqxrTLBoYcffth9++23bvbs2W769OlR9WxVqpaR+vNyadrHnnvuGa360ksvuYYNG0bzNqE+NvVhZWqqSh0O7xYGh9SX1ahRo2hz9WfF+xe1jvrAVCGdhgACCCCAAAIIIIAAAgjUtIA975ZFcEhv6OgB09pmm20WjeVty/L5VLlclZYNW1iZxZaHY5HbMgUrFG5JCg7tsssu/sHR1tVn/Nj1JtKll14arVLV4NCUKVNSKqpox2effXZU9lfzOlYNWRUObVVdwSGFWlRZSEEqa/EglJarapI6I6xpCDMNvxY2C9rILx5wue+++1LepFJ4SA5hs8o14bJ00/HrVtV7buLEiSn3nMJeCjpZ09tLun/CZtWZtCwpOKQOIB2nNV1nBdfU2WQtDNnpLa533nnHvvJvjvXq1Suaf/31193AgQOjeU1Yx5Y6d/R2WdhU3We//faLFr3xxht+KLRowYoJnXeHDh0yVhyy9RV2C4NDCvIpOGQt3kmk5eeee64Pzml62rRpfvg6TavpHn/qqadSKpMp+KS/67CdddZZKfdO+B3TCCCAAAIIIIAAAggggAACCCBQ+wWsI7XYwSG9mHXNNdf4ocRMbejQof7FLptP+sw1OJTUN6KX7bp06eKWLFnih6cPgy7qWzvggAOSfjJxmSolDRgwIKr4rJfUVPFZrTLBIfUbhf1U9qNhVXdblukzfBFszTXXdAokJTUdu/qHMjUNtxb2h4XBoUzbhd+pj00vRSaFl8L1mEYAAQQQQAABBBBAAAEEii1gz7tlERyKP8BtsskmbsKECZU2VlBg6tSp0fZhJaFo4YoJhQw0jFLYFDrSw3hScEgVXfTWT9j0MKoHa2t6G0VVUKxVNTikYMbNN99su/NhCYWJ4i0cg1vfVUdwSL+hSjurrLJKyuEkHUtoopWtoyLc8Pjjj3cnnHCCr9g0Y8aM6CsFjuJD2enL+HBm+QSHCn3PRQf7+4TuH/0Rz58/382aNctpGLF4NZ9MwSHZWjWfcN+jR4/2lZnCZc8995zT21TxpipB8+bN88egkta6T8POJa1vwaGwc8j2Ex8uUNdMgS5VD7Km+12VpjJVHLJ1swWHtF58KLIdd9zRjRo1yu8i/jafhobT3zoNAQQQQAABBBBAAAEEEEAAAQTKW8A6UosVHPr11199P82VV16Z0rdy2GGH+SBOtqo/uQaH1H+jocrU1Dekl6PU32Vt8eLFTn1D6jOyds899zhV+Mmlhf2M8dBR2DeU61Bl8Rfz7Bh07Ko8rhcsc2nqB7QK7pmCQ/Fh7pP2nSk4tOuuu7pmzZqlbKa+srlz5/rh4cIvOnfu7Kt3169fP1zMNAIIIIAAAggggAACCCBQrQL2vFsWwSE9jKpKjTU9XCaFY+z7bJ/xt11OPfVUpwf5eNNDv4IJYbMKJ0nBIauuEq6v4Z/0do61QgeH4sEkDRt1/fXX289FnxqHe9CgQdF83DBufOCBB6asrw1V9SmscKNlGmJrtdVW06QfaktDhFkLQx22TJ9JFYPC79NN2zFp+K2wepJCPvFqONqHhiULA0X5BIfiHnGvdMeYafnPP//sK+Co8ybb20/aT6bgULoKSCrVrHs0bApq2T8UCqop3PP0009XCCqF29i0BYcUTtP9bU1vs2k4v1xboYJDSRW2FLpSOC0e9rKQX67HyHoIIIAAAggggAACCCCAAAIIIFA3BawjtRjBIVXDUYBHoR5rqgB9zjnnuB49etiijJ+5BodUcUjVpBVmUd9Q0vmo/0nhIqverorX8Rcjkw5G+9Uw8mrqv7ziiiv88PO2bmWCQ7atqiHJSX2kb775pi3281bRKFqYMKGX31S929rLL7+cMtyYLX/22Wf9C3I2b5/qTzKPiy66yO288872VUp/UqZKSHpRThXrFQ6zphfW9OIaDQEEEEAAAQQQQAABBBCoKQF73rU8QCGOo96KB6D/XyqkinvUmxhqSQ+w+e46HgDR9umqqNi+Naa1qrmErW3btn7Yoh122CFc7IdsUhglqcVDKieffLIfDiopOJT0cFns4JDe/tGwXtb0ZsyFF15os9FnWNJXC+NBmHhQxkI60Q5WTOQbHNpnn33ckCFDwl346XgloAorpFmgylDaX/xtpPgDv22uENPll19us/4NLN1LubRC3nMaB14hNB2nAju5tkzBoXyqZNkQb+qk0fBmH330Ua6HEFUcUjWt8NjTBdTS7bhQwaGkId3UYbPRRhu5fffdN/p5ddApIMVbXxEJEwgggAACCCCAAAIIIIAAAgiUrYB1pBain9IQFdC59dZb3e23326L/KeGkT/mmGNckyZNUpZnmsk1OJRpH+F3YcgnXR9SuL6mFS6yatjqw2rdunXKKgrr6OU0Nb2A2bNnTz+tzulcz1VmeoHz7bff9tvKSn2t2Zq6rMP+XAV42rRpk22z6Pthw4b5at9aEO9HDF9ES+rbjXby+4TCT+rvVdtuu+18haffv+IDAQQQQAABBBBAAAEEEKh2AXveLYvgkIa7uuSSS1KQrfJPysJgZtKkSU5DF4Xt7rvvduuvv37Kg7C+Vxnbo446KlzVT2vope7du6csP//8850euEslOKRgTFjlJ92wXarYpHCQtWzBoaRhnl5//XU3cOBA24X/zFRxKCl8pI3iFYdUYjh8+E/5gWCmY8eOTmGk+LoKE2l5vMWHRMun4lCh7zkNrRcfXk/XQG84derUyYea9JbS559/Hp1GpuDQ9ttvn/KGk200c+ZMd9xxx9ms/9T90apVK98REw7xpi9VOUhBLA3/p2kNBRc2qzgUHwJNJa5V6jrXVqjgkH5vxIgRvtKV/bbuMx3/yJEjbZEP9+XS8RRtwAQCCCCAAAIIIIAAAggggAACCNRZAetILVRwSC8sqo/MAjCCU//KgAEDnF5czLcVOjgUDu2V68tfvXv3ThlmLddzuOaaa9zmm2/u/u///s9voj4o9dOka0899VTUh6N+1/Clv3TbaLmqKFlVp6FDh7p0L4Em7SMeiqpMxSHb7wcffOCOP/54P6sX1xTSoiGAAAIIIIAAAggggAACNSVgz7tlERz68MMP3bHHHptirbDJ/fff7ysIpXyxYkYhBT08qmyvtTDooKDJiy++aF/54chGjRoVzdtE+CBoy6x6S6kEh8K3XOwYVRlm9dVXt1n/qRDKo48+Gi2LB4fiYYykYcbuu+8+p86AsFUmOBQ/5qTfCn8jPq1gjAIy1lSqOD48l74bPHiwe+WVV2y1vCoOFfqeiw+Pp2M+++yzo7LKSSG1TMEhdUwo3FSvXr3o/DSh4cNUHjtsMliwYIHr27dvuNiPJb/HHntEy+JVqfSFBYf0t6bwUNjiZaFV0SgM72hdlbjeYIMN/N9kr169ws3d+PHj3aabbhot69evny9bbQt0T/bp08dmo8+pU6c6haysyUKdYKpCZk1v/G288cY2yycCCCCAAAIIIIAAAggggAACCJSxgHWkFiI4pKrSqmCjl/TU1MemfikFbyrbcg0OKZyjqj0rrbSS69q1a9qfU5+NhkpTU3+Q+qCytaoEh/Qy2sEHH+x/ItuLe2+88YY7/fTT/brpXoxLOtbwRUT1806ePDmnStOqohQO1VbVikNhvxTBoaQrxTIEEEAAAQQQQAABBBCoTgF73i2L4JBghw8fnhIM0DKFBRRU0EOaNQ1lpBCQAi1hUyUVextEoQKFFsJ22223OVW0saZOAL29EgaM9J0FZUolOKRQiAIyYdNDtwxsmKakSkHx4FC8ooxMVammYcOGftd6k0qVmcLAjr4wD03Hqx+lqzgUdl5oO7V7773Xrbfeev+dWfH/8tUQVMuXL4+WKXyjTpH4sFlaIV6iOCn0la3jIvqh3ycKdc8pwBbvPNJQaDoea3pjSm9OhS1TcEjraZi6Aw44INpEZirxrM4maxaYe+2119ygQYNssf+MB39UkWvMmDEp61hwKOwUsRXkEwaPwjLYto5dl8pUHIrv3/apDjL9bhgMtO/0qc4qBaiS2nfffed0LmFTBavVVlstXMQ0AggggAACCCCAAAIIIIAAAgjUIQHrSC1EcGjatGm+spB41L+mPsWwT6sybLkGh8IX09T3oT6QpBaGbNQfpD66bE1D26vPJV179dVX3U033eS/1lBs9oKYzl1Bph49ekSbPv/8826VVVaJ5sMJ9QFee+21fpH6auPVr8N1w2kdn4YVs6b+wXh/m31nn+rfVUXqsDJUVYNDV1xxhe+H1G9069bNXXXVVfZzfCKAAAIIIIAAAggggAAC1S5gz7tlExzSEE5JD7l6QNeDavv27d0XX3zhQwHhcE+6MlpHwyqpVK7aokWLKpSz1TpnnHGGL6X7/fff+yCLHnLDpvG7NQ63WqkEh1SpRg/NNga5Ha8qMnXp0sWHK8KqO/a9znfKlCk2632uu+66aF4TKrGsyjgKW6i6zZtvvpnyvWYqExyS/0EHHZQS/FDARR0FKm2saj/qRFBHTNgmTpzoOnTo4MI3k+x7na/KQWv89Y8//tgHYOLBknyDQ4W655YtWxaN+27Hq6HxFMTS9VOlHw3FF7+GWmbjxWuM9aSOCP1NqJNCgTkNhaZOlLCpbLbe+Jo+fbo75ZRTwq/8/mxbdf7obbl403Vv0aKF37+Gg4ub6o06VfbRNdGbXrq21sJh8yoTHFLVsP79+/u/38aNG9tu/adCZQ8++GDKMpsJQ4K2zD6TqipRnch0+EQAAQQQQAABBBBAAAEEEECgbgpYR2ohgkPhC4nqn9ALfFVtuQaHwtCK+oTUlxmvRv3ee++lhHFURTocPk19OL/99ps/ZPUdrrzyyjkdvvpJre/ovPPOc3vvvXfKduFQYieeeGKF6vFa+csvv/T9mNa/pPPRC125Nr2Id8cdd0SrZwpFqTL2DTfc4F+MjDZYMVHZ4JBCSLfffrtTNXpruYSXbF0+EUAAAQQQQAABBBBAAIFiCNjzbtkEh4SoBzO9xZNvu/766311onC7O++80z88hssyTStoo/BGs2bN/GqlEhzSwSRV8Ml0LvouHhyaO3eur1aTbbv495UJDmkfCi2df/758d2lndcbRHoYt6ZqOwq75NPyDQ5p34W65xTuiod6VNVJgRrrLImfSzhUV7rgUHybcF77V7imUaNGiWE5ravAVjywFO7DKgZpWb7HoLfHttlmG7+7XIJDSRWetPFmm22W0imjZTNmzPDBK03Hmyonrb/++vHFfp7gUCILCxFAAAEEEEAAAQQQQAABBBCo0wLWkVqI4JBeXLOX3fTSUzy4kwSpoditjyTp+1yDQ2G1I+1n11139UOR6YXKb7/91mkoM4WZrKkytSruWNMLbN27d7dZd99997lcTbIFh+IVzzVEmvrztH+9pKljV5DH+sHUTzd27FjXoEGD6HiyTejFOb3YGVbb1ktxW265pdtggw38C4Xz5893c+bMcer7td8K95spOKSXKONVqVUNXS8Xzp49O2V/qoSvlzCt4nv4G0wjgAACCCCAAAIIIIAAAtUlYM+7ZRUcEq4qoKgSS65tyJAhTpVS4k1VYFTB5bHHHot/VWFeIRuV+A0f8EspOKS3hBTquPrqqyscuy1Q5Robe13L4sEhLVPFGv0vXVMFGT0kh62ywSHtIyybHO4zPq0OAIVomjZtGn2lPwBV04lXl4pWWDGx9dZbp1RJqkxwSPsrxD33wgsv+KHvwuMLp3U9LOBjy/fdd99oPPp8QzvaxzXXXOO23XZb252/P9IN4aWVkkJEYXhJfzP6e8olsBX/u8slOJQuTJYUHNJbXvq7Disc6RzCKkeajzeCQ3ER5hFAAAEEEEAAAQQQQAABBBCo+wLWkZprSCadiIbyUrgk33baaae5Qw89NO1muQaHtIO77rrLB27S7uz3LxRsueyyy1L604oZHNLP6iXBsMJ5umNUP5gqB6Ubai3ddlquvsmLL77YzZw5M9Nq0XdyUMVsVeRWyxQcijbKMqHwkqpka4g2GgIIIIAAAggggAACCCBQkwL2vFt2wSGhf/rppz48pKoj6Zoe4hUsadOmTbpV/PJnn33WXX755SlvjIQbKHihMrw2zJl9pzdcdtllF5v1nw8//LDTkFlhGzdunJs0aVK0aMcdd3SjRo2K5vU20J577hnNa+LJJ590a6yxRrQsXlEoXTji/fffdyo/rJLEVkWmY8eOTsNiKXQjD2uqRvPEE0/YbPSp4JCq1MQDGSo3fMghh/g3maKVV0zIr0mTJn5RfPgorX/66aeHq1eY1ttICjzZ8cZXUGUhvaGU9PaOhlBTFaKpU6fGN3PaTmOsh+fcuXNnp+tRmVaIe07hH/1+/G0ndWAMHjzYd6yMHz8+5fDsza94cEj3kYIz6gCKXytdc7no/MP2448/OlXfShriS2/Iyeqss87y1Xxsu/h9osCOhv3TfpKa1h85cmSFCl9JHWvxIcJ0fHobTPsPjZKCQ/rt+N+WlmkoQXXepGsffPCBO/7441O+tiHwUhYygwACCCCAAAIIIIAAAggggAACdUbAOlKrGhxSaOXoo4/O2yXTkFra2cKFC6MXHzfZZJOML/ZpfYVzNGxXuhfqVGXo8MMPr9CfFg8O6QWzXMM7eiFRQ5SpDR061KkvKd70cqOqV4dVj+LraIg1DTO/+uqrx7/KeV7noX7YpH4224n6aNU3qcCQqhrpuun66UXGMPxly227pE/1d+m6qKqRXizt2rVr0mosQwABBBBAAAEEEEAAAQSqXcCed8syOGTaChoocKIhoATSsmVLXzVFlVPiQR/bJulTYQiVsf3kk0/8/1SSViV+NdxRWOUmadtSXaYyugrbWOBGD9NhYEnhkkzDvmnMcXkowKROFb0JVKwmf3V0fPzxx/5TD/bt2rXzv9uwYcOsP6vOFb1ltHjxYn/d9BBfrDd+qnrPKeykY9WbZK1bt/bDcFnwKtOJJgWHdD11PArDfP311/5e3WijjfzfQaZ9yVolm3Uscta9kItzuE9z0D2ie0V/K/ptBfVyKdEd7is+rX3rfPSpjh39Dcoq3lR+W2/shS0pvBd+zzQCCCCAAAIIIIAAAggggAACCJSfgHWkVjU4VEpyCs/MmjXL94mqb2zttdf2/aJt27YtWr9YruevSu3qs9WLeAsWLHAK3she/1P/baGaDNSnqz5F/U/9W+oTloWCPtYvWqjfYz8IIIAAAggggAACCCCAQKkJ2PNuWQeHSu2i1MTxqLrQ5MmTU35aFVdUpcWahonSm0aqSGRN45mr0hKtdgikCw7VjqMvzlGOHTvWl+e2vW+33XZu9OjRNssnAggggAACCCCAAAIIIIAAAggg4AWsI7UuBYe4tAgggAACCCCAAAIIIIAAAgjY8y7BoTK/F7744gu3//77pyioxLCCQnqDR5VtHnnkETdt2rSUddKVFE5ZiZmSESA49N9LoZLSGqJw7ty5vvR1eIEuueQS17Nnz3AR0wgggAACCCCAAAIIIIAAAggggICvyiMGgkPcDAgggAACCCCAAAIIIIAAAnVJgOBQXbqaVTyXAQMGVAgGZdqlgkWTJk1yjRs3zrQa35WQAMGh/16Mu+66y6nSULxpeLsHHngg7yHX4vthHgEEEEAAAQQQQAABBBBAAAEE6p6AdaQSHKp715YzQgABBBBAAAEEEEAAAQTKWcCed6k4VM53we/nvmTJEjdw4MCUocjSsWiIsuHDh7umTZumW4XlJShAcOi/FyVdcGjEiBGuT58+JXjlOCQEEEAAAQQQQAABBBBAAAEEEKhpAetIJThU01eC30cAAQQQQAABBBBAAAEEECikgD3vEhwqpGot3tfy5cvd1KlT3aOPPuoDRIsWLfJns+qqq7oNN9zQtW/f3nXq1Mntsccern79+rX4TMvz0KdMmeLGjx8fnfy2227rBg0aFM2Xy8Tdd9/txowZ40/X7u0jjjjCKRBHQwABBBBAAAEEEEAAAQQQQAABBJIErCOV4FCSDssQQAABBBBAAAEEEEAAAQRqq4A97xIcqq1XsMjH/csvv7ilS5e61VZbrci/xO4RqF4B3dtqDRo0qN4f5tcQQAABBBBAAAEEEEAAAQQQQKBWClhHKsGhWnn5OGgEEEAAAQQQQAABBBBAAIE0Ava8S3AoDRCLEUAAAQQQQAABBBBAAAEEEEAAAQQQQAAB60glOMS9gAACCCCAAAIIIIAAAgggUJcE7HmX4FBduqqcCwIIIIAAAggggAACCCCAAAIIIIAAAggUVMA6UgkOFZSVnSGAAAIIIIAAAggggAACCNSwgD3vEhyq4QvBzyOAAAIIIIAAAggggAACCCCAAAIIIIBA6QpYRyrBodK9RhwZAggggAACCCCAAAIIIIBA/gL2vEtwKH87tkAAAQQQQAABBBBAAAEEEEAAAQQQQACBMhGwjlSCQ2VywTlNBBBAAAEEEEAAAQQQQKBMBOx5l+BQmVxwThMBBBBAAAEEEEAAAQQQQAABBBBAAAEE8hewjlSCQ/nbsQUCCCCAAAIIIIAAAggggEDpCtjzLsGh0r1GHBkCCCCAAAIIIIAAAggggAACCCCAAAII1LCAdaQSHKrhC8HPI4AAAggggAACCCCAAAIIFFTAnncJDhWUlZ0hgAACCCCAAAIIIIAAAggggAACCCCAQF0SsI5UgkN16apyLggggAACCCCAAAIIIIAAAva8S3CIewEBBBBAAAEEEEAAAQQQQAABBBBAAAEEEEgjYB2pBIfSALEYAQQQQAABBBBAAAEEEECgVgrY8y7BoVp5+ThoBBBAAAEEEEAAAQQQQAABBBBAAAEEEKgOAetIJThUHdr8BgIIIIAAAggggAACCCCAQHUJ2PMuwaHqEud3EEAAAQQQQAABBBBAAAEEEEAAAQQQQKDWCVhHKsGhWnfpOGAEEEAAAQQQQAABBBBAAIEMAva8S3AoAxJfIYAAAggggAACCCCAAAIIIIAAAggggEB5C1hHKsGh8r4POHsEEEAAAQQQQAABBBBAoK4J2PMuwaG6dmU5HwQQQAABBBBAAAEEEEAAAQQQQAABBBAomIB1pBIcKhgpO0IAAQQQQAABBBBAAAEEECgBAXveJThUAheDQ0AAAQQQQAABBBBAAAEEEEAAAQQQQACB0hSwjlSCQ6V5fTgqBBBAAAEEEEAAAQQQQACBygnY8y7Bocr5sRUCCCCAAAIIIIAAAggggAACCCCAAAIIlIGAdaQSHCqDi80pIoAAAggggAACCCCAAAJlJGDPuwSHyuiic6oIIIAAAggggAACCCCAAAIIIIAAAgggkJ+AdaQSHMrPjbURQAABBBBAAAEEEEAAAQRKW8CedwkOlfZ14ugQQAABBBBAAAEEEEAAAQQQQAABBBBAoAYFrCOV4FANXgR+GgEEEEAAAQQQQAABBBBAoOAC9rxLcKjgtOwQAQQQQAABBBBAAAEEEEAAAQQQQAABBOqKgHWkEhyqK1eU80AAAQQQQAABBBBAAAEEEJCAPe8SHOJ+QAABBBBAAAEEEEAAAQQQQAABBBBAAAEE0ghYRyrBoTRALEYAAQQQQAABBBBAAAEEEKiVAva8S3CoVl4+DhoBBBBAAAEEEEAAAQQQQAABBBBAAAEEqkPAOlIJDlWHNr+BAAIIIIAAAggggAACCCBQXQL2vFvywaHqAuF3EEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBAoJwGCQ+V0tTlXBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQR+Fyj54FAhD5CrjgACCCCAAAIIIIAAAggggAACCCCAAAII5CMwd+5cvzr9lPmosS4CCCCAAAIIIIAAAggggECpCxTjebfebytaoU68GAdYqGNjPwgggAACCCCAAAIIIIAAAggggAACCCBQHgL0U5bHdeYsEUAAAQQQQAABBBBAAIFyEyjG8y7BoXK7izhfBBBAAAEEEEAAAQQQQAABBBBAAAEE6rhAMTpS6zgZp4cAAggggAACCCCAAAIIIFALBIrxvEtwqBZceA4RAQQQQAABBBBAAAEEEEAAAQQQQAABBHIXKEZHau6/zpoIIIAAAggggAACCCCAAAIIFEegGM+7BIeKc63YKwIIIIAAAggggAACCCCAAAIIIIAAAgjUkEAxOlJr6FT4WQQQQAABBBBAAAEEEEAAAQQigWI87xIciniZQAABBBBAAAEEEEAAAQQQQAABBBBAAIG6IFCMjtS64MI5IIAAAggggAACCCCAAAII1G6BYjzvEhyq3fcER48AAggggAACCCCAAAIIIIAAAggggAACMYFidKTGfoJZBBBAAAEEEEAAAQQQQAABBKpdoBjPuwSHqv0y8oMIIIAAAggggAACCCCAAAIIIIAAAgggUEyBYnSkFvN42TcCCCCAAAIIIIAAAggggAACuQgU43mX4FAu8qyDAAIIIIAAAggggAACCCCAAAIIIIAAArVGoBgdqbXm5DlQBBBAAAEEEEAAAQQQQACBOitQjOddgkN19nbhxBBAAAEEEEAAAQQQQAABBBBAAAEEEChPgWJ0pJanJGeNAAIIIIAAAggggAACCCBQSgLFeN4lOFRKV5hjQQABBBBAAAEEEEAAAQQQQAABBBBAAIEqCxSjI7XKB8UOEEAAAQQQQAABBBBAAAEEEKiiQDGedwkOVfGisDkCCCCAAAIIIIAAAggggAACCCCAAAIIlJZAMTpSS+sMORoEEEAAAQQQQAABBBBAAIFyFCjG8y7BoXK8kzhnBBBAAAEEEEAAAQQQQAABBBBAAAEE6rBAMTpS6zAXp4YAAggggAACCCCAAAIIIFBLBIrxvEtwqJZcfA4TAQQQQAABBBBAAAEEEEAAAQQQQAABBHITKEZHam6/zFoIIIAAAggggAACCCCAAAIIFE+gGM+7BIeKd73YMwIIIIAAAggggAACCCCAAAIIIIAAAgjUgEAxOlJr4DT4SQQQQAABBBBAAAEEEEAAAQRSBIrxvEtwKIWYGQQQQAABBBBAAAEEEEAAAQQQQAABBBCo7QLF6Eit7SYcPwIIIIAAAggggAACCCCAQO0XKMbzLsGh2n9fcAYIIIAAAggggAACCCCAAAIIIIAAAgggEAgUoyM12D2TCCCAAAIIIIAAAggggAACCNSIQDGedwkO1cil5EcRQAABBBBAAAEEEEAAAQQQQAABBBBAoFgCxehILdaxsl8EEEAAAQQQQAABBBBAAAEEchUoxvMuwaFc9VkPAQQQQAABBBBAAAEEEEAAAQQQQAABBGqFQDE6UmvFiXOQCCCAAAIIIIAAAggggAACdVqgGM+7BIfq9C3DySGAAAIIIIAAAggggAACCCCAAAIIIFB+AsXoSC0/Rc4YAQQQQAABBBBAAAEEEECg1ASK8bxLcKjUrjLHgwACCCCAAAIIIIAAAggggAACCCCAAAJVEihGR2qVDoiNEUAAAQQQQAABBBBAAAEEECiAQDGedwkOFeDCsAsEEEAAAQQQQAABBBBAAAEEEEAAAQQQKB2BYnSkls7ZcSQIIIAAAggggAACCCCAAALlKlCM512CQ+V6N3HeCCCAAAIIIIAAAggggAACCCCAAAII1FGBYnSk1lEqTgsBBBBAAAEEEEAAAQQQQKAWCRTjeZfgUC26AThUBBBAAAEEEEAAAQQQQAABBBBAAAEEEMguUIyO1Oy/yhoIIIAAAggggAACCCCAAAIIFFegGM+7BIeKe83YOwIIIIAAAggggAACCCCAAAIIIIAAAghUs0AxOlKr+RT4OQQQQAABBBBAAAEEEEAAAQQqCBTjebdWBYcWLFjgpk+f7mbPnu0+/PBD9/nnn7vmzZu7li1bui5durjtt9/erbvuuhXgSnXBTz/95J577jn32WefuYULFzrN77TTTq5Xr15u1qxZ7rHHHosOXed18MEHR/NMJAvI7a233oq+bN26tevZs2c0n23ioYcecj/88EO21VzDhg3dGmus4e+3jh07ugYNGmTdhhWqT+C3337z/1boftC/FTNnzvQ/vuaaa/prpn8rttpqK7fyyitX30HVol+67777/L9Ldsj6d2nrrbe2WT4RQAABBBBAAAEEEEAAAQQQKHmBYnSkFuuk33vvPTdnzpyMu2/UqJFr166dW3/99Z2maQgggAACCCCAAAIIIIAAAuUpUIzn3VoRHFII4NFHH3UjR47MeuX33HNPN3DgQNeiRYus69bkCvPmzXP9+vVzX331VcphHHvssW7AgAHu+eefd4MHD46+23LLLd0tt9wSzTORLHDXXXe50aNHR1927tzZTZgwIZrPNtGjRw+3dOnSbKulfK97bdSoUT68lvJFGc3oH6fvv/8+OmN1Yq2++urRfHVOfPHFF+7iiy92r776asaf1XU799xz3c4775xxvXL88s9//rObMWNGdOr6N/XII4+M5vU3EnZorrLKKm6jjTaKvmcCAQQQQAABBBBAAAEEEEAAgZoWKEZHqs5JLzLec889/iUlveTYvn17/3LSFlts4TbbbDNXv379vE/96quvdpMnT855O70QNWLECP9CZc4bsSICCCCAAAIIIIAAAggggECdECjG827JB4eWLFnizjvvvKwhgPAKKxAwceJEt84664SLS2p62LBh7umnn65wTASHKpDktaAmgkN2gEOHDnV9+/a12bL6PO6449w777wTnbOCO7vttls0X10TU6ZM8WGgfH7vhBNOcP37989nkzq/brbg0Ouvv+5OPvnkyEGVnJ588slongkEEEAAAQQQQAABBBBAAAEEalqgGB2pjzzySMYXGw8//HA3aNCgvE893+CQfkDP4tdff7374x//mPfvsQECCCCAAAIIIIAAAggggEDtFSjG827JB4euuOIKd++99+Z91dq2beu3UyWMUmxdu3atcFgKPCk4pE4GKg5V4MlpQU0Gh3SAqoxVyoG1nBArsVIpBIf0xt+f/vSnShy9c+eff36lt63UD5b4RgSHSvwCcXgIIIAAAggggAACCCCAAAJZBQrdkfryyy+7M844I+V3Naz3p59+mlJRXBXG//KXv6Ssl20mDA7pRZ2ePXumbLJs2TL37bff+hcr1fdlbZtttnHjxo2zWT4RQAABBBBAAAEEEEAAAQTKQKDQz7siK+ngkMb3Pvrooytc2t13393pf3qj5uuvv3b//Oc/3U033VRhvdNOOy1x+worVvOCxYsXu1122SXlVy+44AK3zz77RMu0zuzZs6N5DfvEUEARR9qJQgeH+vTpU6FyznfffeevjSpGffPNNynHcswxx7hTTz01ZVk5zJRCcOjMM890L774Ygr3iIc7ogAAEJ9JREFUqquu6k4//XTXqVMnpxDhxx9/7If8++CDDyqs99RTT7kmTZqkLC/XmX/9619O97m1du3apQTiqDhkMnwigAACCCCAAAIIIIAAAgiUqkAhO1J/+eUXt//++7t58+b50+3du7cfKqxRo0Z+ftq0ae7EE0+MKK677jqn4cRybWFw6K9//avbc8890276zDPP+OrstsKzzz7LkGWGwScCCCCAAAIIIIAAAgggUAYChXzeNa6SDQ7pgVwhjPh/4NdbN8cff7wdf/T59ttvV1iu0MALL7zgGjRoEK1nE6pOomDOnDlz3EcffeQUzNlwww19GKlDhw4+ZGDr2udvv/3mdBF+/fVXv0j7VXhJ86+99prf348//uhatWrlNthgA9e5c2fb1H9qPf3HeHUy6NzCdtttt7n11lvP/64CDtqPxkm3po4IlSBOajoXOek8Gjdu7M9DbzzpeD/88MNok5VWWiklfKTt9LaSNR13/Df0RlMYYKpXr57beOONnT7VvvzyS7dw4ULbhVt33XW9pfYtEwW7FJKSR9gWLVrkZs2a5f+n8I0qRGm/ugYrr7xyuGqFad0bn3zyiT9nWWrbLl26+GBDoYNDAwYM8FWgKhzEigU6hgMOOCDlq+22286NHTs2ZZlmdO3tfvv3v//t/XS++l/c3Db+z3/+4+bPn2+zrlmzZn5declWxtpW11oGYdN27777rvvss8/c2muv7YMz8XXC9TX9008/+ftF97j+LnRvKDCia6f/6drGm45x+fLlTk7h3+o555zjFLrStdTfYdgqYxFunzT90ksvVSgFrvO94YYbKtj8/PPPTsenbcI2ePBgd/DBB4eL/LQMdc30qfu9efPm3l1/323atKmwfvxvRn+T7du3d7pvX331VffxivBS/fr1va2uvyqNZWry1TZ6g1HHIfPWrVv731blsmxhJ/07oGPXvw+6B7W/tdZay/+9bLnllon/Ptrv2HHpHtJ5a1v9voJDGkLSms7hvvvu87NNmzb1f9f6XWstW7ZMtNK/D1999ZWt5uL/RkVfMIEAAggggAACCCCAAAIIIIBAngKF7EgNwzrqC7zjjjsq9B1OnjzZKQCkpuCPAkC5tnyCQ+pfUP+TtUmTJrlNNtnEZvlEAAEEEEAAAQQQQAABBBCo4wKFfN41qpINDiUFgfQf6idMmGDHXuHz3HPPdVOmTElZrv+YHY71rYfrm2++OeN+FDi47LLLKjx06z+Yx0sFa2gqhSb0H9rjTf9Rfvjw4e4Pf/iD/yr+9lF8fc2r6pCqD8WHKtM+HnzwwZRNFPQYOXKkU+WbeNt00039cSloZU0BjjAsoaCEfseaQhNaFjaFew477LBwkd9GQSu1Sy65JOW4dOxLly51GmLO2pVXXhm5KTTywAMPuFGjRtnXFT4VqlJZ56Rh5hSm0nWeMWNGhe0U4tG101td1rLdM7aeffbo0cMfv81nCg5pnfiQTknXSaENuSQds/ah+1P3yeabb67ZqD322GP+7TVboLfZevXq5YYOHWqLok/tY/z48T6oo8o7qsIVb+mCMVpP11lhmqT72Paj6l8nnXSSD3fYsn333Td6286WhZ86J4XirFXWwrZP93nKKadUOGeFhrbddtvETZLuawWdLr300mh9hfw0rzf30jWFtoYNG+ZDf7ZOfN+6J7Qf+dubibauPlUZ7cgjj/RhonC5pvWPvq639pmuaXg27Tvp70XBnBEjRvjAUtL2Cp6pfLquY9ji9/XAgQP9MYYdpeH64bT+/vQ3qn8HrHXr1s2NGTPGZqNPDRH35JNPRvPxvx/9+x2eu0KdBx10ULQ+EwgggAACCCCAAAIIIIAAAgikEyhkR6qerdVPo5auf0X9CDvvvHN0OK+88kris3q0QjCRT3BIL+rstNNO0XP3I488UuGlqWDXTCKAAAIIIIAAAggggAACCNQxgUI+7xpNyQaH4qEJHXAYQLETCD9VUSP8j9D6Tv9RXZV81PQAr1DF9OnT/Xy2/9N/1Nb21pKCQ/oP3ZnCFhpe7Pbbb/eBjjfffDPrGOe5Boe+//57HzhIF0axYw4/qyM4tOOOOzp1jITNrptCWxoLfurUqeHXidMq5zx69OiUaij6A+jfv3+F4cESd/D7wmIHhxSm0ZB61uz62Xx8SCdbnvSp4fYURLEW/xvQ9QvDGLaefcreqtrYsvinAh0HHnhgymIN0aXgUi5NQSB1Zqn6jFo+waGqWGQ7tl133TXlvlCQ6t57740qYyVtr1BKWNVLVXVUdlxNVXAU5okPRZe0H1XbUTjKKjrFg0PaRutk2pc6/K666qqU3auqlIJruTTdNwrmhNW6VHFKQa9M94ztW9WD7Ny1rCrBoeuvv95Xevvb3/5mu/efGkYurI6kEKGCmOHxxSvKKVQV/nuhYJ+9vZmyc2YQQAABBBBAAAEEEEAAAQQQiAkUsiM17HfQi33qD0xq6kuwisy33nqr22KLLZJWq7Asn+DQO++84zRkvJr6itJVW6/wIyxAAAEEEEAAAQQQQAABBBCoEwKFfN41kJINDmm4p7BSiQ5Yw/xoKJvKtmuuucbdeeedeW2uikLrrLOO3yYpOJTLzmxscoV89B/kMzX9x3v9R/xsFYdkkzQkVqZ9V0dwKOn3LTj08MMPu4suuihplcRlVuXEvjz11FPTVk6xdeKfxQwOvfHGGz6YEf6m3kDba6+9/CJVhOrbt2/KUEzhuvFpXR9VdLHKMfHgUHz9ysyrwoz2a8P3KTizxx575LUrVXxRdSI13a+ZgnPmX1WLTAeoEJ2CN2FTwOyEE04IF+U1ffnll7v7778/523CwEtScCiXHalKlypKqSlUo4BXJtv4Pm+55RanKme2vUJt1lkZXzdpXkGf9ddf33+VLjikim4Kn2VqqvSk+0uVjMKmYFR4nd5//3131FFHhav46mVh5yvBoRQeZhBAAAEEEEAAAQQQQAABBPIQKFRHqoZyVxVotXjfWvxwVMFclbbVFAbSCzC5tGzBIQ0driHM9Syt37CXcNJVP8rlN1kHAQQQQAABBBBAAAEEEECgdgoU6nk3PPuSDQ6dffbZ/o0ZO1hV7FCoorJNQwTFh+NRhRAFWTQO+JIlS5wqkOg/voctHJM8XXBIQ2upMlHTpk39f/hW5ZiwHXvssSmVQxYuXOh22223cBV/bjpHa5mCQ+oc2H333aNOAttGYQlVHlm0aJFT1Y946CDeuaHOhUIPVWbHok9VW1IVF4UQVPVp7733TjlmhUoUtGnVqpUfQu3GG29MOWZdH4WN6tWr55KGrpOXwkVt2rTx2z/00EMp+9cxWHBF07m0+FBlqrATVgHSPuSrP0a94RU2+eqtM52Pmt4sU4gibKpipWunIbtUSvquu+4Kv/ZDv+ktNrWk4NDGG2/sK03pXlPQQ/uIN+3/8MMP99WHNJRcONST1lUgpn379n4zDaEVrwyjwM0hhxziNBydqilpKK74EFvqBGvXrp1TeWz9T9c49FBYTseha6f/VdUifo7h/MyZM90RRxwRLvJDw6n6U2Vb+Cah9qF9KSylUJeqEakSkP2DrO/DSjjpgkMqV66/+0aNGvnhBeNDDCrUpaBiw4YN3SeffOI09F7YFNhRUEveOmcFg8I2aNAgf921LKmKlI5RQR0Lj2lou7CFnY3pgkNaX6EmVY8KqyFpn48//rjfXf369f29p0BaWGVJIbpwmL2JEyf6f6fsGHRvx4OdBIdMh08EEEAAAQQQQAABBBBAAIF8Bey5XVWJq9IU2NEzrVp8iO34fseNG+cmTJjgF6s/Zb/99ouvkjgfBoe0QthHqPnw+VrzapmGPv/vGvw/AggggAACCCCAAAIIIIBAXRQo1PNuaFOywaHDDjssJfCgAEe8AlF4ItmmNYyPhgwL2xNPPOFDJ+GyUaNG+QBRuOzvf/+7a9mypUsKDvXp08cpfBG2+LGrioiqiViranBIASpVJQrbkCFDUoIGCkIpeBB2LFRXcGibbbbxw8qFwxLFg1A69smTJ7sOHTpEp6EhidTpETaFCzp16uRGjhxZISSjoJANQ6dtXn75ZT8UWrh9VYND4b4yTSs4obBM165do9Xi1Xji94FWjA91piHarrvuOr+PpOBQWAEraagnha0U6rHKXEmBK/0d6e9JQ/cpzBK2gw8+2ClAErYvv/wyqqJkyw899FB31lln2awvkR0Ghy6++OKUcFxVLaIfSph46aWXnEIzYVNw7n/+53/CRTlPWzDPNlBY6I477nBrrbWWLXJhR6AWqhNSwUO1pOBQt27d3LXXXusUqlHTtdPfsCr4hM2q/qialYZVtKaOyUmTJqUMvaay5KG5wmLmoGo/GhrRmu6Lu+++278Zacvi4aBwuLT4d/HqX/Fh53T/x4eJ1H0sN2vq9FRYygz69euXMmykzlf/doZNYcLwHPX3fOKJJ4arMI0AAggggAACCCCAAAIIIIBAokChOlL1UpW9vJOtn0kvxKjiuVpYnTjxAIOF8eBQ8FXaSfXh6Pm/WbNmadfhCwQQQAABBBBAAAEEEEAAgbonUKjn3VCmZINDevi1E9YBb7rppin/ETo8iVym9R++X3nllWjVsJJQtHDFRFKlj5tvvtlttdVWicEhVXSxCjG2H1XRUejDWhgG0bKqBof05pKCC9bigSBbrqCUAlPW4usVo+KQfiMcbst+O+lYFKQI2y+//OJUNSlsCheoEyQeMlA1IwUh4i0+nFm2Dp349vGKQ/Hv081rDHv9tg0BpnPZbrvtUlZX5akddtghZZmCQOG9oi8VhJFjPDikZS+++GJKeCQe8FBVpwsvvDD6jaRwkKogbbvttu7dd991qpYVNlVMCoeKsu9UKSkMhqgKU1hZKx5iCYNDhbCw40j6lEkYstE6VQkOxX9DIR9VXJo/f76viKUwVmih9bMFh1RZqmPHjim7TgrTpTvuZcuW+ZLkOgb9u6hhGxUuClsYHIpXTDrppJMqDJOojk+VOLfWvHnzKEgWv68qExxKGopMQSL9W550X8pUASQaAggggAACCCCAAAIIIIAAAoUQsH7FqlYc0os5Nhz3jjvuGAWDko5RL8kpBKRW2eCQ+lw23HDDlN1rqDJVFp8zZ06FlwT1ElLr1q1T1mcGAQQQQAABBBBAAAEEEECg7goU6nk3FCrZ4FA81BIPvYQnkcu0gkIaYsha/D+E23KFBBSqCJuVFk6qOJQUCIhX2ih0cCgeTIqHOOzY4xV84oZx46RqM0nVUxR40DBWagpOKWxiLRyyyZbpM6liUPh9umk7pngQQm96xasTaR9jx45NqUxVXcEh/XYY3Pj888/98HVanm+zqkLx4FBSOez4NdQwY2H46scff3Tq1AqbBYcU1FAgKGyqJGMVYcLlqnYTBr3i91Km4FAhLMJjiU8n3aPDhw+vMDRhfLts89qvroGq5ISVu5K2yxYcUjWsxo0bp2yqfcZDh/HKYQo76jrlMkyj3X8//PCD6969e8pvJQUcU1aIzRQiOKQh1VSSPRzmTvem7tF//OMfKRWr0v0bFjssZhFAAAEEEEAAAQQQQAABBBDIWaBQHalhv0O251e9ZHXLLbf4Y1SlYVVgzqWFFYdU0Vr9mOna9OnT/XDq1lfRu3dvd9lll6VbneUIIIAAAggggAACCCCAAAJ1TKBQz7shy/8DAAD//+gFJdgAAEAASURBVO3dB7QURdrG8UIEE2IARHQNLGYXRcG0ChhWjIA5IZhQXIIBFQMqKKjAroKCAuoawCyomDO46iLGFUyoa2LNmFBMoHw+9fn21vTtmTvDTF/m3vnXOTA93dXp1z2cU8Vbb9Vb9FtxJSrvvPOOP9If//jHoo94+eWXu2uvvTbjOE8++aRbbrnlMtaFX3788Uf30UcfhavcH/7wB9egQQO31VZbZaw/99xzXZcuXTLW2ZdOnTq5L7/80r66fv36uSOOOMJ99913bscdd4zWa+Huu+92a6yxRsa6yy67zE2YMCFa9+c//9lpnZUvvvjC7bbbbvbVfz788MNu1VVXjdY9/vjjbsCAAdH3tdde291xxx3++wknnOD+9a9/Rdt23313N3To0Oi7LcyaNcsdddRR9tUtv/zy7p///Gf0XcfXeawcdNBBGefU+rfeessdeuihVsV/ap/GjRv75QsvvDC6Lq3o2rWrO+ecc/y28C8ZTp8+PVyV1/Kee+7pj7fddttl1B82bJj7y1/+krFOX2Ska7Ky+eabu3/84x/2tdrPDh06uO+//z6qd+SRR7rDDjss+q6fy1dffeVefPFFN2LEiGi9LTzxxBNuhRVWcDNnznRHH320rS7oc/LkyW6dddZx99xzjzvvvPOifcN3wFbGn+Gxxx7revXqZZudfhM77LBD9F0LY8eO9b+HiRMnuksvvTTapvdP72FS0Xt+/vnnZ2x69tln3VJLLeXX6T3T+2blggsuiN7xUljYcZM+9bz03MLSs2dPd/zxx4erqix//PHH7ocffojW67k1b97cf3/jjTfc4YcfHm2rbkH/5t12222+Wvw3k8s1/r6deOKJrnv37v44d911V+LvOtu16D3t37+/e//9993++++fUe3KK690W265Zca6XF+OOeYY9/LLL0dVTjrppAwPPfvevXtH21dbbTV3//33R99t4eqrr3bjxo2zr26jjTZyN9xwg9Pvd9KkSdF6/ZuhfzsoCCCAAAIIIIAAAggggAACCJRKoFT9lHPnznXqe1NZf/313c0335z1EocPH+5uv/12v139RjvvvHPWuuGGSy65xN10001+lfpf1B+Wq8T7/GbMmOHq16+faxe2IYAAAggggAACCCCAAAII1BGBUrV3Q4565Ro4lBSocPbZZ7t99tknvP6M5euuu86NGTMmY53+c3rdddd1++23n/vggw+ibX379nUKComXX375xW2zzTYZq63BXi6BQ/FgnWydFvHgo+oChw488EB3+umnZ9z7M88842QVllyBQ0nBR9p3yJAhbsqUKdFhFGjQvn376Hu2hY033tgHFMQDv7IFGlx//fVu9OjR0eGKDRzK9p7oBAqIUGBEWBQwtskmm/gAtnhgmjp9cgW+2XH69OnjA7PSDhx69NFH3RlnnGGn9Z/ZOpoUxKdgPivxYJhcgUMK5ivWws6b7TMe7Kf3S++bggaTioKN1OkXBolZAN7nn3/uA2/CbTpG69atnYIA9U6q0zAM1ssVOKR9wyArfbfSrl07W/Sf+m3rXuKBOVZJgUbaZ9NNN3X33nuvu/POO22TD3BT4JCCHnWMsKjjcpdddglX5VwuVeBQUhCTAowUlBUGZz722GNupZVWynlNbEQAAQQQQAABBBBAAAEEEECgEIFSdaQuXLjQbbvttv7U6lubNm1aNJAqfj1nnnmme+SRR/zqQgbxFBo4FO+/VP/AWmutFb8cviOAAAIIIIAAAggggAACCNRBgVK1d0Oasg0cmj17tuvWrVt4rU7BAMrC0bBhw4z1+vLTTz+5XXfdNSMQIMzQctppp7mpU6dG++k/4NUoj5fXXnvN9ejRI2O1giYUNFAugUMKTgkzGOliw2Aeu3hlfQkDC+KBQ8q6FGYJSTLRKKqLL77YDuk/w3PFg5iyBQ7FrznpXBkniX1RoIGywFjZd9993cCBA+1r9KnAiTCrUpqBQ0899ZRTJpawKHBNnUnxDhzVUYYfBZ3kW9IKHFLAk4JPkn5j2a4x7qoMUGGAVq7AoVJYVGcW/32rfq5AQ2UHimeMspGADz30UMa7pd+NslYpQM9KPFtTdYFD+ndL2c/CkuRv/9bEf1fK1HPFFVdEmb50nHjmMcs4pKxY8UC7pAA4ZefSO2ylVatWTkFrKqUKHNKx4r9dXaeNotT2Qv8t0D4UBBBAAAEEEEAAAQQQQAABBKoTKGVHqgaDffbZZ/6U6lNIyrau/o+ddtop6pu0rNTVXae2Fxo4NG/evIxsRkn9DvmclzoIIIAAAggggAACCCCAAAK1T6CU7V27+7INHNIFhqN07ILbtm3rLrroooxpvTTdkNaFQTCqrymbNHWTylVXXeXGjx/vl+0vTZmj/5C38uuvv/qMO2GAkbZZoEy5BA4pMEaBHGHRdFTqZLCpo5IyBcUDh/7+97+7W265JTqMssjIcOmll/brNM2VpnwKA3a0wTy0HA9wyBY4pM6SU045RbtERYELCu6yIl9NYaSRXFaUKWrrrbd28SAnbVfGldVXX92quqSgrzQDh5LOp2egQAgVWdiPVt8POOCAKhl+5B0GOmm6LJvqLa3AIf0O9DtKmuJLHVzKTmPvka47adouZesKM1HFA4c0xdpee+2l3X0p1sKOk+0zPj2Y1bNsYfZdn8rmc+qpp0YdebZNz0G/kfg0icpAFmZbUn0FjIVBN9UFDmlaPb3bVvRvzVlnneWU9SksDz74oGvatKkPXtT7ZUUBPTK2on/zNN1hmBXJAodUJx6so9+ZgnWWXXZZfwidXxmWwqw/mt5M/+aqFBo4FM9A5Q/y+19JwYfhdsuyFK6z5eeffz7qmNU6/d4LmXLNjsMnAggggAACCCCAAAIIIIBA5QlYn0xSkE+hGmFfgdrMf/3rX6scIswerIzF8UF/VXYIVhQaOKSp5tWvYCVbBmnbzicCCCCAAAIIIIAAAggggEDdEShle9dUyjpw6MMPP/TTVNnF2qf+c1/T7qjh//HHH7unn37aqW5YVGfy5MmuWbNmfnXS9D2qo0wlmlpq/vz5/j/W4/+Rr6xHJ598sj9GuQQOaQSTphULp17TBSojk/5TXcEEYTCKv/jf/tL9husVSKCOibAocETZfL7++munwJXnnnsu3OyXFydwSP6aZi4MdFAww/HHH+8U3KPsKzfeeKN74YUXMs6ndRtuuGHi1E26XwVw6FM/jlGjRmUcXwdKM3AoKWOMsjN17NjR34MCdBSwFha9Twr40LOQo7LIhEX+f/vb3/yqtAOHdJKxY8f6bDrhNSjIRUEoCmJS0JACicLnproPPPBA9NvS93jgUOfOnX1gke5T07MVa6FzVFcUPKjffLwoSKpNmzY+U5nuJx4YqPrHHXec/6NlTT+nrExhUZrxVVZZxX3xxRdOAW/xIMQwu1m2ICa9F3r2jRo1cpqKMf5vjbZbdi8FPYa/BWU803XVr1/fvfvuu/656f0JyyGHHOIDorQu229bdZSdTSMR4/uPHDkymj6w0MAhnVO/1ebNm/spx+rVq6dVvmhEpkZmZisWsJW0PZ5ViexESUqsQwABBBBAAAEEEEAAAQQQSBIoZUdqvK2vgXjrrbdedFoN8FEb9qWXXvLrFNSjwXD5lkICh9SOVpZl66sJBwLlez7qIYAAAggggAACCCCAAAII1F6BUrZ3TaGsA4d0kfoPfP2HeaHFsqqE+11//fUZ0yuF25KWFfSg4I2VVlrJby6XwCFdTFIGn6R7CNfFA4f+85//uIMPPjisktfy4gQO6cDx0VDVnaxTp04+o5HViwcR2Ppcn2kGDr3//vtOnTNh0fRwCg5R+fnnn33mF/vhhvWyLSs7i02JlVbg0JVXXhllbfn22299ZiDrbMp2XeH6MMjG1idlB9M2Bbxo+q1iLew8uT7nzp3rO+UKuRcdTxmFNApQQTkq06ZNiwJw/Irf/1JwUDxYz7YreM0ynsU7E61OdZ/hsw87DG0//X6VMSjMEmTb9KkOSRttqGxhXbp0yVo33E/L+p3o31kL+KkucCjXvx0KhlOWsLBoJGZSEKKyHg0dOjSsmrEc/80TOJTBwxcEEEAAAQQQQAABBBBAAIEcAtYfU4qMQzqNsgErs4/Kmmuu6bp37+6ngv/oo4/8oKyXX3452qbpzJZZZhn/PZ+/wn4A9aW0bNmyym6ffvqpU59D2C+gDMAayGMDJ6vsxAoEEEAAAQQQQAABBBBAAIE6J1Dq9q6Ayj5wSBep7ByacijfoqmeunbtWqW6gheUPWXKlClVtsVX6D/pNZVX+B/g5RQ4tGjRInf77be7ESNGxC89+r7zzjtnZBWJBw6pooJI9CdbURCLOiXCsriBQzrGkCFD8vJXSmcF4ay44orRqefMmeOz2MSzS0UVflvYaqutMgIU0gwcSnofFIQyZsyYKABDWYk0XV4+wSx6lnpmVmoicEjnevHFF/0UamHHk11D/FOZYxScYlNe2fZsQWEWOKR6xVjYear7VHabQYMGZbwDufbR+6EMT+pos6J/J44++ugqU/TZdn0mBRHpebVo0cL/Xg499NCwus+KpWvLVnr37u3Padv/+9//+gxd9j3pM34N6gjV9IsNGzb01TWVmjJyVVf0jPRvXZMmTaKq1QUOqaL+jU36Ler933bbbaNjaSHbv+FhlqOMHX7/QuBQkgrrEEAAAQQQQAABBBBAAAEE8hEodUeqspVrYEw4tXjSdSj79BZbbJG0Keu6MHAoa6XYBvVBqZ8t7NOIVeErAggggAACCCCAAAIIIIBAHRQodXtXRLUicEgXqkwfagxbyl+tixdN83TKKae41VdfPb4p4/tDDz3kg1KyBXOo4T148OAqo3VUXxkvwqIsI8o2Ehb9x/l1110XrYpnyfjqq6/crrvuGm3XgqYtWnnllaN18YxCCuBRRpJ4UWeF0iO/8sorUTaUjTbayB155JF+SqS+fftGu6gjQQEe8aLAoUmTJmWMWFIdZZZRAIRcw6Jr0zRWKsOGDfP72nZNcdW/f3/7mvj5zDPP+ICnbNlbBgwY4A444AC31FJLVdl/3rx57txzz3UKiogX7adgivCe1VETny4svl/4XVmOwgAaBS706NEjrJKxnBQ8oVFl4Wg2XbOyuWj6qKSi4BW922ussUbGZr1bulcrOqaOHZZ4ph9NcaVAJSualmr77be3r/4zzDhkG3TPCs5LcrU6CsjRFGRJRRlulNFLo9zC31UYOKT9Ftci6ZzZ1v3666/+t6KsY+G1hPUVRHf66ae7PfbYI/E902hBPZN4lhz9hvSOH3TQQVX+LZCNjJIyDun3pTTimiotLLoO/YYUKBcvzz//vL+GeHCO3nH9xjQFnJ53WOLPSMF2ekfi57V9FAg2cODAKqMg41Ol6Xy677D8+9//9lOmhVOqaXtS4JCeexgUp3q6d/27Z4FOWhcvOm84vWI4lV+8Lt8RQAABBBBAAAEEEEAAAQQQCAXS6Ej95ptvnPqf4m1hnVft9fPOO89nXw6vI5/lUaNG+cFAueqqHb3hhhu6DTbYwG266aZOWXyT+s5yHYNtCCCAAAIIIIAAAggggAACtV8gjfZurQkcssenAAVNEaWpcpSVo2nTpm6dddbxjfNC0vIquEDBAe+9957/07hxY58GeN11183IcmPnrQ2fCxcu9B0G1mlwxx13ZEz1pYAiZSTJVj755BNvscoqq7i11lrL/8d+trrFrpe/AiLeffdd/9m8eXPvr/MuvfTS1R7+iy++cK+//rpTh43SNyuwqkGDBtXutyQrKHhC767uecGCBf66FQwUBowtyevTuZVFSb8J/WMjW/225KugJpvKK9c16vf5+eefux9++ME/x0aNGlUJrNP+NWGhrFy6Ft3L22+/7adL0+9bHXn6kytgxe5RWZL0zH755Ren34/2t+m8rE78MylwSEFAuh69s/p3R++qnr1Sm9vvNX4cfVf2I+2j34pSnG+22WZVAhqT9gvXmYOeq949nVudjOutt16VzFHhfvkua4o4vSv6TesaFUgZz0ilY8WzGB144IE+eCvf81APAQQQQAABBBBAAAEEEEAAgUIE0uhItfOr/2TmzJk+u7L6ddR/0qZNm5xtfNuXTwQQQAABBBBAAAEEEEAAAQSKEUijvVvrAoeKAawr+7766qtu4sSJGbfTrVu3jBFNyjSjzDNh+uSOHTu6iy++OGM/viCAQGkFsgUOlfYstetoCuBSdqewXHPNNT4QKlzHMgIIIIAAAggggAACCCCAAAKlEkijI7VU18ZxEEAAAQQQQAABBBBAAAEEEFhcgTTauwQOLe7TWIL7ffzxx1WmjFLmkn79+rkmTZq4zz77zCnbUDxtcnwaoyV4C5wagTorQODQ/x7tXXfd5bMR3XfffT7zk21RtqVbb7212uxNVp9PBBBAAAEEEEAAAQQQQAABBAoVSKMjtdBroD4CCCCAAAIIIIAAAggggAACpRZIo71L4FCpn1INHa9Xr15VAoNynVqBRTfffHOq04/lOj/bEKgUAQKH/vek27Vr978vwdK5557runTpEqxhEQEEEEAAAQQQQAABBBBAAIHSCqTRkVraK+RoCCCAAAIIIIAAAggggAACCBQukEZ7l8Chwp9DWewxb94817dv34ypyLJdmKYoGzx4sFtxxRWzVWE9AgiUSIDAof9BJgUOrb322j7bUIMGDf5XkSUEEEAAAQQQQAABBBBAAAEESiyQRkdqiS+RwyGAAAIIIIAAAggggAACCCBQsEAa7V0Chwp+DOWzw8KFC91TTz3lNB3Qa6+95r788kt/ccsvv7xbf/31XatWrVzr1q3dXnvt5ZZaaqnyuXCuBIE6LDBnzhx34oknRne4zDLL+Gxf0YoKWggDhxQwtM0227jjjjvOrbLKKhWkwK0igAACCCCAAAIIIIAAAggsCYE0OlKXxH1wTgQQQAABBBBAAAEEEEAAAQRCgTTauwQOhcK1fPmXX35x8+fPd40bN67ld8LlI4BAXRBYtGiR079L9evXd/Xq1asLt8Q9IIAAAggggAACCCCAAAII1BKBNDpSa8mtc5kIIIAAAggggAACCCCAAAJ1WCCN9i6BQ3X4heHWEEAAAQQQQAABBBBAAAEEEEAAAQQQqESBNDpSK9GRe0YAAQQQQAABBBBAAAEEECgvgTTauwQOldcz5moQQAABBBBAAAEEEEAAAQQQQAABBBBAoEiBNDpSi7wkdkcAAQQQQAABBBBAAAEEEECgaIE02rsEDhX9WDgAAggggAACCCCAAAIIIIAAAggggAACCJSTQBodqeV0f1wLAggggAACCCCAAAIIIIBAZQqk0d4lcKgy3yXuGgEEEEAAAQQQQAABBBBAAAEEEEAAgTorkEZHap3F4sYQQAABBBBAAAEEEEAAAQRqjUAa7V0Ch2rN4+dCEUAAAQQQQAABBBBAAAEEEEAAAQQQQCAfgTQ6UvM5L3UQQAABBBBAAAEEEEAAAQQQSFMgjfYugUNpPjGOjQACCCCAAAIIIIAAAggggAACCCCAAAI1LpBGR2qN3wQnRAABBBBAAAEEEEAAAQQQQCAmkEZ7l8ChGDJfEUAAAQQQQAABBBBAAAEEEEAAAQQQQKB2C6TRkVq7Rbh6BBBAAAEEEEAAAQQQQACBuiCQRnuXwKG68GZwDwgggAACCCCAAAIIIIAAAggggAACCCAQCaTRkRodnAUEEEAAAQQQQAABBBBAAAEElpBAGu1dAoeW0MPktAgggAACCCCAAAIIIIAAAggggAACCCCQjkAaHanpXClHRQABBBBAAAEEEEAAAQQQQCB/gTTauwQO5e9PTQQQQAABBBBAAAEEEEAAAQQQQAABBBCoBQJpdKTWgtvmEhFAAAEEEEAAAQQQQAABBOq4QBrtXQKH6vhLw+0hgAACCCCAAAIIIIAAAggggAACCCBQaQJpdKRWmiH3iwACCCCAAAIIIIAAAgggUH4CabR3CRwqv+fMFSGAAAIIIIAAAggggAACCCCAAAIIIIBAEQJpdKQWcTnsigACCCCAAAIIIIAAAggggEBJBNJo7xI4VJJHw0EQQAABBBBAAAEEEEAAAQQQQAABBBBAoFwE0uhILZd74zoQQAABBBBAAAEEEEAAAQQqVyCN9i6BQ5X7PnHnCCCAAAIIIIAAAggggAACCCCAAAII1EmBNDpS6yQUN4UAAggggAACCCCAAAIIIFCrBNJo7xI4VKteAS4WAQQQQAABBBBAAAEEEEAAAQQQQAABBKoTSKMjtbpzsh0BBBBAAAEEEEAAAQQQQACBtAXSaO8SOJT2U+P4CCCAAAIIIIAAAggggAACCCCAAAIIIFCjAml0pNboDXAyBBBAAAEEEEAAAQQQQAABBBIE0mjvEjiUAM0qBBBAAAEEEEAAAQQQQAABBBBAAAEEEKi9Aml0pNZeDa4cAQQQQAABBBBAAAEEEECgrgik0d4lcKiuvB3cBwIIIIAAAggggAACCCCAAAIIIIAAAgh4gTQ6UqFFAAEEEEAAAQQQQAABBBBAYEkLpNHeJXBoST9Vzo8AAggggAACCCCAAAIIIIAAAggggAACJRVIoyO1pBfIwRBAAAEEEEAAAQQQQAABBBBYDIE02rupBA4txr2xCwIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCFQj8Mc//rGaGvlvJnAofytqIoAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCwRAXKPnDoD3/4wxIF4uQIIIAAAggggAACCCCAAAIIIIAAAgggULkC//3vf/3N009Zue8Ad44AAggggAACCCCAAAII1EUBa+8SOFQXny73hAACCCCAAAIIIIAAAggggAACCCCAAAIlEbCOVAKHSsLJQRBAAAEEEEAAAQQQQAABBMpEwNq7BA6VyQPhMhBAAAEEEEAAAQQQQAABBBBAAAEEEECg/ASsI5XAofJ7NlwRAggggAACCCCAAAIIIIDA4gtYe5fAocU3ZE8EEEAAAQQQQAABBBBAAAEEEEAAAQQQqOMC1pFK4FAdf9DcHgIIIIAAAggggAACCCBQYQLW3iVwqMIePLeLAAIIIIAAAggggAACCCCAAAIIIIAAAvkLWEcqgUP5m1ETAQQQQAABBBBAAAEEEECg/AWsvUvgUPk/K64QAQQQQAABBBBAAAEEEEAAAQQQQAABBJaQgHWkEji0hB4Ap0UAAQQQQAABBBBAAAEEEEhFwNq7BA6lwstBEUAAAQQQQAABBBBAAAEEEEAAAQQQQKAuCFhHKoFDdeFpcg8IIIAAAggggAACCCCAAAImYO1dAodMhE8EEEAAAQQQQAABBBBAAAEEEEAAAQQQQCAmYB2pBA7FYPiKAAIIIIAAAggggAACCCBQqwWsvUvgUK1+jFw8AggggAACCCCAAAIIIIAAAggggAACCKQpYB2pBA6lqcyxEUAAAQQQQAABBBBAAAEEalrA2rsEDtW0POdDAAEEEEAAAQQQQAABBBBAAAEEEEAAgVojYB2pBA7VmkfGhSKAAAIIIIAAAggggAACCOQhYO1dAofywKIKAggggAACCCCAAAIIIIAAAggggAACCFSmgHWkEjhUmc+fu0YAAQQQQAABBBBAAAEE6qqAtXcJHKqrT5j7QgABBBBAAAEEEEAAAQQQQAABBBBAAIGiBawjlcChoik5AAIIIIAAAggggAACCCCAQBkJWHuXwKEyeihcCgIIIIAAAggggAACCCCAAAIIIIAAAgiUl4B1pBI4VF7PhatBAAEEEEAAAQQQQAABBBAoTsDauwQOFefI3ggggAACCCCAAAIIIIAAAggggAACCCBQhwWsI5XAoTr8kLk1BBBAAAEEEEAAAQQQQKACBay9S+BQBT58bhkBBBBAAAEEEEAAAQQQQAABBBBAAAEE8hOwjlQCh/LzohYCCCCAAAIIIIAAAggggEDtELD2LoFDteN5cZUIIIAAAggggAACCCCAAAIIIIAAAgggsAQErCOVwKElgM8pEUAAAQQQQAABBBBAAAEEUhOw9i6BQ6kRc2AEEEAAAQQQQAABBBBAAAEEEEAAAQQQqO0C1pFK4FBtf5JcPwIIIIAAAggggAACCCCAQChg7V0Ch0IVlhFAAAEEEEAAAQQQQAABBBBAAAEEEEAAgUDAOlIJHApQWEQAAQQQQAABBBBAAAEEEKj1AtbeJXCo1j9KbgABBBBAAAEEEEAAAQQQQAABBBBAAAEE0hKwjlQCh9IS5rgIIIAAAggggAACCCCAAAJLQsDauwQOLQl9zokAAggggAACCCCAAAIIIIAAAggggAACtULAOlIJHKoVj4uLRAABBBBAAAEEEEAAAQQQyFPA2rsEDuUJRjUEEEAAAQQQQAABBBBAAAEEEEAAAQQQqDwB60glcKjynj13jAACCCCAAAIIIIAAAgjUZQFr7xI4VJefMveGAAIIIIAAAggggAACCCCAAAIIIIAAAkUJWEcqgUNFMbIzAggggAACCCCAAAIIIIBAmQlYe5fAoTJ7MFwOAggggAACCCCAAAIIIIAAAggggAACCJSPgHWkEjhUPs+EK0EAAQQQQAABBBBAAAEEEChewNq7BA4Vb8kREEAAAQQQQAABBBBAAAEEEEAAAQQQQKCOClhHKoFDdfQBc1sIIIAAAggggAACCCCAQIUKWHuXwKEyfQF+/PFHt+yyy5bp1XFZCCCAAAIIIIAAAggggAACCCCAAAIIVIaAdaQSOFQZz5u7RAABBBBAAAEEEEAAAQQqRcDauxUROPTrr7+6yZMnu19++SXn823YsKFr0qSJa9WqlVtzzTVdvXr1stZ/8skn3eeff551e7YNq6yyittpp50yNi9atMg98cQT7vHHH3dz5szxf77//nu3/PLLu5YtW7oNNtjA7b///m5xH9bcuXPdo48+mnHOjh07uhYtWmSssy96OZ566in76j87derkVl111Yx15fzl7bffdjNnzowusWnTpq5Dhw7RdxYQQAABBBBAAAEEEEAAAQQQQAABBBDIR8A6UgkcykcrnToLFixwDz/8cMbB1de34oorZqzL9mX69Onuyy+/jDZvuummbt11142+qy/0m2++ib4nLTRu3Njvs8Yaa7j69esnVSl43fz5893QoUOd7m/QoEF530/BJ6qQHW677Tb37LPPus6dOzv1f1MQQAABBBBAAAEEEEAgt4C1dxc3FiXp6PV+C4BZlLRhcda98847frdSNMjV8Cq0obDhhhu64447zm233XaJl//Xv/7Vvfzyy4nbcq1cb7313IQJE6IqCnD5+9//nhHkEm2MLSh456STTnIrr7xybEvury+++KLr27dvRiWd889//nPGOvsybdo0d9ZZZ9lX/3nVVVc5NahrS7n55pvd6NGjo8vdbLPN3Lhx46LvLCCAAAIIIIAAAggggAACCCCAAAIIIJCPgHWklqKfUuf74Ycf3LXXXpvPqX2fZlKfnAZI3nfffe65555z6kf99NNP3VprreUDW3bccceC+0LtYkpxbXYs+zz33HP9QMm9997bD4609YV8fvXVV26vvfbK2OWMM85wXbp0yViX9EWZ3XfeeeeMTSeccII75JBDonXdunVz7777bvS9ugXVP/7444sOIFL/pfoxt9lmGzdy5Mgqp/3oo4/cpEmT3Jtvvuk0OHSdddZxbdq0cerr3Hjjjd1SSy1VZZ98V+gd0kBWDWj94IMPXIMGDfxxdWydQwNgsxWZ/utf/3JvvPGG79fWYNjVV1/dv4Pt2rVzu+22mz9etv1/+ukn995777mFCxf6Abz59HdfdNFF3kGfOldSeemll1yfPn38gNw77rjDKdiLggACCCCAAAIIIIAAAtkFrL1L4FB2I7/l6KOPdj179qxSqxSBQ2rwHn744U6f+ZbNN9/cXXbZZTkbXvFjETjkfKOXwKH4m8F3BBBAAAEEEEAAAQQQQAABBBBAAIHqBKwjtVSBQwoCOfLII6s7rd8+YMAAt88++2TU1fWcc845bvbs2Rnrwy9t27Z1F1xwQcGBE8VeW3gNWta1HnTQQX61+kF79+4dr5LX96TAoS233NKNGTOm2v0VGHPmmWdm1Cs2cEgH0/mHDx/uVlhhhYxj5/vlP//5j+vevbuvPnbsWKd+37Dcc889TkEy2YoCn3Qfi1PmzZvnTjzxxKzvkDLhX3PNNW7ttdeucngFDSloS5l9spXVVlvNjRgxwmfSD+soWOmmm25yut+w7LDDDu7000/3swGE623Z+rc1EFYDYnMVDaBV/a5du/pj5qrLNgQQQAABBBBAAAEEKl3A2rsEDuXxJpx66qluv/32y6hZisAhHVcjM5KKUt5qRElSOeCAA1z//v2TNiWus4ZVuJGMQ6EGywgggAACCCCAAAIIIIAAAggggAACCCQLWEdqqQKHkrJ9J5/ZuXjgkDKrK0t6GDSkICFlbFHmoTBrTvv27d2wYcNcvXr1sh2+yvpiri1+MGWTUTCIpglTKXXgkI559913u6ZNm2oxa1HGo0cffTRje67AoYsvvrhKRhtlYvrss8/cAw884J588snoWL169XJHHHFE9D3fBSXuVxCVMtonBUA9/fTT7rTTTss4nOrpXdR1WMk26NW2J33quahv+9VXX402qy9aWXzUj2xFGYeuvvpq16JFC1vllClIfdovvPBCtE5BQi1btnQzZsyI1mlBx7zuuutco0aNovUKhFJAVFLR+W655ZYq07X9+uuv7phjjvHv/MSJE12rVq2Sdo/WWdYhrRg/frxr3bp1tI0FBBBAAAEEEEAAAQQQyBSw9m7FBg4p+EZpU1XUUNMc12pYK5AnHrCjERb333+/a9iwYaQYDxzS6AsbIRJVSlhYaaWVnEZQfPvttz5la1hFKXOV5naDDTbwaW41x7XSDcenDdM+aqDmO5c2gUNkHArfM5YRQAABBBBAAAEEEEAAAQQQQAABBPIXsI7UUgUO3Xjjje7yyy/3F6BAiWWWWSbrxagvcdlll422K1BGwUAqCthQxp3wuhQgo0AZK1dccYWfdsq+V/dZzLXp2OpXlZemsFJ/qqbAspJG4FDSgE87nz4V8LPLLruEq/xyrsChyZMnZwTLhDurH1lTimn6MBUFzMis0KJgqlNOOcXvdskll7htt902OoSy8hx88MFRH7X6jJVhyt6TMDBGO8X3jw6UZeHBBx90559/frQ1zHakbELnnXeen75MFTQd2j/+8Y+o7kMPPeS3a4X6zC+99FJnU+n9/PPP7vnnn/fv3/fff+/3CQPftM0yJOmZKDOQgop0PQrWUtHgXT3TsNg73blz5yqZo8J64fJRRx3lA4023HDDvKcFDPdnGQEEEEAAAQQQQACBShGw9m7FBg6pAbTrrrtWed4K1hk0aFCVTEBqcO++++5R/XjgkIKBlH4135IUzKO0uZpLOl6S0ulqtIYCjPIpSecqVcahTz75xM2aNSsa7aJ5o5s3b+7WX3/9vEZzfPjhh05pefVHI6bU2NZ9aU726uboViNanQ9KoazjrLnmmv6cGs2iucE1R7gVzc3NVGWmwScCCCCAAAIIIIAAAggggAACCCCAQL4C1pEaBujku29SPU1vNWXKFKcMK/fdd19SlazrNP2Y7aPMQ0lTnimQxAJb+vTp4wcqZj1gbEMx16ZDaSqpbKVUgUMawGkBSdX1+U2dOtUNHDjQX1K43+IGDulACopSph8r2TLK2/b4p4KPevbs6V5//XUf/HXHHXdk9INaoIz2U1+pAnfC4DGtV8DZZZddpkXfZx0Gi/mVOf7SgFobOKt+8N122y2jtga8aho0TQ+nonPZlGUnn3xylFkoW/9yGJi07777RpmTFMR2ww03+Hu6/vrr3dJLLx2dV/eoP/HfhIKRNNWdsizddddd3ivaKceCfiP6ragouE7ZmigIIIAAAggggAACCCBQVcDauwQOVbVx3333nevUqVPGlvXWW89NmDAhWlds4FA4OkMH1QgNrUvKIqSRHuoIUKCMFTX6czXErZ4+0wgc0mgdNfBCk/CcWlbDVil127RpE9/kNOpEI6vuvPPOKtu0Qh4adaNgraR0ynPnznVnn322mzlzZpX91SBV8JCN3FKFpE4EjR566623ov0VsBSfki7ayAICCCCAAAIIIIAAAggggAACCCCAQEUKWEdqqQKHlGlF/XXbbLONz15TCKqmbFLAiUq2gYVhViJlSFc/Zr6lmGvTOXL1V5YqcEj9fMpUYwE76l/UQMakov7Dxx9/3G9SsI6m3lIpJnBIU8LpXlTigS5+ZTV/aRCmpjhT0bOJZ7EPg8P69+/vFOgTL/Fs9rrHeHBRfB9917usQByVXP3RFuSjeuF0bPvss080VdrDDz+cMQ2Z6qpokKcFtIXvuDIJ6ZkpKMkyD/3/Hs738R5//PH+a3jc2267zY0aNcopg9Cxxx5r1av9DDNNdezY0WmKNAoCCCCAAAIIIIAAAghUFbD2LoFDVW38msGDBzs1UsISNsCKDRxSA18N/bDsuOOOTg0kG8ERbitmOY3AoaS5wbNd4zXXXOM22mijaLNGiCgQKpyPO9oYWwgbprZJU8qpE8NGvdj6XJ9JgUMKTLI51rVvoVmjcp2PbQgggAACCCCAAAIIIIAAAggggAACdUPAOlJLFTi01157+X6tpACK6sQ0ZdVjjz3mq2m6qb/85S9Vdgn7Nc844wzXpUuXKnWyrSjm2nTMzz//PGPwozKpazorlVIFDmlqLAXTKKO8ykknnRQFw/gVv/+lgYvms/nmm7u99947ykJTTOCQsuYosEalffv2TlmaCikKXlJ/qYqy7Gg6sLDYM9C6W2+91WdmD7fbsk3Hpe/jx4/PK/v7Aw884IYMGeIPoQGb2TIVKcDHpgwLg3/Un61Brsr8ni37fpgxKcw4pAxFyq609dZb+2Aguw993nPPPT64R8FMjzzyiB9IquAo7a+ibEMKFiukhP33yjxlU70VcgzqIoAAAggggAACCCBQ1wWsvUvgUJYnrWw0NreyVdEIB+sgCBse2q75kuMjJWw/+9TIF2XCUVEDS/NTJxUFDqlBpkw9f/rTn1yzZs2SquW9rtSBQ9OmTXNnnXVWxvnVqFMDXJ0Db7/9dsa2tm3bZkwbFnZeZFTM8kWN6U022STaGqbEjVZWs0DgUDVAbEYAAQQQQAABBBBAAAEEEEAAAQQQSBSwjlTrF0yslOfKMJjFMs3Mnj3bvfLKKz6wQYPvFBijrDBJgRLhNFDqj1NGHfUjLrfccj5gR/2Xo0ePjq5G2YeaNm0afc+1UOy1JR07DCIpVeCQ+mE1TZdNsaXAGwXgxEt4bgXBaGqsYcOG+WqFBg4pg83777/vnnnmGXfllVdGp7r22mt9v3C0Io+FMOBHgVUNGjSI9vr666/dnnvu6b/r+eoeshULxNF2BS8piKm6IjdNPaYyYMAApwxCSUUDPm1bPBN/Un1bpynQZGtToV144YVOg2VVwgz8/fr185mHlGleU79pkKjev3Bg59ixY93EiRN9Vvr999/fH6OQv2z6M+2j+27Xrl0hu1MXAQQQQAABBBBAAIGKELD2LoFDWR63sgup4R0WTX21xRZb+FXxwKGwXrbleONYjfibb745W/VovYKNlFK1Q4cOfsqtpKm7osoJC6UOHAobpTpdfGRNOMe2XY5GQqkD4+WXX66SHlnTgynT0k8//eQbw1dddZVvKNq+2m4jXMJUvrZdKYHVIFWA1dNPP+3niFdDMywEDoUaLCOAAAIIIIAAAggggAACCCCAAAII5CtgHamlCBzSgLsePXpUe2r1d2ng3vbbb59Rd9GiRT5jjAKIwqJgGgV7hBm6lZFn1113DavlXC722pIOHgbvxPtGk+pnW6f7UiYeFd2rAnZOP/109+STT/p1kyZNigZs+hW//XXmmWc6BeaoKIBKWXTyCRxSwE6YnUZ9lvG+RtVR5p7tttvOHz/fv8L70CBMBceERe+aTSWmwaUW5BPWsWUFMGm6OpV8M0sNHTrU3X///X4fTYm20047+eX4X+FUX7kCmOSiLELz58/3U5SF2d01aFZG1pf966+/ut69e/tpyXS+1VZbzU+v9sEHH0SntwxLn376qc82pH5x9Z+HwVVR5WoWnnvuOXfiiSf6WsW8e9Wchs0IIIAAAggggAACCNRqAWvvEjiU5TGqkaOprMKigBmbp7sUgUO//PKL7wCwBm54rmzLalCq0VvIdGalDhxSkM/MmTOjS9S1qNGp0ScqalhqFIc6MqwcffTRvjE4cuRId/vtt9tqP5e2UuSGjT8bTWKV1Di97777fINd81ErdW1YwkxQWq/godNOOy2s4gOuxo0bl7FOAUqvvvpqtK5169ZVpo+LNrKAAAIIIIAAAggggAACCCCAAAIIIFCRAtaRWorAIfUDKuAlLMqYo341BUs8++yz4SYfIKNAmbCoT7Fnz55OmYqyFU3RNWjQIFe/fv1sVaqsL8W1xQ+aZuBQeGxlrDnssMOi03/33XeuU6dO/vuWW27pxowZ44OH8gkcig5SzYKyomsqLWUyyrcou476SVU6d+7s+3nDfV9//fWofzJpIGRYVwE1ll2qV69e7ogjjgg3Jy5rcKYCqFSqy8Jj/eCqq3cj6V1SZiFNGxcvLVu29EFNYZ+v6nzxxRc+O9JTTz2VsYuCiPS+2qBdBTWpPzhXcFPGARK+hEFYCvCKzy6QsAurEEAAAQQQQAABBBCoOAFr7xI4lOXRa6SEAoXCormnlS5YpRSBQzqORlr885//9MEw4YgMbctWNNJCo0mS0hUn7VPqwCGNVlJq2XhRA2+rrbbyU5apQW7TsoX14tOMaU5rpecNy3vvvVdlbnDNha5jx92zpcqNn6e6hnZ4fpYRQAABBBBAAAEEEEAAAQQQQAABBBAwAetILUXgUBjsoX6tSy65JGMqsXfffdcHUCj7j4oG7Gm6JgvAUKDGOeec4xRgYkWD7po3b+60b1jUH6bja3s+pdhrSzpHGNxTTNaXMFOPZRxSthsFSKnYOruGRx55xDvqu2XjUdahfAKHFHC00kor2aH8p871zjvvZLhrgwZ5Kkv9UkstlVE/2xcFhp100kl+c7du3VyfPn0yqob9uArcifdPh5WVnUd9pirHHXecO/LII/1yrr/CvlVN5aWgtWwln8Chzz//3B166KFVMjLpmHq/FQzUqlWrjFNosOlrr73mMxTJVf9BIUd7Ty3zla7t6quv9hmLtM/DDz/sZsyY4adB0/R7bdq0cV27do1+Gxkn+e3LN9984/bYYw+/OlsfcnwfviOAAAIIIIAAAgggUGkC1t4lcCjLk1ejWiluwxLOCR42slRHQTOWLjfcJ1xWA0iBMtmKMvW89NJL7t///rf/DLPhxPcJp++Kb4t/Dxucti3MnmTr7HPatGk+E5J916ey82h+dZWpU6e6gQMH+uVcf6m+goLCRqbmxlba5EKLGpmat1zGYcrlpAa2jq3sQhMmTIhOQ+BQRMECAggggAACCCCAAAIIIIAAAggggEABAtaRWorAIU3LpAAf9QO2b9/erbDCClWuJJ7FJRzMeOyxx0YZtBVUpOnM1O+logGKCpj529/+FgVyKKijX79+Vc6RtKLYa0s6ZpqBQzqfgqgee+wxf2pN62VZ2sNpzO6991636qqr5p1xaPLkya5FixZJt+MzrN95550ZAT3K4qO+2nyKBmNqUKZKUpYgC5rRdsuUpOWkoqAavRsqAwYMcOp3ra7ofVHfr8oVV1zhg2/8l9hfP/74o9NUYyoK6NFzzFX07qnP9plnnvFZsvQOq2jKvbvuuitrcE/SMW1AqLLSqz9d16IM/AoaihcFBCmTULNmzeKbnDJz6TemoutQBiMKAggggAACCCCAAAIIZApYe5fAoUyX6Fs8MEgbwpSs8e077LCDGzFiRLR/KRY04kJBRGpcxdO3qhGca47r8PyzZs3yDdFwnaYWs8ZfuF7LYQPWtmne8DAt8vXXX+/Gjx9vm3N+du/e3WcKUgNSTotT1KhV4FDHjh0zds92H1OmTMnIWkTgUAYbXxBAAAEEEEAAAQQQQAABBBBAAAEE8hSwjtRSBA7leUp3/vnnuwcffNBXV7/Y3nvv7QOGFDhk5fbbb3drrrmmfY0+FbzRv3//6PsTTzxRUOBGtGOWhaRry1LVB5yce+65fnOpMw7poOEAyOOPP9716NHDffvtt74fUds1iHPUqFFaLEngkD/Qb3/deOONPtOQvm+zzTZu5MiRtinnp4KSbMosZR466KCDMuprKi9NYaZSXZYcDQxV1nyVCy+80O24445+Oddf4WBZBZhtv/32idXnzp3runTp4rcV0g+tHT755BOnZ62+bZVcA1h9heCv5557zp144ok+4Gf48OF+i6ZUs37wtm3b+qz0b731VhQwJn/dV7169YIj/f9iOKDVpmirUokVCCCAAAIIIIAAAghUsIC1dwkcSngJ4qN6VEXZc5R1x0qxgUMapaEpuaysu+66UWpdWxd+ho06W6/RNMstt5x9zfqp0UKHHHJIxvbTTjvNz8GdsfL3L2qIqUEWFo2kUcrjsChzkBqnaqDrHLmKjRCJZxxSQE88XW3Scfbcc0+3ySabVGnMarSJNabD/W644QY/asbWEThkEnwigAACCCCAAAIIIIAAAggggAACCBQiYB2pNRk4FA7as2molDlHASIqu+yyixsyZEjW2zjggAP8lE6qoCnI1llnnax1C92QdG3ZjpF2xqEwM07Lli19QE84KFJZ0y1LfL5TleXKOGT3+cYbb7ijjz7afy0km80999zjLrroIr+fgsCUrT0sCxcudB06dPCrlOlH03NlmwYtzLaUK3tQePzrrrvOXXnllX6Vgsv0niQVTcum4B8Vy3z08ccfO5tCT/25a6yxRtKuft0FF1wQZfhRBqEDDzwwa13boEGnmm5N57jpppuc+svDILCDDz7YnXDCCVGAkPqrFfykkjTtWjiIVbMFaHAuBQEEEEAAAQQQQAABBDIFrL1L4FCmi1uwYIHT6JRwnnBVCRuZ+l5s4JBGoWhUkBU1BB944IGso3/Chq3to4Cdhg0b2tesn/PmzXO77757xnbN/63RQUklTOVr2x9//HG37LLL+pTH3333na32n40bN/YjSV5++WX3/PPPR43CsJIa0j179nSnnHKKmz59erRJI0jU6Mu3qDE9e/bsqLpGvmie8nhRet4wSxOBQ3EhviOAAAIIIIAAAggggAACCCCAAAII5CNgHanFBg5pejJlF1dp0qSJ22CDDbKePsyyoj489eUpAGj06NF+n+qmIFM2m2effdbXTQqqiJ+42GuLH8++px04pPNo6i8FC6lYJiDLLnP//fe7lVde2W8L+1cVgBIOtOzWrZufQk4V8wkc0vE1RZlKIYFD6s9VBikVnV/XES/hwEvdjwKi4kXTcCk7u2X10RR1SdPexfcLg6pyBZ9psKgyBalYv66m+lJAkMoxxxzj//gvCX+F06ipP3j//fdPqJW5yq5t3333dRr0qqL+ZvXDqyiIqlGjRn5Zfy1atMhn4tIUaeoftgxJVkF92J06dfJfN954Yx9cZNv4RAABBBBAAAEEEEAAgf8XsPYugUO/vxEKGNKUXhqpYPNihy+LGrkK7rFSbOCQgoTio4Kyjb5Qw10jQNRQslJIY0eNKI3qsLml7RhJKWzVyBw0aJBV8Z9hOlrNwa6GdFjCeda1Xg1WnU+NNisa2aPgq7DTQ9uUclejXBSUZEWBQRMnTrSv/lOdHU2bNs1I02wV4tmQwhE/VicpcOjFF190yppkRRmVtthiC/vKJwIIIIAAAggggAACCCCAAAIIIIAAAs46UosNHPr000+jDODKgKI+raTplRQUooF21penLDEKMgqDVcIpuJIeURh8kk9QSbHXlnQNWlcTgUMaPKhBhCrKoDNp0iS/vN1220XTgmlFKQOHlOlGz08lfh6/Mstfb775ps+qo80a6GnTuIXVx40b5yZMmOBXKQOPMk7Fi03ppfWFnD/M4KO+7ttuu82tuuqq8cP7gbUzZ87065WFX9n49V0DblWqe//CQZ1jxozxWYv8jln++umnn9x+++3n+5OVlUmBdSrhfU6dOtUts8wyGUewzFoK4tL+YdGUabZuhx12cCNGjAg3s4wAAggggAACCCCAAAK/CVh7t2IDhzQSRA10K2EWG1tnn0lpY4sNHJozZ05iph01YjS3dIsWLZxGRXz44YduypQpUUeBXVOhc4JrZJH+xItGlmy00UZ+hIYCbpRZKF50r927d/erFWDVsWPHjCqaK1pThlmDTtOWaSSKjXhRZUupHDb27CDaX/N5K2XyjBkznAKRwoAeNWI14qR+/foZjUXbX8+xb9++rlmzZn76N428Cs+tekmBQ/HsRzQgTZRPBBBAAAEEEEAAAQQQQAABBBBAAAETsI7UYgOHdDwLdNDy0KFD3c4776zFjHLLLbf4wXdaqQF9yjqjfrEwuEfbLr30UrfVVltpMaNY5hat1HRSFkij7wpGsvtRX1qYzaaYa9Oxk0pNBA79/PPPbs8996zSH6ipvPbYY4/oskoROKTprzRVm4JprFhGKPue6zPMFJRtYKim6urRo0d0GA2w1NRgVuKDTJU9vmvXrrbZ9ym/9tpr/rsCbTbffPNomxbOPvvsqA84KSt9GIilPnQF8mi6tDDoSMe5/PLLEwdhhgFuqhdmfdL3pKKpyRRgpIz1NgWc6n3xxReuc+fOfhcNLFUfshVltVf/rsrYsWOr3KcGjarPWCVbdie/kb8QQAABBBBAAAEEEKhgAWsfVmzgUL7PXqMnLr74Yt84D/cpNnBIx1Kj/ZJLLgkPm9eyOgyuvfZat9xyy+VVX5UUiHPYYYdVaUBXdwA1DjXCxYKCVD+cPzvcX3U1OiQetKM6YQM3HJET7p9tOd74jQf8ZNsvXE/gUKjBMgIIIIAAAggggAACCCCAAAIIIIBAvgLWkVqKwCENmNM0TlY0YNEGEX788cdO00HdfvvtttlPF6VBd1Y0zZWmu7LSu3dvv78ydX/55Zd+8J0yFFmJZxzXNmX/VlGwjYJIrBR7bXac8LMmAod0Pt3nvffeG57aW6y44orRunwDhzRoMtxPB1i4cKEPulJQT9j3ueWWW/ogLwXW5FvCbDzxLPd2jHCqOQV/qV+3bdu2Tu+InqFlA9I2BZaFmXg0HZ7eCxX11+qdCku4Xes1xZem9GrcuLEftGnT4Wmb3re9995bi77o3dH5rfTr188HD+n9U5CP7kfXY6W6Kc1U75tvvommMtMzCrP+a3v4ziu4Tf28GrhqvyNltNeA2QYNGqh6VNQfrYAiFU27Fv6OokosIIAAAggggAACCCBQ4QLW3iVwKMeLoEaTGmnhNFpWvRSBQzqWGi2aMzrfosbeFVdc4bPz5LuP1QtHWdi6XJ9qpI0cOdK1bt06o5pGlygDUZgVKKNC7ItlG7LV8+fP93NTq6FdXdF5lAI3TNusl1fPxdI1Jx1DDekXXngh2kTgUETBAgIIIIAAAggggAACCCCAAAIIIIBAAQLWkVqKwCFlnFHmbmV1qa4kZUlRv5qCMRQ4UV1RMMjgwYMzqinoI1vgULHXlnGi37/UVODQM8884/r37x9dQlJm8XwDh6KDVLNw6KGH+n7LeMBKNbu5W2+91WeLUj0F6agfM170nE844QT3+uuvxzdlfE/KtBMGBiUFDukAmr5u0KBBGceKf5HhsGHDfLYh26YM+Zo+LVe/rNXV9Gbqx67ORwY333yziw8eteN89dVXvi856Z1X//X48eMzMjLZfmGA1mOPPVZE4y/WAAAPF0lEQVTQIFw7Bp8IIIAAAggggAACCNR1AWvvVkTgkEaEdOjQodpnqoaGRmm0adPGp7FVuthsRaMpwsCU9u3bu+HDh2ernnO9pgjTiJ5cHQa6tqOOOsqPvkgKZMp5gmCjUhor7asaS7mKOhb69Onjp/9KqqcRJGqUxUfyhHWVGUlBQzvttFNG4I/qqCNCU7Cp8RiO0rH91agdMmRI1vmv582b55QGWKlv40UZidZaay0fXGTblJLXRpjYurDxqHUaTXTRRRfZZj4RQAABBBBAAAEEEEAAAQQQQAABBBCIpvYqReCQOH/88UefVUhTXiX1i6kfUP1e2TKkaFCfpnbS/klF+yvoRJli4plwwiws2q5sLmEp9trCY2l56tSpbuDAgX61pt/SAMHFKcpKY9OOJU3xtWDBAr/dPBUUs9tuu2WcSv2YykykoiAjZa+xomurbpCj+it1bk0bpini2rVrZ7sX9PnOO++4ww8/3O+jabk0PVdS0T3LToNB40X9rsoKr+CceJk1a5br1auXX50tcEgb5TFq1KjEd1BBa8patPTSS8cP7zRVmvqX77zzzirbtELvn57zPvvsk7h/uJMCkPQcdD833HBD1vp6L/Xuqi949uzZvr78ZbfyyiuHh/TLusZddtnFL2+33XZ+RoEqlViBAAIIIIAAAggggAACUXu3IgKHasvzVjphpZtVcI/+NGzY0LVo0SL6U0zAUNxAjehXX33VZw3SqA11IqghqbSyf/rTn/IegaFRJm+++aZ77733fMpeXfNqq63m/+jlindOxK9DU5vNmTPH76/rUAeM5lZv3rx5lWCj+L76rgAmNRbVkNZ+arhXN4ol6TisQwABBBBAAAEEEEAAAQQQQAABBBBAIEnARmCWKnDIzqGsMgpW0fEVGKGBcPqjfrHq+tR0jK+//tq99dZbfn/1K2o/BWCoT65Ro0Z2msX6LPbaFuukFbSTDUpVf6wyIdWvXz/r3av/9ZVXXvF9sAqS0TNWdvV83pGsB/19w6JFi3y/rIKNFJCmvmgNwGzSpEl1u7q5c+f6fl1lAlJfc7NmzXzfrq5PwUP5FA2oVbYoBWIlBUHlc4ykOpqe7YILLvCbFBy19dZbJ1VjHQIIIIAAAggggAACFS9g7V0Chyr+VQAAAQQQQAABBBBAAAEEEEAAAQQQQAABBLIJWEdqqQOHsp2P9XVfYPr06U5Z01WUxV7Z7CmlE9B0fprmLSk7VenOwpEQQAABBBBAAAEEEKj9AtbeJXCo9j9L7gABBBBAAAEEEEAAAQQQQAABBBBAAAEEUhKwjlQCh1ICrtDDnnzyyW7GjBk+e9C4ceMqVKH0t62p3fr27esPPHr0aNe2bdvSn4QjIoAAAggggAACCCBQRwSsvUvgUB15oNwGAggggAACCCCAAAIIIIAAAggggAACCJRewDpSCRwqvW0lH3HOnDnu4IMP9gSXXnqpn66rkj1Kde82DVynTp3c4MGDS3VYjoMAAggggAACCCCAQJ0UsPYugUN18vFyUwgggAACCCCAAAIIIIAAAggggAACCCBQCgHrSCVwqBSaHCMUmDBhglO2oc0228x/httYLlxg1qxZrlevXn7Hu+++2zVt2rTwg7AHAggggAACCCCAAAIVJGDtXQKHKuihc6sIIIAAAggggAACCCCAAAIIIIAAAgggUJiAdaQSOFSYG7WrF1iwYIEbPny4mz9/vjv11FNdkyZNqt+JGlkFFCw0ffp017FjR7f77rtnrccGBBBAAAEEEEAAAQQQ+H8Ba+8SOMQbgQACCCCAAAIIIIAAAggggAACCCCAAAIIZBGwjlQCh7IAsRoBBBBAAAEEEEAAAQQQQKBWClh7l8ChWvn4uGgEEEAAAQQQQAABBBBAAAEEEEAAAQQQqAkB60glcKgmtDkHAggggAACCCCAAAIIIIBATQlYe5fAoZoS5zwIIIAAAggggAACCCCAAAIIIIAAAgggUOsErCOVwKFa9+i4YAQQQAABBBBAAAEEEEAAgRwC1t4lcCgHEpsQQAABBBBAAAEEEEAAAQQQQAABBBBAoLIFrCOVwKHKfg+4ewQQQAABBBBAAAEEEECgrglYe5fAobr2ZLkfBBBAAAEEEEAAAQQQQAABBBBAAAEEECiZgHWkEjhUMlIOhAACCCCAAAIIIIAAAgggUAYC1t4lcKgMHgaXgAACCCCAAAIIIIAAAggggAACCCCAAALlKWAdqQQOlefz4aoQQAABBBBAAAEEEEAAAQQWT8DauwQOLZ4feyGAAAIIIIAAAggggAACCCCAAAIIIIBABQhYRyqBQxXwsLlFBBBAAAEEEEAAAQQQQKCCBKy9S+BQBT10bhUBBBBAAAEEEEAAAQQQQAABBBBAAAEEChOwjlQChwpzozYCCCCAAAIIIIAAAggggEB5C1h7l8Ch8n5OXB0CCCCAAAIIIIAAAggggAACCCCAAAIILEEB60glcGgJPgROjQACCCCAAAIIIIAAAgggUHIBa+8SOFRyWg6IAAIIIIAAAggggAACCCCAAAIIIIAAAnVFwDpSCRyqK0+U+0AAAQQQQAABBBBAAAEEEJCAtXcJHOJ9QAABBBBAAAEEEEAAAQQQQAABBBBAAAEEsghYRyqBQ1mAWI0AAggggAACCCCAAAIIIFArBay9S+BQrXx8XDQCCCCAAAIIIIAAAggggAACCCCAAAII1ISAdaQSOFQT2pwDAQQQQAABBBBAAAEEEECgpgSsvUvgUE2Jcx4EEEAAAQQQQAABBBBAAAEEEEAAAQQQqHUC1pFK4FCte3RcMAIIIIAAAggggAACCCCAQA4Ba+8SOJQDiU0IIIAAAggggAACCCCAAAIIIIAAAgggUNkC1pFK4FBlvwfcPQIIIIAAAggggAACCCBQ1wSsvUvgUF17stwPAggggAACCCCAAAIIIIAAAggggAACCJRMwDpSCRwqGSkHQgABBBBAAAEEEEAAAQQQKAMBa+8SOFQGD4NLQAABBBBAAAEEEEAAAQQQQAABBBBAAIHyFLCOVAKHyvP5cFUIIIAAAggggAACCCCAAAKLJ2DtXQKHFs+PvRBAAAEEEEAAAQQQQAABBBBAAAEEEECgAgSsI5XAoQp42NwiAggggAACCCCAAAIIIFBBAtbeJXCogh46t4oAAggggAACCCCAAAIIIIAAAggggAAChQlYRyqBQ4W5URsBBBBAAAEEEEAAAQQQQKC8Bay9S+BQeT8nrg4BBBBAAAEEEEAAAQQQQAABBBBAAAEElqCAdaQSOLQEHwKnRgABBBBAAAEEEEAAAQQQKLmAtXcJHCo5LQdEAAEEEEAAAQQQQAABBBBAAAEEEEAAgboiYB2pBA7VlSfKfSCAAAIIIIAAAggggAACCEjA2rtlHzjE40IAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAAAQQQQAABBBBAAIHSCxA4VHpTjogAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAQNkLlH3gUCkvsOyfBheIAAIIIIAAAggggAACCCCAAAIIIIAAAmUl8M477/jroZ+yrB4LF4MAAggggAACCCCAAAIIIFCkQBrt3XqLfitFXle0exoXGB2cBQQQQAABBBBAAAEEEEAAAQQQQAABBBBAIA8B+inzQKIKAggggAACCCCAAAIIIIBArRNIo71L4FCtew24YAQQQAABBBBAAAEEEEAAAQQQQAABBBDIJZBGR2qu87ENAQQQQAABBBBAAAEEEEAAgZoQSKO9S+BQTTw5zoEAAggggAACCCCAAAIIIIAAAggggAACNSaQRkdqjV08J0IAAQQQQAABBBBAAAEEEEAgi0Aa7V0Ch7JgsxoBBBBAAAEEEEAAAQQQQAABBBBAAAEEaqdAGh2ptVOCq0YAAQQQQAABBBBAAAEEEKhLAmm0dwkcqktvCPeCAAIIIIAAAggggAACCCCAAAIIIIAAAi6NjlRYEUAAAQQQQAABBBBAAAEEEFjSAmm0dwkcWtJPlfMjgAACCCCAAAIIIIAAAggggAACCCCAQEkF0uhILekFcjAEEEAAAQQQQAABBBBAAAEEFkMgjfYugUOL8SDYBQEEEEAAAQQQQAABBBBAAAEEEEAAAQTKVyCNjtTyvVuuDAEEEEAAAQQQQAABBBBAoFIE0mjvEjhUKW8P94kAAggggAACCCCAAAIIIIAAAggggECFCKTRkVohdNwmAggggAACCCCAAAIIIIBAGQuk0d4lcKiMHziXhgACCCCAAAIIIIAAAggggAACCCCAAAKFC6TRkVr4VbAHAggggAACCCCAAAIIIIAAAqUVSKO9S+BQaZ8RR0MAAQQQQAABBBBAAAEEEEAAAQQQQACBJSyQRkfqEr4lTo8AAggggAACCCCAAAIIIICAS6O9S+AQLxYCCCCAAAIIIIAAAggggAACCCCAAAII1CmBNDpS6xQQN4MAAggggAACCCCAAAIIIFArBdJo7xI4VCtfBS4aAQQQQAABBBBAAAEEEEAAAQQQQAABBLIJpNGRmu1crEcAAQQQQAABBBBAAAEEEECgpgTSaO8SOFRTT4/zIIAAAggggAACCCCAAAIIIIAAAggggECNCKTRkVojF85JEEAAAQQQQAABBBBAAAEEEMghkEZ7l8ChHOBsQgABBBBAAAEEEEAAAQQQQAABBBBAAIHaJ5BGR2rtU+CKEUAAAQQQQAABBBBAAAEE6ppAGu1dAofq2lvC/SCAAAIIIIAAAggggAACCCCAAAIIIFDhAml0pFY4KbePAAIIIIAAAggggAACCCBQBgJptHcJHCqDB8slIIAAAggggAACCCCAAAIIIIAAAggggEDpBNLoSC3d1XEkBBBAAAEEEEAAAQQQQAABBBZPII32LoFDi/cs2AsBBBBAAAEEEEAAAQQQQAABBBBAAAEEylQgjY7UMr1VLgsBBBBAAAEEEEAAAQQQQKCCBNJo7xI4VEEvELeKAAIIIIAAAggggAACCCCAAAIIIIBAJQik0ZFaCW7cIwIIIIAAAggggAACCCCAQHkLpNHeJXCovJ85V4cAAggggAACCCCAAAIIIIAAAggggAACBQqk0ZFa4CVQHQEEEEAAAQQQQAABBBBAAIGSC6TR3v0/jfGT8TDQeIYAAAAASUVORK5CYII=\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CJd-_xeU8wTH"
},
"source": [
"Remove the folder `my_dir`"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:47.292574Z",
"iopub.status.busy": "2024-10-13T09:40:47.292063Z",
"iopub.status.idle": "2024-10-13T09:40:49.065322Z",
"shell.execute_reply": "2024-10-13T09:40:49.064592Z"
},
"id": "fug-vO6d8y00",
"outputId": "cdf39571-e507-4d86-c84e-743a194609e8"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Deleted hdfs://localhost:8902/my_dir\r\n"
]
}
],
"source": [
"!hdfs dfs -rm -r hdfs://localhost:8902/my_dir"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uJnm8IIdVbXw"
},
"source": [
"Now the DFS used should be back to ~$4$MB."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CU3gbnQSTL47"
},
"source": [
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DQQHj14wVnum"
},
"source": [
"## The default filesystem\n",
"\n",
"By default, `hdfs` will use the local filesystem so you need to prepend `hdfs://...` if you want to use the HDFS filesystem.\n",
"\n",
"If you do not want to use the prefix `hdfs://localhost:8902/` in the filenames, you could set the property `fs.defaultFS` in `core-site.xml` or else use the option `-fs` like this:\n",
"\n",
" hdfs dfs -fs hdfs://localhost:8902/"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:49.068293Z",
"iopub.status.busy": "2024-10-13T09:40:49.067973Z",
"iopub.status.idle": "2024-10-13T09:40:50.748751Z",
"shell.execute_reply": "2024-10-13T09:40:50.748078Z"
},
"id": "iTvSHQntiReX",
"outputId": "132c3dd4-56ac-415a-988b-157fdefc611b"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found 2 items\r\n",
"drwxrwxrwx - runner supergroup 0 2024-10-13 09:40 /home\r\n",
"drwxr-xr-x - runner supergroup 0 2024-10-13 09:40 /user\r\n"
]
}
],
"source": [
"!hdfs dfs -fs hdfs://localhost:8902/ -ls /"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PEteAOIhWOXo"
},
"source": [
"This is the same as"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:50.751301Z",
"iopub.status.busy": "2024-10-13T09:40:50.751082Z",
"iopub.status.idle": "2024-10-13T09:40:52.550408Z",
"shell.execute_reply": "2024-10-13T09:40:52.549647Z"
},
"id": "oh2ddzKYWQEE",
"outputId": "3a19d10c-d679-4c54-ebb7-521f8a475765"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found 2 items\r\n",
"drwxrwxrwx - runner supergroup 0 2024-10-13 09:40 hdfs://localhost:8902/home\r\n",
"drwxr-xr-x - runner supergroup 0 2024-10-13 09:40 hdfs://localhost:8902/user\r\n"
]
}
],
"source": [
"!hdfs dfs -ls hdfs://localhost:8902/"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iv1KeqOYWkRx"
},
"source": [
"And also the same as"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:52.552885Z",
"iopub.status.busy": "2024-10-13T09:40:52.552664Z",
"iopub.status.idle": "2024-10-13T09:40:54.336100Z",
"shell.execute_reply": "2024-10-13T09:40:54.335377Z"
},
"id": "Q67buUSoWmwS",
"outputId": "65dd651d-22d2-4001-aa28-3c3ae31efb08"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found 2 items\r\n",
"drwxrwxrwx - runner supergroup 0 2024-10-13 09:40 /home\r\n",
"drwxr-xr-x - runner supergroup 0 2024-10-13 09:40 /user\r\n"
]
}
],
"source": [
"!hdfs dfs -D fs.defaultFS=hdfs://localhost:8902/ -ls /"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hofC7_S1mmvm"
},
"source": [
"With the option `-D` we can set any variable on the fly (in this case we set `fs.defaultFS` to be the HDFS filesystem).\n",
"\n",
"**Note:** the `-D` option should come before any other option."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ImKd8-1QW5X5"
},
"source": [
"If you configure the property `fs.defaultFS` in `core-site.xml`, you can also use `hdfs dfs -ls /`."
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {
"execution": {
"iopub.execute_input": "2024-10-13T09:40:54.338842Z",
"iopub.status.busy": "2024-10-13T09:40:54.338412Z",
"iopub.status.idle": "2024-10-13T09:40:54.342004Z",
"shell.execute_reply": "2024-10-13T09:40:54.341476Z"
},
"id": "ed3EKZeuBH--"
},
"outputs": [],
"source": [
"with open(os.environ['HADOOP_HOME']+'/etc/hadoop/core-site.xml', 'w') as file:\n",
" file.write(\"\"\"\n",
" \n",
" fs.defaultFS\n",
" hdfs://localhost:8902/\n",
" \n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:54.343961Z",
"iopub.status.busy": "2024-10-13T09:40:54.343660Z",
"iopub.status.idle": "2024-10-13T09:40:54.470677Z",
"shell.execute_reply": "2024-10-13T09:40:54.470050Z"
},
"id": "CTzAAUV3eaHg",
"outputId": "23f60115-c315-47da-b3aa-35f108ff2003"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\r\n",
" \r\n",
" fs.defaultFS\r\n",
" hdfs://localhost:8902/\r\n",
" \r\n",
""
]
}
],
"source": [
"!cat $HADOOP_HOME/etc/hadoop/core-site.xml"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:54.472801Z",
"iopub.status.busy": "2024-10-13T09:40:54.472394Z",
"iopub.status.idle": "2024-10-13T09:40:56.245612Z",
"shell.execute_reply": "2024-10-13T09:40:56.244857Z"
},
"id": "9NUz2NQpXHzj",
"outputId": "f985d802-d777-49d5-f5cc-af7a93d03f02"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found 2 items\r\n",
"drwxrwxrwx - runner supergroup 0 2024-10-13 09:40 /home\r\n",
"drwxr-xr-x - runner supergroup 0 2024-10-13 09:40 /user\r\n"
]
}
],
"source": [
"!hdfs dfs -ls /"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eg1FRosv9Pp5"
},
"source": [
"Let us set the local filesystem as default (`file:///` means the local filesystem `file://` and the extra slash `/` indicates the root folder)."
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {
"execution": {
"iopub.execute_input": "2024-10-13T09:40:56.248456Z",
"iopub.status.busy": "2024-10-13T09:40:56.248006Z",
"iopub.status.idle": "2024-10-13T09:40:56.251855Z",
"shell.execute_reply": "2024-10-13T09:40:56.251298Z"
},
"id": "Eak0oQdk9Xap"
},
"outputs": [],
"source": [
"with open(os.environ['HADOOP_HOME']+'/etc/hadoop/core-site.xml', 'w') as file:\n",
" file.write(\"\"\"\n",
"\n",
" \n",
" fs.defaultFS\n",
" file:///\n",
" \n",
"\"\"\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vzOLLTaP9-3a"
},
"source": [
"Now run `!hdfs dfs -ls /` as before. This time we are going to be listing the local filesystem and not the HDFS."
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:56.254125Z",
"iopub.status.busy": "2024-10-13T09:40:56.253497Z",
"iopub.status.idle": "2024-10-13T09:40:57.305441Z",
"shell.execute_reply": "2024-10-13T09:40:57.304767Z"
},
"id": "uftFbN7P-Mfb",
"outputId": "ccabc915-adb4-4b89-85a4-45083b9904c1"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found 25 items\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"drwxr-xr-x - root root 69632 2024-10-07 16:10 /bin\r\n",
"drwxr-xr-x - root root 4096 2024-10-02 02:17 /boot\r\n",
"drwxr-xr-x - root root 4096 2024-10-13 09:35 /data\r\n",
"drwxr-xr-x - root root 3980 2024-10-13 09:30 /dev\r\n",
"drwxr-xr-x - root root 12288 2024-10-13 09:30 /etc\r\n",
"drwxr-xr-x - root root 4096 2024-10-07 16:07 /home\r\n",
"drwxr-xr-x - root root 4096 2024-10-07 06:58 /imagegeneration\r\n",
"drwxr-xr-x - root root 4096 2024-10-07 06:25 /lib\r\n",
"drwxr-xr-x - root root 4096 2024-10-07 05:51 /lib32\r\n",
"drwxr-xr-x - root root 4096 2024-10-07 06:17 /lib64\r\n",
"drwxr-xr-x - root root 4096 2024-10-02 02:10 /libx32\r\n",
"drwx------ - root root 16384 2024-10-02 02:14 /lost+found\r\n",
"drwxr-xr-x - root root 4096 2024-10-02 02:10 /media\r\n",
"drwxr-xr-x - root root 4096 2024-10-13 09:30 /mnt\r\n",
"drwxrwxrwx - root root 4096 2024-10-13 09:30 /opt\r\n",
"dr-xr-xr-x - root root 0 2024-10-13 09:30 /proc\r\n",
"drwx------ - root root 4096 2024-10-07 06:56 /root\r\n",
"drwxr-xr-x - root root 1080 2024-10-13 09:30 /run\r\n",
"drwxr-xr-x - root root 20480 2024-10-07 16:10 /sbin\r\n",
"drwxr-xr-x - root root 4096 2024-10-02 02:17 /snap\r\n",
"drwxr-xr-x - root root 4096 2024-10-02 02:10 /srv\r\n",
"dr-xr-xr-x - root root 0 2024-10-13 09:30 /sys\r\n",
"drwxrwxrwt - root root 12288 2024-10-13 09:40 /tmp\r\n",
"drwxr-xr-x - root root 4096 2024-10-02 02:10 /usr\r\n",
"drwxr-xr-x - root root 4096 2024-10-07 05:51 /var\r\n"
]
}
],
"source": [
"!hdfs dfs -ls /"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EFLRoQ9K-vNo"
},
"source": [
"I advise to get used to the fact that Hadoop interprets a file path as HDFS (`hdfs://`) vs. local (`file://`) depending on the setting of the variable `fs.defaultFS`, since this is often a source of confusion."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-X2Gcpm_rsT3"
},
"source": [
"## Show status of HDFS with `hdfs dfsadmin`\n",
"\n",
"The command `hdfs dfsadmin` allows to run administration tasks on the Hadoop filesystem."
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:57.307975Z",
"iopub.status.busy": "2024-10-13T09:40:57.307543Z",
"iopub.status.idle": "2024-10-13T09:40:58.215817Z",
"shell.execute_reply": "2024-10-13T09:40:58.215158Z"
},
"id": "tVjiDpX6F04B",
"outputId": "de550d62-4cd9-4aca-c90a-df1e8bb28351"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"h: Unknown command\r\n",
"Usage: hdfs dfsadmin\r\n",
"Note: Administrative commands can only be run as the HDFS superuser.\r\n",
"\t[-report [-live] [-dead] [-decommissioning] [-enteringmaintenance] [-inmaintenance] [-slownodes]]\r\n",
"\t[-safemode ]\r\n",
"\t[-saveNamespace [-beforeShutdown]]\r\n",
"\t[-rollEdits]\r\n",
"\t[-restoreFailedStorage true|false|check]\r\n",
"\t[-refreshNodes]\r\n",
"\t[-setQuota ...]\r\n",
"\t[-clrQuota ...]\r\n",
"\t[-setSpaceQuota [-storageType ] ...]\r\n",
"\t[-clrSpaceQuota [-storageType ] ...]\r\n",
"\t[-finalizeUpgrade]\r\n",
"\t[-rollingUpgrade []]\r\n",
"\t[-upgrade ]\r\n",
"\t[-refreshServiceAcl]\r\n",
"\t[-refreshUserToGroupsMappings]\r\n",
"\t[-refreshSuperUserGroupsConfiguration]\r\n",
"\t[-refreshCallQueue]\r\n",
"\t[-refresh [arg1..argn]\r\n",
"\t[-reconfig ]\r\n",
"\t[-printTopology]\r\n",
"\t[-refreshNamenodes datanode_host:ipc_port]\r\n",
"\t[-getVolumeReport datanode_host:ipc_port]\r\n",
"\t[-deleteBlockPool datanode_host:ipc_port blockpoolId [force]]\r\n",
"\t[-setBalancerBandwidth ]\r\n",
"\t[-getBalancerBandwidth ]\r\n",
"\t[-fetchImage ]\r\n",
"\t[-allowSnapshot ]\r\n",
"\t[-disallowSnapshot ]\r\n",
"\t[-provisionSnapshotTrash [-all]]\r\n",
"\t[-shutdownDatanode [upgrade]]\r\n",
"\t[-evictWriters ]\r\n",
"\t[-getDatanodeInfo ]\r\n",
"\t[-metasave filename]\r\n",
"\t[-triggerBlockReport [-incremental] [-namenode ]]\r\n",
"\t[-listOpenFiles [-blockingDecommission] [-path ]]\r\n",
"\t[-help [cmd]]\r\n",
"\r\n",
"Generic options supported are:\r\n",
"-conf specify an application configuration file\r\n",
"-D define a value for a given property\r\n",
"-fs specify default filesystem URL to use, overrides 'fs.defaultFS' property from configurations.\r\n",
"-jt specify a ResourceManager\r\n",
"-files specify a comma-separated list of files to be copied to the map reduce cluster\r\n",
"-libjars specify a comma-separated list of jar files to be included in the classpath\r\n",
"-archives specify a comma-separated list of archives to be unarchived on the compute machines\r\n",
"\r\n",
"The general command line syntax is:\r\n",
"command [genericOptions] [commandOptions]\r\n",
"\r\n"
]
}
],
"source": [
"!hdfs dfsadmin -h"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZAo3uxzrB7ud"
},
"source": [
"The comand `hdfs dfsadmin -report` shows the current status of the Hadoop filesystem. In order to run it on our MiniCluster HDFS we need to pass the option\n",
"\n",
" -fs hdfs://localhost:8902/\n",
"\n",
"\n",
" \n",
"Alternatively, we can configure the default filesystem (the URI of the namenode) in `core-site.xml` (see discussion in [The default filesystem)](#scrollTo=The_default_filesystem)."
]
},
{
"cell_type": "code",
"execution_count": 57,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:58.218378Z",
"iopub.status.busy": "2024-10-13T09:40:58.217835Z",
"iopub.status.idle": "2024-10-13T09:40:59.983313Z",
"shell.execute_reply": "2024-10-13T09:40:59.982589Z"
},
"id": "ZEHoUeoxinbv",
"outputId": "b9610113-c5f3-4d9f-e2f4-9855cfdaba8e"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Configured Capacity: 155702509568 (145.01 GB)\r\n",
"Present Capacity: 31647035461 (29.47 GB)\r\n",
"DFS Remaining: 31644712960 (29.47 GB)\r\n",
"DFS Used: 2322501 (2.21 MB)\r\n",
"DFS Used%: 0.01%\r\n",
"Replicated Blocks:\r\n",
"\tUnder replicated blocks: 0\r\n",
"\tBlocks with corrupt replicas: 0\r\n",
"\tMissing blocks: 0\r\n",
"\tMissing blocks (with replication factor 1): 0\r\n",
"\tLow redundancy blocks with highest priority to recover: 0\r\n",
"\tPending deletion blocks: 0\r\n",
"Erasure Coded Block Groups: \r\n",
"\tLow redundancy block groups: 0\r\n",
"\tBlock groups with corrupt internal blocks: 0\r\n",
"\tMissing block groups: 0\r\n",
"\tLow redundancy blocks with highest priority to recover: 0\r\n",
"\tPending deletion blocks: 0\r\n",
"\r\n",
"-------------------------------------------------\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Live datanodes (1):\r\n",
"\r\n",
"Name: 127.0.0.1:38697 (localhost)\r\n",
"Hostname: 127.0.0.1\r\n",
"Decommission Status : Normal\r\n",
"Configured Capacity: 155702509568 (145.01 GB)\r\n",
"DFS Used: 2322501 (2.21 MB)\r\n",
"Non DFS Used: 124021919675 (115.50 GB)\r\n",
"DFS Remaining: 31644712960 (29.47 GB)\r\n",
"DFS Used%: 0.00%\r\n",
"DFS Remaining%: 20.32%\r\n",
"Configured Cache Capacity: 0 (0 B)\r\n",
"Cache Used: 0 (0 B)\r\n",
"Cache Remaining: 0 (0 B)\r\n",
"Cache Used%: 100.00%\r\n",
"Cache Remaining%: 0.00%\r\n",
"Xceivers: 0\r\n",
"Last contact: Sun Oct 13 09:40:57 UTC 2024\r\n",
"Last Block Report: Sun Oct 13 09:40:09 UTC 2024\r\n",
"Num of Blocks: 3\r\n",
"\r\n",
"\r\n"
]
}
],
"source": [
"!hdfs dfsadmin -fs hdfs://localhost:8902/ -report"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_5KPExSrasE9"
},
"source": [
"The information displayed by `hdfs dfsadmin` corresponds to what is presented in the [NameNode Web UI](#scrollTo=Start_the_NameNode_Web_UI)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "v1sqv4BItwPZ"
},
"source": [
"# Test MapReduce\n",
"\n",
"Find the MapReduce examples that come with the Hadoop distribution."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VFrcfoxvo7em"
},
"source": [
"## Run the `pi` example"
]
},
{
"cell_type": "code",
"execution_count": 58,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:40:59.986127Z",
"iopub.status.busy": "2024-10-13T09:40:59.985729Z",
"iopub.status.idle": "2024-10-13T09:41:00.189916Z",
"shell.execute_reply": "2024-10-13T09:41:00.189171Z"
},
"id": "Y6l_P08ovfZ9",
"outputId": "9770c765-69ce-4028-eabb-b7e5307d9422"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"./hadoop-3.4.0/share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-3.4.0-sources.jar\r\n",
"./hadoop-3.4.0/share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-3.4.0-test-sources.jar\r\n",
"./hadoop-3.4.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.0.jar\r\n",
"./spark-3.5.3-bin-hadoop3/examples/jars/spark-examples_2.12-3.5.3.jar\r\n",
"./spark-3.5.3-bin-hadoop3/work/app-20241013093607-0000/0/spark-examples_2.12-3.5.3.jar\r\n",
"./spark-3.5.3-bin-hadoop3/work/app-20241013093626-0000/0/spark-examples_2.12-3.5.3.jar\r\n",
"./spark-3.5.3-bin-hadoop3/work/app-20241013093635-0001/0/spark-examples_2.12-3.5.3.jar\r\n"
]
}
],
"source": [
"!find . -name \"*examples*.jar\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "15uYXe4hgiZl"
},
"source": [
"Check if the cluster is still running, if not you will need to restart it! (from the cell cell [Configure the MiniCluster's ports](#scrollTo=Configure_the_MiniCluster_s_ports))."
]
},
{
"cell_type": "code",
"execution_count": 59,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"execution": {
"iopub.execute_input": "2024-10-13T09:41:00.192527Z",
"iopub.status.busy": "2024-10-13T09:41:00.192074Z",
"iopub.status.idle": "2024-10-13T09:41:00.375433Z",
"shell.execute_reply": "2024-10-13T09:41:00.374653Z"
},
"id": "RLf3KzXGglLz",
"outputId": "729403fd-ce8e-494f-ebfd-09bfd2d1ce9c"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\r\n",
"java 5129 runner 341u IPv4 84793 0t0 TCP 127.0.0.1:8901 (LISTEN)\r\n",
"java 5129 runner 354u IPv4 79164 0t0 TCP 127.0.0.1:8902 (LISTEN)\r\n",
"java 5129 runner 364u IPv4 87194 0t0 TCP 127.0.0.1:38697 (LISTEN)\r\n",
"java 5129 runner 367u IPv4 87197 0t0 TCP 127.0.0.1:41527 (LISTEN)\r\n",
"java 5129 runner 420u IPv4 87216 0t0 TCP 127.0.0.1:38035 (LISTEN)\r\n",
"java 5129 runner 421u IPv4 87220 0t0 TCP 127.0.0.1:36527 (LISTEN)\r\n",
"java 5129 runner 444u IPv4 93129 0t0 TCP *:8031 (LISTEN)\r\n",
"java 5129 runner 473u IPv4 88768 0t0 TCP *:10033 (LISTEN)\r\n",
"java 5129 runner 483u IPv4 89164 0t0 TCP *:19888 (LISTEN)\r\n",
"java 5129 runner 485u IPv4 94167 0t0 TCP *:8088 (LISTEN)\r\n",
"java 5129 runner 499u IPv4 88773 0t0 TCP 127.0.0.1:8900 (LISTEN)\r\n",
"java 5129 runner 509u IPv4 94174 0t0 TCP *:8033 (LISTEN)\r\n",
"java 5129 runner 529u IPv4 93134 0t0 TCP *:8030 (LISTEN)\r\n",
"java 5129 runner 539u IPv4 89169 0t0 TCP 127.0.0.1:8903 (LISTEN)\r\n",
"java 5129 runner 549u IPv4 93135 0t0 TCP 127.0.0.1:44085 (LISTEN)\r\n",
"java 5129 runner 559u IPv4 88778 0t0 TCP 127.0.0.1:38203 (LISTEN)\r\n",
"java 5129 runner 569u IPv4 89174 0t0 TCP *:45391 (LISTEN)\r\n",
"java 5129 runner 570u IPv4 93138 0t0 TCP 127.0.0.1:42883 (LISTEN)\r\n"
]
}
],
"source": [
"!lsof -n -i -P +c0 -sTCP:LISTEN -ac java"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-aMDAVoXiGiF"
},
"source": [
"Here's ChatGPT 3.5's poem inspired by\n",
"`lsof -n -i -P +c0 -sTCP:LISTEN`:"
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {
"cellView": "form",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 297
},
"execution": {
"iopub.execute_input": "2024-10-13T09:41:00.377895Z",
"iopub.status.busy": "2024-10-13T09:41:00.377498Z",
"iopub.status.idle": "2024-10-13T09:41:00.383459Z",
"shell.execute_reply": "2024-10-13T09:41:00.382796Z"
},
"id": "y6JlmhVvh_AC",
"outputId": "2765cd39-c64f-4488-d9f2-31dd6e28aeb8"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
\n",
"\n",
"\n",
"\n",
"
\n",
"In the realm of networks, where processes twine, \n",
"A command unfolds, a symphony of lines. \n",
"\"Lsof,\" it whispers, with a mystic hum, \n",
"A dance of flags, each one has its own drum. \n",
"
\n",
"\"-n -i -P,\" the conductor commands, \n",
"Navigate swiftly, across distant lands. \n",
"\"+c0\" echoes softly, a chorus of glee, \n",
"Embrace all processes, as far as eyes can see. \n",
"
\n",
"\"-sTCP:LISTEN,\" a stanza profound, \n",
"Seeking the echoes of ports, a network's sound. \n",
"Processes in repose, in a state so keen, \n",
"A tapestry of LISTEN, a poetic scene. \n",
"
\n",
"In the realm of networks, where processes twine, \n",
"A command unfolds, a symphony of lines. \n",
"\"Lsof,\" it whispers, with a mystic hum, \n",
"A dance of flags, each one has its own drum. \n",
"
\n",
"\"-n -i -P,\" the conductor commands, \n",
"Navigate swiftly, across distant lands. \n",
"\"+c0\" echoes softly, a chorus of glee, \n",
"Embrace all processes, as far as eyes can see. \n",
"
\n",
"\"-sTCP:LISTEN,\" a stanza profound, \n",
"Seeking the echoes of ports, a network's sound. \n",
"Processes in repose, in a state so keen, \n",
"A tapestry of LISTEN, a poetic scene. \n",
"
![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
![](\"https://github.com/groda/big_data/blob/master/logo_bdb.png?raw=true\"){kind=logo}