{ "cells": [ { "cell_type": "markdown", "source": [ "# Simple Load and Store with Aerospike Data on Spark\n", "This notebook shows how to load data from and store processed data to Aerospike Database on Spark. The data transfer is enabled by the Aerospike Connector for Spark." ], "metadata": {} }, { "cell_type": "markdown", "source": [ "## Setup\n", "Execute the code cells in this section to set up Aerospike Server, Spark Server, and Spark Connector." ], "metadata": {} }, { "cell_type": "markdown", "source": [ "### Ensure Database Is Running\n", "This notebook requires that Aerospike Database is running." ], "metadata": {} }, { "cell_type": "code", "execution_count": 1, "source": [ "!asd >& /dev/null\n", "!pgrep -x asd >/dev/null && echo \"Aerospike database is running!\" || echo \"**Aerospike database is not running!**\"" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Aerospike database is running!\r\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "### Initialize Spark\n", "We will be using Spark functionality in this notebook." ], "metadata": {} }, { "cell_type": "markdown", "source": [ "#### Initialize Paths and Env Variables" ], "metadata": {} }, { "cell_type": "code", "execution_count": 2, "source": [ "# directory where spark notebook requisites are installed\n", "SPARK_NB_DIR = '/opt/spark-nb'\n", "SPARK_DIR = 'spark-dir-link'\n", "SPARK_HOME = SPARK_NB_DIR + '/' + SPARK_DIR\n", "AEROSPIKE_JAR = 'aerospike-jar-link'\n", "AEROSPIKE_JAR_PATH = SPARK_NB_DIR + '/' + AEROSPIKE_JAR" ], "outputs": [], "metadata": {} }, { "cell_type": "code", "execution_count": 3, "source": [ "# IP Address or DNS name for one host in your Aerospike cluster\n", "AS_HOST =\"localhost\"\n", "# Name of one of your namespaces. Type 'show namespaces' at the aql prompt if you are not sure\n", "AS_NAMESPACE = \"test\" \n", "AS_PORT = 3000 # Usually 3000, but change here if not\n", "AS_CONNECTION_STRING = AS_HOST + \":\"+ str(AS_PORT)" ], "outputs": [], "metadata": {} }, { "cell_type": "code", "execution_count": 4, "source": [ "# Locate the Spark installation using the SPARK_HOME parameter.\n", "import findspark\n", "findspark.init(SPARK_HOME)" ], "outputs": [], "metadata": {} }, { "cell_type": "code", "execution_count": 5, "source": [ "# Specify the Aerospike Spark Connector jar in the command used to interact with Aerospike.\n", "import os \n", "os.environ[\"PYSPARK_SUBMIT_ARGS\"] = '--jars ' + AEROSPIKE_JAR_PATH + ' pyspark-shell'" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "#### Configure Spark Session\n", "Please visit [Configuring Aerospike Connect for Spark](https://docs.aerospike.com/docs/connect/processing/spark/configuration.html) for more information about the properties used on this page." ], "metadata": {} }, { "cell_type": "code", "execution_count": 6, "source": [ "# imports\n", "import pyspark\n", "from pyspark.context import SparkContext\n", "from pyspark.sql.context import SQLContext\n", "from pyspark.sql.session import SparkSession\n", "from pyspark.sql.types import StringType, StructField, StructType, ArrayType, IntegerType, MapType, LongType, DoubleType" ], "outputs": [], "metadata": {} }, { "cell_type": "code", "execution_count": 7, "source": [ "sc = SparkContext.getOrCreate()\n", "conf=sc._conf.setAll([(\"aerospike.namespace\",AS_NAMESPACE),(\"aerospike.seedhost\",AS_CONNECTION_STRING)])\n", "sc.stop()\n", "sc = pyspark.SparkContext(conf=conf)\n", "spark = SparkSession(sc)\n", "sqlContext = SQLContext(sc)" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "## Store Data into Aerospike\n", "We will first store simple generated data to Aerospike, and then show how to load data from Aerospike." ], "metadata": {} }, { "cell_type": "markdown", "source": [ "### Create Data\n", "We create simple age-salary data with a specified distribution and the following structure.\n", "- id: integer \n", "- name: string\n", "- age: integer\n", "- salary: integer" ], "metadata": {} }, { "cell_type": "code", "execution_count": 8, "source": [ "# We create age vs salary data, using three different Gaussian distributions\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import pandas as pd\n", "import math\n", "\n", "# Make sure we get the same results every time this workbook is run\n", "# Otherwise we are occasionally exposed to results not working out as expected\n", "np.random.seed(12345)\n", "\n", "# Create covariance matrix from std devs + correlation\n", "def covariance_matrix(std_dev_1,std_dev_2,correlation):\n", " return [[std_dev_1 ** 2, correlation * std_dev_1 * std_dev_2], \n", " [correlation * std_dev_1 * std_dev_2, std_dev_2 ** 2]]\n", "\n", "# Return a bivariate sample given means/std dev/correlation\n", "def age_salary_sample(distribution_params,sample_size):\n", " mean = [distribution_params[\"age_mean\"], distribution_params[\"salary_mean\"]]\n", " cov = covariance_matrix(distribution_params[\"age_std_dev\"],distribution_params[\"salary_std_dev\"],\n", " distribution_params[\"age_salary_correlation\"])\n", " return np.random.multivariate_normal(mean, cov, sample_size).T\n", "\n", "# Define the characteristics of our age/salary distribution\n", "age_salary_distribution_1 = {\"age_mean\":25,\"salary_mean\":50000,\n", " \"age_std_dev\":1,\"salary_std_dev\":5000,\"age_salary_correlation\":0.3}\n", "\n", "age_salary_distribution_2 = {\"age_mean\":45,\"salary_mean\":80000,\n", " \"age_std_dev\":4,\"salary_std_dev\":8000,\"age_salary_correlation\":0.7}\n", "\n", "age_salary_distribution_3 = {\"age_mean\":35,\"salary_mean\":70000,\n", " \"age_std_dev\":2,\"salary_std_dev\":9000,\"age_salary_correlation\":0.1}\n", "\n", "distribution_data = [age_salary_distribution_1,age_salary_distribution_2,age_salary_distribution_3]\n", "\n", "# Sample age/salary data for each distributions\n", "sample_size_1 = 100;\n", "sample_size_2 = 120;\n", "sample_size_3 = 80;\n", "sample_sizes = [sample_size_1,sample_size_2,sample_size_3]\n", "group_1_ages,group_1_salaries = age_salary_sample(age_salary_distribution_1,sample_size=sample_size_1)\n", "group_2_ages,group_2_salaries = age_salary_sample(age_salary_distribution_2,sample_size=sample_size_2)\n", "group_3_ages,group_3_salaries = age_salary_sample(age_salary_distribution_3,sample_size=sample_size_3)\n", "\n", "ages=np.concatenate([group_1_ages,group_2_ages,group_3_ages])\n", "salaries=np.concatenate([group_1_salaries,group_2_salaries,group_3_salaries])\n", "\n", "print(\"Data created\")" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Data created\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "### Display Data" ], "metadata": {} }, { "cell_type": "code", "execution_count": 9, "source": [ "# Plot the sample data\n", "group_1_colour, group_2_colour, group_3_colour ='red','blue', 'pink'\n", "plt.xlabel('Age',fontsize=10)\n", "plt.ylabel(\"Salary\",fontsize=10) \n", "\n", "plt.scatter(group_1_ages,group_1_salaries,c=group_1_colour,label=\"Group 1\")\n", "plt.scatter(group_2_ages,group_2_salaries,c=group_2_colour,label=\"Group 2\")\n", "plt.scatter(group_3_ages,group_3_salaries,c=group_3_colour,label=\"Group 3\")\n", "\n", "plt.legend(loc='upper left')\n", "plt.show()" ], "outputs": [ { "output_type": "display_data", "data": { "text/plain": [ "
" ], "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAAEGCAYAAACpXNjrAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA4M0lEQVR4nO2de5QdVZ3vP7/uThObECFNJsMkpDsO4SFMBNJiEAedRF6BK6jcUaaBXMdrBgIOc+cuNA7Li4MTx9eMhotEgzwCaUVEGbjeCMagzOAFtBMxvE0gDzoDIekAASIk6fzuH7UrXX26qk7VOXVO1en+fdY665za9dj7HMj+9v69tqgqhmEYhpElTXkPwDAMwxh5mLgYhmEYmWPiYhiGYWSOiYthGIaROSYuhmEYRua05D2AonDooYdqZ2dn3sMwDMNoKFavXr1dVSeWtpu4ODo7O+nt7c17GIZhGA2FiGwKazezmGEYhpE5Ji6GYRhG5pi4GIZhGJljPpcY9uzZQ19fH2+++WbeQyk8Y8eOZcqUKYwZMybvoRiGUQBMXGLo6+vjoIMOorOzExHJeziFRVXp7++nr6+PadOm5T0cwzAKgJnFYnjzzTdpb283YSmDiNDe3m4rPMNoIHp6oLMTmpq8956ebJ9vK5cymLAkw34nw2gcenpg/nzYtcs73rTJOwbo7s6mD1u5GIZhjDKuumpQWHx27fLas8LEpeBs3bqVv/qrv+Id73gHM2fO5OSTT+auu+6q6xh++MMfcuyxx9LU1GSJpoYxAti8OV17JZi4FBhV5bzzzuPUU0/lueeeY/Xq1dx+++309fUNu3bv3r01G8dxxx3Hj3/8Y0499dSa9WEYRv2YOjVdeyWYuGRJxh6y+++/n9bWVi655JL9bR0dHXz6058G4JZbbuFDH/oQs2fPZs6cOezYsYPzzjuPGTNmMGvWLNauXQvAF77wBb7+9a/vf8Zxxx3Hxo0b2bhxI0cffTTd3d0cc8wxnH/++ewqXSsDxxxzDEcddVRV38UwjOKwaBG0tQ1ta2vz2rPCxCUrfA/Zpk2gOughq0JgnnjiCU488cTYa9asWcOdd97JAw88wNVXX80JJ5zA2rVr+dKXvsTFF19cto9nnnmGBQsW8NRTTzF+/Hiuv/76isdrGEZj0N0NS5dCRweIeO9Ll2bnzAcTl+yog4fssssu413vehfvfve797eddtppTJgwAYAHH3yQiy66CIDZs2fT39/Pzp07Y595+OGHc8oppwBw4YUX8uCDD2Y2XsMwikt3N2zcCPv2ee9ZCguYuGRHDTxkxx57LGvWrNl//K1vfYtVq1axbdu2/W0HHnhg2ee0tLSwb9++/cfBfJTSEGILKTYMIwtMXLKiBh6y2bNn8+abb7JkyZL9bWE+EZ8///M/p8eZ4X75y19y6KGHMn78eDo7O/eL1Jo1a9iwYcP+ezZv3sxDDz0EwPe+9z3e9773VTxewzAMHxOXrKiBh0xE+Ld/+zceeOABpk2bxkknncS8efP4yle+Enr9F77wBVavXs2MGTNYuHAhy5YtA+CjH/0oO3bs4Nhjj+W6667jyCOP3H/PUUcdxbe+9S2OOeYYXn75ZS699NJhz73rrruYMmUKDz30EGeffTZnnHFGxd/JMIzRgahq3mMoBF1dXVqaw/HUU09xzDHHJH9IT4/nY9m82VuxLFqUvSEzQzZu3Mg555zD448/nsnzUv9ehmE0PCKyWlW7Stut/EuWdHcXWkwMwzDqhZnFRjGdnZ2ZrVoMwzCCmLgYhmEYmWPiYhiGYWROzcRFRG4SkZdE5PFA2wQRWSki69z7Ia5dRORaEVkvImtF5MTAPfPc9etEZF6gfaaIPObuuVZcgkZUH4ZhGEb9qOXK5RbgzJK2hcAqVZ0OrHLHAGcB091rPrAEPKEArgbeA5wEXB0QiyXApwL3nVmmD8MwDKNO1ExcVPXfgR0lzecCy9znZcB5gfZb1eNh4GAROQw4A1ipqjtU9WVgJXCmOzdeVR9WL5b61pJnhfXRkBSh5P6VV17J0UcfzYwZM/jwhz/MK6+8Utf+DcNoPOrtc5mkqi+4zy8Ck9znycDzgev6XFtce19Ie1wfwxCR+SLSKyK9wZIqRaEoJfdPO+00Hn/8cdauXcuRRx7JP//zP9esL8MwRga5OfTdiqOmGZzl+lDVparapapdEydOrLq/rPekLkrJ/dNPP52WFi8latasWaHiZhiGEaTe4rLVmbRw7y+59i3A4YHrpri2uPYpIe1xfdSUGlTcL2TJ/Ztuuomzzjor1fcwDGP0UW9xuQfwI77mAXcH2i92UWOzgFedaes+4HQROcQ58k8H7nPndorILBcldnHJs8L6qCn12JM675L7ixYtoqWlhW6rQmAYRhlqGYr8feAh4CgR6RORTwJfBk4TkXXAB90xwArgOWA9cAOwAEBVdwBfBH7jXte4Ntw133X3PAv81LVH9VFTarEndZFK7t9yyy385Cc/oaenx8ryG0aArM3hI4VaRotdoKqHqeoYVZ2iqjeqar+qzlHV6ar6QV8oXJTYZar6p6r6Z6raG3jOTap6hHvdHGjvVdXj3D2XO/8KUX3UmlrsSV2Ukvv33nsvX/3qV7nnnntoK638bBijmFqYw0cKlqGfEbXYk7ooJfcvv/xyXnvtNU477TSOP/74IQEGhjGaqYc5vByFXTmpqr1UmTlzppby5JNPDmuLY/ly1Y4OVRHvffnyVLfXnQ0bNuixxx6b2fPS/l6GUXTK/ZsWUfXWLENfIvUbX1vb0L7b2uo79wC9GjKn2solQ2q9J7VhGPUjicmrFubwNBRh5RSFicsoxkruG0Y0SSbuWpjD01CLQKKsMHExDMMIIcnE3d0NS5dCRweIeO9Ll9bPapH3yikOExfDMIwQkk7ceZrD8145xWHiYhiGEUKRJ26fNCunekeVmbgYhlFXChs6W0LeJq+kJFk55ZGPY+JScIpQcv/zn/88M2bM4Pjjj+f000/nP//zP+vavzFyaLSkw5ESAZpHVJmJS4HRgpTcv/LKK1m7di2PPvoo55xzDtdcc03N+jJGNlGT3Lx5xRWYkUAeUWUmLlmytR8eXgsP9HrvW/urelxRSu6PHz9+/+c33njDaosZFRM1mQ0MFHsF0+jkEVVm4pIVW/vh95vgrd3e8Vu7veMqBKZIJfevuuoqDj/8cHp6emzlYlRM3GRWlOS/kUgewQkmLlmxYYtnmA2yb5/XnhF5ltxftGgRzz//PN3d3Vx33XVZfB1jFBI2yQUpQvJfHI0SjFBKHsEJJi5Z4a9YkrYnoEgl9326u7v50Y9+VLZPwwjDn+Sam8PPFyH5L4pGC0Yopd7BCSYuWXFAa7r2BBSl5P66dev2f7777rs5+uijK/5OhtHdDcuWFT+HpJQi1/EqIiYuWTFtsrdWDtLU5LVXSFFK7i9cuJDjjjuOGTNm8LOf/YzFixdX/J0MAxonhyRIket4FRHxKiYbXV1d2tvbO6Ttqaee4phjjkn+kK39no/lrd3eimXaZJjUnvFIs2Pjxo2cc845mRWvTP17GUYD0dnpmcJK6ejwzEyjFRFZrapdpe22csmSSe0wawa8v8t7L7CwGEbeNIpz3B9nmLAAzJ1b1+E0DCYuoxgruV8hGeczjUYaxTkeHGcUK1ake14jCGoWmLiUwcyGyRg1v1MN8pnyohYTXdJnNopzPGycpST1uTSKoGZFLuIiIleIyOMi8oSI/J1rmyAiK0VknXs/xLWLiFwrIutFZK2InBh4zjx3/ToRmRdonykij7l7rpUKU8rHjh1Lf3//6Jk4K0RV6e/vZ+zYsXkPpfbUIZ+pHtRiokvzzCyd47VcDSQdT5K+G0VQs6LuDn0ROQ64HTgJ2A3cC1wCzAd2qOqXRWQhcIiqflZE5gKfBuYC7wEWq+p7RGQC0At0AQqsBmaq6ssi8mvgb4FHgBXAtar607hxhTn09+zZQ19f35C8ECOcsWPHMmXKFMaMGZP3UGrLA73R594/zKdZWGrhnE7zzKz69wWtdNJub4fFi6uPPovztYTR1hYd9dbU5IluKSLD/15pJKIc+i05jOUY4BFV3QUgIg8AHwHOBT7grlkG/BL4rGu/VT0VfFhEDhaRw9y1K1V1h3vOSuBMEfklMF5VH3bttwLnAbHiEsaYMWOYNm1aRV/SGKEc0BqeGFtFPlMe1CKsNs0zFy0aLgqV5LlEma36+73nQ3UCEzbOOPyVSFifU6eGC1WRE0erIQ+z2OPAn4tIu4i04a1IDgcmqeoL7poXgUnu82Tg+cD9fa4trr0vpN0wqqcG+Ux5UItChmmeGZfnksbMFSeGWZicSseZhKgxNcLmY1lSd3FR1aeArwA/wzOJPQoMlFyjeKaumiIi80WkV0R6gyVVDCOSSe1wZMfgSuWAVu+4wcLOazHRpX1mWDmStL6gcmK4aVNyP0yUqAXH2dFR/jlRY2rExNGqUNVcX8CXgAXAM8Bhru0w4Bn3+TvABYHrn3HnLwC+E2j/jms7DHg60D7kuqjXzJkz1TBGE8uXq3Z0qIp478uXZ/dMUG1u9t7TPNu/t/TV0RHdX1tb+D3BV1tb/BjCnuPfE/yd2ttVW1sr72ckAvRq2Nwe1ljrF/BH7n0q8DRwMPA1YKFrXwh81X0+G89fIsAs4NeufQKwATjEvTYAE9y5X7trxd07t9yYTFwMo3qWL/cm4EonXZHwSVskfZ9JBUo1WtTa24eLzpgxXrsvNv7nrAS60SiauPwH8CTwO2COa2sHVgHrgJ8HhEKAbwHPAo8BXYHn/DWw3r0+EWjvwvPtPAtch4uKi3uZuBhGdZRbRcRN7j5pVy6l/UfdX06gokStmu8yWogSF6st5ggLRTaMYTRY/bh6Ui5stzTktqfHc7hv3uz5KXzfTFgUWRrfRLkw57B+r7oqXchxo4cPZ4nVFjOMahlB2flZUq72lk/Q0R3luIfqnd5xgQVR/c6dG35Pe8TfDSM1fDhL8shzMYzGJC47f5SuXqKSGEspjRqLy1avdiMr/97S1Ul3tyeCYf2uWOGJWNKV1EgNH84SM4s5zCxmlGWEZOdnSZIVS1i2fF7Z6pX0G2ZGG7HhwxVgZjHDqJYa7Dba6MQlMXZ0wPLlsH378Mm4FkmcSaik33pvDzxSMHExjKSMkOz8LImalH3nedREnFe2+mjLks8TExejsann3iojJDs/S8Ima5HymfF5ZauPuiz5HDFxMRqXPKK3Ruluo3GlUfzJGrwJ2/dpBEu3FGmTrFIz169+BS0t3thbWmDBgvzGNpIwh77DHPoNyMNroysUz5pR//GMUMIiwsJyT6Kc++3t8Ic/DL2/tdUToT174p9ZaxYsgCVLhrdfeilcf339xtHIRDn0TVwcJi4NiEVv1YWke69ERWKlobkZli2rn8C0tMDAwPD25mbYu7c+Y2h0LFrMGHlY9FZdSLpPSxaRXgMD9d36N0xY4tqN5Ji4GI1LLaO3SgMFfr+xfoEDBSNp+G5UJFZUlnsUu3bBFVd4KybfDyJSG19Nc3O6diM5Ji5G41Kr6K2wQIEXtlcWOFDPaLYqiXK6Jw3fjYrEWrx4+P3l6O8fNMX5q4hye7tUgl9yJmm7kRzzuTjM52LsJypQoJRygQO+SAVTv5uaChm+XM5pX0mWevCeCRPgrbfg9derH2upr6daFizwvufAgLdimT/fnPlpMId+GUxcjP3EBQqUEhc40EDRbEmd9kkJE6tgmHI1WEXiYmEOfcNIStKAgHLXRa1+kqyK6kxSp30YQXPaoYd6rwsvHF4gMomwJNmn3ioSNwYmLoZRSligQClJAgcaKJqt0lpfpSXs+/u9V6UkEaC5cyt/vk+RkjpHKiYuRnEoivM7LFDgsEPTBw40UC2ySmtuhZXOjyPJyqQcK1ZUd3/Uni4mMNliPheH+VxypoGc36looJ0r0zjt/WvT7N7Y1gbz5sEdd8SvbsIy+oNU63PJ2r802jGfi1Fs4jbiKgKVrqrqXIusGnNP0tLywb/8k+KHJV9/vVeCf/ny8PyXtjYvdHnp0uhck2p9LtX4l0YK9TALmrgYxaDIzu8G2d64XuaeNKawtjZPSErFqrt7UGSiKhQffHD489KUxw+bRPPaS6Yo1Ov/ExMXoxgU2fld9FWVI27r4HKk+Us26V/448aVL0QZtlryJ79S01l7e7rCllGT6Ny5o3tPl2r+P0lDLuIiIv9DRJ4QkcdF5PsiMlZEponIIyKyXkR+ICKt7toD3PF6d74z8JzPufZnROSMQPuZrm29iCzM4SsaaSmy87vIq6oAlZp70v4lm/Qv/DfeGHx+GhNM1Mpo3Lh0BS2jJtEVK0b3ni71MgvW3aEvIpOBB4F3quofROQOYAUwF/ixqt4uIt8GfqeqS0RkATBDVS8RkY8DH1bVj4nIO4HvAycBfwL8HDjSdfN74DSgD/gNcIGqPhk3LnPoF4CiOr/rnQxZ4e8QV/J+3LhoR31aB3dYgmQUYc75MWNg/HjYsSN8PJXscx9GVs8ZaWQd0FA0h34L8DYRaQHagBeA2cCd7vwy4Dz3+Vx3jDs/R0TEtd+uqm+p6gZgPZ7QnASsV9XnVHU3cLu71ig6Rd2Iq56rqir8O2HhxGPGwGuvxa9K0v4lW1pDLK7IY3//cBHas8drjxpP1MpowoTofsIY7b6VKOq11XPdxUVVtwBfBzbjicqrwGrgFVX1d1DoA/x/uZOB5929e9317cH2knui2ochIvNFpFdEerdt21b9lzOKRVZ5M/Xc3rgK/05Y4cjx42F3yaKr1L6eZhL2TVwXXeQd33abt/9KNfkru3Z5Gf3+DpCLFnmbiZWyc2c6p3PUFsxZJGE2MvXa6rnu4iIih+CtJKbhmbMOBM6s9zgAVHWpqnapatfEiRPzGIJRK7KO8KrXqqpK/47vIL/tNu84Kp9k06ZBH8gRR4Rfs337UD9JlG8G4JJLhgtM2nL7S5Z4AtPdDQcdNPz8nj3pnM7d3V5eTXBcqp4YJhGpkZzFnzTsvBryMIt9ENigqttUdQ/wY+AU4GBnJgOYAvh/qm0BDgdw598O9AfbS+6JajdGEw0S4TWMDKLmkuah+AJx//3h5994Y6iIXHFFdJTR9dd7glZtuf2lS733HTvCz6d1Oq9YMdzvkiQyyrL4qycPcdkMzBKRNuc7mQM8CfwCON9dMw+4232+xx3jzt+vXhTCPcDHXTTZNGA68Gs8B/50F33WCnzcXWuMJpKsAIpSbiZIBv6dtCVZksT07NoVvQryJ/ywv4ZLTTDt7d7mX1H4e7ek9ZdErTIqjYyqV7juSCYPn8sjeI75NcBjbgxLgc8Cfy8i6/F8Kje6W24E2l373wML3XOeAO7AE6Z7gctUdcD5ZS4H7gOeAu5w1xqjiXIrgDolRqY2rWTg36l3pnk5B3lQdBYvjvfP+MEBUX6RsPa4VUalTn3L4s8AVbWXKjNnzlRjBPHidtV/X636y98Mvv59tdeuqvrQ74ae818P/S6zISxfrtrWpupNed6rrc1rryUdHUP79F8dHdHnkrza28t/n+XLvT5EvPfS71qu/0svLf8d0nzfSv8bpOl/tAP0asicahn6xsik3AqgDomReZlW4kJNw84lIVjzKyzKqKdncB+X0hXEBz/omcJE4v1Al14Kp5wSnYcB4SuHuFVGpZFR9QrXHdGEKc5ofNnKZZTx4JrwlcuDazLrQiT8r1+RzLqI5NJLVZubvf6amwdXBKrDVxft7fGrCZGh95cStjpI+4paZVS7cqmGcqswwwNbuRijgqRO+igndgLndlLySuLr6fHCbX3n+MDA0PDbUsd7uYgu1eF7qAR9SfPmpQsgKMVfEZQLRIhaOdRqlVGPcN2RjImLUSyqieBK46T3Z96k7RWQl2klyhx3xRXhwQVB01EUQdNTqQO90p+s1EwV5yzv6PBE7Kqr4sc/GmuFFRXbLMxhtcUKQLUbhqWpARZ1rX99sJ5XoNbX63tb+YcbJnPdHe1lN9SCwU21Tpnez1cu2cLk9t3I2NrWTYuqqVVKW9vwSThJ3ak4n0hSmpth796hbXF9L1o0vJ5Z2PiN+lNVbTERiakeZBgZUW3iYxonfVg+SfB6f8VTshoa17Kbb/7NBq799MZEiXXd3bDxkX56/tcmphy62wvDrfF+MEnNbmHBBUlWW1mE4/qZ/Un7tryTxiOpWWydiHzNVSI2jNpQbQRXmuz20miyUvbtg/Wb4ekNwwSvqQkWnLedC+b0J5vg6lwtIE1EWKlQJDExRYlXc/NgouSYMdF9zpnjZfSXEte35Z00HknF5V14Zey/KyIPu4KP42s4LmM0Um3pk7TZ7X69sCj2RjsTmprgS5/yxKHsBBcnmhlWBggWlXzb2wbresUlLYYJRTlHdtQKY9ky757t2+Hmmwd9OH5iZEeHt/Pkz38ePZ7S2mgXXuiFMUeZ+UZ7heMiE1OIYRBVfQ24AbhBRN4PfA/4hojcCXxRVdfXcIzGaGHa5HCfS9LSJ74PI24vlLC9Ug5orSi/ZeofefeUneDinu+byILjr4DSPVb6+wcLR0aVbak0uMAXm6uuit4jxi/9Ugml3yUqYMDyTopNYp+LiHxIRO4Cvgn8C/AO4P/gbfRlGNWTRWn7uOrFUdFkYyvbSnnzS63JJrg4/w5kYiKL8klECQsMTYBMW/23dHUD2VUQTlIbzSLCik+ilQuwDq+w5NdU9f8F2u8UkVOzH5YxapnUXl0UVdwujlG+j1dfT93NG2828a93TU42wZWuqMKosjJAWt9DR8fQ/er9yTxYRj/tXvXVPCNIue8iUtmOiUZ9KbtycZFit6jqJ0uEBQBV/duajMww0lIuzyWr0i4HtHLg8R1ce3t78snTX1FlUFI/jCjTXHt7fPRX2iissFVO1pFc5cyM5mdpDMqKi6oOAOfUYSyGUR3lorLSTuBNTYP3NDdDS7KI/FgzU8qgA79ml4j3OvTQcJNTXITYvHnR0V9porCiqg9H5bxUmgsT913Mz9I4JDWL/UpErgN+ALzhN6rqmpqMyjAg3sQVRjmT07TJXmhxUvbt81Yb/opoYN/g8yKc8GVNREmCDgLP+uu/HrpNcX8/fOITgecx9PMVVwz1s/T3e1FcUea7qVPDRSC4OvBXJ2HXxflGmivMjuvuhl/9Cr797aFRYiKeUJqfpTFIlKEvIr8IaVZVnZ39kPLBMvQLRrls/TDhifJpNDfD+07wPv/qt7EhxsN4f1fZzH9/8t282RtiWHRTMMM9iuBzpk6F11+PdshHPS9Jhn1pn3GZ72Hn01BpAZC038PIj6gMfSv/4jBxKRhxE3pUyPKkCfDC9vDnNTfDdPfneOm9Ufii9ED0/xc9fV2JJl+R+C7TTuJRz4sq/RLXf6moBcOKk5R6aW6uXFCjqOR7GPlQVfkX94CzReQzIvK//Fe2QzSMAHEmrijfyo6d0X6RgYFBU1ZcZn4QX4xinPBJtxQu54ROuzVx2orLcf2XJi5edNGgr6hc5FZbG3zgA8MTNdP6Rkr9VBMmhF9nzvzGIWmey7eBjwGfBgT4r0BMDVXDqJK4qKo44YkzefnO/WAuTBy+H2Ta5OGzpwhMm8zmzXDBnH423L6Wgft72XD7Wi6YM9SWlWSiTRNKPGZM9PMqrcQc5ayPmuRhsFLxQw9V5xsJ63vnTmgt+V/AnPmNRdKVy3tV9WLgZVX9R+Bk4MjaDcsY0SQpqx8XVRUnPOVWJG/tHtpflNe5tL3URuOOL//Lfm64chOdf7zb+6v7j3dzw5Wb6D6tP1X596R/kbe3e6VVop5Xafn5qHBiCBer5cu91c6KFcPvC9v/JW3fe/bAQQdZGf1GJqlD/xFVfY+IPAx8BOgHnlDVI2o9wHphPpc6kaasflS0WNwzoHxEWLC/KAd/SzOc4oIAYvw/r7/hVUou5fW9rYybE1O3rIRyPpf2dq9mV62I83Hcdlu0TyYL30i9/StxPiYjPdX6XH4iIgcDXwPWABuB71c4kKNE5NHAa6eI/J2ITBCRlSKyzr0f4q4XEblWRNaLyFoROTHwrHnu+nUiMi/QPlNEHnP3XCsSV7rPqCtpKgRHlXKJKxOTJLs/2F+UGS3YHmOGCxMWCBecOPwVRxRxZVyyIGrl1NTk+WDAE5nSQpZZ7LZZzx07o8x/1ZSrMcJJJC6q+kVVfUVVf4TnazlaVT9fSYeq+oyqHq+qxwMzgV3AXcBCYJWqTgdWuWOAs4Dp7jUfWAIgIhOAq4H3ACcBV/uC5K75VOC+MysZq1EDsip/EldDLImz3u8vScZ8JWa4FAmbwWrGeRGVuDgwED8JZ7HbZj137LR9YepHrLiIyEdKX8DZwBz3uVrmAM+q6ibgXGCZa18GnOc+nwvcqh4PAweLyGHAGcBKVd2hqi8DK4Ez3bnxqvqweja/WwPPMvKmRuVPhlCuUGSwvyQZ83HXpC3zX0LpX9JRtCdYkFVDd7fnhI9b44dNwllsMVzJMyoptgm2L0w9KZeh/19izinw4yr7/ziD5rVJqvqC+/wiMMl9ngw8H7inz7XFtfeFtA9DRObjrYaYajGO9aHasvpJKFcoMthfkoz5JNekqSQQIEkI8pgxsHhxoscB6X0KcRn4pYRNwtWU16/kGdUUykxSkcDIhlhxUdVP1KpjEWkFPgR8LqRfFZGaZ3eq6lJgKXgO/Vr3Z5Cq/EnV/fjPLFdGJomvJu6aKio5x/3FLJLe4Zx24k2bvFmESTjOtFXud1q0KLwigYU4Z0/S2mKIyNnAscBYv01Vr6mi77OANaq61R1vFZHDVPUFZ9p6ybVvAQ4P3DfFtW0BPlDS/kvXPiXkeqMoVFtWv+j9pSDqL+lKs9vTTrxpkjeLMglXY9pKstGZkQ15JlFewNCIs3sAP+JrHnB3oP1iFzU2C3jVmc/uA04XkUOcI/904D53bqeIzHJRYhcHnmUYhSJrZ3baiTfJ3ilQrDyTaqPLym3jbGRD0pXLe1V1hoisVdV/FJF/AX5aaaciciBwGvA3geYvA3eIyCeBTcBfuvYVwFxgPV5k2ScAVHWHiHwR+I277hpV3eE+LwBuAd7mxlnxWI0CkqRactqKymWu9/0Sp0zv59q/3cyEgwa8ibelGY6YWvHKKOu/pNP6FKKuB09QivhXvZm2GoOk4vIH975LRP4E2AEcVmmnqvoG0F7S1o8XPVZ6rQKXRTznJuCmkPZe4LhKx2cUmNIEytLy91v7Yd3moZUU/Wtefc2rP1YuKbPkmb5f4tyT+7nxMxuG7oq8dwCe2TjYfwLCHO5ZVfpNO/FGXV+UVUoYZtpqDNImUX4VWA1soMIkSsOoirgkzP37roQkRu7b51VMDtulMuqZT2+AB3p5/9i1nHtyP1/61JahwuKjOiQJNC5MttZJfGnDerMIJc4DM20Vn9jyLyLybuB5VX3RHV8MXAg8DXwhYIZqeKz8S4MQU/4+tqhllff4/0xiaz28v6vs/ii2T4kx0qi0/Mt3gN3uAafi+UW+A7yKC+E1akClGWKjgUqqJcfhm8jK4G8zXG5c5TLALYnPGC2UE5fmwOrkY8BSVf2RK/0yYopWFgorfhRPJdWS4/B9L+Uy+uNw5fehvHjUs46WYeRJWXEREd/pPwe4P3AucY6MkQIrfhRPXNHKuCx/kWhRKn1mGVQDpVpamuGozv3O/HLiUc86WoaRJ+UE4vvAAyKyHS9i7D8AROQIPNOYkTVmNylPVFLkpPbocvuqngis3zxY8VjEiyALhiAfPc1ri9ouGZCxrV6xzBDKRWtZpJMxWihX/mWRiKzCCzv+mQ56/5vwEiqNrBntxY/S5qeUEuV78Vcl+wIBLAMDQ0Xkrd1eWHFcBckyddCSiEcWtbgMo+iUNW25SsSlbb+vzXCMUZ0hVi6HJQlxhTHDQo5LiROWhGJn4mEY5jcpHqPZbhKXwxI1oYetdI7sCF/9lNuhshy+QD29wdsGWfBMbLUqvmkYDYyJSxEZrX/6JtlILCgmLc3Dd4z8/SZPXMJ8IpWGK/sExSmsAgCYwBiGo4r4S8PImHIbiflmM18gwrYojtoyGWDC+OrHGEVcv4YxCrGVi1Ecym0klsRnAoPiE1zlNDcnu1fEiyqrxIRWzarIMEYYJi5GcSi3kVjSyfuA1uHBAWH1xsKIc+gn6dcwDMDExSgacRt7JfGZpIkMi2LDFm+lk1SQgv0ahgGYz8VoJMLKtIh4QgBDs/WrMVG9tTtamPy+mpu9gILSfg3DAGzlYjQS5cxmQaqNDAszj7U0wyknVP5Mn1psdmYYBcPExWgs4sxmQcKCA0p5+zh4bdfwAIKoe/YOwMNrq5vokySKZpFMahg5Y2axImIl96vHL0YZx6uvDxUS37wV55gPbjJWCXGJommuMYyCY+JSNKzkfnak+Ss/WCG5XAn+aib6JImiSa4xjIJjZrGiEVdyfyRn7dfKx5DU9xIsM1Pq2wkj7plx36VcYc2k1xhGwbGVS9EYjSX3SzPvqzU9BUmxEdi+N3cPLhAntXslZMpVDSil3HeJ2+wsbswW6mw0GLmIi4gcLCJ3isjTIvKUiJwsIhNEZKWIrHPvh7hrRUSuFZH1IrJWRE4MPGeeu36diMwLtM8UkcfcPdeKxG5QWyyy2qqwkfw2WfkYtvZ7DvcHer33rf2Dvhc/hDiGzVtbh1sgo0rGRLWX+y5xm535JLnGMApOXiuXxcC9qno08C7gKWAhsEpVpwOr3DHAWcB095oPLAEQkQnA1cB7gJOAq31Bctd8KnDfmXX4TtlQyVaFpUKyYEFj+W2y8DHErRgmtcP7TvA2AotYcbzxZhP/cMPk4Zt+7tgZ3l9Ue5nv0tMDne9pp+m9M+ic10XPszOiNz6bNQPe3+W9m7AYDUbdxUVE3g6cCtwIoKq7VfUV4FxgmbtsGXCe+3wucKt6PAwcLCKHAWcAK1V1h6q+DKwEznTnxqvqw25zs1sDzyo+3d2wdCl0dHgJgh0d3nGUvyUsAODb326srZLTmp7CSLL6CU7Yhx3K3r3eT7Z3L9y8YgLfX+VN4EMskGmFL+a71DpWo5EWq8bIJ4+VyzRgG3CziPxWRL4rIgcCk1T1BXfNi8Ak93ky8Hzg/j7XFtfeF9I+DBGZLyK9ItK7bdu2Kr9WhnR3w8aN3uS4cWO8Iz8sACCqPlYlfpswU1PWZOFjSCMCW/th6w5aWjz9bmmBT8zdwQVzvO82xAKZUPj8ib3785PZ9Vb4d4mL1agWCzI0ikYe4tICnAgsUdUTgDcYNIEB4FYcVVQQTIaqLlXVLlXtmjhxYq27qw1pBCOt36aWjvYgWfgY0qx+QlY5B47dx5c+tWW4BTKB8AUn9u/9vJ3//tUONm9t9TQ+8F1qGatRS+EyjErIQ1z6gD5VfcQd34knNludSQv3/pI7vwU4PHD/FNcW1z4lpL2xSGrjSCoYlWyVXM9kvmp9DGlWPxGrnKmTdg+3QCYQvtKJ/fur2un42Aym/beh3yWrWI0wRmOQoVFs6i4uqvoi8LyIHOWa5gBPAvcAfsTXPOBu9/ke4GIXNTYLeNWZz+4DTheRQ5wj/3TgPndup4jMclFiFwee1RiksXEsWgStIX+dNzdDe3syv00UjZTMl2b1E7HKaRrbGv4TlRG+pBN7JbEaSamlcBlGJeQVLfZpoEdE1gLHA18CvgycJiLrgA+6Y4AVwHPAeuAGYAGAqu4Avgj8xr2ucW24a77r7nkW+Gntv1KGpLFxdHfDQQcNbx8YgHHjkvltosjC0V5Pkq5+onJf9g5UZPJLOrGnjdVIQy2FyzAqQbSazZFGEF1dXdrb25v3MDyamsKd8iLhRRXTXp+U0gKKfl8jIediaz+s3zx8q+QKvp+/0Az+PdDWlp1wpBnHVVd5K6apUz1hGclFHYxiICKrVbWrtN0y9ItIGhtHT090Bnq1NpGRnMw3qT08sbICn1ItVyRpx5E0yNAwao3VFisCpX9yHnGE9zm4Gmlrg7lzPee+f93cubBsWfiOiVnZRJKWuC86YfW+MvQpdXfbZG4YQUxc8qbUprJpk/cqpbPTE5Lgdd/+dnROy9veVpPhNiRR+6NEbWVcVJ+SYTQQJi55E+a8D+PJJ4e3xfnL+vs90QL7kzoqpLqlefjmYFYg0jAywXwueVPLRATLovOIMnPtHRi5PiXDyBkTlzwIJkgmLAdfMZs2WbGpuJBqKxBpGDXBxKVe+IIiAhddNJggGWbzj6J054CkOwmM9mJTtj+KYdQdE5d6EMy4h3BfSbn9RkRg9uyh8a5pc5RGq5lsJIdUG0ZBsSRKR02TKDs7wyPAqqHdTYz9KTPKq02szJNabYWcd1+G0cBEJVFatFg9qIXTPk5U4oSnUYtNRYUTQ/aTfj37MowRipnF6sGECfXtr78fdu4cXtCykYtN1bNCcz37MowRiolLrenpgddeq3+/e/Z4BS3zrkmSFfWs0NxI1aANo6CYWazWXHUV7A6ZlEqT92rBjh2wfXtt+6gXB7SGT+61yKavZ1+GMUIxcak1Uf6Wffu8FUVcQEW1AtSo/pUwpk0Or9CcdTjx1v7hlZLT9GWBAIYBmFms9kRN8B0dcMkl0fdVKyyN7F8Jox7hxL4jvzT3qKU5WV/12hbaMBoAE5daM3dudPv118OBB4af37dvMOorCePGDSZVNjfDvHmN61+JotbZ9GGOfPB+zyR9WSCAYezHxKXWrFgR3d7TA2+8EX9/2BbGpVx6qTeJ+Sa2gQGvgvJozMavhmod+RYIYBj7MXGpNXEbrJfLlu/vh717y/exdGn0tsjBOmajub5YEqrd1rnRtoU2jBpi4lJr4naVTJK1n8TvElWfzK8n5tcxG831xZJQbQ0yq2FmGPsxcak1ixZ5zvUgItmWg4mqS9bcHL2iMYZTbdCA1TAzjP3kIi4islFEHhORR0Wk17VNEJGVIrLOvR/i2kVErhWR9SKyVkRODDxnnrt+nYjMC7TPdM9f7+5NWD64CqLMT8EN1r3BpS84WY6WkIjytrboFU0t95BpdKoNGrAS/oYB5Lty+QtVPT5Q8GwhsEpVpwOr3DHAWcB095oPLAFPjICrgfcAJwFX+4LkrvlU4L4za/pNglWPw8xP3d2wcWNllYyT8NZbQ49FvGgxX9BKGUn5L4ZhFJIimcXOBZa5z8uA8wLtt6rHw8DBInIYcAawUlV3qOrLwErgTHduvKo+rF7J51sDz6oNYVsVh5mf6rViUIU77oDXXx9+bqTlvxiGUUjyEhcFfiYiq0XEbfTOJFV9wX1+EZjkPk8Gng/c2+fa4tr7QtqHISLzRaRXRHq3bdtW+beJiwgLUs8VQ3//8KrI7e2NXV/MMIyGIS9xeZ+qnohn8rpMRE4NnnQrjppvNKOqS1W1S1W7Jk6cWPmD4iLCgqRdMYwdW9l4ohg3zoTFMIy6kIu4qOoW9/4ScBeez2SrM2nh3l9yl28BDg/cPsW1xbVPCWmvHWERYWPGeGapoIO/uztd1v2bb2Y6THPkG4ZRL+ouLiJyoIgc5H8GTgceB+4B/IivecDd7vM9wMUuamwW8Kozn90HnC4ihzhH/unAfe7cThGZ5aLELg48qzYEI8JEPAER8cxSpQ7+xYuHC1FbW3QZmLQ0N0cLmDnyDcOoE3msXCYBD4rI74BfA/9XVe8FvgycJiLrgA+6Y4AVwHPAeuAGYAGAqu4Avgj8xr2ucW24a77r7nkW+GnNv5UfEbZvn2d+Ki2z7zv4S4Woo8OL7CpXBiYJbW1e2ZcoATNHvmEYdUK0FqGxDUhXV5f29vZm87CmpvCQ49L963t6PMHJKqFy+fJBn4r/7M2bvRXLokXmbzEMI3NEZHUgpWQ/RQpFHjnEOfj9ZEsRuOii7ISludl7XtC/46+kNm40YTEMo66YuNSCuJIvQUHJctU4MGD1wwzDKAwmLrUgruRLVoLi+2zC6opZ/TDDMHLGxKVW1Lrky9y5MGGC1Q8zDKOQhFQ8NDKlVpP8kiXx5y3s2DCMHLGVS63JY5IXid5e2TAMow6YuNQKPyps06bBve3jaGryQomjKhmnQdW2OTYMI1dMXGpBsAQ/eJN9OYHZt8+7Z+5caM1gW1xz6huGkSMmLrUgrAR/Eqf+rl1eqfysAgDMqW8YRk6YuGRJ0BRWKf39sGdPNuMxp75hGDlh4pIVpaawMNrbvWrJWeGHOS9fbrXEDMMoFCYuWRFmCgvS1uYVlLz55mQO/lJK7wlGhIUVw7RNwQzDyBErXOmounBlVLFK8Cb7YOHIuGvDEIHZs+H++4fe19ZmImIYRq5Y4cpaE+Xf6OgYXjgyrS9EFdavHy5IFhFmGEZBMXHJirBilb7fw3f0NzXBoYfC9u3pnt3RER35ZRFhhmEUEBOXrIjye8Cgo1/ViwZLszFYa6snUHFl/A3DMAqG1RbLku7u4f6Pzs54R38c7e1eEID/zPnzhz7LIsIMwygoJi61plKzlchQ85kvMLa7pGEYDYCZxWpNpWarsPvidpcM+nX83SgNwzBywsSl1oQ5+oO0tg5PrExr7gomcNpulIZhFIDcxEVEmkXktyLyE3c8TUQeEZH1IvIDEWl17Qe44/XufGfgGZ9z7c+IyBmB9jNd23oRWVj3Lxek1NHf3u69fKf/TTd5iZXVJECGJXBamLJhGDmS58rlCuCpwPFXgG+o6hHAy8AnXfsngZdd+zfcdYjIO4GPA8cCZwLXO8FqBr4FnAW8E7jAXZsfQXPW9u2ek37qVM934gtAlLkrCRambBhGwchFXERkCnA28F13LMBs4E53yTLgPPf5XHeMOz/HXX8ucLuqvqWqG4D1wEnutV5Vn1PV3cDt7tpiUAsTloUpG4ZRMPJauXwT+Aywzx23A6+o6l533AdMdp8nA88DuPOvuuv3t5fcE9U+DBGZLyK9ItK7bdu2Kr9SQmphwopL4DQMw8iBuouLiJwDvKSqq+vddymqulRVu1S1a+LEidk9OC5yqxYmLCtcaRhGwcgjz+UU4EMiMhcYC4wHFgMHi0iLW51MAba467cAhwN9ItICvB3oD7T7BO+Jaq89vtnLX534Zi/wJvupU8PL8ldrwgpL4DQMw8iJuq9cVPVzqjpFVTvxHPL3q2o38AvgfHfZPOBu9/ked4w7f796pZzvAT7uosmmAdOBXwO/Aaa76LNW18c9NfkyYSuUcmavRYuGhx6PGWMmLMMwRhRFytD/LHC7iPwT8FvgRtd+I3CbiKwHduCJBar6hIjcATwJ7AUuU9UBABG5HLgPaAZuUtUnMh9t1AolqtRL0OwVtjeLYRjGCML2c3Gk3s8lajvj5mYYGBje7pfej7rPP28YhtFA2H4uWRPlgB8YiI/cspwUwzBGASYulRK3OVhc5JblpBiGMQowcamUuNySuAKTlpNiGMYowMSlUirNLbGcFMMwRgHm0HekdugbhmEY5tA3DMMw6oeJi2EYhpE5Ji6GYRhG5pi4GIZhGJlj4mIYhmFkjkWLOURkGxBSlyVXDgW25z2IlDTamBttvGBjrgeNNl7Ib8wdqjpszxITlwIjIr1hIX5FptHG3GjjBRtzPWi08ULxxmxmMcMwDCNzTFwMwzCMzDFxKTZL8x5ABTTamBttvGBjrgeNNl4o2JjN52IYhmFkjq1cDMMwjMwxcTEMwzAyx8SlIIjI4SLyCxF5UkSeEJErXPsXRGSLiDzqXnPzHiuAiIwVkV+LyO/ceP/RtU8TkUdEZL2I/EBEWvMeq0/MmG8RkQ2B3/j4nIc6BBFpFpHfishP3HFhf2OfkDEX/TfeKCKPubH1urYJIrJSRNa590PyHqdPxHgLNVeYuBSHvcD/VNV3ArOAy0Tkne7cN1T1ePdakd8Qh/AWMFtV3wUcD5wpIrOAr+CN9wjgZeCT+Q1xGFFjBrgy8Bs/mtcAI7gCeCpwXOTf2Kd0zFDs3xjgL9zY/FyRhcAqVZ0OrHLHRaJ0vFCgucLEpSCo6guqusZ9fg3vH+bkfEcVjXq87g7HuJcCs4E7Xfsy4Lz6jy6cmDEXFhGZApwNfNcdCwX+jWH4mBuYc/F+Xyjg71x0TFwKiIh0AicAj7imy0VkrYjcVLClebOIPAq8BKwEngVeUdW97pI+CiaQpWNWVf83XuR+42+IyAH5jXAY3wQ+A+xzx+0U/Ddm+Jh9ivobg/dHxs9EZLWIzHdtk1T1Bff5RWBSPkMLJWy8UKC5wsSlYIjIOOBHwN+p6k5gCfCneGacF4B/yW90Q1HVAVU9HpgCnAQcne+IylM6ZhE5Dvgc3tjfDUwAPpvfCAcRkXOAl1R1dd5jSUrMmAv5Gwd4n6qeCJyFZ5I+NXhSvZyNIq1yw8ZbqLnCxKVAiMgYPGHpUdUfA6jqVjch7gNuwJvEC4WqvgL8AjgZOFhEWtypKcCWvMYVR2DMZzqTpKrqW8DNFOc3PgX4kIhsBG7HM4ctpti/8bAxi8jyAv/GAKjqFvf+EnAX3vi2ishhAO79pfxGOJSw8RZtrjBxKQjOln4j8JSq/mug/bDAZR8GHq/32MIQkYkicrD7/DbgNDw/0S+A891l84C7cxlgCBFjfjowgQieXb0Qv7Gqfk5Vp6hqJ/Bx4H5V7abAv3HEmC8s6m8MICIHishB/mfgdLzx3YP3+0KBfueo8RZtrmgpf4lRJ04BLgIecz4BgH8ALnBhmwpsBP4mj8GFcBiwTESa8f5IuUNVfyIiTwK3i8g/Ab/FE8yiEDXm+0VkIiDAo8AlOY4xCZ+luL9xFD0F/o0nAXd5ukcL8D1VvVdEfgPcISKfxNuO4y9zHGOQqPHeVqS5wsq/GIZhGJljZjHDMAwjc0xcDMMwjMwxcTEMwzAyx8TFMAzDyBwTF8MwDCNzTFwMowCIyHkioiJS+CoHhpEEExfDKAYXAA+6d8NoeExcDCNnXD259+GVzv+4a2sSketF5Gm3l8gKETnfnZspIg+4ooX3lWRmG0YhMHExjPw5F7hXVX8P9IvITOAjQCfwTrzKDSfD/vpz/xs4X1VnAjcBi/IYtGHEYeVfDCN/LsArSAlesccL8P5t/tAVIXxRRH7hzh8FHAesdOU/mvEq4BpGoTBxMYwcEZEJeNWO/0xEFE8sFK/SbegtwBOqenKdhmgYFWFmMcPIl/OB21S1Q1U7VfVwYAOwA/io871MAj7grn8GmCgi+81kInJsHgM3jDhMXAwjXy5g+CrlR8Af4+0y+SSwHFgDvKqqu/EE6Ssi8ju8CsPvrdtoDSMhVhXZMAqKiIxT1ddFpB34NXCKqr6Y97gMIwnmczGM4vITt7lZK/BFExajkbCVi2EYhpE55nMxDMMwMsfExTAMw8gcExfDMAwjc0xcDMMwjMwxcTEMwzAy5/8DrtZ0jXKlhxYAAAAASUVORK5CYII=" }, "metadata": { "needs_background": "light" } } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "### Save Data\n", "We save the generated data in the set \"salary_data\" defined in `aerospike.set` parameter below and in the namespace \"test\" that was specified in the Spark context above as `aerospike.namespace`." ], "metadata": {} }, { "cell_type": "code", "execution_count": 10, "source": [ "# Turn the above records into a Data Frame\n", "# First of all, create an array of arrays\n", "inputBuf = []\n", "\n", "for i in range(0, len(ages)) :\n", " id = i + 1 # Avoid counting from zero\n", " name = \"Individual: {:03d}\".format(id)\n", " # Note we need to make sure values are typed correctly\n", " # salary will have type numpy.float64 - if it is not cast as below, an error will be thrown\n", " age = float(ages[i])\n", " salary = int(salaries[i])\n", " inputBuf.append((id, name,age,salary))\n", "\n", "# Convert to an RDD \n", "inputRDD = spark.sparkContext.parallelize(inputBuf)\n", " \n", "# Convert to a data frame using a schema\n", "schema = StructType([\n", " StructField(\"id\", IntegerType(), True),\n", " StructField(\"name\", StringType(), True),\n", " StructField(\"age\", DoubleType(), True),\n", " StructField(\"salary\",IntegerType(), True)\n", "])\n", "\n", "inputDF=spark.createDataFrame(inputRDD,schema)\n", "\n", "#Write the data frame to Aerospike, the id field is used as the primary key\n", "inputDF \\\n", ".write \\\n", ".mode('overwrite') \\\n", ".format(\"aerospike\") \\\n", ".option(\"aerospike.set\", \"salary_data\")\\\n", ".option(\"aerospike.updateByKey\", \"id\") \\\n", ".save()" ], "outputs": [], "metadata": {} }, { "cell_type": "markdown", "source": [ "### View Stored Data\n", "Use the Aerospike AQL utility to view the stored data." ], "metadata": {} }, { "cell_type": "code", "execution_count": 11, "source": [ "!aql -c \"select * from test.salary_data\"" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "select * from test.salary_data\n", "+-------------------+-----+-------------------+--------+\n", "| age | id | name | salary |\n", "+-------------------+-----+-------------------+--------+\n", "| 25.08233874902072 | 10 | \"Individual: 010\" | 58345 |\n", "| 54.98712625322746 | 160 | \"Individual: 160\" | 97029 |\n", "| 45.18908097916793 | 120 | \"Individual: 120\" | 80007 |\n", "| 22.79485298523146 | 34 | \"Individual: 034\" | 49882 |\n", "| 31.29085325544791 | 243 | \"Individual: 243\" | 70754 |\n", "| 44.89546664470054 | 159 | \"Individual: 159\" | 82870 |\n", "| 24.94527795295446 | 13 | \"Individual: 013\" | 47114 |\n", "| 42.71790399344467 | 128 | \"Individual: 128\" | 83366 |\n", "| 25.27733154496998 | 56 | \"Individual: 056\" | 47356 |\n", "| 26.87692699227393 | 47 | \"Individual: 047\" | 49425 |\n", "| 22.91486461859545 | 66 | \"Individual: 066\" | 43879 |\n", "| 23.62771625204453 | 60 | \"Individual: 060\" | 53345 |\n", "| 33.77525922313378 | 271 | \"Individual: 271\" | 59730 |\n", "| 25.65434773386738 | 32 | \"Individual: 032\" | 50250 |\n", "| 42.46289000913426 | 209 | \"Individual: 209\" | 69066 |\n", "| 34.72684230210607 | 232 | \"Individual: 232\" | 74119 |\n", "| 24.10398532641521 | 52 | \"Individual: 052\" | 49836 |\n", "| 25.385411666593 | 70 | \"Individual: 070\" | 48493 |\n", "| 24.39423332104133 | 24 | \"Individual: 024\" | 51889 |\n", "| 24.52123876549696 | 28 | \"Individual: 028\" | 56635 |\n", "| 35.68815148712056 | 238 | \"Individual: 238\" | 65318 |\n", "| 34.96677276547636 | 244 | \"Individual: 244\" | 61334 |\n", "| 24.47428614857203 | 91 | \"Individual: 091\" | 47162 |\n", "| 36.24668002275217 | 257 | \"Individual: 257\" | 80994 |\n", "| 24.90196021709543 | 74 | \"Individual: 074\" | 48638 |\n", "| 39.07899498554285 | 207 | \"Individual: 207\" | 72197 |\n", "| 42.70354738516746 | 119 | \"Individual: 119\" | 71501 |\n", "| 45.11193363577878 | 217 | \"Individual: 217\" | 76815 |\n", "| 24.04479361358855 | 9 | \"Individual: 009\" | 39991 |\n", "| 26.5744693830482 | 93 | \"Individual: 093\" | 56049 |\n", "| 35.42212899746488 | 291 | \"Individual: 291\" | 67829 |\n", "| 48.65041867984886 | 173 | \"Individual: 173\" | 80760 |\n", "| 24.76376081231679 | 81 | \"Individual: 081\" | 47918 |\n", "| 24.99476474249944 | 40 | \"Individual: 040\" | 46368 |\n", "| 37.41393522454604 | 240 | \"Individual: 240\" | 52542 |\n", "| 24.8206346967664 | 53 | \"Individual: 053\" | 47587 |\n", "| 35.36149798456427 | 299 | \"Individual: 299\" | 73583 |\n", "| 26.28828703337766 | 85 | \"Individual: 085\" | 59603 |\n", "| 35.18898568512165 | 186 | \"Individual: 186\" | 60407 |\n", "| 24.55643080984997 | 29 | \"Individual: 029\" | 42254 |\n", "| 36.20944651871691 | 278 | \"Individual: 278\" | 61692 |\n", "| 44.36528540877659 | 138 | \"Individual: 138\" | 80612 |\n", "| 42.797912685033 | 130 | \"Individual: 130\" | 69916 |\n", "| 24.06564069303854 | 6 | \"Individual: 006\" | 55035 |\n", "| 23.34541624240619 | 99 | \"Individual: 099\" | 44895 |\n", "| 50.8215535658812 | 106 | \"Individual: 106\" | 91658 |\n", "| 34.41467659655716 | 236 | \"Individual: 236\" | 70734 |\n", "| 25.30086646117202 | 7 | \"Individual: 007\" | 51374 |\n", "| 34.65124481816264 | 270 | \"Individual: 270\" | 63552 |\n", "| 46.86205839212964 | 199 | \"Individual: 199\" | 74419 |\n", "| 24.59740890434833 | 30 | \"Individual: 030\" | 53791 |\n", "| 44.62237914959798 | 164 | \"Individual: 164\" | 90424 |\n", "| 26.26436315509618 | 54 | \"Individual: 054\" | 55476 |\n", "| 45.84967817942239 | 125 | \"Individual: 125\" | 85134 |\n", "| 25.000494245766 | 12 | \"Individual: 012\" | 66244 |\n", "| 44.64230533331592 | 115 | \"Individual: 115\" | 77199 |\n", "| 35.03714220167261 | 248 | \"Individual: 248\" | 76750 |\n", "| 25.12969642655245 | 80 | \"Individual: 080\" | 56099 |\n", "| 36.19305472236984 | 295 | \"Individual: 295\" | 60250 |\n", "| 35.07305643561117 | 269 | \"Individual: 269\" | 77824 |\n", "| 35.13007434858572 | 258 | \"Individual: 258\" | 57509 |\n", "| 26.41998520350787 | 82 | \"Individual: 082\" | 40776 |\n", "| 31.80357885327798 | 223 | \"Individual: 223\" | 60439 |\n", "| 27.04404109036838 | 43 | \"Individual: 043\" | 55053 |\n", "| 48.36055753204719 | 114 | \"Individual: 114\" | 88859 |\n", "| 24.82054826105547 | 38 | \"Individual: 038\" | 55007 |\n", "| 23.9936736589131 | 51 | \"Individual: 051\" | 42171 |\n", "| 39.64510250993715 | 192 | \"Individual: 192\" | 72664 |\n", "| 46.53337694047583 | 101 | \"Individual: 101\" | 89019 |\n", "| 45.57430980213641 | 194 | \"Individual: 194\" | 94548 |\n", "| 25.24920420954561 | 31 | \"Individual: 031\" | 54312 |\n", "| 25.59043077849547 | 14 | \"Individual: 014\" | 51513 |\n", "| 42.56064799325679 | 142 | \"Individual: 142\" | 80357 |\n", "| 50.30396237031055 | 132 | \"Individual: 132\" | 78746 |\n", "| 43.67491677796684 | 141 | \"Individual: 141\" | 79076 |\n", "| 43.06512046705784 | 140 | \"Individual: 140\" | 78500 |\n", "| 43.66775304181341 | 145 | \"Individual: 145\" | 73062 |\n", "| 49.15999176564866 | 201 | \"Individual: 201\" | 87532 |\n", "| 44.35304300125181 | 198 | \"Individual: 198\" | 77081 |\n", "| 49.63878026576844 | 170 | \"Individual: 170\" | 84917 |\n", "| 45.86378643346934 | 151 | \"Individual: 151\" | 93977 |\n", "| 36.35641301228243 | 246 | \"Individual: 246\" | 64237 |\n", "| 24.59843481162659 | 90 | \"Individual: 090\" | 52175 |\n", "| 35.90122037525296 | 252 | \"Individual: 252\" | 54238 |\n", "| 24.99849612471949 | 20 | \"Individual: 020\" | 44682 |\n", "| 23.99306685642435 | 16 | \"Individual: 016\" | 46432 |\n", "| 40.9227899001288 | 168 | \"Individual: 168\" | 74485 |\n", "| 33.43000260281838 | 256 | \"Individual: 256\" | 64426 |\n", "| 24.32137934236393 | 49 | \"Individual: 049\" | 50590 |\n", "| 46.58598376039249 | 219 | \"Individual: 219\" | 80532 |\n", "| 34.94841606011511 | 284 | \"Individual: 284\" | 59297 |\n", "| 26.43387466557501 | 67 | \"Individual: 067\" | 58173 |\n", "| 37.83366632235595 | 218 | \"Individual: 218\" | 77712 |\n", "| 50.4687163424899 | 162 | \"Individual: 162\" | 96742 |\n", "| 26.28219492010451 | 15 | \"Individual: 015\" | 50004 |\n", "| 34.38071207570569 | 226 | \"Individual: 226\" | 61805 |\n", "| 32.03178607840392 | 285 | \"Individual: 285\" | 65147 |\n", "| 35.88192426220356 | 296 | \"Individual: 296\" | 83686 |\n", "| 40.94561371543077 | 144 | \"Individual: 144\" | 59101 |\n", "+-------------------+-----+-------------------+--------+\n", "+-------------------+-----+-------------------+--------+\n", "| age | id | name | salary |\n", "+-------------------+-----+-------------------+--------+\n", "| 35.33158066768564 | 273 | \"Individual: 273\" | 71157 |\n", "| 26.08538036032814 | 98 | \"Individual: 098\" | 53605 |\n", "| 50.5714412429367 | 156 | \"Individual: 156\" | 88377 |\n", "| 50.58123004549133 | 203 | \"Individual: 203\" | 91326 |\n", "| 25.62963757719123 | 100 | \"Individual: 100\" | 56483 |\n", "| 34.82926250847292 | 298 | \"Individual: 298\" | 73606 |\n", "| 45.08999506629362 | 111 | \"Individual: 111\" | 76434 |\n", "| 25.65864928914094 | 42 | \"Individual: 042\" | 54083 |\n", "| 36.46912399056626 | 282 | \"Individual: 282\" | 87464 |\n", "| 33.40940020738482 | 255 | \"Individual: 255\" | 85374 |\n", "| 23.30820819245901 | 77 | \"Individual: 077\" | 48478 |\n", "| 36.20663115220678 | 259 | \"Individual: 259\" | 75433 |\n", "| 43.87495649590418 | 188 | \"Individual: 188\" | 81210 |\n", "| 25.95497068258835 | 27 | \"Individual: 027\" | 49940 |\n", "| 25.32113509515113 | 50 | \"Individual: 050\" | 52924 |\n", "| 36.26390557216931 | 288 | \"Individual: 288\" | 72847 |\n", "| 35.9066590731213 | 224 | \"Individual: 224\" | 74906 |\n", "| 31.98857053353701 | 231 | \"Individual: 231\" | 75925 |\n", "| 28.88057978385297 | 261 | \"Individual: 261\" | 58736 |\n", "| 35.14194917811759 | 254 | \"Individual: 254\" | 65494 |\n", "| 26.33199940700781 | 55 | \"Individual: 055\" | 47052 |\n", "| 24.18689142200467 | 88 | \"Individual: 088\" | 43930 |\n", "| 34.61582850993675 | 294 | \"Individual: 294\" | 67240 |\n", "| 41.97248578452533 | 127 | \"Individual: 127\" | 78460 |\n", "| 35.37399618071812 | 262 | \"Individual: 262\" | 75711 |\n", "| 25.76750620371324 | 48 | \"Individual: 048\" | 50148 |\n", "| 36.5934205747499 | 249 | \"Individual: 249\" | 90706 |\n", "| 35.79943069800473 | 234 | \"Individual: 234\" | 67135 |\n", "| 42.54995976189524 | 165 | \"Individual: 165\" | 78846 |\n", "| 25.62534178026618 | 36 | \"Individual: 036\" | 43336 |\n", "| 36.350189394118 | 241 | \"Individual: 241\" | 83611 |\n", "| 39.39998454736205 | 279 | \"Individual: 279\" | 61970 |\n", "| 46.61088172055385 | 181 | \"Individual: 181\" | 83658 |\n", "| 49.27720175120505 | 213 | \"Individual: 213\" | 82175 |\n", "| 43.33014839145317 | 163 | \"Individual: 163\" | 72556 |\n", "| 33.8810753644531 | 293 | \"Individual: 293\" | 67542 |\n", "| 23.6363830959181 | 78 | \"Individual: 078\" | 52134 |\n", "| 23.21107381279437 | 61 | \"Individual: 061\" | 38736 |\n", "| 47.54481718739405 | 190 | \"Individual: 190\" | 69534 |\n", "| 24.85865699390717 | 65 | \"Individual: 065\" | 47914 |\n", "| 35.92189457867195 | 267 | \"Individual: 267\" | 64129 |\n", "| 44.52394611117357 | 148 | \"Individual: 148\" | 80480 |\n", "| 25.27611732265598 | 18 | \"Individual: 018\" | 45696 |\n", "| 50.83291154818824 | 187 | \"Individual: 187\" | 92796 |\n", "| 39.72256297603004 | 133 | \"Individual: 133\" | 72903 |\n", "| 34.48040526116174 | 289 | \"Individual: 289\" | 56548 |\n", "| 52.63646076333807 | 149 | \"Individual: 149\" | 90797 |\n", "| 23.93457888255205 | 39 | \"Individual: 039\" | 46888 |\n", "| 24.31403545898675 | 2 | \"Individual: 002\" | 47402 |\n", "| 49.90462415484514 | 177 | \"Individual: 177\" | 88836 |\n", "| 25.01929121864152 | 86 | \"Individual: 086\" | 61123 |\n", "| 51.04052349344122 | 214 | \"Individual: 214\" | 90306 |\n", "| 45.68587902024383 | 136 | \"Individual: 136\" | 83196 |\n", "| 26.14074137724695 | 83 | \"Individual: 083\" | 46115 |\n", "| 32.39363653301354 | 253 | \"Individual: 253\" | 72542 |\n", "| 33.0655111105172 | 265 | \"Individual: 265\" | 67828 |\n", "| 24.52904962958139 | 72 | \"Individual: 072\" | 52539 |\n", "| 26.25147475955581 | 8 | \"Individual: 008\" | 56764 |\n", "| 35.44696613908334 | 268 | \"Individual: 268\" | 73220 |\n", "| 47.80817585023234 | 110 | \"Individual: 110\" | 90108 |\n", "| 38.46251502919521 | 210 | \"Individual: 210\" | 67302 |\n", "| 56.14454565605458 | 220 | \"Individual: 220\" | 94943 |\n", "| 25.50623242826136 | 97 | \"Individual: 097\" | 54193 |\n", "| 23.64207332999615 | 35 | \"Individual: 035\" | 46737 |\n", "| 25.29664106310324 | 4 | \"Individual: 004\" | 50464 |\n", "| 25.77585254835976 | 62 | \"Individual: 062\" | 51768 |\n", "| 49.67310667314544 | 113 | \"Individual: 113\" | 85003 |\n", "| 34.18772325493474 | 251 | \"Individual: 251\" | 63143 |\n", "| 24.30898150222034 | 58 | \"Individual: 058\" | 44887 |\n", "| 37.55862230964154 | 229 | \"Individual: 229\" | 60490 |\n", "| 41.01749347248462 | 166 | \"Individual: 166\" | 80828 |\n", "| 45.83958306327029 | 157 | \"Individual: 157\" | 79086 |\n", "| 36.24289194006415 | 235 | \"Individual: 235\" | 75146 |\n", "| 35.61575446391777 | 225 | \"Individual: 225\" | 52483 |\n", "| 26.39629960543233 | 92 | \"Individual: 092\" | 45367 |\n", "| 36.66253741915471 | 242 | \"Individual: 242\" | 67597 |\n", "| 23.92403782223526 | 89 | \"Individual: 089\" | 45403 |\n", "| 47.15950850314334 | 107 | \"Individual: 107\" | 76049 |\n", "| 46.60405174417829 | 153 | \"Individual: 153\" | 83040 |\n", "| 44.5656609587089 | 158 | \"Individual: 158\" | 82307 |\n", "| 35.2134790582558 | 237 | \"Individual: 237\" | 63457 |\n", "| 45.84237457281951 | 200 | \"Individual: 200\" | 78791 |\n", "| 25.81033637987936 | 75 | \"Individual: 075\" | 53401 |\n", "| 34.65214128521281 | 239 | \"Individual: 239\" | 61747 |\n", "| 38.84745269824979 | 139 | \"Individual: 139\" | 69645 |\n", "| 33.97918907293992 | 272 | \"Individual: 272\" | 66496 |\n", "| 25.45785726602289 | 76 | \"Individual: 076\" | 46214 |\n", "| 43.1868235157955 | 147 | \"Individual: 147\" | 70158 |\n", "| 24.08476170165959 | 96 | \"Individual: 096\" | 46328 |\n", "| 31.90256940957829 | 263 | \"Individual: 263\" | 81678 |\n", "| 44.41128454132044 | 195 | \"Individual: 195\" | 73805 |\n", "| 43.47469416168524 | 169 | \"Individual: 169\" | 85090 |\n", "| 34.04728104778491 | 300 | \"Individual: 300\" | 67622 |\n", "| 46.70560735944277 | 161 | \"Individual: 161\" | 79156 |\n", "| 46.15744144083992 | 117 | \"Individual: 117\" | 71581 |\n", "| 24.10182248524402 | 21 | \"Individual: 021\" | 38202 |\n", "| 49.1773716696247 | 167 | \"Individual: 167\" | 70600 |\n", "| 46.72555213577564 | 197 | \"Individual: 197\" | 77958 |\n", "| 38.08819977056483 | 281 | \"Individual: 281\" | 75612 |\n", "+-------------------+-----+-------------------+--------+\n", "+-------------------+-----+-------------------+--------+\n", "| age | id | name | salary |\n", "+-------------------+-----+-------------------+--------+\n", "| 42.61365701705083 | 129 | \"Individual: 129\" | 77929 |\n", "| 36.47287618289909 | 118 | \"Individual: 118\" | 75390 |\n", "| 42.95306230074591 | 152 | \"Individual: 152\" | 76974 |\n", "| 26.41972973144746 | 5 | \"Individual: 005\" | 53845 |\n", "| 33.88880875612859 | 290 | \"Individual: 290\" | 72171 |\n", "| 24.45368999722374 | 64 | \"Individual: 064\" | 50538 |\n", "| 25.29309225873203 | 11 | \"Individual: 011\" | 47301 |\n", "| 24.38949434240006 | 87 | \"Individual: 087\" | 53636 |\n", "| 27.1259975101106 | 46 | \"Individual: 046\" | 50941 |\n", "| 32.83361942986529 | 230 | \"Individual: 230\" | 54148 |\n", "| 37.55702968716015 | 184 | \"Individual: 184\" | 67113 |\n", "| 37.8137082474738 | 250 | \"Individual: 250\" | 83404 |\n", "| 24.88105115844521 | 23 | \"Individual: 023\" | 43464 |\n", "| 23.61137828720559 | 22 | \"Individual: 022\" | 42290 |\n", "| 45.63248014149332 | 134 | \"Individual: 134\" | 77862 |\n", "| 42.23627729566275 | 154 | \"Individual: 154\" | 90586 |\n", "| 25.39547052370498 | 1 | \"Individual: 001\" | 48976 |\n", "| 25.25627511005183 | 26 | \"Individual: 026\" | 50349 |\n", "| 36.28570967548751 | 215 | \"Individual: 215\" | 67766 |\n", "| 36.73868279291174 | 247 | \"Individual: 247\" | 64397 |\n", "| 25.83091434262693 | 45 | \"Individual: 045\" | 49342 |\n", "| 34.15486648263456 | 276 | \"Individual: 276\" | 79019 |\n", "| 35.26964423384393 | 260 | \"Individual: 260\" | 72119 |\n", "| 33.88136686744861 | 227 | \"Individual: 227\" | 73672 |\n", "| 44.20600394284013 | 143 | \"Individual: 143\" | 87126 |\n", "| 38.29713989206178 | 137 | \"Individual: 137\" | 59536 |\n", "| 38.46678785970544 | 277 | \"Individual: 277\" | 62013 |\n", "| 33.83509646655938 | 266 | \"Individual: 266\" | 76086 |\n", "| 24.68241178653287 | 73 | \"Individual: 073\" | 50934 |\n", "| 35.34473280302399 | 274 | \"Individual: 274\" | 74003 |\n", "| 24.60518551971716 | 95 | \"Individual: 095\" | 48021 |\n", "| 46.39113119158637 | 150 | \"Individual: 150\" | 76906 |\n", "| 48.0286095157561 | 185 | \"Individual: 185\" | 89932 |\n", "| 24.78489604828937 | 63 | \"Individual: 063\" | 48627 |\n", "| 23.78200189530308 | 57 | \"Individual: 057\" | 54649 |\n", "| 34.14119442749095 | 286 | \"Individual: 286\" | 71691 |\n", "| 24.34404875265196 | 33 | \"Individual: 033\" | 54264 |\n", "| 39.37109603380626 | 103 | \"Individual: 103\" | 70650 |\n", "| 41.21442389919512 | 176 | \"Individual: 176\" | 78635 |\n", "| 45.79820423103285 | 122 | \"Individual: 122\" | 87523 |\n", "| 25.51215670533433 | 44 | \"Individual: 044\" | 45012 |\n", "| 41.60086451199921 | 175 | \"Individual: 175\" | 79559 |\n", "| 25.88749070267593 | 79 | \"Individual: 079\" | 48162 |\n", "| 37.09568187885061 | 112 | \"Individual: 112\" | 72307 |\n", "| 41.55122933798243 | 205 | \"Individual: 205\" | 74759 |\n", "| 45.96865034153888 | 202 | \"Individual: 202\" | 82850 |\n", "| 42.5158131767696 | 105 | \"Individual: 105\" | 73984 |\n", "| 36.27366840833148 | 222 | \"Individual: 222\" | 71161 |\n", "| 56.51623471593592 | 196 | \"Individual: 196\" | 80848 |\n", "| 37.15863767021896 | 104 | \"Individual: 104\" | 58844 |\n", "| 26.13768535640089 | 84 | \"Individual: 084\" | 49447 |\n", "| 37.51982175000117 | 116 | \"Individual: 116\" | 72841 |\n", "| 46.55767726882384 | 180 | \"Individual: 180\" | 74441 |\n", "| 41.81135705787229 | 211 | \"Individual: 211\" | 75883 |\n", "| 47.99445859151342 | 183 | \"Individual: 183\" | 77893 |\n", "| 41.08443489903744 | 121 | \"Individual: 121\" | 69059 |\n", "| 46.0366497775221 | 191 | \"Individual: 191\" | 84516 |\n", "| 43.97255703917137 | 189 | \"Individual: 189\" | 76196 |\n", "| 46.83459080530326 | 204 | \"Individual: 204\" | 83651 |\n", "| 38.40860526256105 | 283 | \"Individual: 283\" | 74555 |\n", "| 24.88165316362767 | 59 | \"Individual: 059\" | 51102 |\n", "| 45.16418356065846 | 135 | \"Individual: 135\" | 76797 |\n", "| 25.87531227356693 | 37 | \"Individual: 037\" | 53618 |\n", "| 41.06986441279633 | 155 | \"Individual: 155\" | 72680 |\n", "| 37.10710655471658 | 292 | \"Individual: 292\" | 71122 |\n", "| 24.10221251354065 | 94 | \"Individual: 094\" | 45128 |\n", "| 45.24321628641496 | 174 | \"Individual: 174\" | 85096 |\n", "| 45.58032665992633 | 212 | \"Individual: 212\" | 79747 |\n", "| 22.5138778236622 | 17 | \"Individual: 017\" | 38148 |\n", "| 41.27090317888393 | 179 | \"Individual: 179\" | 76524 |\n", "| 45.52700851448768 | 124 | \"Individual: 124\" | 83433 |\n", "| 36.05016810737868 | 228 | \"Individual: 228\" | 66749 |\n", "| 48.70499394129772 | 108 | \"Individual: 108\" | 78914 |\n", "| 46.81734082816165 | 216 | \"Individual: 216\" | 82378 |\n", "| 26.91895863598789 | 3 | \"Individual: 003\" | 59828 |\n", "| 44.81905792578452 | 123 | \"Individual: 123\" | 88321 |\n", "| 39.31901069861998 | 264 | \"Individual: 264\" | 79805 |\n", "| 44.55866761589274 | 171 | \"Individual: 171\" | 85346 |\n", "| 44.59708878629509 | 182 | \"Individual: 182\" | 88112 |\n", "| 34.70631177289261 | 287 | \"Individual: 287\" | 65348 |\n", "| 24.73452742986315 | 19 | \"Individual: 019\" | 43670 |\n", "| 32.97228800174097 | 297 | \"Individual: 297\" | 63852 |\n", "| 44.30064433030314 | 131 | \"Individual: 131\" | 71438 |\n", "| 44.35213108442584 | 109 | \"Individual: 109\" | 73225 |\n", "| 23.91100085165325 | 41 | \"Individual: 041\" | 50256 |\n", "| 39.3403878518148 | 208 | \"Individual: 208\" | 71164 |\n", "| 46.77721292571739 | 172 | \"Individual: 172\" | 72735 |\n", "| 24.9717546123257 | 69 | \"Individual: 069\" | 56533 |\n", "| 25.6669656008671 | 68 | \"Individual: 068\" | 52289 |\n", "| 36.46788539299514 | 221 | \"Individual: 221\" | 67791 |\n", "| 34.07230785121148 | 245 | \"Individual: 245\" | 71158 |\n", "| 45.82761289603616 | 126 | \"Individual: 126\" | 91152 |\n", "| 44.21687157148506 | 102 | \"Individual: 102\" | 82474 |\n", "| 51.40563656530654 | 193 | \"Individual: 193\" | 97698 |\n", "| 26.01254219983841 | 71 | \"Individual: 071\" | 45880 |\n", "| 26.38695770425844 | 25 | \"Individual: 025\" | 51656 |\n", "| 33.17865867212326 | 275 | \"Individual: 275\" | 72272 |\n", "| 35.07469885610691 | 233 | \"Individual: 233\" | 63830 |\n", "| 35.85881568977177 | 280 | \"Individual: 280\" | 62640 |\n", "+-------------------+-----+-------------------+--------+\n", "+-------------------+-----+-------------------+--------+\n", "| age | id | name | salary |\n", "+-------------------+-----+-------------------+--------+\n", "| 44.76894184915611 | 146 | \"Individual: 146\" | 76503 |\n", "| 51.28350713525773 | 178 | \"Individual: 178\" | 90077 |\n", "| 56.6362187203851 | 206 | \"Individual: 206\" | 105414 |\n", "+-------------------+-----+-------------------+--------+\n", "300 rows in set (0.190 secs)\n", "\n", "OK\n", "\n", "\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "## Load Data from Aerospike\n", "We will show multiple ways of loading data into a data frame." ], "metadata": {} }, { "cell_type": "markdown", "source": [ "### Load data without schema \n", "The Aerospike Spark Connector can infer the schema by reading a number of records. This method returns data as well as metadata fields such as \\_\\_key, \\_\\_digest, \\_\\_expiry, \\_\\_generation, and \\_\\_ttl." ], "metadata": {} }, { "cell_type": "code", "execution_count": 12, "source": [ "# Create a Spark DataFrame by using the Connector Schema inference mechanism\n", "# The fields preceded with __ are metadata fields - key/digest/expiry/generation/ttl\n", "# By default you just get everything, with no column ordering, which is why it looks untidy\n", "# Note we don't get anything in the 'key' field as we have not chosen to save as a bin.\n", "# Use .option(\"aerospike.sendKey\", True) to do this\n", "\n", "loadedDFWithoutSchema = (\n", " spark.read.format(\"aerospike\") \\\n", " .option(\"aerospike.set\", \"salary_data\") \\\n", " .load()\n", ")\n", "\n", "loadedDFWithoutSchema.show(10)" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "+-----+--------------------+---------+------------+-------+------------------+---------------+------+---+\n", "|__key| __digest| __expiry|__generation| __ttl| age| name|salary| id|\n", "+-----+--------------------+---------+------------+-------+------------------+---------------+------+---+\n", "| null|[03 50 2E 7F 70 9...|401502284| 4|2591999|34.652141285212814|Individual: 239| 61747|239|\n", "| null|[0F 10 1A 93 B1 E...|401502284| 4|2591999| 45.57430980213641|Individual: 194| 94548|194|\n", "| null|[04 C0 5E 9A 68 5...|401502284| 4|2591999| 46.53337694047583|Individual: 101| 89019|101|\n", "| null|[1A E0 A8 A0 F2 3...|401502284| 4|2591999| 25.24920420954561|Individual: 031| 54312| 31|\n", "| null|[23 20 78 35 5D 7...|401502284| 4|2591999| 38.84745269824979|Individual: 139| 69645|139|\n", "| null|[35 00 8C 78 43 F...|401502284| 4|2591999| 25.59043077849547|Individual: 014| 51513| 14|\n", "| null|[37 00 6D 21 08 9...|401502284| 4|2591999| 42.56064799325679|Individual: 142| 80357|142|\n", "| null|[59 00 4B C7 6D 9...|401502284| 4|2591999| 33.97918907293992|Individual: 272| 66496|272|\n", "| null|[6C 50 7F 9B FD C...|401502284| 4|2591999| 43.1868235157955|Individual: 147| 70158|147|\n", "| null|[61 50 89 B1 EC 0...|401502284| 4|2591999|25.457857266022888|Individual: 076| 46214| 76|\n", "+-----+--------------------+---------+------------+-------+------------------+---------------+------+---+\n", "only showing top 10 rows\n", "\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "### Load data using schema \n", "The schema can be explicitly specified, and only these fields will be returned. Note the `schema` used here was constructed earlier with id, name, age, and salary fields." ], "metadata": {} }, { "cell_type": "code", "execution_count": 13, "source": [ "# If we explicitly set the schema, using the previously created schema object\n", "# we effectively type the rows in the Data Frame\n", "\n", "loadedDFWithSchema=spark \\\n", ".read \\\n", ".format(\"aerospike\") \\\n", ".schema(schema) \\\n", ".option(\"aerospike.set\", \"salary_data\").load()\n", "\n", "loadedDFWithSchema.show(5)" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "+---+---------------+------------------+------+\n", "| id| name| age|salary|\n", "+---+---------------+------------------+------+\n", "|101|Individual: 101| 46.53337694047583| 89019|\n", "|239|Individual: 239|34.652141285212814| 61747|\n", "|194|Individual: 194| 45.57430980213641| 94548|\n", "| 31|Individual: 031| 25.24920420954561| 54312|\n", "|139|Individual: 139| 38.84745269824979| 69645|\n", "+---+---------------+------------------+------+\n", "only showing top 5 rows\n", "\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "### Pushing Down Query Predicate\n", "In order to get the best performance, it is important to minimize the amount of data retrieved to Spark. This is achieved by \"pushing down\" the query predicate or by processing filters in the database.\n", "\n", "The Spark Connector allows the \"pushdown expressions\" option for specifying filters to be processed on Aerospike. Note that it cannot be used in `load` together with the `where` clause.\n", "\n", "Below, we have used the Base64 encoding of a simple expression `id % 5 = 0` to get records where the id field is divisble by 5. Please see the notebook [Pushdown Expressions for Spark Connector](https://github.com/aerospike-examples/interactive-notebooks/blob/main/notebooks/spark/resources/pushdown-expressions.ipynb) for details on constructing an expresion and obtaining its Base64 repreentation." ], "metadata": {} }, { "cell_type": "markdown", "source": [ "**Compute Base64 Representation of Predicate**" ], "metadata": {} }, { "cell_type": "code", "execution_count": 14, "source": [ "import aerospike\n", "from aerospike_helpers import expressions as exp\n", "\n", "# Configure the client connection\n", "config = {\n", " 'hosts': [ ('127.0.0.1', 3000) ]\n", "}\n", "# Connect to the server\n", "try:\n", " client = aerospike.client(config).connect()\n", "except ex.ClientError as e:\n", " print(\"Error: {0} [{1}]\".format(e.msg, e.code))\n", " sys.exit(1)\n", "\n", "# Build the expression for id % 5 = 0\n", "expr = exp.Eq(\n", " exp.IntBin(\"id\") % 5, \n", " 0).compile()\n", "# Get Base64 representation of the expression for use in a pushdown-expression request.\n", "pushdown_expr = client.get_expression_base64(expr)\n", "client.close()\n", "print('The base64 representation of the expression \"id % 5 = 0\" is', pushdown_expr)" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Base64 representation of the expression \"id % 5 = 0\" is kwGTGpNRAqJpZAUA\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "**Load Using Pushdown Expressions**" ], "metadata": {} }, { "cell_type": "code", "execution_count": 15, "source": [ "dfWithPushdownExpr = spark \\\n", " .read \\\n", " .format(\"aerospike\") \\\n", " .schema(schema) \\\n", " .option(\"aerospike.set\", \"salary_data\") \\\n", " . option(\"aerospike.pushdown.expressions\", pushdown_expr) \\\n", " .load()\n", "\n", "dfWithPushdownExpr.show(10)" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "+---+---------------+------------------+------+\n", "| id| name| age|salary|\n", "+---+---------------+------------------+------+\n", "| 10|Individual: 010|25.082338749020725| 58345|\n", "|140|Individual: 140| 43.06512046705784| 78500|\n", "|160|Individual: 160| 54.98712625322746| 97029|\n", "|120|Individual: 120|45.189080979167926| 80007|\n", "|205|Individual: 205| 41.55122933798243| 74759|\n", "|195|Individual: 195| 44.41128454132044| 73805|\n", "|145|Individual: 145|43.667753041813405| 73062|\n", "|300|Individual: 300|34.047281047784914| 67622|\n", "|105|Individual: 105| 42.5158131767696| 73984|\n", "|170|Individual: 170| 49.63878026576844| 84917|\n", "+---+---------------+------------------+------+\n", "only showing top 10 rows\n", "\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "### Using Secondary Index\n", "First we create a secondary index on the salary field, and then retrieve data using the secondary index." ], "metadata": {} }, { "cell_type": "code", "execution_count": 16, "source": [ "# create a secondary index on salary\n", "import aerospike \n", "from aerospike import exception as ex\n", "\n", "client = aerospike.client({\"hosts\": [AS_HOST]}).connect()\n", "# create a secondary index on salary\n", "index_name = \"idx_salary_int\"\n", "try:\n", " client.index_integer_create('test', 'salary_data', \"salary\", index_name)\n", "except ex.IndexFoundError as e:\n", " pass" ], "outputs": [], "metadata": {} }, { "cell_type": "code", "execution_count": 17, "source": [ "# automatically selects appropriate secindary index\n", "dfWithSecIdx = spark \\\n", " .read \\\n", " .format(\"aerospike\") \\\n", " .schema(schema) \\\n", " .option(\"aerospike.set\", \"salary_data\") \\\n", " .option(\"aerospike.sindex.enable\", \"true\") \\\n", " .load() \\\n", " .where(\"salary >= 100000\")\n", "\n", "dfWithSecIdx.show()" ], "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "+---+---------------+----------------+------+\n", "| id| name| age|salary|\n", "+---+---------------+----------------+------+\n", "|206|Individual: 206|56.6362187203851|105414|\n", "+---+---------------+----------------+------+\n", "\n" ] } ], "metadata": {} }, { "cell_type": "markdown", "source": [ "## Next Steps\n", "To learn more about the Spark Connector, check out the [multiple tutorials](https://developer.aerospike.com/tutorials/spark). [Aerospike Connect for Spark Tutorial for Python](https://developer.aerospike.com/tutorials/spark/spark-py) is a good reference for the connector capabilities." ], "metadata": {} } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.6" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 4 }