{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Learning about spatial data and maps for archaeology (and other things)\n",
    "\n",
    "### Spatial Thinking and Skills Exercise 1 for Theory and Practice\n",
    "\n",
    "#### Made by Rachel Opitz, Archaeology, University of Glasgow\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Archaeologists regularly work with maps and data about where sites, samples and objects are found. Archaeological survey in particular relies on the collection and analysis of spatial data to understand patterns across whole areas of the landscape. Are most sites in a given landscape from a particular period? Are there more cairns than anything else? Are sites from all periods represented equally? These are the kinds of questions one might as using survey data. \n",
    "\n",
    "\n",
    "The aim of this exercise is for you to:\n",
    "* learn to investigate patterns in archaeological survey data using spatial analytical tools\n",
    "* learn to learn to make maps that illustrate patterns in archaeological survey data\n",
    "* start thinking about the meaning of the patterns of sites and features in the landscape\n",
    "\n",
    "You'll do this using data available from Canmore that describes the location and type of monuments surveyed throughout the Shetlands. Canmore houses survey data on sites and monuments recorded throughout Scotland.\n",
    "\n",
    "As you may recall from Archaeology of Scotland, to start working with spatial data and maps, you need to put together your toolkit. You're currently working inside something called a jupyter notebook. It's a place to keep notes, pictures, code and maps together. You can add tools and data into your jupyter notebook and then use them to ask spatial questions and make maps and visualisations that help answer those questions. \n",
    "\n",
    "\n",
    "### Let's get started... Hit 'Ctrl'+'Enter' to run the code in any cell in the page."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "[Canmore spatial data can be found here.](https://canmore.org.uk/content/data-downloads)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### We'll start by adding some of the tools we will need. They're not quite like these tools...\n",
    "\n",
    "![They're not quite like these tools...](http://www.truelles-pixels.mom.fr/english/episode01/knowmore_files/outils.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "# Matplotlib is your tool for drawing graphs and basic maps. You need this!\n",
    "\n",
    "import pandas as pd\n",
    "import requests\n",
    "import fiona\n",
    "import geopandas as gpd\n",
    "import pysal as ps\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn import cluster\n",
    "from sklearn.preprocessing import scale\n",
    "sns.set_style(\"white\")\n",
    "from shapely.ops import nearest_points\n",
    "\n",
    "# These are what we call prerequisites. They are basic toosl you need to get started.\n",
    "# Pandas manipulate data. Geo-pandas manipulate geographic data. They're also black and white and like to eat bamboo... \n",
    "# You need these to manipulate your data!\n",
    "# Fiona helps with geographic data.\n",
    "# Requests are for asking for things. It's good to be able to ask for things.\n",
    "# ipywidgets supports interactivity.\n",
    "\n",
    "\n",
    "# Remember to hit Ctrl+Enter to make things happen!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Now that we have the basic tools loaded up we need to load the Canmore data. Canmore provides polygons showing the site extents. I've converted them to a set of points located at the centre of each polygon to make things a little simpler in this exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "url = 'http://ropitz.github.io/digitalantiquity/data/CanmoreShetlandPoints.geojson'\n",
    "# This is where I put the data. It's in a format called geojson, used to represent geometry (shapes) and attributes (text).\n",
    "request = requests.get(url)\n",
    "# Please get me the data at that web address (url)\n",
    "b = bytes(request.content)\n",
    "# I will use the letter 'b' to refer to the data, like a nickname\n",
    "with fiona.BytesCollection(b) as f:\n",
    "    crs = f.crs\n",
    "    canmore_shetland = gpd.GeoDataFrame.from_features(f, crs=crs)\n",
    "    print(canmore_shetland.head())\n",
    "# I will use the fiona tool to wrap up all the data from 'b', check the coordinate system (crs) listed in the features\n",
    "# and print out the first few lines of the file so I can check everything looks ok. \n",
    "# Don't worry if you don't understand all the details of this part!\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Does that look right? \n",
    "\n",
    "You should see descriptions of different types of monuments and notes on them and links to the original data on Canmore. Get help if this isn't what you are seeing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Let's visualise the data to double check that all is well\n",
    "\n",
    "canmore_shetland_map = canmore_shetland.plot(column='CLASS', cmap='Pastel2', edgecolor='grey', figsize=(10, 10));\n",
    "# 'plot' means draw me an image showing the geometry of each feature in my data. \n",
    "# We want to control things like the color of different types of sites on our map. \n",
    "# I used the pastel colorscale command (cmap stands for 'colour map') \n",
    "# and asked it to draw the points differently based on the type they were assigned."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Everything good?\n",
    "\n",
    "If you see a bunch of pastel dots, you are on the right track.Once the data is loaded properly, we can start exploring it and seeing if there are patterns within this national survey dataset. \n",
    "\n",
    "Let's start by printing out the attributes of our points as a table. You'll be exploring the data based on it's attributes, so it is important to have a sense of what they are. Scroll down and start to ask yourself some basic questions. What are the feature types assigned in this dataset? Do they mostly have periods assigned to them?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "canmore_shetland"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmmm... If you're me at this point you are a bit concerned about all the sites with a 'PERIOD UNASSIGNED' note. Let's try and see how much of our dataset is actually dated in some way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#How many features do we have in total. Let's count them.\n",
    "canmore_shetland.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Now let's count the ones that have 'PERIOD UNASSIGNED' as part of the 'TYPE' field.\n",
    "\n",
    "canmore_shetland_undated = canmore_shetland[canmore_shetland['TYPE'].str.contains('PERIOD UNASSIGNED')]\n",
    "canmore_shetland_undated.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hmmm... quite a few. Almost 85% of the data! I have questions... \n",
    "# Let's look at the distribution of the points that do have dates. The ~ symbol means 'opposite of'.\n",
    "\n",
    "canmore_shetland_dated = canmore_shetland[~canmore_shetland['TYPE'].str.contains('PERIOD UNASSIGNED')]\n",
    "canmore_shetland_dated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "canmore_shetland_dated.plot(column='CLASS', cmap='Pastel2', edgecolor='grey', figsize=(10, 10));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Well, there are still some sites everywhere. Compare visually with the earlier map... \n",
    "# Make the dated sites red and undated sites blue\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "canmore_shetland_undated.plot(ax=ax, color='blue',)\n",
    "canmore_shetland_dated.plot(ax=ax, color='red')\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do you think the dated sites are representing the whole dataset well? If yes, you might feelc omfortable working with only the dated sites. If not, you might want to continue to work with all the sites. \n",
    "\n",
    "What are the implications of this choice?\n",
    "\n",
    "I'm going to start by exploring only the dated sites and then compare the results with all the sites. I've noticed there are site from 'prehistory' and 'norse' periods. I'm wondering if in general preservation is better in some parts of the Shetlands, and so we will find most of the concentrations of these older sites in one area. To explore this question, I'm going to use something called a 'kde plot'. KDE plots show areas where things concentrate or cluster together. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "canmore_shetland_prehistory = canmore_shetland[canmore_shetland['TYPE'].str.contains('PREHISTORIC')]\n",
    "canmore_shetland_norse = canmore_shetland[canmore_shetland['TYPE'].str.contains('NORSE')]\n",
    "\n",
    "# prehistoric sites will be in blue\n",
    "fig, ax = plt.subplots(figsize=(15,15))\n",
    "\n",
    "sns.kdeplot(canmore_shetland_prehistory['X'], canmore_shetland_prehistory['Y'], shade=True, cmap='Blues');\n",
    "\n",
    "# Norse sites will be in red\n",
    "sns.kdeplot(canmore_shetland_norse['X'], canmore_shetland_norse['Y'], shade=True, cmap='Reds', alpha=0.4);\n",
    "\n",
    "#All the dated sites will appear as green dots, so we can see where the clusters are within the whole set of sites.\n",
    "canmore_shetland_dated.plot(ax=ax, color='Green',markersize=3)\n",
    "plt.show();\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you see? Tp me there seem to be two clusters of prehistoric sites.\n",
    "The norse sites seem to match with the more northern of those clusters.\n",
    "\n",
    "Now let's look at the pattern for 19th-20th c. sites. Looking at the table, all these sites have the word 'century' in their period description and nothing else seems to match this pattern."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#sanity check yourself...\n",
    "canmore_shetland_modern = canmore_shetland_dated[canmore_shetland_dated['TYPE'].str.contains('CENTURY')]\n",
    "canmore_shetland_modern\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#and make a map\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(15,15))\n",
    "\n",
    "#modern site clusters will be in blue\n",
    "sns.kdeplot(canmore_shetland_modern['X'], canmore_shetland_modern['Y'], shade=True, cmap='Blues');\n",
    "\n",
    "#All the dated sites will appear as green dots, so we can see where the clusters are within the whole set of sites.\n",
    "canmore_shetland_dated.plot(ax=ax, color='Green',markersize=3, alpha =0.3)\n",
    "plt.show();\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Hmm. Our map appears to be squished up into the top right corner. What could be happening? I suggest we have a few points in our data that are not actually in the Shetlands or otherwise have dodgy coordinates. Let's investigate by sorting by the X coordinate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "canmore_shetland_modern.sort_values('X')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#Two dodgy points! They have significantly smaller x and y values. Let's get rid of them and replot.\n",
    "\n",
    "canmore_shetland_modern_clean = canmore_shetland_modern[~canmore_shetland_modern['OBJECTID'].isin([99342,127836])]\n",
    "fig, ax = plt.subplots(figsize=(15,15))\n",
    "\n",
    "#modern site clusters will be in blue\n",
    "sns.kdeplot(canmore_shetland_modern_clean['X'], canmore_shetland_modern_clean['Y'], shade=True, cmap='Blues');\n",
    "\n",
    "#All the dated sites will appear as green dots, so we can see where the clusters are within the whole set of sites.\n",
    "canmore_shetland_dated.plot(ax=ax, color='Green',markersize=3, alpha =0.3)\n",
    "plt.show();\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "I'd say the modern sites are clustered in the same places as the prehistoric sites, with only the norse ones so far having a different distribution. But so far we have only been looking at the dated sites. Let's check the clustering of all the sites."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "canmore_shetland_clean = canmore_shetland[~canmore_shetland['OBJECTID'].isin([99342,127836])]\n",
    "fig, ax = plt.subplots(figsize=(15,15))\n",
    "sns.kdeplot(canmore_shetland_clean['X'], canmore_shetland_clean['Y'], shade=True, cmap='Reds');\n",
    "\n",
    "plt.show();\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "OK, after all that we are pretty convinced there are two areas that are dominating the pattern of sites we are seeing. Let's look inside the Norse clusters and explore further. WE can start by seeing what kinds of Norses sites have been identified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "norse = canmore_shetland_norse.groupby(['TYPE']).count()\n",
    "norse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, so clearly there is *some* inconsistency in how site types have been named in Canmore. Let's grab everything that is a house, which seems a common category, and compare with farmsteads."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "norse_houses = canmore_shetland_norse[canmore_shetland_norse['TYPE'].str.contains('HOUSE')]\n",
    "norse_farms = canmore_shetland_norse[canmore_shetland_norse['TYPE'].str.contains('FARM')]\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(15,15))\n",
    "\n",
    "sns.kdeplot(norse_houses['X'], norse_houses['Y'], shade=True, cmap='Blues');\n",
    "\n",
    "\n",
    "sns.kdeplot(norse_farms['X'], norse_farms['Y'], shade=True, cmap='Reds', alpha=0.5);\n",
    "norse_houses.plot(ax=ax, color='Blue',markersize=5, alpha =0.7);\n",
    "norse_farms.plot(ax=ax, color='Red',markersize=5, alpha =0.7)\n",
    "plt.show();\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The patterns seem rather different, there is one shared cluster, but then we see variation... Do we expect to find houses and farms together? How might we explain the pattern we are seeing? Let's look at the types of sites found closest to farms. This is done through a 'nearest neighbour' calculation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "norse_otherf = canmore_shetland_norse[~canmore_shetland_norse['TYPE'].str.contains('FARM')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "pd.options.mode.chained_assignment = None  # default='warn'\n",
    "neighbours = norse_otherf.geometry.unary_union\n",
    "def near(point, pts=neighbours):\n",
    "    # find the nearest point and return the corresponding Place value\n",
    "    nearest = norse_otherf.geometry == nearest_points(point, pts)[1]\n",
    "    return norse_otherf[nearest].TYPE.get_values()[0]\n",
    "norse_farms['Nearest'] = norse_farms.apply(lambda row: near(row.geometry), axis=1)\n",
    "norse_farms.sort_values('Nearest')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Let's use the count function to see how many of each type of norse site appears near a norse farm.\n",
    "norse_farms.groupby('Nearest').count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What do we conclude? \n",
    "\n",
    "### This ends the tutorial. You can practice exploring patterns in survey data further on your own.\n",
    "\n",
    "Hopefully you learned to:\n",
    "* learn to investigate patterns in archaeological survey data using spatial analytical tools\n",
    "* learn to learn to make maps that illustrate patterns in archaeological survey data\n",
    "* start thinking about the meaning of the patterns of sites and features in the landscape\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}