{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Observing change in a web page over time\n", "\n", "

New to Jupyter notebooks? Try Using Jupyter notebooks for a quick introduction.

\n", "\n", "This notebook explores what we can find when you look at all captures of a single page over time." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

Work in progress – this notebook isn't finished yet. Check back later for more...

" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import re\n", "\n", "import altair as alt\n", "import pandas as pd\n", "import requests" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "def query_cdx(url, **kwargs):\n", " params = kwargs\n", " params[\"url\"] = url\n", " params[\"output\"] = \"json\"\n", " response = requests.get(\n", " \"http://web.archive.org/cdx/search/cdx\",\n", " params=params,\n", " headers={\"User-Agent\": \"\"},\n", " )\n", " response.raise_for_status()\n", " return response.json()" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "url = \"http://nla.gov.au\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Getting the data\n", "\n", "In this example we're using the IA CDX API, but this could easily be adapted to use [Timemaps](find_all_captures.ipynb) from a range of repositories. " ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "data = query_cdx(url)\n", "\n", "# Convert to a dataframe\n", "# The column names are in the first row\n", "df = pd.DataFrame(data[1:], columns=data[0])\n", "\n", "# Convert the timestamp string into a datetime object\n", "df[\"date\"] = pd.to_datetime(df[\"timestamp\"])\n", "df.sort_values(by=\"date\", inplace=True, ignore_index=True)\n", "\n", "# Convert the length from a string into an integer\n", "df[\"length\"] = df[\"length\"].astype(\"int\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As noted in the notebook [comparing the CDX API with Timemaps](getting_all_snapshots_timemap_vs_cdx.ipynb), there are a number of duplicate snapshots in the CDX results, so let's remove them." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Before: 4451\n", "After: 4350\n" ] } ], "source": [ "print(f\"Before: {df.shape[0]}\")\n", "df.drop_duplicates(\n", " subset=[\"timestamp\", \"original\", \"digest\", \"statuscode\", \"mimetype\"],\n", " keep=\"first\",\n", " inplace=True,\n", ")\n", "print(f\"After: {df.shape[0]}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The basic shape" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Timestamp('1996-10-19 06:42:23')" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[\"date\"].min()" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Timestamp('2022-04-10 18:38:06')" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[\"date\"].max()" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "count 4350.000000\n", "mean 8318.689655\n", "std 7854.281544\n", "min 235.000000\n", "25% 533.000000\n", "50% 5699.000000\n", "75% 14852.750000\n", "max 30062.000000\n", "Name: length, dtype: float64" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[\"length\"].describe()" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "200 2948\n", "301 775\n", "- 315\n", "302 309\n", "503 3\n", "Name: statuscode, dtype: int64" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[\"statuscode\"].value_counts()" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "text/html 4033\n", "warc/revisit 315\n", "unk 2\n", "Name: mimetype, dtype: int64" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[\"mimetype\"].value_counts()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Plotting snapshots over time" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "

\n", "" ], "text/plain": [ "alt.Chart(...)" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# This is just a bit of fancy customisation to group the types of errors by color\n", "# See https://altair-viz.github.io/user_guide/customization.html#customizing-colors\n", "domain = [\"-\", \"200\", \"301\", \"302\", \"404\", \"503\"]\n", "# green for ok, blue for redirects, red for errors\n", "range_ = [\"#888888\", \"#39a035\", \"#5ba3cf\", \"#125ca4\", \"#e13128\", \"#b21218\"]\n", "\n", "alt.Chart(df).mark_point().encode(\n", " x=\"date:T\",\n", " y=\"length:Q\",\n", " color=alt.Color(\"statuscode\", scale=alt.Scale(domain=domain, range=range_)),\n", " tooltip=[\"date\", \"length\", \"statuscode\"],\n", ").properties(width=700, height=300)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Looking at domains, protocols, and redirects\n", "\n", "Looking at the chart above, it's hard to understand why a request for a page is sometimes redirected, and sometimes not. To understand this we have to look a bit closer at what pages are actually being archived. Let's look at the breakdown of values in the `original` column. These are the urls being requested by the archiving bot." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "https://www.nla.gov.au/ 1508\n", "http://www.nla.gov.au/ 1178\n", "http://www.nla.gov.au:80/ 868\n", "http://nla.gov.au/ 588\n", "http://nla.gov.au:80/ 77\n", "https://nla.gov.au/ 62\n", "http://www.nla.gov.au// 21\n", "http://www.nla.gov.au 11\n", "http://www2.nla.gov.au:80/ 10\n", "https://www.nla.gov.au 10\n", "http://Trove@nla.gov.au/ 6\n", "http://www.nla.gov.au:80/? 2\n", "http://www.nla.gov.au./ 2\n", "http://nla.gov.au 1\n", "http://mailto:media@nla.gov.au/ 1\n", "http://cmccarthy@nla.gov.au/ 1\n", "http://mailto:development@nla.gov.au/ 1\n", "http://mailto:www@nla.gov.au/ 1\n", "http://www.nla.gov.au:80// 1\n", "http://www.nla.gov.au/? 1\n", "Name: original, dtype: int64" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[\"original\"].value_counts()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ah ok, so there's actually a mix of things in here – some include the 'www' prefix and some don't, some use the 'https' protocol and some just plain old 'http'. There's also a bit of junk in there from badly parsed `mailto` links. To look at the differences in more detail, let's create new columns for `subdomain` and `protocol`." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "www 3602\n", " 738\n", "www2 10\n", "Name: subdomain, dtype: int64" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "base_domain = re.search(r\"https*:\\/\\/(\\w*)\\.\", url).group(1)\n", "df[\"subdomain\"] = df[\"original\"].str.extract(\n", " r\"^https*:\\/\\/(\\w*)\\.{}\\.\".format(base_domain), flags=re.IGNORECASE\n", ")\n", "df[\"subdomain\"].fillna(\"\", inplace=True)\n", "df[\"subdomain\"].value_counts()" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "http 2770\n", "https 1580\n", "Name: protocol, dtype: int64" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[\"protocol\"] = df[\"original\"].str.extract(r\"^(https*):\")\n", "df[\"protocol\"].value_counts()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Change in protocol\n", "\n", "Let's look to see how the proportion of requests using each of the protocols changes over time. Here we're grouping the rows by year." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "" ], "text/plain": [ "alt.Chart(...)" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "alt.Chart(df).mark_bar().encode(\n", " x=\"year(date):T\",\n", " y=alt.Y(\"count()\", stack=\"normalize\"),\n", " color=\"protocol:N\",\n", " # tooltip=['date', 'length', 'subdomain:N']\n", ").properties(width=700, height=200)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "No real surprise there given the increased use of https generally." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Status codes by subdomain\n", "\n", "Let's now compare the proportion of status codes between the bare `nla.gov.au` domain and the `www` subdomain." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "" ], "text/plain": [ "alt.Chart(...)" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "alt.Chart(\n", " df.loc[(df[\"statuscode\"] != \"-\") & (df[\"subdomain\"] != \"www2\")]\n", ").mark_bar().encode(\n", " x=\"year(date):T\",\n", " y=alt.Y(\"count()\", stack=\"normalize\"),\n", " color=alt.Color(\"statuscode\", scale=alt.Scale(domain=domain, range=range_)),\n", " row=\"subdomain\",\n", " tooltip=[\"year(date):T\", \"statuscode\"],\n", ").properties(\n", " width=700, height=100\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "I think we can start to see what's going on. Around about 2004, requests to `nla.gov.au` started to be redirected to `www.nla.gov.au` giving a [302](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/302) response, indicating that the page had been moved temporarily. But why the growth in [301](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/301) (moved permanently) responses from both domains after 2018? If we look at the chart above showing the increased use of the `https` protocol, I think we could guess that `http` requests in both domains are being redirected to `https`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Status codes by protocol\n", "\n", "Let's test that hypothesis by looking at the distribution of status codes by protocol." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "" ], "text/plain": [ "alt.Chart(...)" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "alt.Chart(\n", " df.loc[(df[\"statuscode\"] != \"-\") & (df[\"subdomain\"] != \"www2\")]\n", ").mark_bar().encode(\n", " x=\"year(date):T\",\n", " y=alt.Y(\"count()\", stack=\"normalize\"),\n", " color=alt.Color(\"statuscode\", scale=alt.Scale(domain=domain, range=range_)),\n", " row=\"protocol\",\n", " tooltip=[\"year(date):T\", \"protocol\", \"statuscode\"],\n", ").properties(\n", " width=700, height=100\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that by 2019, all requests using `http` are being redirected to `https`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Looking for major changes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we understand what's going on with the different domains and status codes, I think we can focus on just the 'www' domain and the '200' responses." ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "" ], "text/plain": [ "alt.Chart(...)" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_200 = df.copy().loc[\n", " (df[\"statuscode\"] == \"200\") & (df[\"subdomain\"] == \"www\") & (df[\"length\"] > 1000)\n", "]\n", "\n", "alt.Chart(df_200).mark_point().encode(\n", " x=\"date:T\", y=\"length:Q\", tooltip=[\"date\", \"length\"]\n", ").properties(width=700, height=300)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Pandas makes it easy to calculate the difference between two adjacent values, so lets find the absolute difference in length between each capture." ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "df_200[\"change_in_length\"] = abs(df_200[\"length\"].diff())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can look at the captures that varied most in length from their predecessor." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
urlkeytimestamporiginalmimetypestatuscodedigestlengthdatesubdomainprotocolchange_in_length
3656au,gov,nla)/20210701042826https://www.nla.gov.au/text/html2006PC4KROOPHEBXIC6A3GRFFHBMO2EJ4F7292152021-07-01 04:28:26wwwhttps13933.0
4134au,gov,nla)/20220202054835https://www.nla.gov.au/text/html200O6VCJWSKLU3EAFDSH4IGOIW264Y6IIXZ274952022-02-02 05:48:35wwwhttps4648.0
3954au,gov,nla)/20220105025646https://www.nla.gov.au/text/html200MRIUSTANGOWT3CT5QSSRJ7NJPEN2RSEN272732022-01-05 02:56:46wwwhttps4463.0
4058au,gov,nla)/20220121065839https://www.nla.gov.au/text/html200UAGVH7YN6ZPJYQUIZTP32G4GF3JJ2N7J229482022-01-21 06:58:39wwwhttps4394.0
4417au,gov,nla)/20220405063728https://www.nla.gov.au/text/html200HXSLRIVPKI3ECPC5V6NNEJOIHTDKZ5JJ226822022-04-05 06:37:28wwwhttps4375.0
3921au,gov,nla)/20211228211936https://www.nla.gov.au/text/html200FQZBN2C7DPFPC26F6HX3KDNGWCWVOWWX228312021-12-28 21:19:36wwwhttps4374.0
3946au,gov,nla)/20220103064507https://www.nla.gov.au/text/html200PUJLOJI7OUJ4XFKUDI47HOQLFOMEQLKJ229172022-01-03 06:45:07wwwhttps4367.0
4322au,gov,nla)/20220323022916https://www.nla.gov.au/text/html200WEP3PTHC7CAEF22S3NDIMZHFDAQIK65J269192022-03-23 02:29:16wwwhttps4359.0
3939au,gov,nla)/20220102132710https://www.nla.gov.au/text/html200SBM5ATRRZWVT7HYA6J3BMMOXG4HTDZFD272512022-01-02 13:27:10wwwhttps4352.0
4215au,gov,nla)/20220224211329https://www.nla.gov.au/text/html200WCIDXIQ22M35PWXUGK7GCXG2LEJUAJ5J232082022-02-24 21:13:29wwwhttps4351.0
\n", "
" ], "text/plain": [ " urlkey timestamp original mimetype \\\n", "3656 au,gov,nla)/ 20210701042826 https://www.nla.gov.au/ text/html \n", "4134 au,gov,nla)/ 20220202054835 https://www.nla.gov.au/ text/html \n", "3954 au,gov,nla)/ 20220105025646 https://www.nla.gov.au/ text/html \n", "4058 au,gov,nla)/ 20220121065839 https://www.nla.gov.au/ text/html \n", "4417 au,gov,nla)/ 20220405063728 https://www.nla.gov.au/ text/html \n", "3921 au,gov,nla)/ 20211228211936 https://www.nla.gov.au/ text/html \n", "3946 au,gov,nla)/ 20220103064507 https://www.nla.gov.au/ text/html \n", "4322 au,gov,nla)/ 20220323022916 https://www.nla.gov.au/ text/html \n", "3939 au,gov,nla)/ 20220102132710 https://www.nla.gov.au/ text/html \n", "4215 au,gov,nla)/ 20220224211329 https://www.nla.gov.au/ text/html \n", "\n", " statuscode digest length date \\\n", "3656 200 6PC4KROOPHEBXIC6A3GRFFHBMO2EJ4F7 29215 2021-07-01 04:28:26 \n", "4134 200 O6VCJWSKLU3EAFDSH4IGOIW264Y6IIXZ 27495 2022-02-02 05:48:35 \n", "3954 200 MRIUSTANGOWT3CT5QSSRJ7NJPEN2RSEN 27273 2022-01-05 02:56:46 \n", "4058 200 UAGVH7YN6ZPJYQUIZTP32G4GF3JJ2N7J 22948 2022-01-21 06:58:39 \n", "4417 200 HXSLRIVPKI3ECPC5V6NNEJOIHTDKZ5JJ 22682 2022-04-05 06:37:28 \n", "3921 200 FQZBN2C7DPFPC26F6HX3KDNGWCWVOWWX 22831 2021-12-28 21:19:36 \n", "3946 200 PUJLOJI7OUJ4XFKUDI47HOQLFOMEQLKJ 22917 2022-01-03 06:45:07 \n", "4322 200 WEP3PTHC7CAEF22S3NDIMZHFDAQIK65J 26919 2022-03-23 02:29:16 \n", "3939 200 SBM5ATRRZWVT7HYA6J3BMMOXG4HTDZFD 27251 2022-01-02 13:27:10 \n", "4215 200 WCIDXIQ22M35PWXUGK7GCXG2LEJUAJ5J 23208 2022-02-24 21:13:29 \n", "\n", " subdomain protocol change_in_length \n", "3656 www https 13933.0 \n", "4134 www https 4648.0 \n", "3954 www https 4463.0 \n", "4058 www https 4394.0 \n", "4417 www https 4375.0 \n", "3921 www https 4374.0 \n", "3946 www https 4367.0 \n", "4322 www https 4359.0 \n", "3939 www https 4352.0 \n", "4215 www https 4351.0 " ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "top_ten_changes = df_200.sort_values(by=\"change_in_length\", ascending=False)[:10]\n", "top_ten_changes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's try visualising this by highlighting the major changes in length." ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "" ], "text/plain": [ "alt.LayerChart(...)" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "points = (\n", " alt.Chart(df_200)\n", " .mark_point()\n", " .encode(x=\"date:T\", y=\"length:Q\", tooltip=[\"date\", \"length\"])\n", " .properties(width=700, height=300)\n", ")\n", "\n", "lines = (\n", " alt.Chart(top_ten_changes)\n", " .mark_rule(color=\"red\")\n", " .encode(x=\"date:T\", tooltip=[\"date\"])\n", " .properties(width=700, height=300)\n", ")\n", "\n", "points + lines" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Rather than just a raw number, perhaps the percentage change in length would be more useful. Once again, Pandas makes this easy to calculate. This calculates the percentage change from the previous value – so length2 - length1 / length1." ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "df_200[\"pct_change_in_length\"] = abs(df_200[\"length\"].pct_change())" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
urlkeytimestamporiginalmimetypestatuscodedigestlengthdatesubdomainprotocolchange_in_lengthpct_change_in_length
3656au,gov,nla)/20210701042826https://www.nla.gov.au/text/html2006PC4KROOPHEBXIC6A3GRFFHBMO2EJ4F7292152021-07-01 04:28:26wwwhttps13933.00.911726
13au,gov,nla)/19980205162107http://www.nla.gov.au:80/text/html200LIXK3YXSUFO5KPOO22XIMQGPWXKNHV6X19201998-02-05 16:21:07wwwhttp757.00.650903
79au,gov,nla)/20011003175018http://www.nla.gov.au:80/text/html200BWGDP6NTGVOI2TBA62P7IZ2PPWRLOODN33672001-10-03 17:50:18wwwhttp1004.00.424884
1519au,gov,nla)/20160901112433http://www.nla.gov.au/text/html200MZD7NTLMH5HBXSFQIHTGTC6IELQDYBN2115412016-09-01 11:24:33wwwhttp2738.00.311030
1184au,gov,nla)/20130211044309http://www.nla.gov.au/text/html200QWCVHAK2Y6WXLDNIMTJLZT5RY6YCJ7UN85212013-02-11 04:43:09wwwhttp1698.00.248864
1067au,gov,nla)/20110611064218http://www.nla.gov.au/text/html200Z7PQG2MVOOQUZ62ASNRAWDFLSYFHATQT56012011-06-11 06:42:18wwwhttp1739.00.236921
2049au,gov,nla)/20181212014241https://www.nla.gov.au/text/html200C3YSGGHG52WIG6U6X7C3XOL5LOHVIINM148132018-12-12 01:42:41wwwhttps2831.00.236271
786au,gov,nla)/20061107083938http://www.nla.gov.au:80/text/html200HOW52ARISTA4HTCLFPYNCQWR6NK2N2NF56622006-11-07 08:39:38wwwhttp1561.00.216115
4134au,gov,nla)/20220202054835https://www.nla.gov.au/text/html200O6VCJWSKLU3EAFDSH4IGOIW264Y6IIXZ274952022-02-02 05:48:35wwwhttps4648.00.203440
3864au,gov,nla)/20211121150551https://www.nla.gov.au/text/html200GEPXW6EKI7GBG22SMUFIIXQW3KIQXNIY259122021-11-21 15:05:51wwwhttps4316.00.199852
\n", "
" ], "text/plain": [ " urlkey timestamp original mimetype \\\n", "3656 au,gov,nla)/ 20210701042826 https://www.nla.gov.au/ text/html \n", "13 au,gov,nla)/ 19980205162107 http://www.nla.gov.au:80/ text/html \n", "79 au,gov,nla)/ 20011003175018 http://www.nla.gov.au:80/ text/html \n", "1519 au,gov,nla)/ 20160901112433 http://www.nla.gov.au/ text/html \n", "1184 au,gov,nla)/ 20130211044309 http://www.nla.gov.au/ text/html \n", "1067 au,gov,nla)/ 20110611064218 http://www.nla.gov.au/ text/html \n", "2049 au,gov,nla)/ 20181212014241 https://www.nla.gov.au/ text/html \n", "786 au,gov,nla)/ 20061107083938 http://www.nla.gov.au:80/ text/html \n", "4134 au,gov,nla)/ 20220202054835 https://www.nla.gov.au/ text/html \n", "3864 au,gov,nla)/ 20211121150551 https://www.nla.gov.au/ text/html \n", "\n", " statuscode digest length date \\\n", "3656 200 6PC4KROOPHEBXIC6A3GRFFHBMO2EJ4F7 29215 2021-07-01 04:28:26 \n", "13 200 LIXK3YXSUFO5KPOO22XIMQGPWXKNHV6X 1920 1998-02-05 16:21:07 \n", "79 200 BWGDP6NTGVOI2TBA62P7IZ2PPWRLOODN 3367 2001-10-03 17:50:18 \n", "1519 200 MZD7NTLMH5HBXSFQIHTGTC6IELQDYBN2 11541 2016-09-01 11:24:33 \n", "1184 200 QWCVHAK2Y6WXLDNIMTJLZT5RY6YCJ7UN 8521 2013-02-11 04:43:09 \n", "1067 200 Z7PQG2MVOOQUZ62ASNRAWDFLSYFHATQT 5601 2011-06-11 06:42:18 \n", "2049 200 C3YSGGHG52WIG6U6X7C3XOL5LOHVIINM 14813 2018-12-12 01:42:41 \n", "786 200 HOW52ARISTA4HTCLFPYNCQWR6NK2N2NF 5662 2006-11-07 08:39:38 \n", "4134 200 O6VCJWSKLU3EAFDSH4IGOIW264Y6IIXZ 27495 2022-02-02 05:48:35 \n", "3864 200 GEPXW6EKI7GBG22SMUFIIXQW3KIQXNIY 25912 2021-11-21 15:05:51 \n", "\n", " subdomain protocol change_in_length pct_change_in_length \n", "3656 www https 13933.0 0.911726 \n", "13 www http 757.0 0.650903 \n", "79 www http 1004.0 0.424884 \n", "1519 www http 2738.0 0.311030 \n", "1184 www http 1698.0 0.248864 \n", "1067 www http 1739.0 0.236921 \n", "2049 www https 2831.0 0.236271 \n", "786 www http 1561.0 0.216115 \n", "4134 www https 4648.0 0.203440 \n", "3864 www https 4316.0 0.199852 " ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "top_ten_changes_pct = df_200.sort_values(by=\"pct_change_in_length\", ascending=False)[\n", " :10\n", "]\n", "top_ten_changes_pct" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "" ], "text/plain": [ "alt.LayerChart(...)" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "lines = (\n", " alt.Chart(top_ten_changes_pct)\n", " .mark_rule(color=\"red\")\n", " .encode(x=\"date:T\", tooltip=[\"date\"])\n", " .properties(width=700, height=300)\n", ")\n", "\n", "points + lines" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By focusing on percentage difference we can see that more prominence is given to the change in 2001. But rather than just the top 10, should we look at changes greater than 10% or some other threshold?" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "" ], "text/plain": [ "alt.LayerChart(...)" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "lines = (\n", " alt.Chart(df_200.loc[df_200[\"pct_change_in_length\"] > 0.1])\n", " .mark_rule(color=\"red\")\n", " .encode(x=\"date:T\", tooltip=[\"date\"])\n", " .properties(width=700, height=300)\n", ")\n", "\n", "points + lines" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Other possibilities to explore\n", "\n", "* Rate of change – what proportion of the snapshots each year are *different*?\n", "* Use similarity measures to identify changes." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Comparing individual captures\n", "\n", "Once major changes, such as those above, have been identified, we can use some of the other notebooks in this repository to compare individual captures. For example:\n", "\n", "* [Compare two versions of an archived web page](show_diffs.ipynb)\n", "* [Create and compare full page screenshots from archived web pages](save_screenshot.ipynb)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "Created by [Tim Sherratt](https://timsherratt.org) for the [GLAM Workbench](https://glam-workbench.github.io). Support me by becoming a [GitHub sponsor](https://github.com/sponsors/wragge)!\n", "\n", "Work on this notebook was supported by the [IIPC Discretionary Funding Programme 2019-2020](http://netpreserve.org/projects/)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.12" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 4 }