{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# SysFlow support for Kubernetes environments" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Starting with version 0.5.0, SysFlow records contain more information related to containers in case they are part of a Kubernetes (k8s) or OpenShift environment. Specifically, there is a new record type, `KE`, that captures and exposes the Kubernetes events in the new `k8s.*` attributes. Furthermore, all other record types are extended with Kubernetes pod information in the new `pod.*` attributes.\n", "\n", "These new attributes are, in more detail:" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "
\n",
    "k8s.action                     K8s Event Action\n",
    "k8s.kind                       K8s Event Component Type\n",
    "k8s.msg                        K8s Event Message\n",
    "\n",
    "pod.id                         Pod Identifier\n",
    "pod.name                       Pod Name\n",
    "pod.nname                      Pod Node Name\n",
    "pod.hostip                     Pod Host IP\n",
    "pod.internalip                 Pod Internal IP\n",
    "pod.ns                         Pod Namespace\n",
    "pod.rstrtcnt                   Pod Restart Count\n",
    "pod.services                   Pod Services\n",
    "\n",
    "
" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "In this notebook we will look into the new information using data from a test setup running Instana's [robot-shop application](https://github.com/instana/robot-shop), with a special eye to the new information that is available with respect to cluster-relevant IP addresses.\n", "\n", "First, we describe the experimental setup that was used to create our test data. Next, after loading the experimental SysFlow data, we look at the new k8s event data, the new information related to pods as avaible per SysFlow record, and compare the newly available cluster-level IP address that augment the observed network activity in the regular NF objects." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Experimental setup and timeline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The experimental setup is based on the installation of: \n", "- minikube v1.25.2 on Ubuntu 18.04 (kvm/amd64) with\n", "- kubernetes version v1.23.3 using the\n", "- virtualbox driver\n", "as a small base Kubernetes test environment.\n", "\n", "The experiment consists in the installation of Instana's [robot-shop application](https://github.com/instana/robot-shop). To have sufficient resources to run this multi-container application, slightly more than the default minimal configuration should be used, e.g., here we are using 4 CPUs and 16GB of memory for a virtualbox VM." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```\n", "$ minikube start\n", "* minikube v1.25.2 on Ubuntu 18.04 (kvm/amd64)\n", "* Using the virtualbox driver based on user configuration\n", "* Starting control plane node minikube in cluster minikube\n", "* Creating virtualbox VM (CPUs=4, Memory=16000MB, Disk=100000MB) ...\n", "* Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...\n", " - kubelet.housekeeping-interval=5m\n", " - Generating certificates and keys ...\n", " - Booting up control plane ...\n", " - Configuring RBAC rules ...\n", " - Using image gcr.io/k8s-minikube/storage-provisioner:v5\n", "* Verifying Kubernetes components...\n", "* Enabled addons: storage-provisioner, default-storageclass\n", "* Done! kubectl is now configured to use \"minikube\" cluster and \"default\" namespace by default\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The next section contains information about the experiment used to collect the SysFlow data with `sf-collector` 0.5.0." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Timeline of the experiment" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The experiment used to create our data consists of:\n", "- the creation of the namespace `robot-shop`\n", "- the installation of the the `robot-shop` with helm charts provided by the application\n", "- the deletion of some of its containers (`mongodb`, `web`, `user`) - which will be automatically recreated by the application\n", "- deinstall and cleanup of the `robot-shop` application.\n", "\n", "The scripted experiment logs the timestamps of these events so that we have a baseline of the events to compare with the collected SysFlow data." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from sysflow.reader import FlattenedSFReader, SFReader\n", "from sysflow.formatter import SFFormatter\n", "import json\n", "import os\n", "import pprint\n", "import pickle\n", "import gzip\n", "import pandas as pd\n", "import numpy as np\n", "import datetime\n", "import tabulate\n", "import textwrap\n", "import plotly.graph_objects as go\n", "import plotly as pl\n", "import plotly.io as pio\n", "pio.renderers.default = 'iframe'\n", "pd.set_option('display.max_rows', 50)" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "data_dir = 'data/'" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "log_file = data_dir + 'experiment.log'\n", "log_content_selection = ['----- ', 'waiting']" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "-------------------------- -------------------------\n", "2022-03-17 18:48:40.736000 starting experiment\n", "2022-03-17 18:48:40.736000 create project robot-shop\n", "2022-03-17 18:48:40.991000 waiting for 60s\n", "2022-03-17 18:49:41.047000 install robot shop\n", "2022-03-17 18:49:42.531000 waiting for 900s\n", "2022-03-17 19:04:47.423000 kill container mongodb\n", "2022-03-17 19:04:53.940000 waiting for 300s\n", "2022-03-17 19:09:53.947000 kill container web\n", "2022-03-17 19:10:02.620000 waiting for 300s\n", "2022-03-17 19:15:02.660000 kill container user\n", "2022-03-17 19:15:36.016000 waiting for 300s\n", "2022-03-17 19:20:36.117000 delete robot shop\n", "2022-03-17 19:20:37.534000 waiting for 300s\n", "2022-03-17 19:25:37.627000 delete project robot-shop\n", "2022-03-17 19:25:50.305000 waiting for 300s\n", "2022-03-17 19:30:50.346000 experiment ends\n", "-------------------------- -------------------------\n" ] } ], "source": [ "log_selected_lines = []\n", "with open(log_file, 'r') as inp:\n", " for line in inp: \n", " if any([p in line for p in log_content_selection]):\n", " log_selected_lines.append(line.rstrip())\n", "log_events = []\n", "for line in log_selected_lines:\n", " time_str = ' '.join(line.split()[0:2])\n", " tdt = datetime.datetime.strptime(time_str, '%Y-%m-%d %H:%M:%S,%f') #.replace(tzinfo=localtz) \n", " rest = ' '.join(line.split()[4:])\n", " event = rest.replace('----- ', '').replace('... ', '').replace(' seconds', 's')\n", " log_events.append([tdt, event])\n", "print(tabulate.tabulate(log_events))" ] }, { "cell_type": "markdown", "metadata": { "incorrectly_encoded_metadata": "toc-hr-collapsed=true" }, "source": [ "# Experimental SysFlow data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The collected SysFlow data is combined into the accompanying `experiment.sf` (as SysFlow trace files are essentially AVRO files, AVRO tools like `avro-tools concat` can be used to combine multiple SysFlow traces into one file).\n", "This file is read into a Pandas DataFrame for our further evaluation." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "sf_file = data_dir + 'experiment.sf'" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# reading of the SysFlow trace file and conversion to a Pandas DataFrame (with caching onto disk)\n", "df_file = data_dir + 'experiment_df.pkl.gz'\n", "if os.path.exists(df_file):\n", " with gzip.open(df_file, 'rb') as inp:\n", " df = pickle.load(inp)\n", "else: \n", " reader = FlattenedSFReader(sf_file, False)\n", " formatter = SFFormatter(reader)\n", " df = formatter.toDataframe()\n", " # applying some functions to allow for hashing of the more complex data types\n", " df['pod.internalip'] = df['pod.internalip'].apply(tuple)\n", " df['pod.hostip'] = df['pod.hostip'].apply(tuple)\n", " df['pod.services_str'] = df['pod.services'].apply(str)\n", " with gzip.open(df_file, 'wb') as out:\n", " pickle.dump(df, out)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The captured data contains 169324 SysFlow records, describing the activity of 41 containers.\n" ] } ], "source": [ "print(f'The captured data contains {df.shape[0]} SysFlow records, describing the activity of {len(df[\"container.id\"].unique())} containers.')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Kubernetes Events: the new `KE` record type" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let us first look at the new `KE` record type. For this, we subselect the entries of interest into a new DataFrame `df_ke`." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "# k8s.msg fields have still some spurious line-ending\n", "def fix_k8s_msg(msg):\n", " if msg.endswith('\\n\\u0000'):\n", " msg = msg[:-2]\n", " return msg" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "tags": [] }, "outputs": [], "source": [ "# select the KE records, drop all irrelevant columns (empty string or NaN)\n", "df_ke = df[df.type == 'KE'].replace('', np.nan).dropna(axis=1, how='all').reset_index()\n", "# fix k8s.msg\n", "df_ke['k8s.msg'] = df_ke['k8s.msg'].apply(fix_k8s_msg)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The relevant information gathered from the events is shown in the fields:\n", "- `k8s.kind`: the kind of the K8s infrastructure that this event is concerned with (like \"K8S_NODES\", \"K8S_NAMESPACES\", \"K8S_PODS\", \"K8S_REPLICATIONCONTROLLERS\", \"K8S_SERVICES\", \"K8S_EVENTS\", \"K8S_REPLICASETS\", \"K8S_DAEMONSETS\", \"K8S_DEPLOYMENTS\", \"K8S_UNKNOWN\")\n", "- `k8s.action`: the action type (like \"K8S_COMPONENT_ADDED\", \"K8S_COMPONENT_MODIFIED\", \"K8S_COMPONENT_DELETED\", \"K8S_COMPONENT_ERROR\", \"K8S_COMPONENT_NONEXISTENT\", \"K8S_COMPONENT_UNKNOWN\")\n", "- `k8s.msg`: the JSON string of the K8s event\n" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
indexversiontypetsts_utspod.hostippod.internalipnode.idnode.ipfilenameschematagsk8s.actionk8s.kindk8s.msg
004KE2022-03-17T18:47:15.8456311647542835845631000()()minikube192.168.59.100/mnt/data/16475428364()K8S_COMPONENT_ADDEDK8S_NODES{\"apiVersion\":\"v1\",\"items\":[{\"addresses\":[\"192...
114KE2022-03-17T18:47:15.8456311647542835845631000()()minikube192.168.59.100/mnt/data/16475428364()K8S_COMPONENT_ADDEDK8S_NAMESPACES{\"apiVersion\":\"v1\",\"items\":[{\"labels\":{\"kubern...
224KE2022-03-17T18:47:15.8456311647542835845631000()()minikube192.168.59.100/mnt/data/16475428364()K8S_COMPONENT_ADDEDK8S_PODS{\"apiVersion\":\"v1\",\"items\":[{\"containerStatuse...
334KE2022-03-17T18:47:15.8456311647542835845631000()()minikube192.168.59.100/mnt/data/16475428364()K8S_COMPONENT_ADDEDK8S_REPLICATIONCONTROLLERS{\"apiVersion\":\"v1\",\"items\":[],\"kind\":\"Replicat...
444KE2022-03-17T18:47:15.8456311647542835845631000()()minikube192.168.59.100/mnt/data/16475428364()K8S_COMPONENT_ADDEDK8S_SERVICES{\"apiVersion\":\"v1\",\"items\":[{\"clusterIP\":\"10.9...
................................................
1681577074KE2022-03-17T19:25:50.6924441647545150692443689()()minikube192.168.59.100/mnt/data/16475451154()K8S_COMPONENT_MODIFIEDK8S_NAMESPACES{\"apiVersion\":\"v1\",\"items\":[{\"labels\":{\"kubern...
1691577084KE2022-03-17T19:25:50.6924441647545150692443689()()minikube192.168.59.100/mnt/data/16475451154()K8S_COMPONENT_DELETEDK8S_NAMESPACES{\"apiVersion\":\"v1\",\"items\":[{\"labels\":{\"kubern...
1701594054KE2022-03-17T19:26:56.7469721647545216746972249()()minikube192.168.59.100/mnt/data/16475451754()K8S_COMPONENT_MODIFIEDK8S_NODES{\"apiVersion\":\"v1\",\"items\":[{\"addresses\":[\"192...
1711648364KE2022-03-17T19:30:57.8807671647545457880767012()()minikube192.168.59.100/mnt/data/16475454154()K8S_COMPONENT_ADDEDK8S_NODES{\"apiVersion\":\"v1\",\"items\":[{\"addresses\":[\"192...
1721663724KE2022-03-17T19:32:04.9315461647545524931546223()()minikube192.168.59.100/mnt/data/16475454754()K8S_COMPONENT_MODIFIEDK8S_NODES{\"apiVersion\":\"v1\",\"items\":[{\"addresses\":[\"192...
\n", "

173 rows × 15 columns

\n", "
" ], "text/plain": [ " index version type ts ts_uts \\\n", "0 0 4 KE 2022-03-17T18:47:15.845631 1647542835845631000 \n", "1 1 4 KE 2022-03-17T18:47:15.845631 1647542835845631000 \n", "2 2 4 KE 2022-03-17T18:47:15.845631 1647542835845631000 \n", "3 3 4 KE 2022-03-17T18:47:15.845631 1647542835845631000 \n", "4 4 4 KE 2022-03-17T18:47:15.845631 1647542835845631000 \n", ".. ... ... ... ... ... \n", "168 157707 4 KE 2022-03-17T19:25:50.692444 1647545150692443689 \n", "169 157708 4 KE 2022-03-17T19:25:50.692444 1647545150692443689 \n", "170 159405 4 KE 2022-03-17T19:26:56.746972 1647545216746972249 \n", "171 164836 4 KE 2022-03-17T19:30:57.880767 1647545457880767012 \n", "172 166372 4 KE 2022-03-17T19:32:04.931546 1647545524931546223 \n", "\n", " pod.hostip pod.internalip node.id node.ip filename \\\n", "0 () () minikube 192.168.59.100 /mnt/data/1647542836 \n", "1 () () minikube 192.168.59.100 /mnt/data/1647542836 \n", "2 () () minikube 192.168.59.100 /mnt/data/1647542836 \n", "3 () () minikube 192.168.59.100 /mnt/data/1647542836 \n", "4 () () minikube 192.168.59.100 /mnt/data/1647542836 \n", ".. ... ... ... ... ... \n", "168 () () minikube 192.168.59.100 /mnt/data/1647545115 \n", "169 () () minikube 192.168.59.100 /mnt/data/1647545115 \n", "170 () () minikube 192.168.59.100 /mnt/data/1647545175 \n", "171 () () minikube 192.168.59.100 /mnt/data/1647545415 \n", "172 () () minikube 192.168.59.100 /mnt/data/1647545475 \n", "\n", " schema tags k8s.action k8s.kind \\\n", "0 4 () K8S_COMPONENT_ADDED K8S_NODES \n", "1 4 () K8S_COMPONENT_ADDED K8S_NAMESPACES \n", "2 4 () K8S_COMPONENT_ADDED K8S_PODS \n", "3 4 () K8S_COMPONENT_ADDED K8S_REPLICATIONCONTROLLERS \n", "4 4 () K8S_COMPONENT_ADDED K8S_SERVICES \n", ".. ... ... ... ... \n", "168 4 () K8S_COMPONENT_MODIFIED K8S_NAMESPACES \n", "169 4 () K8S_COMPONENT_DELETED K8S_NAMESPACES \n", "170 4 () K8S_COMPONENT_MODIFIED K8S_NODES \n", "171 4 () K8S_COMPONENT_ADDED K8S_NODES \n", "172 4 () K8S_COMPONENT_MODIFIED K8S_NODES \n", "\n", " k8s.msg \n", "0 {\"apiVersion\":\"v1\",\"items\":[{\"addresses\":[\"192... \n", "1 {\"apiVersion\":\"v1\",\"items\":[{\"labels\":{\"kubern... \n", "2 {\"apiVersion\":\"v1\",\"items\":[{\"containerStatuse... \n", "3 {\"apiVersion\":\"v1\",\"items\":[],\"kind\":\"Replicat... \n", "4 {\"apiVersion\":\"v1\",\"items\":[{\"clusterIP\":\"10.9... \n", ".. ... \n", "168 {\"apiVersion\":\"v1\",\"items\":[{\"labels\":{\"kubern... \n", "169 {\"apiVersion\":\"v1\",\"items\":[{\"labels\":{\"kubern... \n", "170 {\"apiVersion\":\"v1\",\"items\":[{\"addresses\":[\"192... \n", "171 {\"apiVersion\":\"v1\",\"items\":[{\"addresses\":[\"192... \n", "172 {\"apiVersion\":\"v1\",\"items\":[{\"addresses\":[\"192... \n", "\n", "[173 rows x 15 columns]" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_ke" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As to be expected given the experiment, the largest activity can be seen around changes in the Pods:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "k8s.kind k8s.action \n", "K8S_NAMESPACES K8S_COMPONENT_ADDED 13\n", " K8S_COMPONENT_DELETED 1\n", " K8S_COMPONENT_MODIFIED 3\n", "K8S_NODES K8S_COMPONENT_ADDED 3\n", " K8S_COMPONENT_MODIFIED 9\n", "K8S_PODS K8S_COMPONENT_ADDED 24\n", " K8S_COMPONENT_DELETED 15\n", " K8S_COMPONENT_MODIFIED 77\n", "K8S_REPLICATIONCONTROLLERS K8S_COMPONENT_ADDED 1\n", "K8S_SERVICES K8S_COMPONENT_ADDED 15\n", " K8S_COMPONENT_DELETED 12\n", "dtype: int64" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_ke.value_counts(['k8s.kind', 'k8s.action'], sort=False)" ] }, { "cell_type": "markdown", "metadata": { "incorrectly_encoded_metadata": "toc-hr-collapsed=true" }, "source": [ "## Unpacking `k8s.msg` data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A deeper understanding of what the KE records tell us about the cluster activity can be found when expanding the `k8s.msg` field of the records.\n", "\n", "The JSON-formatted k8s.msg contains a list of items to which the event relates. Usually this is only one item, but in some cases, the event is related to multiple items, e.g., when multiple items of the same type are added or deleted.\n", "\n", "For this reason, we create a new DataFrame, where events are potentially duplicated for each item if there are multiple. Special consideration is given to extract IP releated data out of `k8s.msg` where avaible. The resulting information is stored in the new DataFrame `df_ke_ext`." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
indexversiontypetsts_utspod.hostippod.internalipnode.idnode.ipfilename...namenamespacets_itemipiptypeporttargetPortportnameprotonodePort
114KE2022-03-17T18:47:15.8456311647542835845631000()()minikube192.168.59.100/mnt/data/1647542836...defaultNone2022-03-17T15:19:35ZNaNNaNNaNNaNNaNNaNNaN
214KE2022-03-17T18:47:15.8456311647542835845631000()()minikube192.168.59.100/mnt/data/1647542836...kube-node-leaseNone2022-03-17T15:19:33ZNaNNaNNaNNaNNaNNaNNaN
314KE2022-03-17T18:47:15.8456311647542835845631000()()minikube192.168.59.100/mnt/data/1647542836...kube-publicNone2022-03-17T15:19:33ZNaNNaNNaNNaNNaNNaNNaN
414KE2022-03-17T18:47:15.8456311647542835845631000()()minikube192.168.59.100/mnt/data/1647542836...kube-systemNone2022-03-17T15:19:33ZNaNNaNNaNNaNNaNNaNNaN
514KE2022-03-17T18:47:15.8456311647542835845631000()()minikube192.168.59.100/mnt/data/1647542836...sysflowNone2022-03-17T15:32:12ZNaNNaNNaNNaNNaNNaNNaN
..................................................................
1871577074KE2022-03-17T19:25:50.6924441647545150692443689()()minikube192.168.59.100/mnt/data/1647545115...robot-shopNone2022-03-17T18:48:40ZNaNNaNNaNNaNNaNNaNNaN
1881577084KE2022-03-17T19:25:50.6924441647545150692443689()()minikube192.168.59.100/mnt/data/1647545115...robot-shopNone2022-03-17T18:48:40ZNaNNaNNaNNaNNaNNaNNaN
1891594054KE2022-03-17T19:26:56.7469721647545216746972249()()minikube192.168.59.100/mnt/data/1647545175...minikubeNone2022-03-17T15:19:33ZNaNNaNNaNNaNNaNNaNNaN
1901648364KE2022-03-17T19:30:57.8807671647545457880767012()()minikube192.168.59.100/mnt/data/1647545415...minikubeNone2022-03-17T15:19:33ZNaNNaNNaNNaNNaNNaNNaN
1911663724KE2022-03-17T19:32:04.9315461647545524931546223()()minikube192.168.59.100/mnt/data/1647545475...minikubeNone2022-03-17T15:19:33ZNaNNaNNaNNaNNaNNaNNaN
\n", "

192 rows × 28 columns

\n", "
" ], "text/plain": [ " index version type ts ts_uts \\\n", "1 1 4 KE 2022-03-17T18:47:15.845631 1647542835845631000 \n", "2 1 4 KE 2022-03-17T18:47:15.845631 1647542835845631000 \n", "3 1 4 KE 2022-03-17T18:47:15.845631 1647542835845631000 \n", "4 1 4 KE 2022-03-17T18:47:15.845631 1647542835845631000 \n", "5 1 4 KE 2022-03-17T18:47:15.845631 1647542835845631000 \n", ".. ... ... ... ... ... \n", "187 157707 4 KE 2022-03-17T19:25:50.692444 1647545150692443689 \n", "188 157708 4 KE 2022-03-17T19:25:50.692444 1647545150692443689 \n", "189 159405 4 KE 2022-03-17T19:26:56.746972 1647545216746972249 \n", "190 164836 4 KE 2022-03-17T19:30:57.880767 1647545457880767012 \n", "191 166372 4 KE 2022-03-17T19:32:04.931546 1647545524931546223 \n", "\n", " pod.hostip pod.internalip node.id node.ip filename \\\n", "1 () () minikube 192.168.59.100 /mnt/data/1647542836 \n", "2 () () minikube 192.168.59.100 /mnt/data/1647542836 \n", "3 () () minikube 192.168.59.100 /mnt/data/1647542836 \n", "4 () () minikube 192.168.59.100 /mnt/data/1647542836 \n", "5 () () minikube 192.168.59.100 /mnt/data/1647542836 \n", ".. ... ... ... ... ... \n", "187 () () minikube 192.168.59.100 /mnt/data/1647545115 \n", "188 () () minikube 192.168.59.100 /mnt/data/1647545115 \n", "189 () () minikube 192.168.59.100 /mnt/data/1647545175 \n", "190 () () minikube 192.168.59.100 /mnt/data/1647545415 \n", "191 () () minikube 192.168.59.100 /mnt/data/1647545475 \n", "\n", " ... name namespace ts_item ip iptype port \\\n", "1 ... default None 2022-03-17T15:19:35Z NaN NaN NaN \n", "2 ... kube-node-lease None 2022-03-17T15:19:33Z NaN NaN NaN \n", "3 ... kube-public None 2022-03-17T15:19:33Z NaN NaN NaN \n", "4 ... kube-system None 2022-03-17T15:19:33Z NaN NaN NaN \n", "5 ... sysflow None 2022-03-17T15:32:12Z NaN NaN NaN \n", ".. ... ... ... ... ... ... ... \n", "187 ... robot-shop None 2022-03-17T18:48:40Z NaN NaN NaN \n", "188 ... robot-shop None 2022-03-17T18:48:40Z NaN NaN NaN \n", "189 ... minikube None 2022-03-17T15:19:33Z NaN NaN NaN \n", "190 ... minikube None 2022-03-17T15:19:33Z NaN NaN NaN \n", "191 ... minikube None 2022-03-17T15:19:33Z NaN NaN NaN \n", "\n", " targetPort portname proto nodePort \n", "1 NaN NaN NaN NaN \n", "2 NaN NaN NaN NaN \n", "3 NaN NaN NaN NaN \n", "4 NaN NaN NaN NaN \n", "5 NaN NaN NaN NaN \n", ".. ... ... ... ... \n", "187 NaN NaN NaN NaN \n", "188 NaN NaN NaN NaN \n", "189 NaN NaN NaN NaN \n", "190 NaN NaN NaN NaN \n", "191 NaN NaN NaN NaN \n", "\n", "[192 rows x 28 columns]" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "table = []\n", "itemcols = ['name', 'namespace', 'podIP', 'hostIP', 'clusterIP']\n", "for ie,e in df_ke.iterrows():\n", " msg = json.loads(e['k8s.msg'])\n", " for item in msg['items']:\n", " d = e.to_dict()\n", " d['msg_hash'] = hash(str(msg))\n", " d['kind'] = msg['kind']\n", " d['typ'] = msg['type']\n", " d['name'] = item.get('name')\n", " d['namespace'] = item.get('namespace')\n", " d['ts_item'] = item.get('timestamp')\n", " if item.get('podIP'):\n", " d['ip'] = item.get('podIP')\n", " d['iptype'] = 'podIP'\n", " # d['podIP'] = item.get('podIP')\n", " table.append(d)\n", " elif item.get('hostIP'):\n", " d['ip'] = item.get('hostIP')\n", " d['iptype'] = 'hostIP'\n", " # d['hostIP'] = item.get('hostIP')\n", " table.append(d)\n", " elif item.get('clusterIP') and not item.get('clusterIP')=='None':\n", " if msg['kind'] != 'Service':\n", " print(f'>>>> WARNING: clusterIP but not a service - investigate! msg: {msg}')\n", " continue\n", " d['ip'] = item.get('clusterIP')\n", " d['iptype'] = 'clusterIP'\n", " # d['clusterIP'] = item.get('clusterIP')\n", " ports = item.get('ports')\n", " for port in ports:\n", " port['portname'] = port['name'] # make sure to avoid overwriting data in existing record\n", " del port['name']\n", " d.update(port)\n", " # fix naming of proto\n", " d['proto'] = d['protocol']\n", " del d['protocol']\n", " table.append(d)\n", " else: \n", " table.append(d)\n", "df_ke_ext = pd.DataFrame(table).reset_index(drop=True).sort_values(['ts','kind'])\n", "df_ke_ext" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally we put the focus onto the subset of data concerned with the robot-shop application. The final DataFrame `df_ke_ext_sel` is a restriction of the KE event information to events related to the robot-shop, with a focus on addition/deletion events." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "kind typ \n", "Pod ADDED 15\n", " DELETED 15\n", "Service ADDED 14\n", " DELETED 14\n", "Namespace ADDED 2\n", " DELETED 1\n", "dtype: int64" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_ke_ext_sel = df_ke_ext[((df_ke_ext.name == 'robot-shop') | (df_ke_ext.namespace =='robot-shop')) & ((df_ke_ext.typ == 'ADDED') | (df_ke_ext.typ == 'DELETED'))]\n", "df_ke_ext_sel.value_counts(['kind', 'typ'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Timeline comparison of experiment with kubernetes events" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next, we compare the timeline of the experiment with the Kubernetes events seen in the SysFlow data." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "fig = go.Figure()\n", "pp = pprint.PrettyPrinter(indent=4, width=80, compact=True)\n", "log_events_cleaned = list(filter(lambda x: 'waiting' not in x[1] and 'starting' not in x[1], log_events))\n", "# fig.add_trace(go.Scatter(x=[e[0] for e in log_events_cleaned], y=['LOGEVENT' for e in log_events_cleaned], text=[e[1] for e in log_events_cleaned], mode='markers', marker_size=10, marker_symbol='diamond-open'))\n", "for e in log_events_cleaned:\n", " fig.add_annotation(x=e[0], xref='x', yref='paper', y=1., text=e[1], xanchor='left', showarrow=True, textangle=-35, arrowwidth=2)\n", " fig.add_shape(dict(type=\"line\", x0=e[0], y0=0, x1=e[0], y1=1, xref='x', yref='paper', line=dict(color=\"RoyalBlue\", width=2)))\n", "\n", "texts = ['' + \n", " df_ke_ext_sel.iloc[i]['k8s.kind'] +'
' +\n", " df_ke_ext_sel.iloc[i]['k8s.action'] +'
' +\n", " pp.pformat(json.loads(df_ke_ext_sel.iloc[i]['k8s.msg'])).replace('\\n', '
') +\n", " ''\n", " for i in range(df_ke_ext_sel.shape[0])]\n", "colors = ['green' if df_ke_ext_sel.iloc[i]['k8s.action'] == 'K8S_COMPONENT_ADDED' else 'red'\n", " for i in range(df_ke_ext_sel.shape[0])]\n", "symbols = ['triangle-up' if df_ke_ext_sel.iloc[i]['k8s.action'] == 'K8S_COMPONENT_ADDED' else 'triangle-down'\n", " for i in range(df_ke_ext_sel.shape[0])]\n", " \n", "fig.add_trace(go.Scatter(x=df_ke_ext_sel.ts, \n", " y=[df_ke_ext_sel.iloc[i]['k8s.kind'] for i in range(df_ke_ext_sel.shape[0])],\n", " text=texts,\n", " mode='markers', marker_size=20, marker_color=colors, marker_symbol=symbols, marker_line_color='black', marker_line_width=1))\n", " \n", "fig.update_layout(height=900, margin=dict(t=200, pad=4), showlegend=False)\n", "fig.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Green triangles represent the creation of a component (`K8S_COMPONENT_ADDED`), whereas red triangles represent the deletion of a component (`K8S_COMPONENT_DELETED`). It is quite clear that we find KE events for all the changes related to the robot-shop application happening in the experiment: creation/deletion of the namespace robot-shop, creation/deletion of services and creation/deletion of pods, on beginning and end, but also seen when we forcibly killed some of the containers of the robot-shop application." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## IP address information in `KE` records" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's now take a quick look into the IP data gathered from the KE records." ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
kindnamenamespaceiptypeipprotoporttargetPortportname
63Serviceshippingrobot-shopclusterIP10.103.83.70TCP8080.08080.0http
64Serviceratingsrobot-shopclusterIP10.104.84.135TCP80.080.0http
58Serviceuserrobot-shopclusterIP10.105.106.134TCP8080.08080.0http
62Servicemysqlrobot-shopclusterIP10.107.7.181TCP3306.03306.0mysql
147Servicepaymentrobot-shopclusterIP10.108.220.250TCP8080.08080.0http
69Servicemongodbrobot-shopclusterIP10.109.105.252TCP27017.027017.0mongo
146Servicecartrobot-shopclusterIP10.109.213.103TCP8080.08080.0http
141Servicerabbitmqrobot-shopclusterIP10.109.218.161TCP4369.04369.0tcp-epmd
60Serviceredisrobot-shopclusterIP10.111.214.104TCP6379.06379.0redis
137Servicecataloguerobot-shopclusterIP10.96.58.129TCP8080.08080.0http
136Servicewebrobot-shopclusterIP10.99.3.147TCP8080.08080.0http
165Podmysql-6d778f4c8f-4bcr7robot-shoppodIP172.17.0.10NaNNaNNaNNaN
162Podshipping-7f6dfbf46f-94trrrobot-shoppodIP172.17.0.11NaNNaNNaNNaN
156Podratings-7ccf67b49f-6qckrrobot-shoppodIP172.17.0.12NaNNaNNaNNaN
174Poddispatch-69b65d89b9-4lgl7robot-shoppodIP172.17.0.13NaNNaNNaNNaN
171Podpayment-5465d9cc79-8ln4brobot-shoppodIP172.17.0.14NaNNaNNaNNaN
150Podredis-0robot-shoppodIP172.17.0.15NaNNaNNaNNaN
168Podcatalogue-998b69bc9-bfnr7robot-shoppodIP172.17.0.4NaNNaNNaNNaN
180Podrabbitmq-785b678f74-mhhtgrobot-shoppodIP172.17.0.5NaNNaNNaNNaN
114Poduser-899b6c7ff-c7wnjrobot-shoppodIP172.17.0.6NaNNaNNaNNaN
183Podcart-7d7745696b-qgb99robot-shoppodIP172.17.0.8NaNNaNNaNNaN
153Podweb-77486f858f-jnf9rrobot-shoppodIP172.17.0.9NaNNaNNaNNaN
97Podmongodb-67c5456f4-d4bgvrobot-shoppodIP172.17.0.9NaNNaNNaNNaN
105Podweb-77486f858f-gpfrnrobot-shophostIP192.168.59.100NaNNaNNaNNaN
159Podmongodb-67c5456f4-ddhnfrobot-shophostIP192.168.59.100NaNNaNNaNNaN
177Poduser-899b6c7ff-qxd47robot-shophostIP192.168.59.100NaNNaNNaNNaN
\n", "
" ], "text/plain": [ " kind name namespace iptype \\\n", "63 Service shipping robot-shop clusterIP \n", "64 Service ratings robot-shop clusterIP \n", "58 Service user robot-shop clusterIP \n", "62 Service mysql robot-shop clusterIP \n", "147 Service payment robot-shop clusterIP \n", "69 Service mongodb robot-shop clusterIP \n", "146 Service cart robot-shop clusterIP \n", "141 Service rabbitmq robot-shop clusterIP \n", "60 Service redis robot-shop clusterIP \n", "137 Service catalogue robot-shop clusterIP \n", "136 Service web robot-shop clusterIP \n", "165 Pod mysql-6d778f4c8f-4bcr7 robot-shop podIP \n", "162 Pod shipping-7f6dfbf46f-94trr robot-shop podIP \n", "156 Pod ratings-7ccf67b49f-6qckr robot-shop podIP \n", "174 Pod dispatch-69b65d89b9-4lgl7 robot-shop podIP \n", "171 Pod payment-5465d9cc79-8ln4b robot-shop podIP \n", "150 Pod redis-0 robot-shop podIP \n", "168 Pod catalogue-998b69bc9-bfnr7 robot-shop podIP \n", "180 Pod rabbitmq-785b678f74-mhhtg robot-shop podIP \n", "114 Pod user-899b6c7ff-c7wnj robot-shop podIP \n", "183 Pod cart-7d7745696b-qgb99 robot-shop podIP \n", "153 Pod web-77486f858f-jnf9r robot-shop podIP \n", "97 Pod mongodb-67c5456f4-d4bgv robot-shop podIP \n", "105 Pod web-77486f858f-gpfrn robot-shop hostIP \n", "159 Pod mongodb-67c5456f4-ddhnf robot-shop hostIP \n", "177 Pod user-899b6c7ff-qxd47 robot-shop hostIP \n", "\n", " ip proto port targetPort portname \n", "63 10.103.83.70 TCP 8080.0 8080.0 http \n", "64 10.104.84.135 TCP 80.0 80.0 http \n", "58 10.105.106.134 TCP 8080.0 8080.0 http \n", "62 10.107.7.181 TCP 3306.0 3306.0 mysql \n", "147 10.108.220.250 TCP 8080.0 8080.0 http \n", "69 10.109.105.252 TCP 27017.0 27017.0 mongo \n", "146 10.109.213.103 TCP 8080.0 8080.0 http \n", "141 10.109.218.161 TCP 4369.0 4369.0 tcp-epmd \n", "60 10.111.214.104 TCP 6379.0 6379.0 redis \n", "137 10.96.58.129 TCP 8080.0 8080.0 http \n", "136 10.99.3.147 TCP 8080.0 8080.0 http \n", "165 172.17.0.10 NaN NaN NaN NaN \n", "162 172.17.0.11 NaN NaN NaN NaN \n", "156 172.17.0.12 NaN NaN NaN NaN \n", "174 172.17.0.13 NaN NaN NaN NaN \n", "171 172.17.0.14 NaN NaN NaN NaN \n", "150 172.17.0.15 NaN NaN NaN NaN \n", "168 172.17.0.4 NaN NaN NaN NaN \n", "180 172.17.0.5 NaN NaN NaN NaN \n", "114 172.17.0.6 NaN NaN NaN NaN \n", "183 172.17.0.8 NaN NaN NaN NaN \n", "153 172.17.0.9 NaN NaN NaN NaN \n", "97 172.17.0.9 NaN NaN NaN NaN \n", "105 192.168.59.100 NaN NaN NaN NaN \n", "159 192.168.59.100 NaN NaN NaN NaN \n", "177 192.168.59.100 NaN NaN NaN NaN " ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_ke_ext_sel.dropna(subset=['iptype'])[['kind', 'name', 'namespace', 'iptype', 'ip', 'proto', 'port', 'targetPort', 'portname']].sort_values('ip').drop_duplicates()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the data, we can recognize quite a bit of IP address information:\n", "- in some cases we see a `hostIP` (that corresponds to the `node.ip` data in the records)\n", "- podIPs in the private range 172.17.0.0/16 showing the main internal IPs of the respective pods\n", "- clusterIPs in the private range 10.0.0.0/8: these are most interesting as they represent the IP addresses of the service endpoints, i.e., this is new information of the cluster level. This is more complex information as it not only includes IP address, but the service is specific also to a port and has additional information!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let us keep track of the pod IPs gleaned from this data for later use in the comparison with the observed network traffic." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'172.17.0.10': {'mysql-6d778f4c8f-4bcr7'},\n", " '172.17.0.11': {'shipping-7f6dfbf46f-94trr'},\n", " '172.17.0.12': {'ratings-7ccf67b49f-6qckr'},\n", " '172.17.0.13': {'dispatch-69b65d89b9-4lgl7'},\n", " '172.17.0.14': {'payment-5465d9cc79-8ln4b'},\n", " '172.17.0.15': {'redis-0'},\n", " '172.17.0.4': {'catalogue-998b69bc9-bfnr7'},\n", " '172.17.0.5': {'rabbitmq-785b678f74-mhhtg'},\n", " '172.17.0.6': {'user-899b6c7ff-c7wnj'},\n", " '172.17.0.8': {'cart-7d7745696b-qgb99'},\n", " '172.17.0.9': {'mongodb-67c5456f4-d4bgv', 'web-77486f858f-jnf9r'}}" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "podips = {}\n", "for irow, row in df_ke_ext_sel.dropna(subset=['iptype'])[['kind', 'name', 'namespace', 'iptype', 'ip', 'proto', 'port', 'targetPort', 'portname']].sort_values('ip').drop_duplicates().iterrows():\n", " if not row['iptype'] == 'podIP': continue\n", " podips.setdefault(row['ip'], set()).add(row['name'])\n", "podips" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# New `pod.*` fields" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "SysFlow records identify the containers they belong to. In the context of a Kubernetes/OpenShift cluster, each container belongs to a *pod*, which in turn is part of a *namespace*. Every SysFlow record now contains this metadata that helps to put the low-level, container-related information into the context of the cluster." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Container vs Pod" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let us first look into the relationship between containers and pods. To make this easier, let's focus again on the containers related to the robot-shop application." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
container.namecontainer.idcontainer.imagepod.name0
0k8s_POD_cart-7d7745696b-qgb99_robot-shop_06111...9f2d50473bb4k8s.gcr.io/pause:3.6:3.62
1k8s_POD_catalogue-998b69bc9-bfnr7_robot-shop_4...eb32dd737f52k8s.gcr.io/pause:3.6:3.62
2k8s_POD_dispatch-69b65d89b9-4lgl7_robot-shop_b...63e3c777b5c7k8s.gcr.io/pause:3.6:3.62
3k8s_POD_mongodb-67c5456f4-d4bgv_robot-shop_646...9d90554ce0a6k8s.gcr.io/pause:3.6:3.62
4k8s_POD_mongodb-67c5456f4-ddhnf_robot-shop_423...a1397d34b86ek8s.gcr.io/pause:3.6:3.62
5k8s_POD_mysql-6d778f4c8f-4bcr7_robot-shop_93bb...cabf4edbc827k8s.gcr.io/pause:3.6:3.62
6k8s_POD_payment-5465d9cc79-8ln4b_robot-shop_39...f4631d398156k8s.gcr.io/pause:3.6:3.62
7k8s_POD_rabbitmq-785b678f74-mhhtg_robot-shop_b...ba13946ea53fk8s.gcr.io/pause:3.6:3.62
8k8s_POD_ratings-7ccf67b49f-6qckr_robot-shop_ee...4d4b1cf9894dk8s.gcr.io/pause:3.6:3.62
9k8s_POD_redis-0_robot-shop_29c707fd-318a-4af4-...b3f1af98e495k8s.gcr.io/pause:3.6:3.62
10k8s_POD_shipping-7f6dfbf46f-94trr_robot-shop_5...c661019b5785k8s.gcr.io/pause:3.6:3.62
11k8s_POD_user-899b6c7ff-c7wnj_robot-shop_6fad1f...7eca18b706efk8s.gcr.io/pause:3.6:3.62
12k8s_POD_user-899b6c7ff-qxd47_robot-shop_b9b155...b59379f4b206k8s.gcr.io/pause:3.6:3.62
13k8s_POD_web-77486f858f-gpfrn_robot-shop_cb8700...43861efc9ee3k8s.gcr.io/pause:3.6:3.62
14k8s_POD_web-77486f858f-jnf9r_robot-shop_9f6687...96f8362d2e24k8s.gcr.io/pause:3.6:3.62
15k8s_cart_cart-7d7745696b-qgb99_robot-shop_0611...668cdd1c7d0csha256:791bdeb9c40842f51f9037086ac06b6e630d6b0...2658
16k8s_catalogue_catalogue-998b69bc9-bfnr7_robot-...e9b881dd05e5sha256:7cdadb2368155c32906c789493f7481975a1ccc...886
17k8s_catalogue_catalogue-998b69bc9-bfnr7_robot-...e9b881dd05e5sha256:7cdadb2368155c32906c789493f7481975a1ccc...catalogue-998b69bc9-bfnr71962
18k8s_dispatch_dispatch-69b65d89b9-4lgl7_robot-s...e04525cc3464sha256:7e63803d2a06df53549d0d089b88b56d242920f...59
19k8s_dispatch_dispatch-69b65d89b9-4lgl7_robot-s...e04525cc3464sha256:7e63803d2a06df53549d0d089b88b56d242920f...dispatch-69b65d89b9-4lgl75716
20k8s_mongodb_mongodb-67c5456f4-d4bgv_robot-shop...cd4ea3f3e8aesha256:621ddd7848a2327f471de8541d8b020d65a58a1...1045
21k8s_mongodb_mongodb-67c5456f4-d4bgv_robot-shop...cd4ea3f3e8aesha256:621ddd7848a2327f471de8541d8b020d65a58a1...mongodb-67c5456f4-d4bgv9311
22k8s_mongodb_mongodb-67c5456f4-ddhnf_robot-shop...4269f06be75asha256:621ddd7848a2327f471de8541d8b020d65a58a1...1315
23k8s_mongodb_mongodb-67c5456f4-ddhnf_robot-shop...4269f06be75asha256:621ddd7848a2327f471de8541d8b020d65a58a1...mongodb-67c5456f4-ddhnf9721
24k8s_mysql_mysql-6d778f4c8f-4bcr7_robot-shop_93...38af625d9ec8sha256:1a9332d91b6161f822552e2c9b3411b9ca75fad...877
25k8s_mysql_mysql-6d778f4c8f-4bcr7_robot-shop_93...38af625d9ec8sha256:1a9332d91b6161f822552e2c9b3411b9ca75fad...mysql-6d778f4c8f-4bcr75429
26k8s_payment_payment-5465d9cc79-8ln4b_robot-sho...eed324f483a1sha256:d7df366a05b47922b99cb6cfe58384f28edf79c...659
27k8s_payment_payment-5465d9cc79-8ln4b_robot-sho...eed324f483a1sha256:d7df366a05b47922b99cb6cfe58384f28edf79c...payment-5465d9cc79-8ln4b363
28k8s_rabbitmq_rabbitmq-785b678f74-mhhtg_robot-s...27a319d7a44dsha256:f6219c413094ec0fbaa87f134dbfb3bc4a28b05...1184
29k8s_rabbitmq_rabbitmq-785b678f74-mhhtg_robot-s...27a319d7a44dsha256:f6219c413094ec0fbaa87f134dbfb3bc4a28b05...rabbitmq-785b678f74-mhhtg14885
30k8s_ratings_ratings-7ccf67b49f-6qckr_robot-sho...2bcca59fd11csha256:a1676b19d3f9142ab46185efe631d1dd9918bc7...1114
31k8s_ratings_ratings-7ccf67b49f-6qckr_robot-sho...2bcca59fd11csha256:a1676b19d3f9142ab46185efe631d1dd9918bc7...ratings-7ccf67b49f-6qckr5777
32k8s_redis_redis-0_robot-shop_29c707fd-318a-4af...9e71b7187518sha256:1e70071f4af45af2cc9e1d1300c675c1ce37ee2...123
33k8s_redis_redis-0_robot-shop_29c707fd-318a-4af...9e71b7187518sha256:1e70071f4af45af2cc9e1d1300c675c1ce37ee2...redis-018127
34k8s_shipping_shipping-7f6dfbf46f-94trr_robot-s...34bdedd4f86dsha256:3bb300680fb6c3257ba84e8a65715ad621dab14...78
35k8s_shipping_shipping-7f6dfbf46f-94trr_robot-s...34bdedd4f86dsha256:3bb300680fb6c3257ba84e8a65715ad621dab14...shipping-7f6dfbf46f-94trr1894
36k8s_user_user-899b6c7ff-c7wnj_robot-shop_6fad1...942c80fd11easha256:8c1e369ddb3a21c7d2b66ab46d66d27ce3a6200...874
37k8s_user_user-899b6c7ff-c7wnj_robot-shop_6fad1...942c80fd11easha256:8c1e369ddb3a21c7d2b66ab46d66d27ce3a6200...user-899b6c7ff-c7wnj1541
38k8s_user_user-899b6c7ff-qxd47_robot-shop_b9b15...35ef3651eb95sha256:8c1e369ddb3a21c7d2b66ab46d66d27ce3a6200...509
39k8s_user_user-899b6c7ff-qxd47_robot-shop_b9b15...35ef3651eb95sha256:8c1e369ddb3a21c7d2b66ab46d66d27ce3a6200...user-899b6c7ff-qxd47917
40k8s_web_web-77486f858f-gpfrn_robot-shop_cb8700...69ca25df3460sha256:e1d19c905d3ba8267a5a41cc19230a26f4bf967...992
41k8s_web_web-77486f858f-jnf9r_robot-shop_9f6687...6765f016049asha256:e1d19c905d3ba8267a5a41cc19230a26f4bf967...660
42k8s_web_web-77486f858f-jnf9r_robot-shop_9f6687...6765f016049asha256:e1d19c905d3ba8267a5a41cc19230a26f4bf967...web-77486f858f-jnf9r39
\n", "
" ], "text/plain": [ " container.name container.id \\\n", "0 k8s_POD_cart-7d7745696b-qgb99_robot-shop_06111... 9f2d50473bb4 \n", "1 k8s_POD_catalogue-998b69bc9-bfnr7_robot-shop_4... eb32dd737f52 \n", "2 k8s_POD_dispatch-69b65d89b9-4lgl7_robot-shop_b... 63e3c777b5c7 \n", "3 k8s_POD_mongodb-67c5456f4-d4bgv_robot-shop_646... 9d90554ce0a6 \n", "4 k8s_POD_mongodb-67c5456f4-ddhnf_robot-shop_423... a1397d34b86e \n", "5 k8s_POD_mysql-6d778f4c8f-4bcr7_robot-shop_93bb... cabf4edbc827 \n", "6 k8s_POD_payment-5465d9cc79-8ln4b_robot-shop_39... f4631d398156 \n", "7 k8s_POD_rabbitmq-785b678f74-mhhtg_robot-shop_b... ba13946ea53f \n", "8 k8s_POD_ratings-7ccf67b49f-6qckr_robot-shop_ee... 4d4b1cf9894d \n", "9 k8s_POD_redis-0_robot-shop_29c707fd-318a-4af4-... b3f1af98e495 \n", "10 k8s_POD_shipping-7f6dfbf46f-94trr_robot-shop_5... c661019b5785 \n", "11 k8s_POD_user-899b6c7ff-c7wnj_robot-shop_6fad1f... 7eca18b706ef \n", "12 k8s_POD_user-899b6c7ff-qxd47_robot-shop_b9b155... b59379f4b206 \n", "13 k8s_POD_web-77486f858f-gpfrn_robot-shop_cb8700... 43861efc9ee3 \n", "14 k8s_POD_web-77486f858f-jnf9r_robot-shop_9f6687... 96f8362d2e24 \n", "15 k8s_cart_cart-7d7745696b-qgb99_robot-shop_0611... 668cdd1c7d0c \n", "16 k8s_catalogue_catalogue-998b69bc9-bfnr7_robot-... e9b881dd05e5 \n", "17 k8s_catalogue_catalogue-998b69bc9-bfnr7_robot-... e9b881dd05e5 \n", "18 k8s_dispatch_dispatch-69b65d89b9-4lgl7_robot-s... e04525cc3464 \n", "19 k8s_dispatch_dispatch-69b65d89b9-4lgl7_robot-s... e04525cc3464 \n", "20 k8s_mongodb_mongodb-67c5456f4-d4bgv_robot-shop... cd4ea3f3e8ae \n", "21 k8s_mongodb_mongodb-67c5456f4-d4bgv_robot-shop... cd4ea3f3e8ae \n", "22 k8s_mongodb_mongodb-67c5456f4-ddhnf_robot-shop... 4269f06be75a \n", "23 k8s_mongodb_mongodb-67c5456f4-ddhnf_robot-shop... 4269f06be75a \n", "24 k8s_mysql_mysql-6d778f4c8f-4bcr7_robot-shop_93... 38af625d9ec8 \n", "25 k8s_mysql_mysql-6d778f4c8f-4bcr7_robot-shop_93... 38af625d9ec8 \n", "26 k8s_payment_payment-5465d9cc79-8ln4b_robot-sho... eed324f483a1 \n", "27 k8s_payment_payment-5465d9cc79-8ln4b_robot-sho... eed324f483a1 \n", "28 k8s_rabbitmq_rabbitmq-785b678f74-mhhtg_robot-s... 27a319d7a44d \n", "29 k8s_rabbitmq_rabbitmq-785b678f74-mhhtg_robot-s... 27a319d7a44d \n", "30 k8s_ratings_ratings-7ccf67b49f-6qckr_robot-sho... 2bcca59fd11c \n", "31 k8s_ratings_ratings-7ccf67b49f-6qckr_robot-sho... 2bcca59fd11c \n", "32 k8s_redis_redis-0_robot-shop_29c707fd-318a-4af... 9e71b7187518 \n", "33 k8s_redis_redis-0_robot-shop_29c707fd-318a-4af... 9e71b7187518 \n", "34 k8s_shipping_shipping-7f6dfbf46f-94trr_robot-s... 34bdedd4f86d \n", "35 k8s_shipping_shipping-7f6dfbf46f-94trr_robot-s... 34bdedd4f86d \n", "36 k8s_user_user-899b6c7ff-c7wnj_robot-shop_6fad1... 942c80fd11ea \n", "37 k8s_user_user-899b6c7ff-c7wnj_robot-shop_6fad1... 942c80fd11ea \n", "38 k8s_user_user-899b6c7ff-qxd47_robot-shop_b9b15... 35ef3651eb95 \n", "39 k8s_user_user-899b6c7ff-qxd47_robot-shop_b9b15... 35ef3651eb95 \n", "40 k8s_web_web-77486f858f-gpfrn_robot-shop_cb8700... 69ca25df3460 \n", "41 k8s_web_web-77486f858f-jnf9r_robot-shop_9f6687... 6765f016049a \n", "42 k8s_web_web-77486f858f-jnf9r_robot-shop_9f6687... 6765f016049a \n", "\n", " container.image \\\n", "0 k8s.gcr.io/pause:3.6:3.6 \n", "1 k8s.gcr.io/pause:3.6:3.6 \n", "2 k8s.gcr.io/pause:3.6:3.6 \n", "3 k8s.gcr.io/pause:3.6:3.6 \n", "4 k8s.gcr.io/pause:3.6:3.6 \n", "5 k8s.gcr.io/pause:3.6:3.6 \n", "6 k8s.gcr.io/pause:3.6:3.6 \n", "7 k8s.gcr.io/pause:3.6:3.6 \n", "8 k8s.gcr.io/pause:3.6:3.6 \n", "9 k8s.gcr.io/pause:3.6:3.6 \n", "10 k8s.gcr.io/pause:3.6:3.6 \n", "11 k8s.gcr.io/pause:3.6:3.6 \n", "12 k8s.gcr.io/pause:3.6:3.6 \n", "13 k8s.gcr.io/pause:3.6:3.6 \n", "14 k8s.gcr.io/pause:3.6:3.6 \n", "15 sha256:791bdeb9c40842f51f9037086ac06b6e630d6b0... \n", "16 sha256:7cdadb2368155c32906c789493f7481975a1ccc... \n", "17 sha256:7cdadb2368155c32906c789493f7481975a1ccc... \n", "18 sha256:7e63803d2a06df53549d0d089b88b56d242920f... \n", "19 sha256:7e63803d2a06df53549d0d089b88b56d242920f... \n", "20 sha256:621ddd7848a2327f471de8541d8b020d65a58a1... \n", "21 sha256:621ddd7848a2327f471de8541d8b020d65a58a1... \n", "22 sha256:621ddd7848a2327f471de8541d8b020d65a58a1... \n", "23 sha256:621ddd7848a2327f471de8541d8b020d65a58a1... \n", "24 sha256:1a9332d91b6161f822552e2c9b3411b9ca75fad... \n", "25 sha256:1a9332d91b6161f822552e2c9b3411b9ca75fad... \n", "26 sha256:d7df366a05b47922b99cb6cfe58384f28edf79c... \n", "27 sha256:d7df366a05b47922b99cb6cfe58384f28edf79c... \n", "28 sha256:f6219c413094ec0fbaa87f134dbfb3bc4a28b05... \n", "29 sha256:f6219c413094ec0fbaa87f134dbfb3bc4a28b05... \n", "30 sha256:a1676b19d3f9142ab46185efe631d1dd9918bc7... \n", "31 sha256:a1676b19d3f9142ab46185efe631d1dd9918bc7... \n", "32 sha256:1e70071f4af45af2cc9e1d1300c675c1ce37ee2... \n", "33 sha256:1e70071f4af45af2cc9e1d1300c675c1ce37ee2... \n", "34 sha256:3bb300680fb6c3257ba84e8a65715ad621dab14... \n", "35 sha256:3bb300680fb6c3257ba84e8a65715ad621dab14... \n", "36 sha256:8c1e369ddb3a21c7d2b66ab46d66d27ce3a6200... \n", "37 sha256:8c1e369ddb3a21c7d2b66ab46d66d27ce3a6200... \n", "38 sha256:8c1e369ddb3a21c7d2b66ab46d66d27ce3a6200... \n", "39 sha256:8c1e369ddb3a21c7d2b66ab46d66d27ce3a6200... \n", "40 sha256:e1d19c905d3ba8267a5a41cc19230a26f4bf967... \n", "41 sha256:e1d19c905d3ba8267a5a41cc19230a26f4bf967... \n", "42 sha256:e1d19c905d3ba8267a5a41cc19230a26f4bf967... \n", "\n", " pod.name 0 \n", "0 2 \n", "1 2 \n", "2 2 \n", "3 2 \n", "4 2 \n", "5 2 \n", "6 2 \n", "7 2 \n", "8 2 \n", "9 2 \n", "10 2 \n", "11 2 \n", "12 2 \n", "13 2 \n", "14 2 \n", "15 2658 \n", "16 886 \n", "17 catalogue-998b69bc9-bfnr7 1962 \n", "18 59 \n", "19 dispatch-69b65d89b9-4lgl7 5716 \n", "20 1045 \n", "21 mongodb-67c5456f4-d4bgv 9311 \n", "22 1315 \n", "23 mongodb-67c5456f4-ddhnf 9721 \n", "24 877 \n", "25 mysql-6d778f4c8f-4bcr7 5429 \n", "26 659 \n", "27 payment-5465d9cc79-8ln4b 363 \n", "28 1184 \n", "29 rabbitmq-785b678f74-mhhtg 14885 \n", "30 1114 \n", "31 ratings-7ccf67b49f-6qckr 5777 \n", "32 123 \n", "33 redis-0 18127 \n", "34 78 \n", "35 shipping-7f6dfbf46f-94trr 1894 \n", "36 874 \n", "37 user-899b6c7ff-c7wnj 1541 \n", "38 509 \n", "39 user-899b6c7ff-qxd47 917 \n", "40 992 \n", "41 660 \n", "42 web-77486f858f-jnf9r 39 " ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[df['container.name'].str.contains('robot-shop')].value_counts(['container.name', 'container.id', 'container.image', 'pod.name'], sort=False).reset_index()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To understand this better, let's take a look at one specific container of the robot-shop setup, picking up the data for the `mongodb` container, as this is also one of the containers that gets killed as part of the experiment and subsequently gets restarted by Kubernetes." ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
container.namecontainer.idpod.namecontainer.imagets
firstminmax
2k8s_mongodb_mongodb-67c5456f4-d4bgv_robot-shop...cd4ea3f3e8aesha256:621ddd7848a2327f471de8541d8b020d65a58a1...2022-03-17T18:50:01.1731242022-03-17T18:50:14.874146
3k8s_mongodb_mongodb-67c5456f4-d4bgv_robot-shop...cd4ea3f3e8aemongodb-67c5456f4-d4bgvsha256:621ddd7848a2327f471de8541d8b020d65a58a1...2022-03-17T18:50:01.2767732022-03-17T19:04:48.912214
0k8s_POD_mongodb-67c5456f4-d4bgv_robot-shop_646...9d90554ce0a6k8s.gcr.io/pause:3.6:3.62022-03-17T19:04:49.4848372022-03-17T19:04:49.485062
4k8s_mongodb_mongodb-67c5456f4-ddhnf_robot-shop...4269f06be75asha256:621ddd7848a2327f471de8541d8b020d65a58a1...2022-03-17T19:04:55.5669712022-03-17T19:05:13.797194
5k8s_mongodb_mongodb-67c5456f4-ddhnf_robot-shop...4269f06be75amongodb-67c5456f4-ddhnfsha256:621ddd7848a2327f471de8541d8b020d65a58a1...2022-03-17T19:04:55.5682212022-03-17T19:20:39.186520
1k8s_POD_mongodb-67c5456f4-ddhnf_robot-shop_423...a1397d34b86ek8s.gcr.io/pause:3.6:3.62022-03-17T19:20:40.3282512022-03-17T19:20:40.328324
\n", "
" ], "text/plain": [ " container.name container.id \\\n", " \n", "2 k8s_mongodb_mongodb-67c5456f4-d4bgv_robot-shop... cd4ea3f3e8ae \n", "3 k8s_mongodb_mongodb-67c5456f4-d4bgv_robot-shop... cd4ea3f3e8ae \n", "0 k8s_POD_mongodb-67c5456f4-d4bgv_robot-shop_646... 9d90554ce0a6 \n", "4 k8s_mongodb_mongodb-67c5456f4-ddhnf_robot-shop... 4269f06be75a \n", "5 k8s_mongodb_mongodb-67c5456f4-ddhnf_robot-shop... 4269f06be75a \n", "1 k8s_POD_mongodb-67c5456f4-ddhnf_robot-shop_423... a1397d34b86e \n", "\n", " pod.name container.image \\\n", " first \n", "2 sha256:621ddd7848a2327f471de8541d8b020d65a58a1... \n", "3 mongodb-67c5456f4-d4bgv sha256:621ddd7848a2327f471de8541d8b020d65a58a1... \n", "0 k8s.gcr.io/pause:3.6:3.6 \n", "4 sha256:621ddd7848a2327f471de8541d8b020d65a58a1... \n", "5 mongodb-67c5456f4-ddhnf sha256:621ddd7848a2327f471de8541d8b020d65a58a1... \n", "1 k8s.gcr.io/pause:3.6:3.6 \n", "\n", " ts \n", " min max \n", "2 2022-03-17T18:50:01.173124 2022-03-17T18:50:14.874146 \n", "3 2022-03-17T18:50:01.276773 2022-03-17T19:04:48.912214 \n", "0 2022-03-17T19:04:49.484837 2022-03-17T19:04:49.485062 \n", "4 2022-03-17T19:04:55.566971 2022-03-17T19:05:13.797194 \n", "5 2022-03-17T19:04:55.568221 2022-03-17T19:20:39.186520 \n", "1 2022-03-17T19:20:40.328251 2022-03-17T19:20:40.328324 " ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[df['container.name'].str.contains('mongodb')].sort_values('ts_uts').groupby(['container.name', 'container.id', 'pod.name']).agg({'container.image': 'first', 'ts': ['min', 'max']}).reset_index().sort_values(by=[('ts', 'min')])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "From this listing we observe that:\n", "- there are 4 `container.id`s involved, 2 each having the same `container.image` - corresponding to our killing of the first `mongodb` container and its restart\n", "- when the container comes up first, we do not see yet related pod information (no `pod.name` here), only slightly after the creation\n", "- when the container gets killed or stopped, we see for a very short time a container with the name `k8s_POD_mongodb-...` using the `pause` image" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As long as we have all information, including the pod data, the relationship between container and pod is unique:" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
container.namepod.name
0k8s_catalogue_catalogue-998b69bc9-bfnr7_robot-...catalogue-998b69bc9-bfnr7
1k8s_coredns_coredns-64897985d-n4jjl_kube-syste...coredns-64897985d-n4jjl
2k8s_dispatch_dispatch-69b65d89b9-4lgl7_robot-s...dispatch-69b65d89b9-4lgl7
3k8s_etcd_etcd-minikube_kube-system_fc45a20ce68...etcd-minikube
4k8s_kube-apiserver_kube-apiserver-minikube_kub...kube-apiserver-minikube
5k8s_kube-controller-manager_kube-controller-ma...kube-controller-manager-minikube
6k8s_kube-proxy_kube-proxy-9g9kt_kube-system_e4...kube-proxy-9g9kt
7k8s_kube-scheduler_kube-scheduler-minikube_kub...kube-scheduler-minikube
8k8s_mongodb_mongodb-67c5456f4-d4bgv_robot-shop...mongodb-67c5456f4-d4bgv
9k8s_mongodb_mongodb-67c5456f4-ddhnf_robot-shop...mongodb-67c5456f4-ddhnf
10k8s_mysql_mysql-6d778f4c8f-4bcr7_robot-shop_93...mysql-6d778f4c8f-4bcr7
11k8s_payment_payment-5465d9cc79-8ln4b_robot-sho...payment-5465d9cc79-8ln4b
12k8s_rabbitmq_rabbitmq-785b678f74-mhhtg_robot-s...rabbitmq-785b678f74-mhhtg
13k8s_ratings_ratings-7ccf67b49f-6qckr_robot-sho...ratings-7ccf67b49f-6qckr
14k8s_redis_redis-0_robot-shop_29c707fd-318a-4af...redis-0
15k8s_sfcollector_sysflowagent-9wslt_sysflow_884...sysflowagent-9wslt
16k8s_sfexporter_sysflowagent-9wslt_sysflow_8845...sysflowagent-9wslt
17k8s_shipping_shipping-7f6dfbf46f-94trr_robot-s...shipping-7f6dfbf46f-94trr
18k8s_sidecar_sysflowagent-9wslt_sysflow_8845ce3...sysflowagent-9wslt
19k8s_storage-provisioner_storage-provisioner_ku...storage-provisioner
20k8s_user_user-899b6c7ff-c7wnj_robot-shop_6fad1...user-899b6c7ff-c7wnj
21k8s_user_user-899b6c7ff-qxd47_robot-shop_b9b15...user-899b6c7ff-qxd47
22k8s_web_web-77486f858f-jnf9r_robot-shop_9f6687...web-77486f858f-jnf9r
\n", "
" ], "text/plain": [ " container.name \\\n", "0 k8s_catalogue_catalogue-998b69bc9-bfnr7_robot-... \n", "1 k8s_coredns_coredns-64897985d-n4jjl_kube-syste... \n", "2 k8s_dispatch_dispatch-69b65d89b9-4lgl7_robot-s... \n", "3 k8s_etcd_etcd-minikube_kube-system_fc45a20ce68... \n", "4 k8s_kube-apiserver_kube-apiserver-minikube_kub... \n", "5 k8s_kube-controller-manager_kube-controller-ma... \n", "6 k8s_kube-proxy_kube-proxy-9g9kt_kube-system_e4... \n", "7 k8s_kube-scheduler_kube-scheduler-minikube_kub... \n", "8 k8s_mongodb_mongodb-67c5456f4-d4bgv_robot-shop... \n", "9 k8s_mongodb_mongodb-67c5456f4-ddhnf_robot-shop... \n", "10 k8s_mysql_mysql-6d778f4c8f-4bcr7_robot-shop_93... \n", "11 k8s_payment_payment-5465d9cc79-8ln4b_robot-sho... \n", "12 k8s_rabbitmq_rabbitmq-785b678f74-mhhtg_robot-s... \n", "13 k8s_ratings_ratings-7ccf67b49f-6qckr_robot-sho... \n", "14 k8s_redis_redis-0_robot-shop_29c707fd-318a-4af... \n", "15 k8s_sfcollector_sysflowagent-9wslt_sysflow_884... \n", "16 k8s_sfexporter_sysflowagent-9wslt_sysflow_8845... \n", "17 k8s_shipping_shipping-7f6dfbf46f-94trr_robot-s... \n", "18 k8s_sidecar_sysflowagent-9wslt_sysflow_8845ce3... \n", "19 k8s_storage-provisioner_storage-provisioner_ku... \n", "20 k8s_user_user-899b6c7ff-c7wnj_robot-shop_6fad1... \n", "21 k8s_user_user-899b6c7ff-qxd47_robot-shop_b9b15... \n", "22 k8s_web_web-77486f858f-jnf9r_robot-shop_9f6687... \n", "\n", " pod.name \n", "0 catalogue-998b69bc9-bfnr7 \n", "1 coredns-64897985d-n4jjl \n", "2 dispatch-69b65d89b9-4lgl7 \n", "3 etcd-minikube \n", "4 kube-apiserver-minikube \n", "5 kube-controller-manager-minikube \n", "6 kube-proxy-9g9kt \n", "7 kube-scheduler-minikube \n", "8 mongodb-67c5456f4-d4bgv \n", "9 mongodb-67c5456f4-ddhnf \n", "10 mysql-6d778f4c8f-4bcr7 \n", "11 payment-5465d9cc79-8ln4b \n", "12 rabbitmq-785b678f74-mhhtg \n", "13 ratings-7ccf67b49f-6qckr \n", "14 redis-0 \n", "15 sysflowagent-9wslt \n", "16 sysflowagent-9wslt \n", "17 shipping-7f6dfbf46f-94trr \n", "18 sysflowagent-9wslt \n", "19 storage-provisioner \n", "20 user-899b6c7ff-c7wnj \n", "21 user-899b6c7ff-qxd47 \n", "22 web-77486f858f-jnf9r " ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[df['pod.name'].astype(bool)][['container.name', 'pod.name']].drop_duplicates().sort_values('container.name').reset_index(drop=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## IP information from k8s metadata in `pod.*` fields" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### IPs of the Pods aka `pod.internalip`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let us start by looking specifically at the `pod.internalip` field first, limiting ourselves to SysFlow records for the `robot-shop` namespace:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
pod.namepod.internalipts
minmax
0catalogue-998b69bc9-bfnr7(172.17.0.4,)2022-03-17T18:49:59.8676362022-03-17T19:21:08.694894
1dispatch-69b65d89b9-4lgl7(172.17.0.13,)2022-03-17T18:50:02.5500972022-03-17T19:21:08.963347
2mongodb-67c5456f4-d4bgv(172.17.0.9,)2022-03-17T18:50:01.2767732022-03-17T19:04:48.912214
3mongodb-67c5456f4-ddhnf(172.17.0.16,)2022-03-17T19:04:55.5682212022-03-17T19:20:39.186520
4mysql-6d778f4c8f-4bcr7(172.17.0.10,)2022-03-17T18:50:01.4529822022-03-17T19:20:41.989241
5payment-5465d9cc79-8ln4b(172.17.0.14,)2022-03-17T18:51:37.2415202022-03-17T19:21:09.370607
6rabbitmq-785b678f74-mhhtg(172.17.0.5,)2022-03-17T18:51:01.0097192022-03-17T19:21:09.582437
7ratings-7ccf67b49f-6qckr(172.17.0.12,)2022-03-17T18:50:02.4018342022-03-17T19:20:40.260521
8redis-0(172.17.0.15,)2022-03-17T18:50:02.5209902022-03-17T19:20:37.733984
9shipping-7f6dfbf46f-94trr(172.17.0.11,)2022-03-17T18:50:02.5713522022-03-17T19:20:40.710991
10user-899b6c7ff-c7wnj(172.17.0.6,)2022-03-17T18:50:00.4973362022-03-17T19:15:33.901413
11user-899b6c7ff-qxd47(172.17.0.7,)2022-03-17T19:15:12.4847462022-03-17T19:21:08.707541
12web-77486f858f-jnf9r(172.17.0.9,)2022-03-17T19:20:13.0273312022-03-17T19:20:38.690758
\n", "
" ], "text/plain": [ " pod.name pod.internalip ts \\\n", " min \n", "0 catalogue-998b69bc9-bfnr7 (172.17.0.4,) 2022-03-17T18:49:59.867636 \n", "1 dispatch-69b65d89b9-4lgl7 (172.17.0.13,) 2022-03-17T18:50:02.550097 \n", "2 mongodb-67c5456f4-d4bgv (172.17.0.9,) 2022-03-17T18:50:01.276773 \n", "3 mongodb-67c5456f4-ddhnf (172.17.0.16,) 2022-03-17T19:04:55.568221 \n", "4 mysql-6d778f4c8f-4bcr7 (172.17.0.10,) 2022-03-17T18:50:01.452982 \n", "5 payment-5465d9cc79-8ln4b (172.17.0.14,) 2022-03-17T18:51:37.241520 \n", "6 rabbitmq-785b678f74-mhhtg (172.17.0.5,) 2022-03-17T18:51:01.009719 \n", "7 ratings-7ccf67b49f-6qckr (172.17.0.12,) 2022-03-17T18:50:02.401834 \n", "8 redis-0 (172.17.0.15,) 2022-03-17T18:50:02.520990 \n", "9 shipping-7f6dfbf46f-94trr (172.17.0.11,) 2022-03-17T18:50:02.571352 \n", "10 user-899b6c7ff-c7wnj (172.17.0.6,) 2022-03-17T18:50:00.497336 \n", "11 user-899b6c7ff-qxd47 (172.17.0.7,) 2022-03-17T19:15:12.484746 \n", "12 web-77486f858f-jnf9r (172.17.0.9,) 2022-03-17T19:20:13.027331 \n", "\n", " \n", " max \n", "0 2022-03-17T19:21:08.694894 \n", "1 2022-03-17T19:21:08.963347 \n", "2 2022-03-17T19:04:48.912214 \n", "3 2022-03-17T19:20:39.186520 \n", "4 2022-03-17T19:20:41.989241 \n", "5 2022-03-17T19:21:09.370607 \n", "6 2022-03-17T19:21:09.582437 \n", "7 2022-03-17T19:20:40.260521 \n", "8 2022-03-17T19:20:37.733984 \n", "9 2022-03-17T19:20:40.710991 \n", "10 2022-03-17T19:15:33.901413 \n", "11 2022-03-17T19:21:08.707541 \n", "12 2022-03-17T19:20:38.690758 " ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_rs = df[df['pod.ns'] == 'robot-shop']\n", "df_rs.sort_values('ts_uts').groupby(['pod.name', 'pod.internalip']).agg({'ts': ['min', 'max']}).reset_index().sort_values('pod.name')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see here that IP addresses are allocated to pods, i.e., if a new pod is respawned after the old one is killed (e.g., for the `mongodb-*` pod), it will receive a new IP address (old ones being reused later on, e.g., `172.17.0.9`)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We use this information to update our list of `podips` with the additional data found here." ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'172.17.0.10': {'mysql-6d778f4c8f-4bcr7'},\n", " '172.17.0.11': {'shipping-7f6dfbf46f-94trr'},\n", " '172.17.0.12': {'ratings-7ccf67b49f-6qckr'},\n", " '172.17.0.13': {'dispatch-69b65d89b9-4lgl7'},\n", " '172.17.0.14': {'payment-5465d9cc79-8ln4b'},\n", " '172.17.0.15': {'redis-0'},\n", " '172.17.0.4': {'catalogue-998b69bc9-bfnr7'},\n", " '172.17.0.5': {'rabbitmq-785b678f74-mhhtg'},\n", " '172.17.0.6': {'user-899b6c7ff-c7wnj'},\n", " '172.17.0.8': {'cart-7d7745696b-qgb99'},\n", " '172.17.0.9': {'mongodb-67c5456f4-d4bgv', 'web-77486f858f-jnf9r'},\n", " '172.17.0.16': {'mongodb-67c5456f4-ddhnf'},\n", " '172.17.0.7': {'user-899b6c7ff-qxd47'}}" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "for irow, row in df_rs.sort_values('ts_uts')[['pod.name', 'pod.internalip']].drop_duplicates().iterrows():\n", " for ip in row['pod.internalip']:\n", " podips.setdefault(ip, set()).add(row['pod.name'])\n", "podips" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### IPs for the robot-shop services" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The information in the `pod.services` attribute shows us the services running in the robot-shop application and gives us details like IP address and port for each service:" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
nameidnamespaceporttargetPortnodePortprotoclusterIPpod.name
0redis81bc1a3e-f067-4b37-a077-54061050c1cbrobot-shop637963790TCP10.111.214.104redis-0
1mongodb09131df1-404c-4a02-bba2-ebe1e4393caarobot-shop27017270170TCP10.109.105.252mongodb-67c5456f4-d4bgv
2userf1686819-1201-4e1b-8d99-8579007104derobot-shop808080800TCP10.105.106.134user-899b6c7ff-c7wnj
3catalogued34b8f78-0865-4b54-a600-c011fe449545robot-shop808080800TCP10.96.58.129catalogue-998b69bc9-bfnr7
4shippinge11145c8-1fe9-4f2d-a9a0-07d039d519ffrobot-shop808080800TCP10.103.83.70shipping-7f6dfbf46f-94trr
5ratings1f438794-721a-4641-8643-8964cba70095robot-shop80800TCP10.104.84.135ratings-7ccf67b49f-6qckr
6mysqld92b4e08-03bf-49a5-954b-026f7744482erobot-shop330633060TCP10.107.7.181mysql-6d778f4c8f-4bcr7
7rabbitmq9a209f55-ed6f-4224-b6d3-2a056f9c7783robot-shop567256720TCP10.109.218.161rabbitmq-785b678f74-mhhtg
8rabbitmq9a209f55-ed6f-4224-b6d3-2a056f9c7783robot-shop15672156720TCP10.109.218.161rabbitmq-785b678f74-mhhtg
9rabbitmq9a209f55-ed6f-4224-b6d3-2a056f9c7783robot-shop436943690TCP10.109.218.161rabbitmq-785b678f74-mhhtg
10payment9eb039d4-dc0d-4837-9d8e-857cb8e5d8ebrobot-shop808080800TCP10.108.220.250payment-5465d9cc79-8ln4b
\n", "
" ], "text/plain": [ " name id namespace port \\\n", "0 redis 81bc1a3e-f067-4b37-a077-54061050c1cb robot-shop 6379 \n", "1 mongodb 09131df1-404c-4a02-bba2-ebe1e4393caa robot-shop 27017 \n", "2 user f1686819-1201-4e1b-8d99-8579007104de robot-shop 8080 \n", "3 catalogue d34b8f78-0865-4b54-a600-c011fe449545 robot-shop 8080 \n", "4 shipping e11145c8-1fe9-4f2d-a9a0-07d039d519ff robot-shop 8080 \n", "5 ratings 1f438794-721a-4641-8643-8964cba70095 robot-shop 80 \n", "6 mysql d92b4e08-03bf-49a5-954b-026f7744482e robot-shop 3306 \n", "7 rabbitmq 9a209f55-ed6f-4224-b6d3-2a056f9c7783 robot-shop 5672 \n", "8 rabbitmq 9a209f55-ed6f-4224-b6d3-2a056f9c7783 robot-shop 15672 \n", "9 rabbitmq 9a209f55-ed6f-4224-b6d3-2a056f9c7783 robot-shop 4369 \n", "10 payment 9eb039d4-dc0d-4837-9d8e-857cb8e5d8eb robot-shop 8080 \n", "\n", " targetPort nodePort proto clusterIP pod.name \n", "0 6379 0 TCP 10.111.214.104 redis-0 \n", "1 27017 0 TCP 10.109.105.252 mongodb-67c5456f4-d4bgv \n", "2 8080 0 TCP 10.105.106.134 user-899b6c7ff-c7wnj \n", "3 8080 0 TCP 10.96.58.129 catalogue-998b69bc9-bfnr7 \n", "4 8080 0 TCP 10.103.83.70 shipping-7f6dfbf46f-94trr \n", "5 80 0 TCP 10.104.84.135 ratings-7ccf67b49f-6qckr \n", "6 3306 0 TCP 10.107.7.181 mysql-6d778f4c8f-4bcr7 \n", "7 5672 0 TCP 10.109.218.161 rabbitmq-785b678f74-mhhtg \n", "8 15672 0 TCP 10.109.218.161 rabbitmq-785b678f74-mhhtg \n", "9 4369 0 TCP 10.109.218.161 rabbitmq-785b678f74-mhhtg \n", "10 8080 0 TCP 10.108.220.250 payment-5465d9cc79-8ln4b " ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "table = []\n", "for irow, row in df_rs.drop_duplicates(subset=['pod.name', 'pod.services_str']).iterrows():\n", " for service in row['pod.services']:\n", " # resolve portList x clusterIP\n", " for cip in service['clusterIP']:\n", " for port in service['portList']:\n", " svc = service.copy()\n", " del svc['portList']\n", " svc.update(port)\n", " del svc['clusterIP']\n", " svc['clusterIP'] = cip\n", " svc['pod.name'] = row['pod.name']\n", " # svc.update(row)\n", " table.append(svc)\n", "df_services = pd.DataFrame(table)\n", "df_services" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Collect the high-level information for later identification in the observed network traffic:" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{('10.111.214.104', 6379): 'redis-6379',\n", " ('10.109.105.252', 27017): 'mongodb-27017',\n", " ('10.105.106.134', 8080): 'user-8080',\n", " ('10.96.58.129', 8080): 'catalogue-8080',\n", " ('10.103.83.70', 8080): 'shipping-8080',\n", " ('10.104.84.135', 80): 'ratings-80',\n", " ('10.107.7.181', 3306): 'mysql-3306',\n", " ('10.109.218.161', 5672): 'rabbitmq-5672',\n", " ('10.109.218.161', 15672): 'rabbitmq-15672',\n", " ('10.109.218.161', 4369): 'rabbitmq-4369',\n", " ('10.108.220.250', 8080): 'payment-8080'}" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "services = {}\n", "for irow, row in df_services.iterrows():\n", " services[(row['clusterIP'], row['port'])] = f\"{row['name']}-{row['port']}\"\n", "services" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Understanding observed network traffic with cluster metadata" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let us take a look at the actually observed network traffic (i.e., the SysFlow NF records) for the robot-shop application, and collect this subset of SysFlow records into `df_rs_traffic`:" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "df_nf = df[df.type == 'NF']\n", "df_rs_traffic = df_nf[df_nf['pod.ns']=='robot-shop'].groupby(['pod.name', 'net.sip', 'net.dip', 'net.dport']).agg({'flow.rops':'sum', 'flow.rbytes':'sum','flow.wops':'sum','flow.wbytes':'sum'}).reset_index()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "... and compare these records with the knowledge about the IP address background that we gathered from the new cluster metadata attributes above (while adding also some 'well-known' background IP information manually):" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "types_source = []\n", "types_destination = []\n", "for irow, row in df_rs_traffic.iterrows():\n", " sip = row['net.sip']\n", " dip = row['net.dip']\n", " dport = row['net.dport']\n", " type_source = ''\n", " if sip == '0.0.0.0': type_source = '\"localhost\"'\n", " elif sip == '127.0.0.1': type_source = '\"localhost\"'\n", " elif sip == '172.17.0.1': type_source = '\"docker network gateway\"'\n", " elif podips.get(sip): type_source = f'POD {podips[sip]}'\n", " type_destination = ''\n", " if dip == '0.0.0.0': type_destination = '\"localhost\"'\n", " elif dip == '127.0.0.1': type_destination = '\"localhost\"'\n", " elif dip == '172.17.0.1': type_destination = '\"docker network gateway\"'\n", " elif podips.get(dip): type_destination = f'POD {podips[dip]}'\n", " elif dport == 42699: type_destination = '\"instana agent\"'\n", " elif sip == dip: \n", " type_source = 'local'\n", " type_destination = 'local'\n", " else:\n", " service = services.get((dip, dport), '')\n", " if service != '': type_destination = f'SERVICE {service}'\n", " if dip == '10.96.0.10' and dport == 53: type_destination = '\"cluster DNS\"'\n", " \n", " types_source.append(type_source)\n", " types_destination.append(type_destination)\n", "df_rs_traffic['type_source'] = types_source\n", "df_rs_traffic['type_destination'] = types_destination\n", "\n", "# for readability\n", "for col in ('net.dport', 'flow.rops', 'flow.rbytes', 'flow.wops', 'flow.wbytes'):\n", " df_rs_traffic[col] = df_rs_traffic[col].apply(int)" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
pod.namenet.sipnet.dipnet.dportflow.ropsflow.rbytesflow.wopsflow.wbytestype_sourcetype_destination
0catalogue-998b69bc9-bfnr7172.17.0.410.109.105.25227017359108558106918992POD {'catalogue-998b69bc9-bfnr7'}SERVICE mongodb-27017
1catalogue-998b69bc9-bfnr7172.17.0.410.96.0.1053148161361486452POD {'catalogue-998b69bc9-bfnr7'}\"cluster DNS\"
4dispatch-69b65d89b9-4lgl7172.17.0.1310.109.218.161567247324691231287POD {'dispatch-69b65d89b9-4lgl7'}SERVICE rabbitmq-5672
5dispatch-69b65d89b9-4lgl7172.17.0.1310.96.0.10532779180224140877440POD {'dispatch-69b65d89b9-4lgl7'}\"cluster DNS\"
9mongodb-67c5456f4-d4bgv172.17.0.1172.17.0.92701792118316344104606\"docker network gateway\"POD {'mongodb-67c5456f4-d4bgv', 'web-77486f858...
10mongodb-67c5456f4-ddhnf172.17.0.1172.17.0.1627017102619986372113133\"docker network gateway\"POD {'mongodb-67c5456f4-ddhnf'}
11mysql-6d778f4c8f-4bcr7172.17.0.1172.17.0.103306413564937116173956\"docker network gateway\"POD {'mysql-6d778f4c8f-4bcr7'}
15rabbitmq-785b678f74-mhhtg127.0.0.1127.0.0.143693547024470\"localhost\"\"localhost\"
16rabbitmq-785b678f74-mhhtg172.17.0.1172.17.0.5567224512872362469\"docker network gateway\"POD {'rabbitmq-785b678f74-mhhtg'}
17rabbitmq-785b678f74-mhhtg172.17.0.5172.17.0.543693937527375POD {'rabbitmq-785b678f74-mhhtg'}POD {'rabbitmq-785b678f74-mhhtg'}
18ratings-7ccf67b49f-6qckr172.17.0.1172.17.0.12806804006847912744694\"docker network gateway\"POD {'ratings-7ccf67b49f-6qckr'}
19ratings-7ccf67b49f-6qckr172.17.0.1210.107.7.1813306106856871133551264POD {'ratings-7ccf67b49f-6qckr'}SERVICE mysql-3306
20ratings-7ccf67b49f-6qckr172.17.0.1210.96.0.10537429163700POD {'ratings-7ccf67b49f-6qckr'}\"cluster DNS\"
21redis-0172.17.0.1172.17.0.15637944237914\"docker network gateway\"POD {'redis-0'}
23shipping-7f6dfbf46f-94trr172.17.0.1172.17.0.1180802582812225835862\"docker network gateway\"POD {'shipping-7f6dfbf46f-94trr'}
24shipping-7f6dfbf46f-94trr172.17.0.1110.107.7.1813306173170859313723POD {'shipping-7f6dfbf46f-94trr'}SERVICE mysql-3306
25shipping-7f6dfbf46f-94trr172.17.0.1110.96.0.105310102010520POD {'shipping-7f6dfbf46f-94trr'}\"cluster DNS\"
27user-899b6c7ff-c7wnj172.17.0.610.109.105.252270172968971088315768POD {'user-899b6c7ff-c7wnj'}SERVICE mongodb-27017
28user-899b6c7ff-c7wnj172.17.0.610.111.214.104637912635114POD {'user-899b6c7ff-c7wnj'}SERVICE redis-6379
29user-899b6c7ff-c7wnj172.17.0.610.96.0.1053363816361944POD {'user-899b6c7ff-c7wnj'}\"cluster DNS\"
33user-899b6c7ff-qxd47172.17.0.710.109.105.2522701765194711913542POD {'user-899b6c7ff-qxd47'}SERVICE mongodb-27017
34user-899b6c7ff-qxd47172.17.0.710.111.214.104637922641114POD {'user-899b6c7ff-qxd47'}SERVICE redis-6379
35user-899b6c7ff-qxd47172.17.0.710.96.0.1053146159841465838POD {'user-899b6c7ff-qxd47'}\"cluster DNS\"
\n", "
" ], "text/plain": [ " pod.name net.sip net.dip net.dport \\\n", "0 catalogue-998b69bc9-bfnr7 172.17.0.4 10.109.105.252 27017 \n", "1 catalogue-998b69bc9-bfnr7 172.17.0.4 10.96.0.10 53 \n", "4 dispatch-69b65d89b9-4lgl7 172.17.0.13 10.109.218.161 5672 \n", "5 dispatch-69b65d89b9-4lgl7 172.17.0.13 10.96.0.10 53 \n", "9 mongodb-67c5456f4-d4bgv 172.17.0.1 172.17.0.9 27017 \n", "10 mongodb-67c5456f4-ddhnf 172.17.0.1 172.17.0.16 27017 \n", "11 mysql-6d778f4c8f-4bcr7 172.17.0.1 172.17.0.10 3306 \n", "15 rabbitmq-785b678f74-mhhtg 127.0.0.1 127.0.0.1 4369 \n", "16 rabbitmq-785b678f74-mhhtg 172.17.0.1 172.17.0.5 5672 \n", "17 rabbitmq-785b678f74-mhhtg 172.17.0.5 172.17.0.5 4369 \n", "18 ratings-7ccf67b49f-6qckr 172.17.0.1 172.17.0.12 80 \n", "19 ratings-7ccf67b49f-6qckr 172.17.0.12 10.107.7.181 3306 \n", "20 ratings-7ccf67b49f-6qckr 172.17.0.12 10.96.0.10 53 \n", "21 redis-0 172.17.0.1 172.17.0.15 6379 \n", "23 shipping-7f6dfbf46f-94trr 172.17.0.1 172.17.0.11 8080 \n", "24 shipping-7f6dfbf46f-94trr 172.17.0.11 10.107.7.181 3306 \n", "25 shipping-7f6dfbf46f-94trr 172.17.0.11 10.96.0.10 53 \n", "27 user-899b6c7ff-c7wnj 172.17.0.6 10.109.105.252 27017 \n", "28 user-899b6c7ff-c7wnj 172.17.0.6 10.111.214.104 6379 \n", "29 user-899b6c7ff-c7wnj 172.17.0.6 10.96.0.10 53 \n", "33 user-899b6c7ff-qxd47 172.17.0.7 10.109.105.252 27017 \n", "34 user-899b6c7ff-qxd47 172.17.0.7 10.111.214.104 6379 \n", "35 user-899b6c7ff-qxd47 172.17.0.7 10.96.0.10 53 \n", "\n", " flow.rops flow.rbytes flow.wops flow.wbytes \\\n", "0 359 108558 1069 18992 \n", "1 148 16136 148 6452 \n", "4 473 2469 123 1287 \n", "5 2779 180224 1408 77440 \n", "9 921 18316 344 104606 \n", "10 1026 19986 372 113133 \n", "11 4135 64937 1161 73956 \n", "15 35 470 24 470 \n", "16 245 1287 236 2469 \n", "17 39 375 27 375 \n", "18 680 40068 479 12744694 \n", "19 1068 56871 1335 51264 \n", "20 742 91637 0 0 \n", "21 4 42 3 7914 \n", "23 258 28122 258 35862 \n", "24 173 17085 93 13723 \n", "25 10 1020 10 520 \n", "27 296 89710 883 15768 \n", "28 1 2635 1 14 \n", "29 36 3816 36 1944 \n", "33 65 19471 191 3542 \n", "34 2 2641 1 14 \n", "35 146 15984 146 5838 \n", "\n", " type_source \\\n", "0 POD {'catalogue-998b69bc9-bfnr7'} \n", "1 POD {'catalogue-998b69bc9-bfnr7'} \n", "4 POD {'dispatch-69b65d89b9-4lgl7'} \n", "5 POD {'dispatch-69b65d89b9-4lgl7'} \n", "9 \"docker network gateway\" \n", "10 \"docker network gateway\" \n", "11 \"docker network gateway\" \n", "15 \"localhost\" \n", "16 \"docker network gateway\" \n", "17 POD {'rabbitmq-785b678f74-mhhtg'} \n", "18 \"docker network gateway\" \n", "19 POD {'ratings-7ccf67b49f-6qckr'} \n", "20 POD {'ratings-7ccf67b49f-6qckr'} \n", "21 \"docker network gateway\" \n", "23 \"docker network gateway\" \n", "24 POD {'shipping-7f6dfbf46f-94trr'} \n", "25 POD {'shipping-7f6dfbf46f-94trr'} \n", "27 POD {'user-899b6c7ff-c7wnj'} \n", "28 POD {'user-899b6c7ff-c7wnj'} \n", "29 POD {'user-899b6c7ff-c7wnj'} \n", "33 POD {'user-899b6c7ff-qxd47'} \n", "34 POD {'user-899b6c7ff-qxd47'} \n", "35 POD {'user-899b6c7ff-qxd47'} \n", "\n", " type_destination \n", "0 SERVICE mongodb-27017 \n", "1 \"cluster DNS\" \n", "4 SERVICE rabbitmq-5672 \n", "5 \"cluster DNS\" \n", "9 POD {'mongodb-67c5456f4-d4bgv', 'web-77486f858... \n", "10 POD {'mongodb-67c5456f4-ddhnf'} \n", "11 POD {'mysql-6d778f4c8f-4bcr7'} \n", "15 \"localhost\" \n", "16 POD {'rabbitmq-785b678f74-mhhtg'} \n", "17 POD {'rabbitmq-785b678f74-mhhtg'} \n", "18 POD {'ratings-7ccf67b49f-6qckr'} \n", "19 SERVICE mysql-3306 \n", "20 \"cluster DNS\" \n", "21 POD {'redis-0'} \n", "23 POD {'shipping-7f6dfbf46f-94trr'} \n", "24 SERVICE mysql-3306 \n", "25 \"cluster DNS\" \n", "27 SERVICE mongodb-27017 \n", "28 SERVICE redis-6379 \n", "29 \"cluster DNS\" \n", "33 SERVICE mongodb-27017 \n", "34 SERVICE redis-6379 \n", "35 \"cluster DNS\" " ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_rs_traffic[df_rs_traffic['flow.rbytes']>0] #[['pod.name', 'net.sip', 'net.dip', 'net.dport', 'type_source', 'type_destination']]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this table we can see the summarized network flows inside the robot-shop application — how often, how many bytes have been read and written — in the context of the application, i.e., recognizing which pods and _services_ that are involved!\n", "\n", "Small further remarks:\n", "- If you look in detail, the above approach is not sufficient with respect to the reuse of IP addresses (i.e., this is a shortcoming of our `podips` set). In order to make this exact, one would have to keep track of the time information in addition, e.g., which IP address is used by a pod during what time interval.\n", "- This shows the extension of the data to the cluster level, the `NF` records themselves still bind that as before to the lower-level details like the process involved etc." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Summary" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this notebook, we have taken a first look at the new cluster metadata for Kubernetes/OpenShift clusters that is available with the recent SysFlow 0.5.0 release. \n", "\n", "In an experiment with Instana's robot-shop, we have seen this new cluster metadata at work and did an initial investigation into the collected data for that experiment, especially into the IP-related information newly available through the newly collected metadata.\n", "\n", "We find that with the new data, the lower-level SysFlow data related to containers are put into the context of the cluster structure, namely pods, nodes and namespaces. \n", "The new `KE` records are in that respect complimentary to existing records, as they report data driven by _cluster events_ like the creation of a new pod. Conversily, the standard SysFlow records contain now the cluster metadata directly attached via the `pod.*` attributes.\n", "\n", "With respect to observed IP addresses, the availability of the endpoint IPs/ports connected to services is especially interesting as it can be used, together with the IP information for the pods, to understand network flows internal to a cluster application.\n", "\n", "Further extensions of these cluster-related metadata are planned, stay tuned!" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.12" }, "toc-autonumbering": true }, "nbformat": 4, "nbformat_minor": 4 }