{ "cells": [ { "cell_type": "markdown", "metadata": { "lc_cell_meme": { "current": "a6fba2a2-14ed-11e7-b31a-0242ac12000d", "history": [ { "current": "a6fba2a2-14ed-11e7-b31a-0242ac12000d", "next": "a6fba4f0-14ed-11e7-b31a-0242ac12000d", "previous": null } ], "next": "a6fba6e4-14ed-11e7-b31a-0242ac12000d", "previous": null } }, "source": [ "# Elastic Stackの構築：状態の診断\n", "----\n", "稼働しているElasticsearchに異常がないか調べ、 異常がある場合の対処手順について説明します。" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "lc_cell_meme": { "current": "a6fba6e4-14ed-11e7-b31a-0242ac12000d", "history": [ { "current": "a6fba6e4-14ed-11e7-b31a-0242ac12000d", "next": "a6fba8ce-14ed-11e7-b31a-0242ac12000d", "previous": "a6fba4f0-14ed-11e7-b31a-0242ac12000d" } ], "next": "a6fba8ce-14ed-11e7-b31a-0242ac12000d", "previous": "a6fba2a2-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "## 準備\n", "本章のコマンドを実行するための設定を行います。 " ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fba8ce-14ed-11e7-b31a-0242ac12000d", "next": "a6fbaaea-14ed-11e7-b31a-0242ac12000d", "previous": "a6fba6e4-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "セットアップ済みの環境のうち、コマンドを送る先であるCoordinating(Client) Nodeのホストアドレスを、次のセルに記入して実行し、保存してください。" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbaaea-14ed-11e7-b31a-0242ac12000d", "next": "a6fbacc0-14ed-11e7-b31a-0242ac12000d", "previous": "a6fba8ce-14ed-11e7-b31a-0242ac12000d" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "env: ES_HOST=XXX.XXX.XXX.231:9200\n" ] } ], "source": [ "%env ES_HOST=XXX.XXX.XXX.231:9200" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbacc0-14ed-11e7-b31a-0242ac12000d", "next": "a6fbae8c-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbaaea-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "診断の対象としたいインデックスを指定してください。" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbae8c-14ed-11e7-b31a-0242ac12000d", "next": "a6fbb058-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbacc0-14ed-11e7-b31a-0242ac12000d" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "env: INDEX=meteorological-data-*\n" ] } ], "source": [ "%env INDEX=meteorological-data-*" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbb058-14ed-11e7-b31a-0242ac12000d", "next": "a6fbb224-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbae8c-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "各サーバにAnsibleコマンドを発行するため、 Ansibleを実行可能なユーザアカウントとそのSSHキーを次のセルで設定してください。" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fbb224-14ed-11e7-b31a-0242ac12000d", "next": "a6fbb404-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbb058-14ed-11e7-b31a-0242ac12000d" } }, "outputs": [], "source": [ "#elasticsearchをインストールするサーバーでansibleコマンドを実行するユーザー\n", "USER='ansible'\n", "\n", "#公開鍵認証を行う場合の秘密鍵のパス\n", "#KEYPATH='/notebooks/etc/key/id_rsa'\n", "KEYPATH = '~/.ssh/ansible_id_rsa'" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbb404-14ed-11e7-b31a-0242ac12000d", "next": "a6fbb5da-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbb224-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "※この後のコマンドは、[01_03_Set_Inventory.ipynb](01_03_Set_Inventory.ipynb) で生成したインベントリに対して発行されるように構築されています。" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "lc_cell_meme": { "current": "a6fbb5da-14ed-11e7-b31a-0242ac12000d", "next": "a6fbb7a6-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbb404-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "## 診断\n", "現在の状態を把握するために、各種の状態を診断します。 \n", "本章の一連のコマンドを実行してください。 " ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbb7a6-14ed-11e7-b31a-0242ac12000d", "next": "a6fbb968-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbb5da-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "「■確認内容」で異常があると診断できた場合、次章の[症状と対処](#症状と対処)の内容に従い状態を正常化してください。" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fbb968-14ed-11e7-b31a-0242ac12000d", "next": "a6fbbb2a-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbb7a6-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "### サーバマシンが稼働しているか？\n", "サーバごとにpingコマンドを実行し、サーバが稼働しているか確認します。" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbbb2a-14ed-11e7-b31a-0242ac12000d", "next": "a6fbbcec-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbb968-14ed-11e7-b31a-0242ac12000d" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/usr/local/lib/python2.7/dist-packages/cffi/model.py:525: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'\n", " % self._get_c_name())\n", "\u001b[0;32mXXX.XXX.XXX.231 | SUCCESS => {\n", " \"changed\": false, \n", " \"ping\": \"pong\"\n", "}\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.233 | SUCCESS => {\n", " \"changed\": false, \n", " \"ping\": \"pong\"\n", "}\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS => {\n", " \"changed\": false, \n", " \"ping\": \"pong\"\n", "}\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.234 | SUCCESS => {\n", " \"changed\": false, \n", " \"ping\": \"pong\"\n", "}\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.235 | SUCCESS => {\n", " \"changed\": false, \n", " \"ping\": \"pong\"\n", "}\u001b[0m\n" ] } ], "source": [ "!ansible all -m ping -i ./hosts -u $USER --private-key=$KEYPATH" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbbcec-14ed-11e7-b31a-0242ac12000d", "next": "a6fbbeae-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbbb2a-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■確認内容**" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbbeae-14ed-11e7-b31a-0242ac12000d", "next": "a6fbc07a-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbbcec-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "サーバごとに次のような応答があれば正常です。" ] }, { "cell_type": "raw", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbc07a-14ed-11e7-b31a-0242ac12000d", "next": "a6fbc232-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbbeae-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "<各サーバのIPアドレス> | SUCCESS => {\n", " \"changed\": false, \n", " \"ping\": \"pong\"\n", "}" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fbc232-14ed-11e7-b31a-0242ac12000d", "next": "a6fbc3f4-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbc07a-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "### プロセスが稼働しているか？\n", "サーバごとにpsコマンドを実行し、Elasticsearchプロセスが稼働しているか確認します。" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbc3f4-14ed-11e7-b31a-0242ac12000d", "next": "a6fbc5c0-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbc232-14ed-11e7-b31a-0242ac12000d" }, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/usr/local/lib/python2.7/dist-packages/cffi/model.py:525: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'\n", " % self._get_c_name())\n", "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS | rc=0 >>\n", "elastic+ 9608 1 0 Apr04 ? 00:10:05 /usr/java/default/bin/java -Xms31g -Xmx31g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-5.0.0.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.233 | SUCCESS | rc=0 >>\n", "elastic+ 8826 1 0 Apr04 ? 00:09:51 /usr/java/default/bin/java -Xms31g -Xmx31g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-5.0.0.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.231 | SUCCESS | rc=0 >>\n", "elastic+ 12091 1 0 Apr04 ? 00:12:50 /usr/java/default/bin/java -Xms31g -Xmx31g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-5.0.0.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.235 | SUCCESS | rc=0 >>\n", "elastic+ 30867 1 0 Apr04 ? 00:09:36 /usr/java/default/bin/java -Xms31g -Xmx31g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-5.0.0.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.234 | SUCCESS | rc=0 >>\n", "elastic+ 32010 1 0 Apr04 ? 00:11:24 /usr/java/default/bin/java -Xms31g -Xmx31g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-5.0.0.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch\n", "\u001b[0m\n" ] } ], "source": [ "!ansible all -m shell -a \"ps -aef |grep Elasticsearch |grep -v grep\" -i ./hosts -u $USER --private-key=$KEYPATH" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbc5c0-14ed-11e7-b31a-0242ac12000d", "next": "a6fbc782-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbc3f4-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■確認内容**" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbc782-14ed-11e7-b31a-0242ac12000d", "next": "a6fbc944-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbc5c0-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "Elastisearchが稼働しているはずのサーバごとに、次のコマンドのプロセスが存在していれば正常です。" ] }, { "cell_type": "raw", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbc944-14ed-11e7-b31a-0242ac12000d", "next": "a6fbcb06-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbc782-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "/bin/java ～～ org.elasticsearch.bootstrap.Elasticsearch" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fbcb06-14ed-11e7-b31a-0242ac12000d", "next": "a6fbccd2-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbc944-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "### ディスクの使用量に余裕があるか？\n", "サーバごとにdfコマンドを実行し、ディスクサイズに余裕があるか確認します。" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbccd2-14ed-11e7-b31a-0242ac12000d", "next": "a6fbce94-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbcb06-14ed-11e7-b31a-0242ac12000d" }, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/usr/local/lib/python2.7/dist-packages/cffi/model.py:525: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'\n", " % self._get_c_name())\n", "\u001b[0;32mXXX.XXX.XXX.231 | SUCCESS | rc=0 >>\n", "Filesystem 1K-blocks Used Available Use% Mounted on\n", "/dev/sda2 480482404 2608712 453443492 1% /\n", "devtmpfs 49405296 0 49405296 0% /dev\n", "tmpfs 49413876 0 49413876 0% /dev/shm\n", "tmpfs 49413876 74352 49339524 1% /run\n", "tmpfs 49413876 0 49413876 0% /sys/fs/cgroup\n", "/dev/sda5 1928003752 68956 1829974880 1% /mnt\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.233 | SUCCESS | rc=0 >>\n", "Filesystem 1K-blocks Used Available Use% Mounted on\n", "/dev/sda2 480482404 2608420 453443784 1% /\n", "devtmpfs 49405296 0 49405296 0% /dev\n", "tmpfs 49413876 0 49413876 0% /dev/shm\n", "tmpfs 49413876 74348 49339528 1% /run\n", "tmpfs 49413876 0 49413876 0% /sys/fs/cgroup\n", "/dev/sda5 1928003752 68956 1829974880 1% /mnt\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS | rc=0 >>\n", "Filesystem 1K-blocks Used Available Use% Mounted on\n", "/dev/sda2 480482404 2608476 453443728 1% /\n", "devtmpfs 49405296 0 49405296 0% /dev\n", "tmpfs 49413876 0 49413876 0% /dev/shm\n", "tmpfs 49413876 74348 49339528 1% /run\n", "tmpfs 49413876 0 49413876 0% /sys/fs/cgroup\n", "/dev/sda5 1928003752 68956 1829974880 1% /mnt\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.235 | SUCCESS | rc=0 >>\n", "Filesystem 1K-blocks Used Available Use% Mounted on\n", "/dev/sda2 480482404 2608644 453443560 1% /\n", "devtmpfs 49405296 0 49405296 0% /dev\n", "tmpfs 49413876 0 49413876 0% /dev/shm\n", "tmpfs 49413876 66148 49347728 1% /run\n", "tmpfs 49413876 0 49413876 0% /sys/fs/cgroup\n", "/dev/sda5 1928003752 68956 1829974880 1% /mnt\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.234 | SUCCESS | rc=0 >>\n", "Filesystem 1K-blocks Used Available Use% Mounted on\n", "/dev/sda2 480482404 3046048 453006156 1% /\n", "devtmpfs 49405296 0 49405296 0% /dev\n", "tmpfs 49413876 0 49413876 0% /dev/shm\n", "tmpfs 49413876 66148 49347728 1% /run\n", "tmpfs 49413876 0 49413876 0% /sys/fs/cgroup\n", "/dev/sda5 1928003752 68956 1829974880 1% /mnt\n", "\u001b[0m\n" ] } ], "source": [ "!ansible all -m shell -a \"df\" -i ./hosts -u $USER --private-key=$KEYPATH" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbce94-14ed-11e7-b31a-0242ac12000d", "next": "a6fbd06a-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbccd2-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■確認内容**" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbd06a-14ed-11e7-b31a-0242ac12000d", "next": "a6fbd240-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbce94-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "Elasticsearchのデフォルトの設定では、ディスク使用率が85%を越えると、そのノードに新たなシャードを割り当てません。そのため、**\"Use%\"**の列に出力されるディスク使用率が85%を越えないようにしてください。 \n", "ディスク使用率に余裕があれば問題ありません。" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fbd240-14ed-11e7-b31a-0242ac12000d", "next": "a6fbd40c-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbd06a-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "### クラスタと通信できるか？\n", "APIを発行し、Elasticsearchのクラウドと通信可能か確認します。" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbd40c-14ed-11e7-b31a-0242ac12000d", "next": "a6fbd5ce-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbd240-14ed-11e7-b31a-0242ac12000d" }, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\r\n", " \"name\" : \"LXj8U7-\",\r\n", " \"cluster_name\" : \"es-cluster\",\r\n", " \"cluster_uuid\" : \"4nC7EdwsRLeMCfKFaIiltA\",\r\n", " \"version\" : {\r\n", " \"number\" : \"5.0.0\",\r\n", " \"build_hash\" : \"253032b\",\r\n", " \"build_date\" : \"2016-10-26T04:37:51.531Z\",\r\n", " \"build_snapshot\" : false,\r\n", " \"lucene_version\" : \"6.2.0\"\r\n", " },\r\n", " \"tagline\" : \"You Know, for Search\"\r\n", "}\r\n" ] } ], "source": [ "!curl -XGET \"http://$ES_HOST/\"" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbd5ce-14ed-11e7-b31a-0242ac12000d", "next": "a6fbd790-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbd40c-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■確認内容**" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbd790-14ed-11e7-b31a-0242ac12000d", "next": "a6fbd95c-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbd5ce-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "JSON形式の応答が表示されれば問題ありません。 \n", "表示されない場合は、クラスタの状態か、クラスタまでの通信経路に問題があります。" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fbd95c-14ed-11e7-b31a-0242ac12000d", "next": "a6fbdb1e-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbd790-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "### クラスタに全Nodeが参加しているか？\n", "クラスタのヘルスチェック用のAPIを発行し、参加しているノードの状態等を確認します。" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbdb1e-14ed-11e7-b31a-0242ac12000d", "next": "a6fbdce0-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbd95c-14ed-11e7-b31a-0242ac12000d" }, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\r\n", " \"cluster_name\" : \"es-cluster\",\r\n", " \"status\" : \"green\",\r\n", " \"timed_out\" : false,\r\n", " \"number_of_nodes\" : 5,\r\n", " \"number_of_data_nodes\" : 5,\r\n", " \"active_primary_shards\" : 0,\r\n", " \"active_shards\" : 0,\r\n", " \"relocating_shards\" : 0,\r\n", " \"initializing_shards\" : 0,\r\n", " \"unassigned_shards\" : 0,\r\n", " \"delayed_unassigned_shards\" : 0,\r\n", " \"number_of_pending_tasks\" : 0,\r\n", " \"number_of_in_flight_fetch\" : 0,\r\n", " \"task_max_waiting_in_queue_millis\" : 0,\r\n", " \"active_shards_percent_as_number\" : 100.0\r\n", "}\r\n" ] } ], "source": [ "!curl -XGET \"http://$ES_HOST/_cluster/health?pretty\"" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbdce0-14ed-11e7-b31a-0242ac12000d", "next": "a6fbdea2-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbdb1e-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■確認内容**" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbdea2-14ed-11e7-b31a-0242ac12000d", "next": "a6fbe078-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbdce0-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "出力結果から次の内容が確認できれば正常です。\n", "\n", "1. \"status\" が \"green\" かどうか？\n", " - **green** \n", " ：正常です。\n", " - **yellow** \n", " ：Primary Shardは割り当てられていますが、Replica Shardが割り当てられていません。\n", " - **red** \n", " ：特定のShardがクラスタに割り当てられていません\n", "2. \"number_of_nodes\" の数が、起動しているはずの全Node数と一致しているか？\n", "3. \"number_of_data_nodes\"の数が、起動しているはずのData Node数と一致しているか？" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fbe078-14ed-11e7-b31a-0242ac12000d", "next": "a6fbe276-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbdea2-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "### エラーログが無いか？\n", "各ElasticsearchプロセスがERRORメッセージをログに出力していないか確認します。" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbe276-14ed-11e7-b31a-0242ac12000d", "next": "a6fbe442-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbe078-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "ログはデフォルトでは次の場所に出力されています。" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbe442-14ed-11e7-b31a-0242ac12000d", "next": "a6fbe60e-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbe276-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "/var/log/elasticsearch/**{クラスタ名}**.log \n" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbe60e-14ed-11e7-b31a-0242ac12000d", "next": "a6fbe7da-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbe442-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "その他のログも含め、ログディレクトリ配下のファイルについてERRORが出力がされていないかgrepします。" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbe7da-14ed-11e7-b31a-0242ac12000d", "next": "a6fbe99c-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbe60e-14ed-11e7-b31a-0242ac12000d" }, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/usr/local/lib/python2.7/dist-packages/cffi/model.py:525: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'\n", " % self._get_c_name())\n", "\u001b[0;32mXXX.XXX.XXX.233 | SUCCESS | rc=0 >>\n", "/var/log/elasticsearch/es-cluster-2017-04-03.log:[2017-04-03T18:17:12,497][ERROR][o.e.b.Bootstrap ] Exception\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.231 | SUCCESS | rc=0 >>\n", "/var/log/elasticsearch/es-cluster-2017-04-03.log:[2017-04-03T18:17:12,476][ERROR][o.e.b.Bootstrap ] Exception\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS | rc=0 >>\n", "/var/log/elasticsearch/es-cluster-2017-04-03.log:[2017-04-03T18:17:12,690][ERROR][o.e.b.Bootstrap ] Exception\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.234 | SUCCESS | rc=0 >>\n", "/var/log/elasticsearch/es-cluster-2017-04-03.log:[2017-04-03T18:17:13,381][ERROR][o.e.b.Bootstrap ] Exception\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.235 | SUCCESS | rc=0 >>\n", "/var/log/elasticsearch/es-cluster-2017-04-03.log:[2017-04-03T18:17:13,712][ERROR][o.e.b.Bootstrap ] Exception\n", "\u001b[0m\n" ] } ], "source": [ "!ansible all -m shell -a 'grep ERROR /var/log/elasticsearch/*.log' -i ./hosts -u $USER --private-key=$KEYPATH" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbe99c-14ed-11e7-b31a-0242ac12000d", "next": "a6fbeb54-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbe7da-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■確認内容**" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbeb54-14ed-11e7-b31a-0242ac12000d", "next": "a6fbed20-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbe99c-14ed-11e7-b31a-0242ac12000d" } }, "source": [ " XXX.XXX.XXX.111 | FAILED | rc=1 >>\n", "\n", "のように出力される場合は、grepした結果でERRORが0行ということなので、問題がありません。 \n", "1行以上、何かの出力がある場合は問題があります。" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fbed20-14ed-11e7-b31a-0242ac12000d", "next": "a6fbeef6-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbeb54-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "### Slowログが無いか？ \n", "各ElasticsearchプロセスがSlowログを出力していないか確認します。" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbeef6-14ed-11e7-b31a-0242ac12000d", "next": "a6fbf0b8-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbed20-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "Slowログはデフォルトでは次の場所に出力されています。" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbf0b8-14ed-11e7-b31a-0242ac12000d", "next": "a6fbf27a-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbeef6-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "/var/log/elasticsearch/**{クラスタ名}**_index_indexing_slowlog.log \n", "/var/log/elasticsearch/**{クラスタ名}**_index_search_slowlog.log " ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbf27a-14ed-11e7-b31a-0242ac12000d", "next": "a6fbf43c-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbf0b8-14ed-11e7-b31a-0242ac12000d" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/usr/local/lib/python2.7/dist-packages/cffi/model.py:525: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'\n", " % self._get_c_name())\n", "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS | rc=0 >>\n", "\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.233 | SUCCESS | rc=0 >>\n", "\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.231 | SUCCESS | rc=0 >>\n", "\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.234 | SUCCESS | rc=0 >>\n", "\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.235 | SUCCESS | rc=0 >>\n", "\n", "\u001b[0m\n" ] } ], "source": [ "!ansible all -m shell -a 'cat /var/log/elasticsearch/*slowlog.log' -i ./hosts -u $USER --private-key=$KEYPATH" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbf43c-14ed-11e7-b31a-0242ac12000d", "next": "a6fbf644-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbf27a-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■確認内容**" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbf644-14ed-11e7-b31a-0242ac12000d", "next": "a6fbf81a-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbf43c-14ed-11e7-b31a-0242ac12000d" } }, "source": [ " XXX.XXX.XXX.111 | SUCCESS | rc=0 >>\n", "\n", "のように出力される場合は、Slowログが0行ということなので、問題がありません。 \n", "1行以上、何かの出力がある場合は問題があります。" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fbf81a-14ed-11e7-b31a-0242ac12000d", "next": "a6fbf9dc-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbf644-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "### クラスタ内のサーバのCPU・メモリが枯渇してないか？\n", "CPU使用率、メモリ使用率が利用できる限界に近づいていないか確認します。" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbf9dc-14ed-11e7-b31a-0242ac12000d", "next": "a6fbfb9e-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbf81a-14ed-11e7-b31a-0242ac12000d" }, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/usr/local/lib/python2.7/dist-packages/cffi/model.py:525: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'\n", " % self._get_c_name())\n", "\u001b[0;32mXXX.XXX.XXX.231 | SUCCESS | rc=0 >>\n", "top - 10:51:43 up 2 days, 17:22, 1 user, load average: 0.00, 0.01, 0.05\n", "Tasks: 296 total, 1 running, 293 sleeping, 0 stopped, 2 zombie\n", "%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st\n", "KiB Mem : 98827752 total, 62325280 free, 35258496 used, 1243976 buff/cache\n", "KiB Swap: 93750000 total, 93750000 free, 0 used. 63183308 avail Mem \n", "\n", " PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND\n", "16115 ansible 20 0 130152 1784 1196 R 6.2 0.0 0:00.01 top\n", " 1 root 20 0 54304 4136 2480 S 0.0 0.0 0:03.72 systemd\n", " 2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd\n", " 3 root 20 0 0 0 0 S 0.0 0.0 0:00.20 ksoftirqd/0\n", " 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:+\n", " 8 root rt 0 0 0 0 S 0.0 0.0 0:00.31 migration/0\n", " 9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh\n", " 10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/0\n", " 11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/1\n", " 12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/2\n", " 13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/3\n", " 14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/4\n", " 15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/5\n", " 16 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/6\n", " 17 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/7\n", " 18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/8\n", " 19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/9\n", " 20 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/10\n", " 21 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/11\n", " 22 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/12\n", " 23 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/13\n", " 24 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/14\n", " 25 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/15\n", " 26 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/16\n", " 27 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/17\n", " 28 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/18\n", " 29 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/19\n", " 30 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/20\n", " 31 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/21\n", " 32 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/22\n", " 33 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/23\n", " 34 root 20 0 0 0 0 S 0.0 0.0 5:57.94 rcu_sched\n", " 35 root 20 0 0 0 0 S 0.0 0.0 0:04.33 rcuos/0\n", " 36 root 20 0 0 0 0 S 0.0 0.0 0:24.10 rcuos/1\n", " 37 root 20 0 0 0 0 S 0.0 0.0 0:11.13 rcuos/2\n", " 38 root 20 0 0 0 0 S 0.0 0.0 0:07.94 rcuos/3\n", " 39 root 20 0 0 0 0 S 0.0 0.0 0:05.56 rcuos/4\n", " 40 root 20 0 0 0 0 S 0.0 0.0 0:04.49 rcuos/5\n", " 41 root 20 0 0 0 0 S 0.0 0.0 0:05.39 rcuos/6\n", " 42 root 20 0 0 0 0 S 0.0 0.0 0:04.52 rcuos/7\n", " 43 root 20 0 0 0 0 S 0.0 0.0 0:08.40 rcuos/8\n", " 44 root 20 0 0 0 0 S 0.0 0.0 0:06.96 rcuos/9\n", " 45 root 20 0 0 0 0 S 0.0 0.0 1:28.10 rcuos/10\n", " 46 root 20 0 0 0 0 S 0.0 0.0 0:05.13 rcuos/11\n", " 47 root 20 0 0 0 0 S 0.0 0.0 0:01.53 rcuos/12\n", " 48 root 20 0 0 0 0 S 0.0 0.0 0:11.72 rcuos/13\n", " 49 root 20 0 0 0 0 S 0.0 0.0 0:02.49 rcuos/14\n", " 50 root 20 0 0 0 0 S 0.0 0.0 0:02.17 rcuos/15\n", " 51 root 20 0 0 0 0 S 0.0 0.0 0:01.81 rcuos/16\n", " 52 root 20 0 0 0 0 S 0.0 0.0 0:02.54 rcuos/17\n", " 53 root 20 0 0 0 0 S 0.0 0.0 0:02.11 rcuos/18\n", " 54 root 20 0 0 0 0 S 0.0 0.0 0:12.20 rcuos/19\n", " 55 root 20 0 0 0 0 S 0.0 0.0 0:03.22 rcuos/20\n", " 56 root 20 0 0 0 0 S 0.0 0.0 0:02.40 rcuos/21\n", " 57 root 20 0 0 0 0 S 0.0 0.0 0:09.34 rcuos/22\n", " 58 root 20 0 0 0 0 S 0.0 0.0 0:01.94 rcuos/23\n", " 59 root rt 0 0 0 0 S 0.0 0.0 0:00.55 watchdog/0\n", " 60 root rt 0 0 0 0 S 0.0 0.0 0:00.54 watchdog/1\n", " 61 root rt 0 0 0 0 S 0.0 0.0 0:00.21 migration/1\n", " 62 root 20 0 0 0 0 S 0.0 0.0 0:00.03 ksoftirqd/1\n", " 64 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:+\n", " 65 root rt 0 0 0 0 S 0.0 0.0 0:00.54 watchdog/2\n", " 66 root rt 0 0 0 0 S 0.0 0.0 0:00.32 migration/2\n", " 67 root 20 0 0 0 0 S 0.0 0.0 0:00.22 ksoftirqd/2\n", " 69 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/2:+\n", " 70 root rt 0 0 0 0 S 0.0 0.0 0:00.51 watchdog/3\n", " 71 root rt 0 0 0 0 S 0.0 0.0 0:00.17 migration/3\n", " 72 root 20 0 0 0 0 S 0.0 0.0 0:00.04 ksoftirqd/3\n", " 74 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:+\n", " 75 root rt 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/4\n", " 76 root rt 0 0 0 0 S 0.0 0.0 0:00.22 migration/4\n", " 77 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/4\n", " 79 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/4:+\n", " 80 root rt 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/5\n", " 81 root rt 0 0 0 0 S 0.0 0.0 0:00.19 migration/5\n", " 82 root 20 0 0 0 0 S 0.0 0.0 0:00.04 ksoftirqd/5\n", " 84 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/5:+\n", " 85 root rt 0 0 0 0 S 0.0 0.0 0:00.52 watchdog/6\n", " 86 root rt 0 0 0 0 S 0.0 0.0 0:00.32 migration/6\n", " 87 root 20 0 0 0 0 S 0.0 0.0 0:00.04 ksoftirqd/6\n", " 89 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/6:+\n", " 91 root rt 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/7\n", " 92 root rt 0 0 0 0 S 0.0 0.0 0:00.31 migration/7\n", " 93 root 20 0 0 0 0 S 0.0 0.0 0:00.49 ksoftirqd/7\n", " 95 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/7:+\n", " 96 root rt 0 0 0 0 S 0.0 0.0 0:00.51 watchdog/8\n", " 97 root rt 0 0 0 0 S 0.0 0.0 0:00.23 migration/8\n", " 98 root 20 0 0 0 0 S 0.0 0.0 0:00.04 ksoftirqd/8\n", " 100 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/8:+\n", " 101 root rt 0 0 0 0 S 0.0 0.0 0:00.52 watchdog/9\n", " 102 root rt 0 0 0 0 S 0.0 0.0 0:00.25 migration/9\n", " 103 root 20 0 0 0 0 S 0.0 0.0 0:00.50 ksoftirqd/9\n", " 105 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/9:+\n", " 106 root rt 0 0 0 0 S 0.0 0.0 0:00.49 watchdog/10\n", " 107 root rt 0 0 0 0 S 0.0 0.0 0:00.21 migration/+\n", " 108 root 20 0 0 0 0 S 0.0 0.0 0:00.16 ksoftirqd/+\n", " 110 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/10+\n", " 111 root rt 0 0 0 0 S 0.0 0.0 0:00.49 watchdog/11\n", " 112 root rt 0 0 0 0 S 0.0 0.0 0:00.24 migration/+\n", " 113 root 20 0 0 0 0 S 0.0 0.0 0:00.05 ksoftirqd/+\n", " 115 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/11+\n", " 116 root rt 0 0 0 0 S 0.0 0.0 0:00.49 watchdog/12\n", " 117 root rt 0 0 0 0 S 0.0 0.0 0:00.20 migration/+\n", " 118 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/+\n", " 120 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/12+\n", " 121 root rt 0 0 0 0 S 0.0 0.0 0:00.52 watchdog/13\n", " 122 root rt 0 0 0 0 S 0.0 0.0 0:00.87 migration/+\n", " 123 root 20 0 0 0 0 S 0.0 0.0 0:00.36 ksoftirqd/+\n", " 125 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/13+\n", " 126 root rt 0 0 0 0 S 0.0 0.0 0:00.47 watchdog/14\n", " 127 root rt 0 0 0 0 S 0.0 0.0 0:00.43 migration/+\n", " 128 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 130 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/14+\n", " 131 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/15\n", " 132 root rt 0 0 0 0 S 0.0 0.0 0:00.21 migration/+\n", " 133 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 135 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/15+\n", " 136 root rt 0 0 0 0 S 0.0 0.0 0:00.47 watchdog/16\n", " 137 root rt 0 0 0 0 S 0.0 0.0 0:00.26 migration/+\n", " 138 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 140 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/16+\n", " 141 root rt 0 0 0 0 S 0.0 0.0 0:00.47 watchdog/17\n", " 142 root rt 0 0 0 0 S 0.0 0.0 0:00.16 migration/+\n", " 143 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 145 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/17+\n", " 146 root rt 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/18\n", " 147 root rt 0 0 0 0 S 0.0 0.0 0:00.28 migration/+\n", " 148 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 149 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/18+\n", " 150 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/18+\n", " 151 root rt 0 0 0 0 S 0.0 0.0 0:00.49 watchdog/19\n", " 152 root rt 0 0 0 0 S 0.0 0.0 0:00.75 migration/+\n", " 153 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/+\n", " 155 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/19+\n", " 156 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/20\n", " 157 root rt 0 0 0 0 S 0.0 0.0 0:00.36 migration/+\n", " 158 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 160 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/20+\n", " 161 root rt 0 0 0 0 S 0.0 0.0 0:00.47 watchdog/21\n", " 162 root rt 0 0 0 0 S 0.0 0.0 0:00.20 migration/+\n", " 163 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 165 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/21+\n", " 166 root rt 0 0 0 0 S 0.0 0.0 0:00.47 watchdog/22\n", " 167 root rt 0 0 0 0 S 0.0 0.0 0:00.25 migration/+\n", " 168 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 170 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/22+\n", " 171 root rt 0 0 0 0 S 0.0 0.0 0:00.47 watchdog/23\n", " 172 root rt 0 0 0 0 S 0.0 0.0 0:00.62 migration/+\n", " 173 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 174 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/23+\n", " 175 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/23+\n", " 176 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper\n", " 177 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs\n", " 178 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns\n", " 179 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback\n", " 180 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd\n", " 181 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset\n", " 182 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd\n", " 184 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khubd\n", " 185 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 md\n", " 187 root 20 0 0 0 0 S 0.0 0.0 0:00.22 khungtaskd\n", " 188 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kswapd0\n", " 189 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kswapd1\n", " 190 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd\n", " 191 root 39 19 0 0 0 S 0.0 0.0 0:01.90 khugepaged\n", " 192 root 20 0 0 0 0 S 0.0 0.0 0:00.00 fsnotify_m+\n", " 193 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto\n", " 201 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kthrotld\n", " 205 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kmpath_rda+\n", " 217 root 20 0 0 0 0 S 0.0 0.0 0:02.82 kworker/12+\n", " 220 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kworker/22+\n", " 223 root 20 0 0 0 0 S 0.0 0.0 0:00.61 kworker/18+\n", " 224 root 20 0 0 0 0 S 0.0 0.0 0:00.05 kworker/20+\n", " 227 root 20 0 0 0 0 S 0.0 0.0 0:00.60 kworker/21+\n", " 230 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kpsmoused\n", " 249 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 deferwq\n", " 279 root 20 0 0 0 0 S 0.0 0.0 0:00.32 kauditd\n", " 412 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0\n", " 413 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_0\n", " 416 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ata_sff\n", " 427 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_1\n", " 428 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_1\n", " 429 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_2\n", " 430 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_2\n", " 431 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_3\n", " 432 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_3\n", " 433 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_4\n", " 434 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_4\n", " 435 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_5\n", " 436 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_5\n", " 437 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_6\n", " 438 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_6\n", " 441 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u5+\n", " 442 root 20 0 0 0 0 S 0.0 0.0 0:02.00 kworker/u5+\n", " 449 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u4+\n", " 510 root 20 0 0 0 0 S 0.0 0.0 0:02.61 jbd2/sda2-8\n", " 511 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-rsv-c+\n", " 512 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-unrsv+\n", " 581 root 20 0 177560 94620 94284 S 0.0 0.1 0:37.21 systemd-jo+\n", " 600 root 20 0 42932 2360 1224 S 0.0 0.0 0:00.15 systemd-ud+\n", " 690 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 edac-poller\n", " 735 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kvm-irqfd-+\n", " 742 root 20 0 0 0 0 S 0.0 0.0 0:00.00 jbd2/sda5-8\n", " 743 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-rsv-c+\n", " 744 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-unrsv+\n", " 758 root 16 -4 116684 1636 1240 S 0.0 0.0 0:01.23 auditd\n", " 781 root 20 0 339468 24192 6880 S 0.0 0.0 0:01.50 firewalld\n", " 784 root 20 0 550144 18188 5628 S 0.0 0.0 0:13.20 tuned\n", " 791 root 20 0 19284 1284 948 S 0.0 0.0 0:29.36 irqbalance\n", " 793 avahi 20 0 28072 1684 1392 S 0.0 0.0 0:00.30 avahi-daem+\n", " 796 root 20 0 328256 52280 49640 S 0.0 0.1 0:20.25 rsyslogd\n", " 797 root 20 0 34684 1720 1408 S 0.0 0.0 0:01.56 systemd-lo+\n", " 798 dbus 20 0 26708 1936 1344 S 0.0 0.0 0:02.85 dbus-daemon\n", " 800 root 20 0 25932 944 744 S 0.0 0.0 0:00.00 atd\n", " 802 root 20 0 126304 1600 984 S 0.0 0.0 0:00.70 crond\n", " 810 root 20 0 110004 836 712 S 0.0 0.0 0:00.00 agetty\n", " 812 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:+\n", " 813 avahi 20 0 27944 444 216 S 0.0 0.0 0:00.00 avahi-daem+\n", " 814 ntp 20 0 31432 2072 1468 S 0.0 0.0 0:00.78 ntpd\n", " 908 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/12+\n", " 982 root 20 0 454020 8944 6812 S 0.0 0.0 0:04.68 NetworkMan+\n", " 1040 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:+\n", " 1132 polkitd 20 0 514436 10564 4740 S 0.0 0.0 0:01.98 polkitd\n", " 1202 root 0 -20 0 0 0 S 0.0 0.0 0:00.08 kworker/2:+\n", " 1266 root 20 0 0 0 0 S 0.0 0.0 0:01.81 kworker/u4+\n", " 1680 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:+\n", " 1681 root 20 0 223228 11088 6412 S 0.0 0.0 2:33.06 snmpd\n", " 1684 root 20 0 358620 25916 4704 S 0.0 0.0 0:16.92 dodai-inst+\n", " 1781 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/21+\n", " 1953 root 20 0 0 0 0 S 0.0 0.0 0:15.55 kworker/13+\n", " 2096 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/5:+\n", " 2138 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/9:0\n", " 2385 root 20 0 159408 9308 2144 S 0.0 0.0 0:08.25 munin-node\n", " 2920 root 20 0 91084 2172 1144 S 0.0 0.0 0:00.87 master\n", " 3008 postfix 20 0 91256 3900 2908 S 0.0 0.0 0:00.16 qmgr\n", " 3102 root 20 0 0 0 0 S 0.0 0.0 0:00.12 kworker/23+\n", " 3453 root 20 0 0 0 0 S 0.0 0.0 0:00.04 kworker/20+\n", " 3612 root 20 0 0 0 0 S 0.0 0.0 0:12.22 kworker/0:0\n", " 4617 root 20 0 0 0 0 S 0.0 0.0 0:07.18 kworker/4:2\n", " 4777 root 20 0 0 0 0 S 0.0 0.0 0:00.04 kworker/16+\n", " 4929 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/1:2\n", " 4930 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/5:0\n", " 5909 root 20 0 0 0 0 S 0.0 0.0 0:05.00 kworker/6:0\n", " 6210 root 20 0 0 0 0 S 0.0 0.0 1:00.66 kworker/u4+\n", " 6906 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0\n", " 8222 root 20 0 0 0 0 S 0.0 0.0 0:00.14 kworker/5:1\n", " 8688 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/4:+\n", " 8689 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/8:+\n", " 8690 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/9:+\n", " 8691 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/10+\n", " 9760 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/14+\n", " 9850 root 20 0 0 0 0 S 0.0 0.0 0:00.22 kworker/17+\n", "11346 root 20 0 0 0 0 S 0.0 0.0 0:00.21 kworker/14+\n", "11846 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/21+\n", "11875 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/11+\n", "12051 root 20 0 0 0 0 S 0.0 0.0 0:02.61 kworker/1:0\n", "12091 elastic+ 20 0 44.131g 0.032t 15828 S 0.0 34.7 12:50.73 java\n", "12643 root 20 0 0 0 0 Z 0.0 0.0 0:00.00 sh\n", "12653 root 20 0 0 0 0 Z 0.0 0.0 0:00.00 sh\n", "12870 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/4:0\n", "13062 postfix 20 0 91188 3884 2888 S 0.0 0.0 0:00.00 pickup\n", "13238 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bond0\n", "13332 root 20 0 0 0 0 S 0.0 0.0 0:06.63 kworker/2:1\n", "13707 root 20 0 0 0 0 S 0.0 0.0 0:00.39 kworker/17+\n", "13869 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/20+\n", "14034 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/7:0\n", "14274 root 20 0 0 0 0 S 0.0 0.0 0:01.69 kworker/19+\n", "14634 root 20 0 0 0 0 S 0.0 0.0 0:02.98 kworker/22+\n", "15160 root 20 0 0 0 0 S 0.0 0.0 0:00.13 kworker/15+\n", "15520 root 20 0 0 0 0 S 0.0 0.0 0:00.26 kworker/16+\n", "15618 root 20 0 135216 4848 3680 S 0.0 0.0 0:00.03 sshd\n", "15620 ansible 20 0 135348 2612 1356 S 0.0 0.0 0:00.03 sshd\n", "16099 ansible 20 0 113116 1196 1020 S 0.0 0.0 0:00.00 sh\n", "16113 ansible 20 0 191232 7992 3584 S 0.0 0.0 0:00.04 python\n", "16114 ansible 20 0 208384 12716 3848 S 0.0 0.0 0:00.07 python\n", "16829 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/15+\n", "18171 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/3:0\n", "19646 root 20 0 0 0 0 S 0.0 0.0 0:02.19 kworker/8:0\n", "19811 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/10+\n", "20196 root 20 0 82508 3644 2788 S 0.0 0.0 0:00.01 sshd\n", "20609 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/19+\n", "20610 root 20 0 0 0 0 S 0.0 0.0 0:01.14 kworker/6:2\n", "21506 root 20 0 0 0 0 S 0.0 0.0 0:07.91 kworker/u4+\n", "22125 root 20 0 0 0 0 S 0.0 0.0 0:01.28 kworker/9:1\n", "22629 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/0:2\n", "23443 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/23+\n", "25416 root 20 0 0 0 0 S 0.0 0.0 0:04.88 kworker/10+\n", "25455 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/11+\n", "25503 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/6:+\n", "25523 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/7:+\n", "27375 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/13+\n", "28210 root 20 0 0 0 0 S 0.0 0.0 0:06.71 kworker/11+\n", "28255 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/8:1\n", "28583 root 20 0 0 0 0 S 0.0 0.0 0:04.10 kworker/3:1\n", "30560 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/14+\n", "31217 root 20 0 0 0 0 S 0.0 0.0 0:08.36 kworker/7:2\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS | rc=0 >>\n", "top - 10:51:43 up 2 days, 17:22, 1 user, load average: 0.00, 0.01, 0.05\n", "Tasks: 297 total, 1 running, 294 sleeping, 0 stopped, 2 zombie\n", "%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st\n", "KiB Mem : 98827752 total, 62652100 free, 34940596 used, 1235056 buff/cache\n", "KiB Swap: 93750000 total, 93750000 free, 0 used. 63501916 avail Mem \n", "\n", " PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND\n", "13415 ansible 20 0 130152 1776 1196 R 6.2 0.0 0:00.01 top\n", " 1 root 20 0 54336 4124 2480 S 0.0 0.0 0:03.72 systemd\n", " 2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd\n", " 3 root 20 0 0 0 0 S 0.0 0.0 0:00.19 ksoftirqd/0\n", " 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:+\n", " 8 root rt 0 0 0 0 S 0.0 0.0 0:00.24 migration/0\n", " 9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh\n", " 10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/0\n", " 11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/1\n", " 12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/2\n", " 13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/3\n", " 14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/4\n", " 15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/5\n", " 16 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/6\n", " 17 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/7\n", " 18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/8\n", " 19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/9\n", " 20 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/10\n", " 21 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/11\n", " 22 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/12\n", " 23 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/13\n", " 24 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/14\n", " 25 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/15\n", " 26 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/16\n", " 27 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/17\n", " 28 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/18\n", " 29 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/19\n", " 30 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/20\n", " 31 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/21\n", " 32 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/22\n", " 33 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/23\n", " 34 root 20 0 0 0 0 S 0.0 0.0 5:34.14 rcu_sched\n", " 35 root 20 0 0 0 0 S 0.0 0.0 0:05.56 rcuos/0\n", " 36 root 20 0 0 0 0 S 0.0 0.0 0:04.91 rcuos/1\n", " 37 root 20 0 0 0 0 S 0.0 0.0 0:06.63 rcuos/2\n", " 38 root 20 0 0 0 0 S 0.0 0.0 0:06.21 rcuos/3\n", " 39 root 20 0 0 0 0 S 0.0 0.0 0:04.79 rcuos/4\n", " 40 root 20 0 0 0 0 S 0.0 0.0 0:05.27 rcuos/5\n", " 41 root 20 0 0 0 0 S 0.0 0.0 0:05.95 rcuos/6\n", " 42 root 20 0 0 0 0 S 0.0 0.0 0:04.48 rcuos/7\n", " 43 root 20 0 0 0 0 S 0.0 0.0 0:09.22 rcuos/8\n", " 44 root 20 0 0 0 0 S 0.0 0.0 0:06.00 rcuos/9\n", " 45 root 20 0 0 0 0 S 0.0 0.0 0:32.02 rcuos/10\n", " 46 root 20 0 0 0 0 S 0.0 0.0 0:08.65 rcuos/11\n", " 47 root 20 0 0 0 0 S 0.0 0.0 1:11.71 rcuos/12\n", " 48 root 20 0 0 0 0 S 0.0 0.0 0:11.60 rcuos/13\n", " 49 root 20 0 0 0 0 S 0.0 0.0 0:02.62 rcuos/14\n", " 50 root 20 0 0 0 0 S 0.0 0.0 0:01.94 rcuos/15\n", " 51 root 20 0 0 0 0 S 0.0 0.0 0:02.68 rcuos/16\n", " 52 root 20 0 0 0 0 S 0.0 0.0 0:02.28 rcuos/17\n", " 53 root 20 0 0 0 0 S 0.0 0.0 0:01.67 rcuos/18\n", " 54 root 20 0 0 0 0 S 0.0 0.0 0:11.56 rcuos/19\n", " 55 root 20 0 0 0 0 S 0.0 0.0 0:02.69 rcuos/20\n", " 56 root 20 0 0 0 0 S 0.0 0.0 0:02.93 rcuos/21\n", " 57 root 20 0 0 0 0 S 0.0 0.0 0:02.77 rcuos/22\n", " 58 root 20 0 0 0 0 S 0.0 0.0 0:02.27 rcuos/23\n", " 59 root rt 0 0 0 0 S 0.0 0.0 0:00.56 watchdog/0\n", " 60 root rt 0 0 0 0 S 0.0 0.0 0:00.56 watchdog/1\n", " 61 root rt 0 0 0 0 S 0.0 0.0 0:00.28 migration/1\n", " 62 root 20 0 0 0 0 S 0.0 0.0 0:00.46 ksoftirqd/1\n", " 64 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:+\n", " 65 root rt 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/2\n", " 66 root rt 0 0 0 0 S 0.0 0.0 0:00.25 migration/2\n", " 67 root 20 0 0 0 0 S 0.0 0.0 0:00.15 ksoftirqd/2\n", " 69 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/2:+\n", " 70 root rt 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/3\n", " 71 root rt 0 0 0 0 S 0.0 0.0 0:00.42 migration/3\n", " 72 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/3\n", " 74 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:+\n", " 75 root rt 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/4\n", " 76 root rt 0 0 0 0 S 0.0 0.0 0:00.25 migration/4\n", " 77 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/4\n", " 79 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/4:+\n", " 80 root rt 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/5\n", " 81 root rt 0 0 0 0 S 0.0 0.0 0:00.25 migration/5\n", " 82 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/5\n", " 84 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/5:+\n", " 85 root rt 0 0 0 0 S 0.0 0.0 0:00.52 watchdog/6\n", " 86 root rt 0 0 0 0 S 0.0 0.0 0:00.24 migration/6\n", " 87 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/6\n", " 89 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/6:+\n", " 90 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u5+\n", " 91 root rt 0 0 0 0 S 0.0 0.0 0:00.51 watchdog/7\n", " 92 root rt 0 0 0 0 S 0.0 0.0 0:00.18 migration/7\n", " 93 root 20 0 0 0 0 S 0.0 0.0 0:00.49 ksoftirqd/7\n", " 95 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/7:+\n", " 96 root rt 0 0 0 0 S 0.0 0.0 0:00.51 watchdog/8\n", " 97 root rt 0 0 0 0 S 0.0 0.0 0:00.25 migration/8\n", " 98 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/8\n", " 100 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/8:+\n", " 101 root rt 0 0 0 0 S 0.0 0.0 0:00.51 watchdog/9\n", " 102 root rt 0 0 0 0 S 0.0 0.0 0:00.23 migration/9\n", " 103 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/9\n", " 105 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/9:+\n", " 106 root rt 0 0 0 0 S 0.0 0.0 0:00.51 watchdog/10\n", " 107 root rt 0 0 0 0 S 0.0 0.0 0:00.25 migration/+\n", " 108 root 20 0 0 0 0 S 0.0 0.0 0:00.39 ksoftirqd/+\n", " 110 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/10+\n", " 111 root rt 0 0 0 0 S 0.0 0.0 0:00.51 watchdog/11\n", " 112 root rt 0 0 0 0 S 0.0 0.0 0:00.20 migration/+\n", " 113 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/+\n", " 115 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/11+\n", " 116 root rt 0 0 0 0 S 0.0 0.0 0:00.49 watchdog/12\n", " 117 root rt 0 0 0 0 S 0.0 0.0 0:00.22 migration/+\n", " 118 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/+\n", " 120 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/12+\n", " 121 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/13\n", " 122 root rt 0 0 0 0 S 0.0 0.0 0:01.00 migration/+\n", " 123 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 125 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/13+\n", " 126 root rt 0 0 0 0 S 0.0 0.0 0:00.57 watchdog/14\n", " 127 root rt 0 0 0 0 S 0.0 0.0 0:00.36 migration/+\n", " 128 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 130 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/14+\n", " 131 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/15\n", " 132 root rt 0 0 0 0 S 0.0 0.0 0:00.22 migration/+\n", " 133 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 135 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/15+\n", " 136 root rt 0 0 0 0 S 0.0 0.0 0:00.47 watchdog/16\n", " 137 root rt 0 0 0 0 S 0.0 0.0 0:00.18 migration/+\n", " 138 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 140 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/16+\n", " 141 root rt 0 0 0 0 S 0.0 0.0 0:00.47 watchdog/17\n", " 142 root rt 0 0 0 0 S 0.0 0.0 0:00.16 migration/+\n", " 143 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 145 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/17+\n", " 146 root rt 0 0 0 0 S 0.0 0.0 0:00.49 watchdog/18\n", " 147 root rt 0 0 0 0 S 0.0 0.0 0:00.21 migration/+\n", " 148 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 150 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/18+\n", " 151 root rt 0 0 0 0 S 0.0 0.0 0:00.49 watchdog/19\n", " 152 root rt 0 0 0 0 S 0.0 0.0 0:00.50 migration/+\n", " 153 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/+\n", " 154 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/19+\n", " 155 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/19+\n", " 156 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/20\n", " 157 root rt 0 0 0 0 S 0.0 0.0 0:00.37 migration/+\n", " 158 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 160 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/20+\n", " 161 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/21\n", " 162 root rt 0 0 0 0 S 0.0 0.0 0:00.25 migration/+\n", " 163 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 165 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/21+\n", " 166 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/22\n", " 167 root rt 0 0 0 0 S 0.0 0.0 0:00.19 migration/+\n", " 168 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 170 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/22+\n", " 171 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/23\n", " 172 root rt 0 0 0 0 S 0.0 0.0 0:00.62 migration/+\n", " 173 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 175 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/23+\n", " 176 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper\n", " 177 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs\n", " 178 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns\n", " 179 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback\n", " 180 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd\n", " 181 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset\n", " 182 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd\n", " 184 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khubd\n", " 185 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 md\n", " 187 root 20 0 0 0 0 S 0.0 0.0 0:00.21 khungtaskd\n", " 188 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kswapd0\n", " 189 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kswapd1\n", " 190 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd\n", " 191 root 39 19 0 0 0 S 0.0 0.0 0:01.88 khugepaged\n", " 192 root 20 0 0 0 0 S 0.0 0.0 0:00.00 fsnotify_m+\n", " 193 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto\n", " 201 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kthrotld\n", " 205 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kmpath_rda+\n", " 214 root 20 0 0 0 0 S 0.0 0.0 0:01.83 kworker/19+\n", " 215 root 20 0 0 0 0 S 0.0 0.0 0:04.67 kworker/18+\n", " 218 root 20 0 0 0 0 S 0.0 0.0 0:00.88 kworker/17+\n", " 227 root 20 0 0 0 0 S 0.0 0.0 0:00.72 kworker/20+\n", " 229 root 20 0 0 0 0 S 0.0 0.0 0:00.17 kworker/23+\n", " 231 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kpsmoused\n", " 250 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 deferwq\n", " 280 root 20 0 0 0 0 S 0.0 0.0 0:00.34 kauditd\n", " 418 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0\n", " 419 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_0\n", " 420 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ata_sff\n", " 431 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_1\n", " 432 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_1\n", " 433 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_2\n", " 434 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_2\n", " 436 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_3\n", " 437 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_3\n", " 438 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_4\n", " 439 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_4\n", " 441 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_5\n", " 442 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_5\n", " 443 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_6\n", " 444 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_6\n", " 449 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u4+\n", " 450 root 20 0 0 0 0 S 0.0 0.0 0:01.49 kworker/u4+\n", " 511 root 20 0 0 0 0 S 0.0 0.0 0:02.54 jbd2/sda2-8\n", " 512 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-rsv-c+\n", " 513 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-unrsv+\n", " 581 root 20 0 177640 94228 93888 S 0.0 0.1 0:36.02 systemd-jo+\n", " 600 root 20 0 42932 2368 1224 S 0.0 0.0 0:00.15 systemd-ud+\n", " 666 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 edac-poller\n", " 732 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kvm-irqfd-+\n", " 741 root 20 0 0 0 0 S 0.0 0.0 0:00.00 jbd2/sda5-8\n", " 742 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-rsv-c+\n", " 743 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-unrsv+\n", " 757 root 16 -4 116684 1632 1240 S 0.0 0.0 0:01.09 auditd\n", " 781 root 20 0 339472 24196 6896 S 0.0 0.0 0:01.44 firewalld\n", " 785 avahi 20 0 28072 1680 1392 S 0.0 0.0 0:00.31 avahi-daem+\n", " 786 root 20 0 550144 16148 5628 S 0.0 0.0 0:13.27 tuned\n", " 792 avahi 20 0 27944 440 216 S 0.0 0.0 0:00.00 avahi-daem+\n", " 794 root 20 0 19284 1284 948 S 0.0 0.0 0:28.98 irqbalance\n", " 799 root 20 0 394084 52168 49528 S 0.0 0.1 0:18.73 rsyslogd\n", " 800 dbus 20 0 26708 1936 1344 S 0.0 0.0 0:02.62 dbus-daemon\n", " 801 root 20 0 34684 1724 1408 S 0.0 0.0 0:01.54 systemd-lo+\n", " 804 root 20 0 25932 944 744 S 0.0 0.0 0:00.00 atd\n", " 805 root 20 0 126300 1596 980 S 0.0 0.0 0:00.70 crond\n", " 811 root 20 0 110004 836 712 S 0.0 0.0 0:00.00 agetty\n", " 813 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/19+\n", " 817 ntp 20 0 31432 2068 1464 S 0.0 0.0 0:00.76 ntpd\n", " 864 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/9:+\n", " 983 root 20 0 527860 11024 6812 S 0.0 0.0 0:04.61 NetworkMan+\n", " 994 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/10+\n", " 1022 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/11+\n", " 1043 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/6:+\n", " 1068 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/7:+\n", " 1079 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/8:+\n", " 1137 polkitd 20 0 514384 10392 4704 S 0.0 0.0 0:01.58 polkitd\n", " 1215 root 20 0 0 0 0 S 0.0 0.0 0:02.25 kworker/u5+\n", " 1233 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/21+\n", " 1569 root 20 0 0 0 0 S 0.0 0.0 0:00.15 kworker/3:1\n", " 1686 root 20 0 223228 11080 6412 S 0.0 0.0 2:33.09 snmpd\n", " 1688 root 20 0 358616 25916 4704 S 0.0 0.0 0:16.89 dodai-inst+\n", " 1931 root 20 0 0 0 0 S 0.0 0.0 0:00.26 kworker/15+\n", " 1959 root 20 0 0 0 0 S 0.0 0.0 0:01.01 kworker/13+\n", " 2215 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/22+\n", " 2499 root 20 0 159408 9308 2144 S 0.0 0.0 0:07.75 munin-node\n", " 2835 root 20 0 91084 2172 1144 S 0.0 0.0 0:00.81 master\n", " 2893 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/6:2\n", " 2909 postfix 20 0 91256 3896 2904 S 0.0 0.0 0:00.14 qmgr\n", " 2921 root 20 0 0 0 0 S 0.0 0.0 0:00.55 kworker/21+\n", " 3400 root 20 0 0 0 0 S 0.0 0.0 0:02.66 kworker/7:0\n", " 3555 root 20 0 0 0 0 S 0.0 0.0 0:15.34 kworker/10+\n", " 4214 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/9:0\n", " 4370 root 20 0 0 0 0 S 0.0 0.0 0:07.89 kworker/0:1\n", " 4719 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/10+\n", " 5844 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:+\n", " 6191 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/8:2\n", " 6678 root 20 0 0 0 0 S 0.0 0.0 0:00.76 kworker/11+\n", " 7382 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/7:1\n", " 7878 root 20 0 0 0 0 S 0.0 0.0 0:01.80 kworker/5:1\n", " 8194 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0\n", " 8686 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:+\n", " 9512 root 20 0 0 0 0 S 0.0 0.0 0:00.04 kworker/8:0\n", " 9606 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/20+\n", " 9608 elastic+ 20 0 43.707g 0.032t 15840 S 0.0 34.3 10:05.08 java\n", "10173 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/3:0\n", "10368 postfix 20 0 91188 3892 2888 S 0.0 0.0 0:00.01 pickup\n", "12642 root 20 0 0 0 0 Z 0.0 0.0 0:00.00 sh\n", "12652 root 20 0 0 0 0 Z 0.0 0.0 0:00.00 sh\n", "12808 root 20 0 0 0 0 S 0.0 0.0 0:00.55 kworker/6:0\n", "12918 root 20 0 135216 4848 3680 S 0.0 0.0 0:00.03 sshd\n", "12920 ansible 20 0 135348 2612 1356 S 0.0 0.0 0:00.03 sshd\n", "12963 root 20 0 0 0 0 S 0.0 0.0 0:00.98 kworker/2:1\n", "13237 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bond0\n", "13399 ansible 20 0 113116 1196 1020 S 0.0 0.0 0:00.00 sh\n", "13413 ansible 20 0 191232 7992 3584 S 0.0 0.0 0:00.04 python\n", "13414 ansible 20 0 208384 12720 3848 S 0.0 0.0 0:00.08 python\n", "13465 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/16+\n", "14600 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/18+\n", "14634 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/20+\n", "14768 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/14+\n", "15598 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/4:1\n", "15991 root 20 0 0 0 0 S 0.0 0.0 0:37.35 kworker/u4+\n", "18106 root 20 0 0 0 0 S 0.0 0.0 0:00.40 kworker/9:1\n", "19068 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0\n", "19723 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:+\n", "19920 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/13+\n", "20196 root 20 0 82508 3648 2788 S 0.0 0.0 0:00.01 sshd\n", "20386 root 20 0 0 0 0 S 0.0 0.0 0:00.76 kworker/4:2\n", "21238 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/11+\n", "21791 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/18+\n", "22071 root 20 0 0 0 0 S 0.0 0.0 0:39.73 kworker/u4+\n", "22885 root 20 0 0 0 0 S 0.0 0.0 0:00.50 kworker/23+\n", "23405 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/0:2\n", "23729 root 20 0 0 0 0 S 0.0 0.0 0:08.91 kworker/1:2\n", "24155 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/14+\n", "24379 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/5:+\n", "24723 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/17+\n", "25379 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/15+\n", "26041 root 20 0 0 0 0 S 0.0 0.0 0:00.36 kworker/5:0\n", "27065 root 0 -20 0 0 0 S 0.0 0.0 0:00.05 kworker/2:+\n", "27430 root 20 0 0 0 0 S 0.0 0.0 0:00.74 kworker/14+\n", "27746 root 20 0 0 0 0 S 0.0 0.0 0:00.52 kworker/12+\n", "28698 root 20 0 0 0 0 S 0.0 0.0 0:00.68 kworker/22+\n", "28881 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/15+\n", "29495 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/4:+\n", "31835 root 20 0 0 0 0 S 0.0 0.0 0:00.62 kworker/16+\n", "32609 root 20 0 0 0 0 S 0.0 0.0 0:00.40 kworker/12+\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.233 | SUCCESS | rc=0 >>\n", "top - 10:51:43 up 2 days, 17:21, 1 user, load average: 0.02, 0.02, 0.05\n", "Tasks: 297 total, 1 running, 294 sleeping, 0 stopped, 2 zombie\n", "%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st\n", "KiB Mem : 98827752 total, 62656732 free, 34935120 used, 1235900 buff/cache\n", "KiB Swap: 93750000 total, 93750000 free, 0 used. 63506656 avail Mem \n", "\n", " PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND\n", "12549 ansible 20 0 130152 1780 1196 R 6.2 0.0 0:00.01 top\n", " 1 root 20 0 54296 4128 2480 S 0.0 0.0 0:03.54 systemd\n", " 2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd\n", " 3 root 20 0 0 0 0 S 0.0 0.0 0:00.03 ksoftirqd/0\n", " 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:+\n", " 7 root 20 0 0 0 0 S 0.0 0.0 0:02.54 kworker/u4+\n", " 8 root rt 0 0 0 0 S 0.0 0.0 0:00.25 migration/0\n", " 9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh\n", " 10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/0\n", " 11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/1\n", " 12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/2\n", " 13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/3\n", " 14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/4\n", " 15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/5\n", " 16 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/6\n", " 17 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/7\n", " 18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/8\n", " 19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/9\n", " 20 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/10\n", " 21 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/11\n", " 22 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/12\n", " 23 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/13\n", " 24 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/14\n", " 25 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/15\n", " 26 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/16\n", " 27 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/17\n", " 28 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/18\n", " 29 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/19\n", " 30 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/20\n", " 31 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/21\n", " 32 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/22\n", " 33 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/23\n", " 34 root 20 0 0 0 0 S 0.0 0.0 5:50.95 rcu_sched\n", " 35 root 20 0 0 0 0 S 0.0 0.0 0:03.64 rcuos/0\n", " 36 root 20 0 0 0 0 S 0.0 0.0 0:03.36 rcuos/1\n", " 37 root 20 0 0 0 0 S 0.0 0.0 0:07.10 rcuos/2\n", " 38 root 20 0 0 0 0 S 0.0 0.0 0:06.11 rcuos/3\n", " 39 root 20 0 0 0 0 S 0.0 0.0 0:05.20 rcuos/4\n", " 40 root 20 0 0 0 0 S 0.0 0.0 0:03.99 rcuos/5\n", " 41 root 20 0 0 0 0 S 0.0 0.0 0:04.15 rcuos/6\n", " 42 root 20 0 0 0 0 S 0.0 0.0 0:04.99 rcuos/7\n", " 43 root 20 0 0 0 0 S 0.0 0.0 0:08.37 rcuos/8\n", " 44 root 20 0 0 0 0 S 0.0 0.0 0:06.44 rcuos/9\n", " 45 root 20 0 0 0 0 S 0.0 0.0 0:04.80 rcuos/10\n", " 46 root 20 0 0 0 0 S 0.0 0.0 0:04.53 rcuos/11\n", " 47 root 20 0 0 0 0 S 0.0 0.0 0:02.51 rcuos/12\n", " 48 root 20 0 0 0 0 S 0.0 0.0 0:15.41 rcuos/13\n", " 49 root 20 0 0 0 0 S 0.0 0.0 0:24.33 rcuos/14\n", " 50 root 20 0 0 0 0 S 0.0 0.0 0:02.35 rcuos/15\n", " 51 root 20 0 0 0 0 S 0.0 0.0 0:01.85 rcuos/16\n", " 52 root 20 0 0 0 0 S 0.0 0.0 0:02.00 rcuos/17\n", " 53 root 20 0 0 0 0 S 0.0 0.0 0:32.03 rcuos/18\n", " 54 root 20 0 0 0 0 S 0.0 0.0 0:12.12 rcuos/19\n", " 55 root 20 0 0 0 0 S 0.0 0.0 0:02.61 rcuos/20\n", " 56 root 20 0 0 0 0 S 0.0 0.0 0:02.22 rcuos/21\n", " 57 root 20 0 0 0 0 S 0.0 0.0 0:56.09 rcuos/22\n", " 58 root 20 0 0 0 0 S 0.0 0.0 0:01.39 rcuos/23\n", " 59 root rt 0 0 0 0 S 0.0 0.0 0:00.56 watchdog/0\n", " 60 root rt 0 0 0 0 S 0.0 0.0 0:00.56 watchdog/1\n", " 61 root rt 0 0 0 0 S 0.0 0.0 0:00.17 migration/1\n", " 62 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/1\n", " 64 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:+\n", " 65 root rt 0 0 0 0 S 0.0 0.0 0:00.53 watchdog/2\n", " 66 root rt 0 0 0 0 S 0.0 0.0 0:00.17 migration/2\n", " 67 root 20 0 0 0 0 S 0.0 0.0 0:00.18 ksoftirqd/2\n", " 69 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/2:+\n", " 70 root rt 0 0 0 0 S 0.0 0.0 0:00.51 watchdog/3\n", " 71 root rt 0 0 0 0 S 0.0 0.0 0:00.17 migration/3\n", " 72 root 20 0 0 0 0 S 0.0 0.0 0:00.03 ksoftirqd/3\n", " 74 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:+\n", " 75 root rt 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/4\n", " 76 root rt 0 0 0 0 S 0.0 0.0 0:00.17 migration/4\n", " 77 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/4\n", " 79 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/4:+\n", " 80 root rt 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/5\n", " 81 root rt 0 0 0 0 S 0.0 0.0 0:00.34 migration/5\n", " 82 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/5\n", " 84 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/5:+\n", " 85 root rt 0 0 0 0 S 0.0 0.0 0:00.51 watchdog/6\n", " 86 root rt 0 0 0 0 S 0.0 0.0 0:00.20 migration/6\n", " 87 root 20 0 0 0 0 S 0.0 0.0 0:00.04 ksoftirqd/6\n", " 89 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/6:+\n", " 91 root rt 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/7\n", " 92 root rt 0 0 0 0 S 0.0 0.0 0:00.19 migration/7\n", " 93 root 20 0 0 0 0 S 0.0 0.0 0:00.04 ksoftirqd/7\n", " 95 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/7:+\n", " 96 root rt 0 0 0 0 S 0.0 0.0 0:00.51 watchdog/8\n", " 97 root rt 0 0 0 0 S 0.0 0.0 0:00.29 migration/8\n", " 98 root 20 0 0 0 0 S 0.0 0.0 0:00.50 ksoftirqd/8\n", " 100 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/8:+\n", " 101 root rt 0 0 0 0 S 0.0 0.0 0:00.51 watchdog/9\n", " 102 root rt 0 0 0 0 S 0.0 0.0 0:00.25 migration/9\n", " 103 root 20 0 0 0 0 S 0.0 0.0 0:00.04 ksoftirqd/9\n", " 105 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/9:+\n", " 106 root rt 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/10\n", " 107 root rt 0 0 0 0 S 0.0 0.0 0:00.24 migration/+\n", " 108 root 20 0 0 0 0 S 0.0 0.0 0:00.52 ksoftirqd/+\n", " 110 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/10+\n", " 111 root rt 0 0 0 0 S 0.0 0.0 0:00.49 watchdog/11\n", " 112 root rt 0 0 0 0 S 0.0 0.0 0:00.33 migration/+\n", " 113 root 20 0 0 0 0 S 0.0 0.0 0:00.04 ksoftirqd/+\n", " 115 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/11+\n", " 116 root rt 0 0 0 0 S 0.0 0.0 0:00.49 watchdog/12\n", " 117 root rt 0 0 0 0 S 0.0 0.0 0:00.30 migration/+\n", " 118 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 120 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/12+\n", " 121 root rt 0 0 0 0 S 0.0 0.0 0:00.49 watchdog/13\n", " 122 root rt 0 0 0 0 S 0.0 0.0 0:00.76 migration/+\n", " 123 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 125 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/13+\n", " 126 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/14\n", " 127 root rt 0 0 0 0 S 0.0 0.0 0:00.36 migration/+\n", " 128 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 130 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/14+\n", " 131 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/15\n", " 132 root rt 0 0 0 0 S 0.0 0.0 0:00.20 migration/+\n", " 133 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 135 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/15+\n", " 136 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/16\n", " 137 root rt 0 0 0 0 S 0.0 0.0 0:00.18 migration/+\n", " 138 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 140 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/16+\n", " 141 root rt 0 0 0 0 S 0.0 0.0 0:00.49 watchdog/17\n", " 142 root rt 0 0 0 0 S 0.0 0.0 0:00.25 migration/+\n", " 143 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 145 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/17+\n", " 146 root rt 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/18\n", " 147 root rt 0 0 0 0 S 0.0 0.0 0:00.18 migration/+\n", " 148 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 150 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/18+\n", " 151 root rt 0 0 0 0 S 0.0 0.0 0:00.53 watchdog/19\n", " 152 root rt 0 0 0 0 S 0.0 0.0 0:00.83 migration/+\n", " 153 root 20 0 0 0 0 S 0.0 0.0 0:00.19 ksoftirqd/+\n", " 155 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/19+\n", " 156 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/20\n", " 157 root rt 0 0 0 0 S 0.0 0.0 0:00.48 migration/+\n", " 158 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 160 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/20+\n", " 161 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/21\n", " 162 root rt 0 0 0 0 S 0.0 0.0 0:00.28 migration/+\n", " 163 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 165 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/21+\n", " 166 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/22\n", " 167 root rt 0 0 0 0 S 0.0 0.0 0:00.19 migration/+\n", " 168 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/+\n", " 170 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/22+\n", " 171 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/23\n", " 172 root rt 0 0 0 0 S 0.0 0.0 0:00.58 migration/+\n", " 173 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 175 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/23+\n", " 176 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper\n", " 177 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs\n", " 178 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns\n", " 179 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback\n", " 180 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd\n", " 181 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset\n", " 182 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd\n", " 184 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khubd\n", " 185 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 md\n", " 187 root 20 0 0 0 0 S 0.0 0.0 0:00.22 khungtaskd\n", " 188 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kswapd0\n", " 189 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kswapd1\n", " 190 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd\n", " 191 root 39 19 0 0 0 S 0.0 0.0 0:01.71 khugepaged\n", " 192 root 20 0 0 0 0 S 0.0 0.0 0:00.00 fsnotify_m+\n", " 193 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto\n", " 201 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kthrotld\n", " 205 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kmpath_rda+\n", " 216 root 20 0 0 0 0 S 0.0 0.0 0:02.57 kworker/22+\n", " 218 root 20 0 0 0 0 S 0.0 0.0 0:00.41 kworker/16+\n", " 220 root 20 0 0 0 0 S 0.0 0.0 0:11.03 kworker/18+\n", " 226 root 20 0 0 0 0 S 0.0 0.0 0:00.73 kworker/15+\n", " 229 root 20 0 0 0 0 S 0.0 0.0 0:02.24 kworker/23+\n", " 230 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kpsmoused\n", " 249 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 deferwq\n", " 279 root 20 0 0 0 0 S 0.0 0.0 0:00.31 kauditd\n", " 391 root 20 0 0 0 0 S 0.0 0.0 0:00.58 kworker/3:0\n", " 406 root 20 0 0 0 0 S 0.0 0.0 0:06.93 kworker/11+\n", " 410 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0\n", " 411 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_0\n", " 413 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ata_sff\n", " 426 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_1\n", " 427 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_1\n", " 428 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_2\n", " 429 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_2\n", " 430 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_3\n", " 431 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_3\n", " 432 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_4\n", " 433 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_4\n", " 434 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_5\n", " 435 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_5\n", " 436 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_6\n", " 437 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_6\n", " 440 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u5+\n", " 441 root 20 0 0 0 0 S 0.0 0.0 0:01.22 kworker/u5+\n", " 448 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u4+\n", " 510 root 20 0 0 0 0 S 0.0 0.0 0:02.50 jbd2/sda2-8\n", " 511 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-rsv-c+\n", " 512 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-unrsv+\n", " 581 root 20 0 177688 94444 94092 S 0.0 0.1 0:39.94 systemd-jo+\n", " 599 root 20 0 42932 2364 1224 S 0.0 0.0 0:00.16 systemd-ud+\n", " 685 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 edac-poller\n", " 734 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kvm-irqfd-+\n", " 740 root 20 0 0 0 0 S 0.0 0.0 0:00.00 jbd2/sda5-8\n", " 741 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-rsv-c+\n", " 742 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-unrsv+\n", " 757 root 16 -4 116684 1636 1240 S 0.0 0.0 0:01.18 auditd\n", " 779 root 20 0 339472 24176 6880 S 0.0 0.0 0:01.44 firewalld\n", " 788 root 20 0 19284 1288 948 S 0.0 0.0 0:28.91 irqbalance\n", " 790 root 20 0 550144 16152 5628 S 0.0 0.0 0:13.05 tuned\n", " 791 root 20 0 394132 52108 49456 S 0.0 0.1 0:20.10 rsyslogd\n", " 794 root 20 0 34684 1724 1408 S 0.0 0.0 0:01.51 systemd-lo+\n", " 795 avahi 20 0 28072 1532 1288 S 0.0 0.0 0:00.29 avahi-daem+\n", " 796 dbus 20 0 26708 1928 1344 S 0.0 0.0 0:02.56 dbus-daemon\n", " 798 root 20 0 126300 1596 980 S 0.0 0.0 0:00.67 crond\n", " 799 root 20 0 25932 940 744 S 0.0 0.0 0:00.00 atd\n", " 810 root 20 0 110004 844 712 S 0.0 0.0 0:00.00 agetty\n", " 811 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:+\n", " 812 avahi 20 0 27944 444 216 S 0.0 0.0 0:00.00 avahi-daem+\n", " 816 ntp 20 0 31432 2076 1468 S 0.0 0.0 0:00.78 ntpd\n", " 935 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/21+\n", " 981 root 20 0 454324 11276 6908 S 0.0 0.0 0:04.47 NetworkMan+\n", " 1122 polkitd 20 0 514436 13444 4704 S 0.0 0.0 0:01.51 polkitd\n", " 1202 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:+\n", " 1333 root 20 0 0 0 0 S 0.0 0.0 0:35.98 kworker/u4+\n", " 1476 root 0 -20 0 0 0 S 0.0 0.0 0:00.06 kworker/2:+\n", " 1686 root 20 0 360108 26536 4704 S 0.0 0.0 0:17.61 dodai-inst+\n", " 1693 root 20 0 223232 11088 6412 S 0.0 0.0 2:32.04 snmpd\n", " 1798 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/14+\n", " 2267 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/8:2\n", " 2359 root 20 0 159408 9304 2144 S 0.0 0.0 0:07.65 munin-node\n", " 2830 root 20 0 91084 2172 1148 S 0.0 0.0 0:00.83 master\n", " 2859 postfix 20 0 91256 3896 2904 S 0.0 0.0 0:00.16 qmgr\n", " 3936 root 20 0 0 0 0 S 0.0 0.0 0:02.01 kworker/8:1\n", " 4020 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/21+\n", " 4363 root 20 0 0 0 0 S 0.0 0.0 1:15.51 kworker/19+\n", " 4463 root 20 0 0 0 0 S 0.0 0.0 0:03.13 kworker/4:0\n", " 5388 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/20+\n", " 5389 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/11+\n", " 6599 root 20 0 0 0 0 S 0.0 0.0 0:00.50 kworker/7:2\n", " 6726 root 20 0 0 0 0 S 0.0 0.0 0:00.19 kworker/6:0\n", " 7076 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/18+\n", " 7366 root 20 0 0 0 0 S 0.0 0.0 0:01.69 kworker/0:0\n", " 8573 root 20 0 0 0 0 S 0.0 0.0 0:00.13 kworker/16+\n", " 8687 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/8:+\n", " 8688 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/9:+\n", " 8689 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/10+\n", " 8764 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:+\n", " 8765 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/4:+\n", " 8770 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/5:+\n", " 8826 elastic+ 20 0 43.644g 0.032t 15844 S 0.0 34.3 9:51.07 java\n", " 9496 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/23+\n", " 9587 postfix 20 0 91188 3884 2888 S 0.0 0.0 0:00.00 pickup\n", " 9750 root 20 0 0 0 0 S 0.0 0.0 0:32.66 kworker/u4+\n", "10714 root 20 0 0 0 0 S 0.0 0.0 0:00.66 kworker/9:0\n", "12029 root 20 0 0 0 0 Z 0.0 0.0 0:00.00 sh\n", "12033 root 20 0 0 0 0 S 0.0 0.0 0:00.64 kworker/6:2\n", "12039 root 20 0 0 0 0 Z 0.0 0.0 0:00.00 sh\n", "12139 root 20 0 135216 4848 3680 S 0.0 0.0 0:00.03 sshd\n", "12141 ansible 20 0 135348 2612 1356 S 0.0 0.0 0:00.03 sshd\n", "12533 ansible 20 0 113116 1192 1020 S 0.0 0.0 0:00.00 sh\n", "12547 ansible 20 0 191232 7992 3584 S 0.0 0.0 0:00.03 python\n", "12548 ansible 20 0 208384 12716 3848 S 0.0 0.0 0:00.08 python\n", "12623 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bond0\n", "12690 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/13+\n", "13495 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/15+\n", "13812 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/7:+\n", "14037 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/11+\n", "14040 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/6:+\n", "15258 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/19+\n", "16342 root 20 0 0 0 0 S 0.0 0.0 0:09.90 kworker/10+\n", "16519 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/19+\n", "16834 root 20 0 0 0 0 S 0.0 0.0 0:00.56 kworker/20+\n", "17835 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/22+\n", "19148 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/22+\n", "19154 root 20 0 0 0 0 S 0.0 0.0 0:00.45 kworker/5:1\n", "19474 root 20 0 82508 3636 2788 S 0.0 0.0 0:00.00 sshd\n", "19978 root 20 0 0 0 0 S 0.0 0.0 0:03.95 kworker/2:0\n", "20473 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/5:0\n", "21284 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0\n", "23134 root 20 0 0 0 0 S 0.0 0.0 0:00.41 kworker/1:2\n", "24914 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/0:2\n", "24921 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kworker/21+\n", "24949 root 20 0 0 0 0 S 0.0 0.0 0:04.28 kworker/17+\n", "25649 root 20 0 0 0 0 S 0.0 0.0 0:00.43 kworker/14+\n", "26267 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/15+\n", "26774 root 20 0 0 0 0 S 0.0 0.0 0:00.62 kworker/12+\n", "27612 root 20 0 0 0 0 S 0.0 0.0 0:01.60 kworker/13+\n", "28056 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/3:1\n", "29214 root 20 0 0 0 0 S 0.0 0.0 0:00.26 kworker/7:0\n", "29215 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/9:2\n", "29873 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/12+\n", "30361 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/10+\n", "30835 root 20 0 0 0 0 S 0.0 0.0 0:00.55 kworker/23+\n", "30836 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/2:1\n", "31329 root 20 0 0 0 0 S 0.0 0.0 0:01.24 kworker/4:1\n", "31564 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/17+\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.235 | SUCCESS | rc=0 >>\n", "top - 10:51:44 up 2 days, 17:13, 1 user, load average: 0.00, 0.01, 0.05\n", "Tasks: 295 total, 1 running, 294 sleeping, 0 stopped, 0 zombie\n", "%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st\n", "KiB Mem : 98827752 total, 62807528 free, 34798252 used, 1221972 buff/cache\n", "KiB Swap: 93750000 total, 93750000 free, 0 used. 63653308 avail Mem \n", "\n", " PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND\n", " 2100 ansible 20 0 130152 1808 1196 R 6.2 0.0 0:00.01 top\n", " 1 root 20 0 54292 4112 2476 S 0.0 0.0 0:03.72 systemd\n", " 2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd\n", " 3 root 20 0 0 0 0 S 0.0 0.0 0:00.45 ksoftirqd/0\n", " 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:+\n", " 7 root 20 0 0 0 0 S 0.0 0.0 0:00.68 kworker/u4+\n", " 8 root rt 0 0 0 0 S 0.0 0.0 0:00.16 migration/0\n", " 9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh\n", " 10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/0\n", " 11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/1\n", " 12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/2\n", " 13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/3\n", " 14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/4\n", " 15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/5\n", " 16 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/6\n", " 17 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/7\n", " 18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/8\n", " 19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/9\n", " 20 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/10\n", " 21 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/11\n", " 22 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/12\n", " 23 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/13\n", " 24 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/14\n", " 25 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/15\n", " 26 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/16\n", " 27 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/17\n", " 28 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/18\n", " 29 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/19\n", " 30 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/20\n", " 31 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/21\n", " 32 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/22\n", " 33 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/23\n", " 34 root 20 0 0 0 0 S 0.0 0.0 5:44.32 rcu_sched\n", " 35 root 20 0 0 0 0 S 0.0 0.0 0:04.38 rcuos/0\n", " 36 root 20 0 0 0 0 S 0.0 0.0 0:06.00 rcuos/1\n", " 37 root 20 0 0 0 0 S 0.0 0.0 0:08.86 rcuos/2\n", " 38 root 20 0 0 0 0 S 0.0 0.0 0:08.07 rcuos/3\n", " 39 root 20 0 0 0 0 S 0.0 0.0 0:05.83 rcuos/4\n", " 40 root 20 0 0 0 0 S 0.0 0.0 0:05.14 rcuos/5\n", " 41 root 20 0 0 0 0 S 0.0 0.0 0:04.98 rcuos/6\n", " 42 root 20 0 0 0 0 S 0.0 0.0 0:04.06 rcuos/7\n", " 43 root 20 0 0 0 0 S 0.0 0.0 0:07.17 rcuos/8\n", " 44 root 20 0 0 0 0 S 0.0 0.0 0:27.90 rcuos/9\n", " 45 root 20 0 0 0 0 S 0.0 0.0 0:08.72 rcuos/10\n", " 46 root 20 0 0 0 0 S 0.0 0.0 0:06.14 rcuos/11\n", " 47 root 20 0 0 0 0 S 0.0 0.0 0:01.93 rcuos/12\n", " 48 root 20 0 0 0 0 S 0.0 0.0 0:12.85 rcuos/13\n", " 49 root 20 0 0 0 0 S 0.0 0.0 1:18.25 rcuos/14\n", " 50 root 20 0 0 0 0 S 0.0 0.0 0:02.50 rcuos/15\n", " 51 root 20 0 0 0 0 S 0.0 0.0 0:02.24 rcuos/16\n", " 52 root 20 0 0 0 0 S 0.0 0.0 0:02.00 rcuos/17\n", " 53 root 20 0 0 0 0 S 0.0 0.0 0:01.60 rcuos/18\n", " 54 root 20 0 0 0 0 S 0.0 0.0 0:11.04 rcuos/19\n", " 55 root 20 0 0 0 0 S 0.0 0.0 0:02.47 rcuos/20\n", " 56 root 20 0 0 0 0 S 0.0 0.0 0:03.60 rcuos/21\n", " 57 root 20 0 0 0 0 S 0.0 0.0 0:01.74 rcuos/22\n", " 58 root 20 0 0 0 0 S 0.0 0.0 0:01.15 rcuos/23\n", " 59 root rt 0 0 0 0 S 0.0 0.0 0:00.54 watchdog/0\n", " 60 root rt 0 0 0 0 S 0.0 0.0 0:00.56 watchdog/1\n", " 61 root rt 0 0 0 0 S 0.0 0.0 0:00.23 migration/1\n", " 62 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/1\n", " 64 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:+\n", " 65 root rt 0 0 0 0 S 0.0 0.0 0:00.52 watchdog/2\n", " 66 root rt 0 0 0 0 S 0.0 0.0 0:00.20 migration/2\n", " 67 root 20 0 0 0 0 S 0.0 0.0 0:00.14 ksoftirqd/2\n", " 69 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/2:+\n", " 70 root rt 0 0 0 0 S 0.0 0.0 0:00.49 watchdog/3\n", " 71 root rt 0 0 0 0 S 0.0 0.0 0:00.31 migration/3\n", " 72 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/3\n", " 74 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:+\n", " 75 root rt 0 0 0 0 S 0.0 0.0 0:00.49 watchdog/4\n", " 76 root rt 0 0 0 0 S 0.0 0.0 0:00.16 migration/4\n", " 77 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/4\n", " 79 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/4:+\n", " 80 root rt 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/5\n", " 81 root rt 0 0 0 0 S 0.0 0.0 0:00.26 migration/5\n", " 82 root 20 0 0 0 0 S 0.0 0.0 0:00.44 ksoftirqd/5\n", " 84 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/5:+\n", " 85 root rt 0 0 0 0 S 0.0 0.0 0:00.52 watchdog/6\n", " 86 root rt 0 0 0 0 S 0.0 0.0 0:00.26 migration/6\n", " 87 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/6\n", " 89 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/6:+\n", " 91 root rt 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/7\n", " 92 root rt 0 0 0 0 S 0.0 0.0 0:00.23 migration/7\n", " 93 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/7\n", " 95 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/7:+\n", " 96 root rt 0 0 0 0 S 0.0 0.0 0:00.51 watchdog/8\n", " 97 root rt 0 0 0 0 S 0.0 0.0 0:00.16 migration/8\n", " 98 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/8\n", " 100 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/8:+\n", " 101 root rt 0 0 0 0 S 0.0 0.0 0:00.51 watchdog/9\n", " 102 root rt 0 0 0 0 S 0.0 0.0 0:00.26 migration/9\n", " 103 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/9\n", " 105 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/9:+\n", " 106 root rt 0 0 0 0 S 0.0 0.0 0:00.49 watchdog/10\n", " 107 root rt 0 0 0 0 S 0.0 0.0 0:00.16 migration/+\n", " 108 root 20 0 0 0 0 S 0.0 0.0 0:00.08 ksoftirqd/+\n", " 110 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/10+\n", " 111 root rt 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/11\n", " 112 root rt 0 0 0 0 S 0.0 0.0 0:00.27 migration/+\n", " 113 root 20 0 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/+\n", " 115 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/11+\n", " 116 root rt 0 0 0 0 S 0.0 0.0 0:00.47 watchdog/12\n", " 117 root rt 0 0 0 0 S 0.0 0.0 0:00.18 migration/+\n", " 118 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 119 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/12+\n", " 120 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/12+\n", " 121 root rt 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/13\n", " 122 root rt 0 0 0 0 S 0.0 0.0 0:00.75 migration/+\n", " 123 root 20 0 0 0 0 S 0.0 0.0 0:00.35 ksoftirqd/+\n", " 125 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/13+\n", " 126 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/14\n", " 127 root rt 0 0 0 0 S 0.0 0.0 0:00.38 migration/+\n", " 128 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/+\n", " 130 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/14+\n", " 131 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/15\n", " 132 root rt 0 0 0 0 S 0.0 0.0 0:00.24 migration/+\n", " 133 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 135 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/15+\n", " 136 root rt 0 0 0 0 S 0.0 0.0 0:00.46 watchdog/16\n", " 137 root rt 0 0 0 0 S 0.0 0.0 0:00.18 migration/+\n", " 138 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 140 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/16+\n", " 141 root rt 0 0 0 0 S 0.0 0.0 0:00.46 watchdog/17\n", " 142 root rt 0 0 0 0 S 0.0 0.0 0:00.34 migration/+\n", " 143 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 145 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/17+\n", " 146 root rt 0 0 0 0 S 0.0 0.0 0:00.49 watchdog/18\n", " 147 root rt 0 0 0 0 S 0.0 0.0 0:00.25 migration/+\n", " 148 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 150 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/18+\n", " 151 root rt 0 0 0 0 S 0.0 0.0 0:00.51 watchdog/19\n", " 152 root rt 0 0 0 0 S 0.0 0.0 0:00.73 migration/+\n", " 153 root 20 0 0 0 0 S 0.0 0.0 0:00.36 ksoftirqd/+\n", " 155 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/19+\n", " 156 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/20\n", " 157 root rt 0 0 0 0 S 0.0 0.0 0:00.36 migration/+\n", " 158 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 160 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/20+\n", " 161 root rt 0 0 0 0 S 0.0 0.0 0:00.47 watchdog/21\n", " 162 root rt 0 0 0 0 S 0.0 0.0 0:00.20 migration/+\n", " 163 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 165 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/21+\n", " 166 root rt 0 0 0 0 S 0.0 0.0 0:00.47 watchdog/22\n", " 167 root rt 0 0 0 0 S 0.0 0.0 0:00.18 migration/+\n", " 168 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 170 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/22+\n", " 171 root rt 0 0 0 0 S 0.0 0.0 0:00.47 watchdog/23\n", " 172 root rt 0 0 0 0 S 0.0 0.0 0:00.55 migration/+\n", " 173 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/+\n", " 175 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/23+\n", " 176 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper\n", " 177 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs\n", " 178 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns\n", " 179 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback\n", " 180 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd\n", " 181 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset\n", " 182 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd\n", " 184 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khubd\n", " 185 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 md\n", " 188 root 20 0 0 0 0 S 0.0 0.0 0:00.21 khungtaskd\n", " 189 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kswapd0\n", " 190 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kswapd1\n", " 191 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd\n", " 192 root 39 19 0 0 0 S 0.0 0.0 0:01.75 khugepaged\n", " 193 root 20 0 0 0 0 S 0.0 0.0 0:00.00 fsnotify_m+\n", " 194 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto\n", " 202 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kthrotld\n", " 206 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kmpath_rda+\n", " 213 root 20 0 0 0 0 S 0.0 0.0 0:01.08 kworker/14+\n", " 214 root 20 0 0 0 0 S 0.0 0.0 0:36.57 kworker/19+\n", " 215 root 20 0 0 0 0 S 0.0 0.0 0:05.54 kworker/20+\n", " 221 root 20 0 0 0 0 S 0.0 0.0 0:00.99 kworker/16+\n", " 224 root 20 0 0 0 0 S 0.0 0.0 0:00.88 kworker/18+\n", " 225 root 20 0 0 0 0 S 0.0 0.0 0:01.34 kworker/21+\n", " 226 root 20 0 0 0 0 S 0.0 0.0 0:00.76 kworker/12+\n", " 229 root 20 0 0 0 0 S 0.0 0.0 0:02.81 kworker/23+\n", " 230 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kpsmoused\n", " 249 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 deferwq\n", " 279 root 20 0 0 0 0 S 0.0 0.0 0:00.30 kauditd\n", " 303 postfix 20 0 91188 3888 2888 S 0.0 0.0 0:00.00 pickup\n", " 422 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0\n", " 423 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_0\n", " 426 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ata_sff\n", " 437 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_1\n", " 438 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_1\n", " 439 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_2\n", " 440 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_2\n", " 441 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_3\n", " 442 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_3\n", " 443 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_4\n", " 444 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_4\n", " 445 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_5\n", " 446 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_5\n", " 447 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_6\n", " 448 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_6\n", " 451 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u5+\n", " 452 root 20 0 0 0 0 S 0.0 0.0 0:03.03 kworker/u5+\n", " 459 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u4+\n", " 507 root 20 0 0 0 0 S 0.0 0.0 0:06.82 kworker/11+\n", " 510 root 20 0 0 0 0 S 0.0 0.0 0:02.33 jbd2/sda2-8\n", " 511 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-rsv-c+\n", " 512 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-unrsv+\n", " 581 root 20 0 161888 92012 91672 S 0.0 0.1 0:35.41 systemd-jo+\n", " 600 root 20 0 42344 1776 1224 S 0.0 0.0 0:00.11 systemd-ud+\n", " 645 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 edac-poller\n", " 716 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kvm-irqfd-+\n", " 729 root 20 0 0 0 0 S 0.0 0.0 0:00.00 jbd2/sda5-8\n", " 730 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-rsv-c+\n", " 731 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-unrsv+\n", " 744 root 16 -4 116684 1632 1240 S 0.0 0.0 0:01.04 auditd\n", " 766 root 20 0 339500 24164 6880 S 0.0 0.0 0:01.18 firewalld\n", " 770 avahi 20 0 28072 1684 1392 S 0.0 0.0 0:00.27 avahi-daem+\n", " 771 root 20 0 19284 1284 948 S 0.0 0.0 0:28.55 irqbalance\n", " 777 avahi 20 0 27944 440 216 S 0.0 0.0 0:00.00 avahi-daem+\n", " 778 root 20 0 314184 50864 48220 S 0.0 0.1 0:18.81 rsyslogd\n", " 779 root 20 0 550144 18168 5628 S 0.0 0.0 0:13.14 tuned\n", " 781 dbus 20 0 26708 1920 1344 S 0.0 0.0 0:02.56 dbus-daemon\n", " 783 root 20 0 34684 1724 1408 S 0.0 0.0 0:01.54 systemd-lo+\n", " 784 root 20 0 25932 944 744 S 0.0 0.0 0:00.00 atd\n", " 785 root 20 0 126304 1600 984 S 0.0 0.0 0:00.63 crond\n", " 799 root 20 0 110004 840 712 S 0.0 0.0 0:00.00 agetty\n", " 800 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/8:+\n", " 801 ntp 20 0 31432 2068 1468 S 0.0 0.0 0:00.76 ntpd\n", " 843 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/10+\n", " 903 root 20 0 454052 8760 6748 S 0.0 0.0 0:03.49 NetworkMan+\n", " 925 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/11+\n", " 970 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/9:+\n", " 1012 polkitd 20 0 514436 11416 4708 S 0.0 0.0 0:01.55 polkitd\n", " 1029 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/6:+\n", " 1071 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kworker/0:0\n", " 1093 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/7:+\n", " 1568 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bond0\n", " 1687 root 20 0 135216 4848 3680 S 0.0 0.0 0:00.03 sshd\n", " 1689 ansible 20 0 135348 2612 1356 S 0.0 0.0 0:00.03 sshd\n", " 1817 root 20 0 223264 11076 6412 S 0.0 0.0 2:31.96 snmpd\n", " 1820 root 20 0 217128 19940 4528 S 0.0 0.0 0:03.58 dodai-inst+\n", " 2084 ansible 20 0 113116 1196 1020 S 0.0 0.0 0:00.00 sh\n", " 2098 ansible 20 0 191232 7992 3584 S 0.0 0.0 0:00.03 python\n", " 2099 ansible 20 0 208384 12716 3848 S 0.0 0.0 0:00.06 python\n", " 2407 root 20 0 155272 9208 2068 S 0.0 0.0 0:08.70 munin-node\n", " 2529 root 20 0 91084 2172 1148 S 0.0 0.0 0:00.86 master\n", " 2575 postfix 20 0 91256 3908 2908 S 0.0 0.0 0:00.16 qmgr\n", " 2906 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/10+\n", " 2966 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/5:+\n", " 3319 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/13+\n", " 4636 root 20 0 0 0 0 S 0.0 0.0 0:00.35 kworker/17+\n", " 4637 root 20 0 0 0 0 S 0.0 0.0 0:03.50 kworker/10+\n", " 4738 root 20 0 0 0 0 S 0.0 0.0 0:00.77 kworker/3:0\n", " 6807 root 0 -20 0 0 0 S 0.0 0.0 0:00.04 kworker/2:+\n", " 6928 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:+\n", " 7109 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/4:+\n", " 7372 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/9:1\n", " 8687 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/4:0\n", " 8904 root 20 0 0 0 0 S 0.0 0.0 0:24.33 kworker/u4+\n", " 9063 root 20 0 82508 3640 2788 S 0.0 0.0 0:00.00 sshd\n", " 9508 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/23+\n", "10028 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/19+\n", "10222 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/16+\n", "10511 root 20 0 0 0 0 S 0.0 0.0 0:01.43 kworker/8:2\n", "10843 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/0:1\n", "11356 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/3:2\n", "11997 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/11+\n", "12188 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0\n", "14571 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:+\n", "14820 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/8:0\n", "15033 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/14+\n", "15232 root 20 0 0 0 0 S 0.0 0.0 0:00.38 kworker/15+\n", "15817 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/21+\n", "16140 root 20 0 0 0 0 S 0.0 0.0 0:00.70 kworker/5:1\n", "16553 root 20 0 0 0 0 S 0.0 0.0 0:00.47 kworker/22+\n", "16652 root 20 0 0 0 0 S 0.0 0.0 0:02.06 kworker/0:2\n", "17048 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/19+\n", "17460 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0\n", "18336 root 20 0 0 0 0 S 0.0 0.0 0:21.93 kworker/u4+\n", "18781 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:+\n", "19184 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/15+\n", "19394 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/20+\n", "19930 root 20 0 0 0 0 S 0.0 0.0 0:00.57 kworker/1:1\n", "22008 root 20 0 0 0 0 S 0.0 0.0 0:24.71 kworker/13+\n", "23266 root 20 0 0 0 0 S 0.0 0.0 0:00.09 kworker/14+\n", "23268 root 20 0 0 0 0 S 0.0 0.0 0:00.78 kworker/6:2\n", "24478 root 20 0 0 0 0 S 0.0 0.0 0:04.01 kworker/7:1\n", "24544 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/22+\n", "24586 root 20 0 0 0 0 S 0.0 0.0 0:01.03 kworker/9:0\n", "24764 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/17+\n", "25394 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/23+\n", "25570 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/20+\n", "27349 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/7:2\n", "28054 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/5:2\n", "29873 root 20 0 0 0 0 S 0.0 0.0 0:02.99 kworker/2:2\n", "30450 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/18+\n", "30579 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/6:0\n", "30867 elastic+ 20 0 43.487g 0.031t 15868 S 0.0 34.2 9:36.72 java\n", "30922 root 20 0 0 0 0 S 0.0 0.0 0:04.05 kworker/4:2\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.234 | SUCCESS | rc=0 >>\n", "top - 10:51:44 up 2 days, 17:14, 1 user, load average: 0.00, 0.01, 0.05\n", "Tasks: 293 total, 1 running, 292 sleeping, 0 stopped, 0 zombie\n", "%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st\n", "KiB Mem : 98827752 total, 62337512 free, 34777856 used, 1712384 buff/cache\n", "KiB Swap: 93750000 total, 93750000 free, 0 used. 63656840 avail Mem \n", "\n", " PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND\n", " 1 root 20 0 54328 4124 2476 S 0.0 0.0 0:03.62 systemd\n", " 2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd\n", " 3 root 20 0 0 0 0 S 0.0 0.0 0:00.03 ksoftirqd/0\n", " 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:+\n", " 7 root 20 0 0 0 0 S 0.0 0.0 0:02.07 kworker/u4+\n", " 8 root rt 0 0 0 0 S 0.0 0.0 0:00.19 migration/0\n", " 9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh\n", " 10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/0\n", " 11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/1\n", " 12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/2\n", " 13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/3\n", " 14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/4\n", " 15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/5\n", " 16 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/6\n", " 17 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/7\n", " 18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/8\n", " 19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/9\n", " 20 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/10\n", " 21 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/11\n", " 22 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/12\n", " 23 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/13\n", " 24 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/14\n", " 25 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/15\n", " 26 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/16\n", " 27 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/17\n", " 28 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/18\n", " 29 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/19\n", " 30 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/20\n", " 31 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/21\n", " 32 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/22\n", " 33 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/23\n", " 34 root 20 0 0 0 0 S 0.0 0.0 5:41.28 rcu_sched\n", " 35 root 20 0 0 0 0 S 0.0 0.0 0:04.93 rcuos/0\n", " 36 root 20 0 0 0 0 S 0.0 0.0 0:05.51 rcuos/1\n", " 37 root 20 0 0 0 0 S 0.0 0.0 0:08.44 rcuos/2\n", " 38 root 20 0 0 0 0 S 0.0 0.0 0:06.34 rcuos/3\n", " 39 root 20 0 0 0 0 S 0.0 0.0 0:05.27 rcuos/4\n", " 40 root 20 0 0 0 0 S 0.0 0.0 0:04.87 rcuos/5\n", " 41 root 20 0 0 0 0 S 0.0 0.0 0:12.52 rcuos/6\n", " 42 root 20 0 0 0 0 S 0.0 0.0 0:04.18 rcuos/7\n", " 43 root 20 0 0 0 0 S 0.0 0.0 0:08.07 rcuos/8\n", " 44 root 20 0 0 0 0 S 0.0 0.0 0:05.46 rcuos/9\n", " 45 root 20 0 0 0 0 S 0.0 0.0 0:04.56 rcuos/10\n", " 46 root 20 0 0 0 0 S 0.0 0.0 0:04.72 rcuos/11\n", " 47 root 20 0 0 0 0 S 0.0 0.0 0:21.14 rcuos/12\n", " 48 root 20 0 0 0 0 S 0.0 0.0 0:11.12 rcuos/13\n", " 49 root 20 0 0 0 0 S 0.0 0.0 1:12.74 rcuos/14\n", " 50 root 20 0 0 0 0 S 0.0 0.0 0:01.89 rcuos/15\n", " 51 root 20 0 0 0 0 S 0.0 0.0 0:01.69 rcuos/16\n", " 52 root 20 0 0 0 0 S 0.0 0.0 0:01.77 rcuos/17\n", " 53 root 20 0 0 0 0 S 0.0 0.0 0:04.58 rcuos/18\n", " 54 root 20 0 0 0 0 S 0.0 0.0 0:12.88 rcuos/19\n", " 55 root 20 0 0 0 0 S 0.0 0.0 0:03.04 rcuos/20\n", " 56 root 20 0 0 0 0 S 0.0 0.0 0:02.36 rcuos/21\n", " 57 root 20 0 0 0 0 S 0.0 0.0 0:02.34 rcuos/22\n", " 58 root 20 0 0 0 0 S 0.0 0.0 0:01.49 rcuos/23\n", " 59 root rt 0 0 0 0 S 0.0 0.0 0:00.53 watchdog/0\n", " 60 root rt 0 0 0 0 S 0.0 0.0 0:00.54 watchdog/1\n", " 61 root rt 0 0 0 0 S 0.0 0.0 0:00.23 migration/1\n", " 62 root 20 0 0 0 0 S 0.0 0.0 0:00.46 ksoftirqd/1\n", " 64 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:+\n", " 65 root rt 0 0 0 0 S 0.0 0.0 0:00.53 watchdog/2\n", " 66 root rt 0 0 0 0 S 0.0 0.0 0:00.21 migration/2\n", " 67 root 20 0 0 0 0 S 0.0 0.0 0:00.65 ksoftirqd/2\n", " 69 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/2:+\n", " 70 root rt 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/3\n", " 71 root rt 0 0 0 0 S 0.0 0.0 0:00.17 migration/3\n", " 72 root 20 0 0 0 0 S 0.0 0.0 0:00.03 ksoftirqd/3\n", " 74 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:+\n", " 75 root rt 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/4\n", " 76 root rt 0 0 0 0 S 0.0 0.0 0:00.33 migration/4\n", " 77 root 20 0 0 0 0 S 0.0 0.0 0:00.03 ksoftirqd/4\n", " 79 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/4:+\n", " 80 root rt 0 0 0 0 S 0.0 0.0 0:00.49 watchdog/5\n", " 81 root rt 0 0 0 0 S 0.0 0.0 0:00.17 migration/5\n", " 82 root 20 0 0 0 0 S 0.0 0.0 0:00.03 ksoftirqd/5\n", " 84 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/5:+\n", " 85 root rt 0 0 0 0 S 0.0 0.0 0:00.53 watchdog/6\n", " 86 root rt 0 0 0 0 S 0.0 0.0 0:00.27 migration/6\n", " 87 root 20 0 0 0 0 S 0.0 0.0 0:00.05 ksoftirqd/6\n", " 89 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/6:+\n", " 91 root rt 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/7\n", " 92 root rt 0 0 0 0 S 0.0 0.0 0:00.34 migration/7\n", " 93 root 20 0 0 0 0 S 0.0 0.0 0:00.04 ksoftirqd/7\n", " 95 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/7:+\n", " 96 root rt 0 0 0 0 S 0.0 0.0 0:00.51 watchdog/8\n", " 97 root rt 0 0 0 0 S 0.0 0.0 0:00.17 migration/8\n", " 98 root 20 0 0 0 0 S 0.0 0.0 0:00.04 ksoftirqd/8\n", " 100 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/8:+\n", " 101 root rt 0 0 0 0 S 0.0 0.0 0:00.51 watchdog/9\n", " 102 root rt 0 0 0 0 S 0.0 0.0 0:00.25 migration/9\n", " 103 root 20 0 0 0 0 S 0.0 0.0 0:00.04 ksoftirqd/9\n", " 105 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/9:+\n", " 106 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/10\n", " 107 root rt 0 0 0 0 S 0.0 0.0 0:00.29 migration/+\n", " 108 root 20 0 0 0 0 S 0.0 0.0 0:00.08 ksoftirqd/+\n", " 110 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/10+\n", " 111 root rt 0 0 0 0 S 0.0 0.0 0:00.49 watchdog/11\n", " 112 root rt 0 0 0 0 S 0.0 0.0 0:00.17 migration/+\n", " 113 root 20 0 0 0 0 S 0.0 0.0 0:00.04 ksoftirqd/+\n", " 115 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/11+\n", " 116 root rt 0 0 0 0 S 0.0 0.0 0:00.49 watchdog/12\n", " 117 root rt 0 0 0 0 S 0.0 0.0 0:00.24 migration/+\n", " 118 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 120 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/12+\n", " 121 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/13\n", " 122 root rt 0 0 0 0 S 0.0 0.0 0:00.66 migration/+\n", " 123 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 125 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/13+\n", " 126 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/14\n", " 127 root rt 0 0 0 0 S 0.0 0.0 0:00.35 migration/+\n", " 128 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/+\n", " 130 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/14+\n", " 131 root rt 0 0 0 0 S 0.0 0.0 0:00.47 watchdog/15\n", " 132 root rt 0 0 0 0 S 0.0 0.0 0:00.20 migration/+\n", " 133 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 135 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/15+\n", " 136 root rt 0 0 0 0 S 0.0 0.0 0:00.47 watchdog/16\n", " 137 root rt 0 0 0 0 S 0.0 0.0 0:00.24 migration/+\n", " 138 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 140 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/16+\n", " 141 root rt 0 0 0 0 S 0.0 0.0 0:00.47 watchdog/17\n", " 142 root rt 0 0 0 0 S 0.0 0.0 0:00.17 migration/+\n", " 143 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 145 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/17+\n", " 146 root rt 0 0 0 0 S 0.0 0.0 0:00.49 watchdog/18\n", " 147 root rt 0 0 0 0 S 0.0 0.0 0:00.19 migration/+\n", " 148 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 149 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/18+\n", " 150 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/18+\n", " 151 root rt 0 0 0 0 S 0.0 0.0 0:00.52 watchdog/19\n", " 152 root rt 0 0 0 0 S 0.0 0.0 0:00.85 migration/+\n", " 153 root 20 0 0 0 0 S 0.0 0.0 0:00.36 ksoftirqd/+\n", " 154 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/19+\n", " 155 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/19+\n", " 156 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/20\n", " 157 root rt 0 0 0 0 S 0.0 0.0 0:00.37 migration/+\n", " 158 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 160 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/20+\n", " 161 root rt 0 0 0 0 S 0.0 0.0 0:00.47 watchdog/21\n", " 162 root rt 0 0 0 0 S 0.0 0.0 0:00.20 migration/+\n", " 163 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 164 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/21+\n", " 165 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/21+\n", " 166 root rt 0 0 0 0 S 0.0 0.0 0:00.47 watchdog/22\n", " 167 root rt 0 0 0 0 S 0.0 0.0 0:00.29 migration/+\n", " 168 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 170 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/22+\n", " 171 root rt 0 0 0 0 S 0.0 0.0 0:00.47 watchdog/23\n", " 172 root rt 0 0 0 0 S 0.0 0.0 0:00.60 migration/+\n", " 173 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/+\n", " 174 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/23+\n", " 175 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/23+\n", " 176 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper\n", " 177 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs\n", " 178 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns\n", " 179 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback\n", " 180 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd\n", " 181 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset\n", " 182 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd\n", " 184 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khubd\n", " 185 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 md\n", " 187 root 20 0 0 0 0 S 0.0 0.0 0:00.21 khungtaskd\n", " 188 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kswapd0\n", " 189 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kswapd1\n", " 190 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd\n", " 191 root 39 19 0 0 0 S 0.0 0.0 0:01.76 khugepaged\n", " 192 root 20 0 0 0 0 S 0.0 0.0 0:00.00 fsnotify_m+\n", " 193 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto\n", " 201 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kthrotld\n", " 205 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kmpath_rda+\n", " 214 root 20 0 0 0 0 S 0.0 0.0 0:01.52 kworker/18+\n", " 216 root 20 0 0 0 0 S 0.0 0.0 1:18.28 kworker/19+\n", " 217 root 20 0 0 0 0 S 0.0 0.0 0:00.87 kworker/22+\n", " 218 root 20 0 0 0 0 S 0.0 0.0 0:01.76 kworker/13+\n", " 220 root 20 0 0 0 0 S 0.0 0.0 0:07.74 kworker/12+\n", " 229 root 20 0 0 0 0 S 0.0 0.0 0:00.95 kworker/23+\n", " 230 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kpsmoused\n", " 249 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 deferwq\n", " 279 root 20 0 0 0 0 S 0.0 0.0 0:00.30 kauditd\n", " 410 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0\n", " 411 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_0\n", " 413 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ata_sff\n", " 425 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_1\n", " 426 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_1\n", " 428 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_2\n", " 429 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_2\n", " 430 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_3\n", " 431 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_3\n", " 432 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_4\n", " 433 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_4\n", " 434 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_5\n", " 435 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_5\n", " 436 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_6\n", " 437 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_6\n", " 440 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u5+\n", " 441 root 20 0 0 0 0 S 0.0 0.0 0:01.94 kworker/u5+\n", " 497 root 20 0 0 0 0 S 0.0 0.0 0:00.97 kworker/21+\n", " 512 root 20 0 0 0 0 S 0.0 0.0 0:02.49 jbd2/sda2-8\n", " 513 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-rsv-c+\n", " 514 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-unrsv+\n", " 582 root 20 0 161860 92048 91700 S 0.0 0.1 0:39.08 systemd-jo+\n", " 601 root 20 0 42432 1868 1224 S 0.0 0.0 0:00.10 systemd-ud+\n", " 647 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 edac-poller\n", " 712 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kvm-irqfd-+\n", " 724 root 20 0 0 0 0 S 0.0 0.0 0:00.00 jbd2/sda5-8\n", " 725 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-rsv-c+\n", " 726 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-unrsv+\n", " 739 root 16 -4 116684 1632 1240 S 0.0 0.0 0:01.10 auditd\n", " 761 root 20 0 339508 24180 6880 S 0.0 0.0 0:01.17 firewalld\n", " 765 root 20 0 19284 1284 948 S 0.0 0.0 0:28.09 irqbalance\n", " 766 avahi 20 0 28072 1684 1392 S 0.0 0.0 0:00.30 avahi-daem+\n", " 770 root 20 0 314156 50824 48184 S 0.0 0.1 0:19.00 rsyslogd\n", " 771 root 20 0 550144 16152 5628 S 0.0 0.0 0:13.31 tuned\n", " 774 dbus 20 0 26708 1924 1344 S 0.0 0.0 0:02.54 dbus-daemon\n", " 776 avahi 20 0 27944 440 212 S 0.0 0.0 0:00.00 avahi-daem+\n", " 777 root 20 0 34684 1732 1408 S 0.0 0.0 0:01.54 systemd-lo+\n", " 779 root 20 0 25932 940 744 S 0.0 0.0 0:00.00 atd\n", " 783 root 20 0 126300 1600 984 S 0.0 0.0 0:00.67 crond\n", " 793 root 20 0 110004 840 712 S 0.0 0.0 0:00.00 agetty\n", " 794 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:+\n", " 795 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:+\n", " 796 ntp 20 0 31432 2076 1468 S 0.0 0.0 0:00.74 ntpd\n", " 905 root 20 0 527808 8764 6748 S 0.0 0.0 0:03.49 NetworkMan+\n", " 922 root 20 0 0 0 0 S 0.0 0.0 0:32.60 kworker/u4+\n", " 955 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/9:+\n", " 973 root 20 0 0 0 0 S 0.0 0.0 0:03.74 kworker/6:1\n", " 976 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/10+\n", " 1007 polkitd 20 0 514404 12436 4708 S 0.0 0.0 0:01.52 polkitd\n", " 1008 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/11+\n", " 1026 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/6:+\n", " 1050 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/7:+\n", " 1071 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 kworker/8:+\n", " 1194 root 20 0 0 0 0 S 0.0 0.0 0:02.91 kworker/5:1\n", " 1396 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/5:+\n", " 1407 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u4+\n", " 1489 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/12+\n", " 1566 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bond0\n", " 1698 postfix 20 0 91188 3888 2888 S 0.0 0.0 0:00.00 pickup\n", " 1816 root 20 0 217012 19936 4528 S 0.0 0.0 0:03.70 dodai-inst+\n", " 1822 root 20 0 223264 11076 6412 S 0.0 0.0 2:32.76 snmpd\n", " 2414 root 20 0 155272 9212 2068 S 0.0 0.0 0:08.82 munin-node\n", " 2597 root 20 0 91084 2168 1144 S 0.0 0.0 0:00.80 master\n", " 2629 postfix 20 0 91256 3908 2908 S 0.0 0.0 0:00.16 qmgr\n", " 3020 root 20 0 135216 4852 3680 S 0.0 0.0 0:00.03 sshd\n", " 3022 ansible 20 0 135348 2616 1356 S 0.0 0.0 0:00.04 sshd\n", " 3115 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/8:1\n", " 3415 ansible 20 0 113116 1196 1020 S 0.0 0.0 0:00.00 sh\n", " 3429 ansible 20 0 191232 7992 3584 S 0.0 0.0 0:00.03 python\n", " 3430 ansible 20 0 208384 12720 3848 S 0.0 0.0 0:00.07 python\n", " 3431 ansible 20 0 130152 1808 1196 R 0.0 0.0 0:00.00 top\n", " 3622 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/22+\n", " 3916 root 20 0 0 0 0 S 0.0 0.0 0:01.79 kworker/10+\n", " 4057 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/23+\n", " 4261 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/18+\n", " 4668 root 20 0 0 0 0 S 0.0 0.0 0:00.19 kworker/16+\n", " 5236 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/10+\n", " 6375 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/19+\n", " 6754 root 20 0 0 0 0 S 0.0 0.0 0:01.02 kworker/20+\n", " 7659 root 20 0 0 0 0 S 0.0 0.0 0:00.85 kworker/15+\n", " 8369 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/4:+\n", " 9062 root 20 0 82508 3644 2788 S 0.0 0.0 0:00.01 sshd\n", " 9541 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/6:0\n", "10954 root 20 0 0 0 0 S 0.0 0.0 0:05.41 kworker/7:1\n", "11158 root 20 0 0 0 0 S 0.0 0.0 0:00.31 kworker/14+\n", "11366 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/13+\n", "11374 root 20 0 0 0 0 S 0.0 0.0 0:01.03 kworker/11+\n", "12815 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/11+\n", "13681 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/16+\n", "13682 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/9:1\n", "15763 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:+\n", "15927 root 20 0 0 0 0 S 0.0 0.0 0:05.03 kworker/u4+\n", "17093 root 20 0 0 0 0 S 0.0 0.0 0:04.57 kworker/9:2\n", "17469 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/4:0\n", "17555 root 20 0 0 0 0 S 0.0 0.0 0:02.01 kworker/17+\n", "17985 root 20 0 0 0 0 S 0.0 0.0 0:02.00 kworker/2:2\n", "19082 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kworker/20+\n", "21446 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/2:1\n", "24163 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/15+\n", "25910 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/5:0\n", "27711 root 20 0 0 0 0 S 0.0 0.0 0:01.41 kworker/8:0\n", "28025 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0\n", "28215 root 20 0 0 0 0 S 0.0 0.0 0:01.26 kworker/4:2\n", "29531 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/7:0\n", "29959 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/14+\n", "30688 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/3:1\n", "30690 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/17+\n", "31205 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0\n", "31694 root 20 0 0 0 0 S 0.0 0.0 0:00.75 kworker/0:1\n", "31703 root 0 -20 0 0 0 S 0.0 0.0 0:00.07 kworker/2:+\n", "32010 elastic+ 20 0 43.455g 0.031t 15852 S 0.0 34.2 11:24.22 java\n", "32296 root 20 0 0 0 0 S 0.0 0.0 0:09.54 kworker/1:1\n", "32583 root 20 0 0 0 0 S 0.0 0.0 0:02.39 kworker/3:2\n", "\u001b[0m\n" ] } ], "source": [ "!ansible all -m shell -a 'top n 1 b ' -i ./hosts -u $USER --private-key=$KEYPATH" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbfb9e-14ed-11e7-b31a-0242ac12000d", "next": "a6fbfd56-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbf9dc-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■確認内容**" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fbfd56-14ed-11e7-b31a-0242ac12000d", "next": "a6fbff22-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbfb9e-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "CPU使用率、メモリ使用率が継続的に100%近い数値になっていないか確認します。 \n", "継続的に100%近い数値になっている場合、リソースが枯渇していると考えられます。" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fbff22-14ed-11e7-b31a-0242ac12000d", "next": "a6fc00e4-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbfd56-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "### ヒープが枯渇していないか？\n", "ノード状態を調査するAPIを発行し、ヒープの利用状況を確認します。" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc00e4-14ed-11e7-b31a-0242ac12000d", "next": "a6fc02a6-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbff22-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "Nodes Stats APIを、JVMの状態を見るオプションを付与して発行します。 \n", "APIの詳細はElasticsearch Referenceの[Nodes Stats](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html)を参照してください。" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc02a6-14ed-11e7-b31a-0242ac12000d", "next": "a6fc045e-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc00e4-14ed-11e7-b31a-0242ac12000d" }, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\r\n", " \"_nodes\" : {\r\n", " \"total\" : 5,\r\n", " \"successful\" : 5,\r\n", " \"failed\" : 0\r\n", " },\r\n", " \"cluster_name\" : \"es-cluster\",\r\n", " \"nodes\" : {\r\n", " \"z0-yqIssTTeGPdynW-UEEw\" : {\r\n", " \"timestamp\" : 1491443504886,\r\n", " \"name\" : \"z0-yqIs\",\r\n", " \"transport_address\" : \"XXX.XXX.XXX.235:9300\",\r\n", " \"host\" : \"XXX.XXX.XXX.235\",\r\n", " \"ip\" : \"XXX.XXX.XXX.235:9300\",\r\n", " \"roles\" : [\r\n", " \"data\",\r\n", " \"ingest\"\r\n", " ],\r\n", " \"jvm\" : {\r\n", " \"timestamp\" : 1491443504886,\r\n", " \"uptime_in_millis\" : 162357336,\r\n", " \"mem\" : {\r\n", " \"heap_used_in_bytes\" : 969223056,\r\n", " \"heap_used_percent\" : 2,\r\n", " \"heap_committed_in_bytes\" : 33128972288,\r\n", " \"heap_max_in_bytes\" : 33128972288,\r\n", " \"non_heap_used_in_bytes\" : 68031648,\r\n", " \"non_heap_committed_in_bytes\" : 72294400,\r\n", " \"pools\" : {\r\n", " \"young\" : {\r\n", " \"used_in_bytes\" : 513253704,\r\n", " \"max_in_bytes\" : 1256259584,\r\n", " \"peak_used_in_bytes\" : 1256259584,\r\n", " \"peak_max_in_bytes\" : 1256259584\r\n", " },\r\n", " \"survivor\" : {\r\n", " \"used_in_bytes\" : 4482560,\r\n", " \"max_in_bytes\" : 157024256,\r\n", " \"peak_used_in_bytes\" : 157024248,\r\n", " \"peak_max_in_bytes\" : 157024256\r\n", " },\r\n", " \"old\" : {\r\n", " \"used_in_bytes\" : 451486792,\r\n", " \"max_in_bytes\" : 31715688448,\r\n", " \"peak_used_in_bytes\" : 451486792,\r\n", " \"peak_max_in_bytes\" : 31715688448\r\n", " }\r\n", " }\r\n", " },\r\n", " \"threads\" : {\r\n", " \"count\" : 114,\r\n", " \"peak_count\" : 118\r\n", " },\r\n", " \"gc\" : {\r\n", " \"collectors\" : {\r\n", " \"young\" : {\r\n", " \"collection_count\" : 3,\r\n", " \"collection_time_in_millis\" : 289\r\n", " },\r\n", " \"old\" : {\r\n", " \"collection_count\" : 1,\r\n", " \"collection_time_in_millis\" : 404\r\n", " }\r\n", " }\r\n", " },\r\n", " \"buffer_pools\" : {\r\n", " \"direct\" : {\r\n", " \"count\" : 50,\r\n", " \"used_in_bytes\" : 397951788,\r\n", " \"total_capacity_in_bytes\" : 397951787\r\n", " },\r\n", " \"mapped\" : {\r\n", " \"count\" : 0,\r\n", " \"used_in_bytes\" : 0,\r\n", " \"total_capacity_in_bytes\" : 0\r\n", " }\r\n", " },\r\n", " \"classes\" : {\r\n", " \"current_loaded_count\" : 9065,\r\n", " \"total_loaded_count\" : 9065,\r\n", " \"total_unloaded_count\" : 0\r\n", " }\r\n", " }\r\n", " },\r\n", " \"kMWX4wbYTC-jYH-UJHmVmg\" : {\r\n", " \"timestamp\" : 1491443504886,\r\n", " \"name\" : \"kMWX4wb\",\r\n", " \"transport_address\" : \"XXX.XXX.XXX.232:9300\",\r\n", " \"host\" : \"XXX.XXX.XXX.232\",\r\n", " \"ip\" : \"XXX.XXX.XXX.232:9300\",\r\n", " \"roles\" : [\r\n", " \"master\",\r\n", " \"data\",\r\n", " \"ingest\"\r\n", " ],\r\n", " \"jvm\" : {\r\n", " \"timestamp\" : 1491443504886,\r\n", " \"uptime_in_millis\" : 162357760,\r\n", " \"mem\" : {\r\n", " \"heap_used_in_bytes\" : 1214249592,\r\n", " \"heap_used_percent\" : 3,\r\n", " \"heap_committed_in_bytes\" : 33128972288,\r\n", " \"heap_max_in_bytes\" : 33128972288,\r\n", " \"non_heap_used_in_bytes\" : 68588872,\r\n", " \"non_heap_committed_in_bytes\" : 72876032,\r\n", " \"pools\" : {\r\n", " \"young\" : {\r\n", " \"used_in_bytes\" : 669358648,\r\n", " \"max_in_bytes\" : 1256259584,\r\n", " \"peak_used_in_bytes\" : 1256259584,\r\n", " \"peak_max_in_bytes\" : 1256259584\r\n", " },\r\n", " \"survivor\" : {\r\n", " \"used_in_bytes\" : 7127584,\r\n", " \"max_in_bytes\" : 157024256,\r\n", " \"peak_used_in_bytes\" : 157024248,\r\n", " \"peak_max_in_bytes\" : 157024256\r\n", " },\r\n", " \"old\" : {\r\n", " \"used_in_bytes\" : 537763360,\r\n", " \"max_in_bytes\" : 31715688448,\r\n", " \"peak_used_in_bytes\" : 537763360,\r\n", " \"peak_max_in_bytes\" : 31715688448\r\n", " }\r\n", " }\r\n", " },\r\n", " \"threads\" : {\r\n", " \"count\" : 115,\r\n", " \"peak_count\" : 120\r\n", " },\r\n", " \"gc\" : {\r\n", " \"collectors\" : {\r\n", " \"young\" : {\r\n", " \"collection_count\" : 3,\r\n", " \"collection_time_in_millis\" : 315\r\n", " },\r\n", " \"old\" : {\r\n", " \"collection_count\" : 1,\r\n", " \"collection_time_in_millis\" : 381\r\n", " }\r\n", " }\r\n", " },\r\n", " \"buffer_pools\" : {\r\n", " \"direct\" : {\r\n", " \"count\" : 63,\r\n", " \"used_in_bytes\" : 501761631,\r\n", " \"total_capacity_in_bytes\" : 501761630\r\n", " },\r\n", " \"mapped\" : {\r\n", " \"count\" : 0,\r\n", " \"used_in_bytes\" : 0,\r\n", " \"total_capacity_in_bytes\" : 0\r\n", " }\r\n", " },\r\n", " \"classes\" : {\r\n", " \"current_loaded_count\" : 9085,\r\n", " \"total_loaded_count\" : 9085,\r\n", " \"total_unloaded_count\" : 0\r\n", " }\r\n", " }\r\n", " },\r\n", " \"LXj8U7-rR4el_IoiVAa8Cg\" : {\r\n", " \"timestamp\" : 1491443504886,\r\n", " \"name\" : \"LXj8U7-\",\r\n", " \"transport_address\" : \"XXX.XXX.XXX.231:9300\",\r\n", " \"host\" : \"XXX.XXX.XXX.231\",\r\n", " \"ip\" : \"XXX.XXX.XXX.231:9300\",\r\n", " \"roles\" : [\r\n", " \"master\",\r\n", " \"data\",\r\n", " \"ingest\"\r\n", " ],\r\n", " \"jvm\" : {\r\n", " \"timestamp\" : 1491443504886,\r\n", " \"uptime_in_millis\" : 162357731,\r\n", " \"mem\" : {\r\n", " \"heap_used_in_bytes\" : 1716765864,\r\n", " \"heap_used_percent\" : 5,\r\n", " \"heap_committed_in_bytes\" : 33128972288,\r\n", " \"heap_max_in_bytes\" : 33128972288,\r\n", " \"non_heap_used_in_bytes\" : 70881272,\r\n", " \"non_heap_committed_in_bytes\" : 75497472,\r\n", " \"pools\" : {\r\n", " \"young\" : {\r\n", " \"used_in_bytes\" : 872606976,\r\n", " \"max_in_bytes\" : 1256259584,\r\n", " \"peak_used_in_bytes\" : 1256259584,\r\n", " \"peak_max_in_bytes\" : 1256259584\r\n", " },\r\n", " \"survivor\" : {\r\n", " \"used_in_bytes\" : 78800,\r\n", " \"max_in_bytes\" : 157024256,\r\n", " \"peak_used_in_bytes\" : 157024256,\r\n", " \"peak_max_in_bytes\" : 157024256\r\n", " },\r\n", " \"old\" : {\r\n", " \"used_in_bytes\" : 844080088,\r\n", " \"max_in_bytes\" : 31715688448,\r\n", " \"peak_used_in_bytes\" : 844080088,\r\n", " \"peak_max_in_bytes\" : 31715688448\r\n", " }\r\n", " }\r\n", " },\r\n", " \"threads\" : {\r\n", " \"count\" : 119,\r\n", " \"peak_count\" : 122\r\n", " },\r\n", " \"gc\" : {\r\n", " \"collectors\" : {\r\n", " \"young\" : {\r\n", " \"collection_count\" : 7,\r\n", " \"collection_time_in_millis\" : 511\r\n", " },\r\n", " \"old\" : {\r\n", " \"collection_count\" : 1,\r\n", " \"collection_time_in_millis\" : 400\r\n", " }\r\n", " }\r\n", " },\r\n", " \"buffer_pools\" : {\r\n", " \"direct\" : {\r\n", " \"count\" : 109,\r\n", " \"used_in_bytes\" : 832603750,\r\n", " \"total_capacity_in_bytes\" : 832603749\r\n", " },\r\n", " \"mapped\" : {\r\n", " \"count\" : 0,\r\n", " \"used_in_bytes\" : 0,\r\n", " \"total_capacity_in_bytes\" : 0\r\n", " }\r\n", " },\r\n", " \"classes\" : {\r\n", " \"current_loaded_count\" : 9292,\r\n", " \"total_loaded_count\" : 9292,\r\n", " \"total_unloaded_count\" : 0\r\n", " }\r\n", " }\r\n", " },\r\n", " \"YKUa3dgMSjSnTvC0UgjI3w\" : {\r\n", " \"timestamp\" : 1491443504887,\r\n", " \"name\" : \"YKUa3dg\",\r\n", " \"transport_address\" : \"XXX.XXX.XXX.234:9300\",\r\n", " \"host\" : \"XXX.XXX.XXX.234\",\r\n", " \"ip\" : \"XXX.XXX.XXX.234:9300\",\r\n", " \"roles\" : [\r\n", " \"data\",\r\n", " \"ingest\"\r\n", " ],\r\n", " \"jvm\" : {\r\n", " \"timestamp\" : 1491443504888,\r\n", " \"uptime_in_millis\" : 162357349,\r\n", " \"mem\" : {\r\n", " \"heap_used_in_bytes\" : 1046726648,\r\n", " \"heap_used_percent\" : 3,\r\n", " \"heap_committed_in_bytes\" : 33128972288,\r\n", " \"heap_max_in_bytes\" : 33128972288,\r\n", " \"non_heap_used_in_bytes\" : 68631448,\r\n", " \"non_heap_committed_in_bytes\" : 72945664,\r\n", " \"pools\" : {\r\n", " \"young\" : {\r\n", " \"used_in_bytes\" : 637270144,\r\n", " \"max_in_bytes\" : 1256259584,\r\n", " \"peak_used_in_bytes\" : 1256259584,\r\n", " \"peak_max_in_bytes\" : 1256259584\r\n", " },\r\n", " \"survivor\" : {\r\n", " \"used_in_bytes\" : 24829928,\r\n", " \"max_in_bytes\" : 157024256,\r\n", " \"peak_used_in_bytes\" : 156694360,\r\n", " \"peak_max_in_bytes\" : 157024256\r\n", " },\r\n", " \"old\" : {\r\n", " \"used_in_bytes\" : 384626576,\r\n", " \"max_in_bytes\" : 31715688448,\r\n", " \"peak_used_in_bytes\" : 384626576,\r\n", " \"peak_max_in_bytes\" : 31715688448\r\n", " }\r\n", " }\r\n", " },\r\n", " \"threads\" : {\r\n", " \"count\" : 114,\r\n", " \"peak_count\" : 118\r\n", " },\r\n", " \"gc\" : {\r\n", " \"collectors\" : {\r\n", " \"young\" : {\r\n", " \"collection_count\" : 3,\r\n", " \"collection_time_in_millis\" : 357\r\n", " },\r\n", " \"old\" : {\r\n", " \"collection_count\" : 1,\r\n", " \"collection_time_in_millis\" : 181\r\n", " }\r\n", " }\r\n", " },\r\n", " \"buffer_pools\" : {\r\n", " \"direct\" : {\r\n", " \"count\" : 45,\r\n", " \"used_in_bytes\" : 363340588,\r\n", " \"total_capacity_in_bytes\" : 363340587\r\n", " },\r\n", " \"mapped\" : {\r\n", " \"count\" : 0,\r\n", " \"used_in_bytes\" : 0,\r\n", " \"total_capacity_in_bytes\" : 0\r\n", " }\r\n", " },\r\n", " \"classes\" : {\r\n", " \"current_loaded_count\" : 9106,\r\n", " \"total_loaded_count\" : 9106,\r\n", " \"total_unloaded_count\" : 0\r\n", " }\r\n", " }\r\n", " },\r\n", " \"hy7ZccVjThKC6O2FXquNZA\" : {\r\n", " \"timestamp\" : 1491443504888,\r\n", " \"name\" : \"hy7ZccV\",\r\n", " \"transport_address\" : \"XXX.XXX.XXX.233:9300\",\r\n", " \"host\" : \"XXX.XXX.XXX.233\",\r\n", " \"ip\" : \"XXX.XXX.XXX.233:9300\",\r\n", " \"roles\" : [\r\n", " \"master\",\r\n", " \"data\",\r\n", " \"ingest\"\r\n", " ],\r\n", " \"jvm\" : {\r\n", " \"timestamp\" : 1491443504888,\r\n", " \"uptime_in_millis\" : 162357766,\r\n", " \"mem\" : {\r\n", " \"heap_used_in_bytes\" : 1204936880,\r\n", " \"heap_used_percent\" : 3,\r\n", " \"heap_committed_in_bytes\" : 33128972288,\r\n", " \"heap_max_in_bytes\" : 33128972288,\r\n", " \"non_heap_used_in_bytes\" : 68259352,\r\n", " \"non_heap_committed_in_bytes\" : 72667136,\r\n", " \"pools\" : {\r\n", " \"young\" : {\r\n", " \"used_in_bytes\" : 644871480,\r\n", " \"max_in_bytes\" : 1256259584,\r\n", " \"peak_used_in_bytes\" : 1256259584,\r\n", " \"peak_max_in_bytes\" : 1256259584\r\n", " },\r\n", " \"survivor\" : {\r\n", " \"used_in_bytes\" : 6667776,\r\n", " \"max_in_bytes\" : 157024256,\r\n", " \"peak_used_in_bytes\" : 157024256,\r\n", " \"peak_max_in_bytes\" : 157024256\r\n", " },\r\n", " \"old\" : {\r\n", " \"used_in_bytes\" : 553397624,\r\n", " \"max_in_bytes\" : 31715688448,\r\n", " \"peak_used_in_bytes\" : 553397624,\r\n", " \"peak_max_in_bytes\" : 31715688448\r\n", " }\r\n", " }\r\n", " },\r\n", " \"threads\" : {\r\n", " \"count\" : 114,\r\n", " \"peak_count\" : 119\r\n", " },\r\n", " \"gc\" : {\r\n", " \"collectors\" : {\r\n", " \"young\" : {\r\n", " \"collection_count\" : 3,\r\n", " \"collection_time_in_millis\" : 327\r\n", " },\r\n", " \"old\" : {\r\n", " \"collection_count\" : 1,\r\n", " \"collection_time_in_millis\" : 414\r\n", " }\r\n", " }\r\n", " },\r\n", " \"buffer_pools\" : {\r\n", " \"direct\" : {\r\n", " \"count\" : 63,\r\n", " \"used_in_bytes\" : 501761631,\r\n", " \"total_capacity_in_bytes\" : 501761630\r\n", " },\r\n", " \"mapped\" : {\r\n", " \"count\" : 0,\r\n", " \"used_in_bytes\" : 0,\r\n", " \"total_capacity_in_bytes\" : 0\r\n", " }\r\n", " },\r\n", " \"classes\" : {\r\n", " \"current_loaded_count\" : 9093,\r\n", " \"total_loaded_count\" : 9093,\r\n", " \"total_unloaded_count\" : 0\r\n", " }\r\n", " }\r\n", " }\r\n", " }\r\n", "}\r\n" ] } ], "source": [ "!curl -XGET \"http://$ES_HOST/_nodes/stats/jvm?pretty\"" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc045e-14ed-11e7-b31a-0242ac12000d", "next": "a6fc0616-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc02a6-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■確認内容**" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc0616-14ed-11e7-b31a-0242ac12000d", "history": [ { "current": "a6fc0616-14ed-11e7-b31a-0242ac12000d", "next": "a6fc07e2-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc045e-14ed-11e7-b31a-0242ac12000d" }, { "current": "a6fc0616-14ed-11e7-b31a-0242ac12000d", "next": "a6fbff22-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc045e-14ed-11e7-b31a-0242ac12000d" }, { "current": "a6fc0616-14ed-11e7-b31a-0242ac12000d", "next": "a6fc07e2-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc045e-14ed-11e7-b31a-0242ac12000d" } ], "next": "a6fbff22-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc045e-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "応答の\"nodes\"部分にNodeごとの状態が出力されます。 \n", "その下の\"jvm\"部分にJVMの状態が出力されます。次を確認してください。\n", "\n", "1. mem/heap_max_in_bytes に表示される最大ヒープサイズ（単位はbyte）がインストール時に指定した通りか？\n", "2. mem/heap_used_percent に表示されるヒープ使用率（単位はパーセント）が100近くになってしまっていないか？\n", "3. gc/collectors/old/collection_count に表示されるGCカウントが、稼働時間に比して大きすぎないか？" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fbff22-14ed-11e7-b31a-0242ac12000d", "history": [ { "current": "a6fbff22-14ed-11e7-b31a-0242ac12000d", "next": "a6fc00e4-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbfd56-14ed-11e7-b31a-0242ac12000d" } ], "next": "a6fc00e4-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc0616-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "### 実行中の重いタスクが無いか？\n", "タスク管理用のAPIを発行し、実行中の重いタスクが無いか確認します。" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc00e4-14ed-11e7-b31a-0242ac12000d", "next": "a6fc02a6-14ed-11e7-b31a-0242ac12000d", "previous": "a6fbff22-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "Task Management APIでクラスタ内のノードで実行中のタスクを確認することができます。 \n", "APIの詳細はElasticsearch Referenceの[Task Management API](https://www.elastic.co/guide/en/elasticsearch/reference/current/tasks.html)を参照してください。\n" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc02a6-14ed-11e7-b31a-0242ac12000d", "next": "a6fc045e-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc00e4-14ed-11e7-b31a-0242ac12000d" }, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\r\n", " \"nodes\" : {\r\n", " \"hy7ZccVjThKC6O2FXquNZA\" : {\r\n", " \"name\" : \"hy7ZccV\",\r\n", " \"transport_address\" : \"XXX.XXX.XXX.233:9300\",\r\n", " \"host\" : \"XXX.XXX.XXX.233\",\r\n", " \"ip\" : \"XXX.XXX.XXX.233:9300\",\r\n", " \"roles\" : [\r\n", " \"master\",\r\n", " \"data\",\r\n", " \"ingest\"\r\n", " ],\r\n", " \"tasks\" : {\r\n", " \"hy7ZccVjThKC6O2FXquNZA:167690\" : {\r\n", " \"node\" : \"hy7ZccVjThKC6O2FXquNZA\",\r\n", " \"id\" : 167690,\r\n", " \"type\" : \"netty\",\r\n", " \"action\" : \"cluster:monitor/tasks/lists[n]\",\r\n", " \"start_time_in_millis\" : 1491443505138,\r\n", " \"running_time_in_nanos\" : 3950610,\r\n", " \"cancellable\" : false,\r\n", " \"parent_task_id\" : \"LXj8U7-rR4el_IoiVAa8Cg:665199\"\r\n", " }\r\n", " }\r\n", " },\r\n", " \"z0-yqIssTTeGPdynW-UEEw\" : {\r\n", " \"name\" : \"z0-yqIs\",\r\n", " \"transport_address\" : \"XXX.XXX.XXX.235:9300\",\r\n", " \"host\" : \"XXX.XXX.XXX.235\",\r\n", " \"ip\" : \"XXX.XXX.XXX.235:9300\",\r\n", " \"roles\" : [\r\n", " \"data\",\r\n", " \"ingest\"\r\n", " ],\r\n", " \"tasks\" : {\r\n", " \"z0-yqIssTTeGPdynW-UEEw:167660\" : {\r\n", " \"node\" : \"z0-yqIssTTeGPdynW-UEEw\",\r\n", " \"id\" : 167660,\r\n", " \"type\" : \"netty\",\r\n", " \"action\" : \"cluster:monitor/tasks/lists[n]\",\r\n", " \"start_time_in_millis\" : 1491443505137,\r\n", " \"running_time_in_nanos\" : 3140838,\r\n", " \"cancellable\" : false,\r\n", " \"parent_task_id\" : \"LXj8U7-rR4el_IoiVAa8Cg:665199\"\r\n", " }\r\n", " }\r\n", " },\r\n", " \"YKUa3dgMSjSnTvC0UgjI3w\" : {\r\n", " \"name\" : \"YKUa3dg\",\r\n", " \"transport_address\" : \"XXX.XXX.XXX.234:9300\",\r\n", " \"host\" : \"XXX.XXX.XXX.234\",\r\n", " \"ip\" : \"XXX.XXX.XXX.234:9300\",\r\n", " \"roles\" : [\r\n", " \"data\",\r\n", " \"ingest\"\r\n", " ],\r\n", " \"tasks\" : {\r\n", " \"YKUa3dgMSjSnTvC0UgjI3w:167664\" : {\r\n", " \"node\" : \"YKUa3dgMSjSnTvC0UgjI3w\",\r\n", " \"id\" : 167664,\r\n", " \"type\" : \"netty\",\r\n", " \"action\" : \"cluster:monitor/tasks/lists[n]\",\r\n", " \"start_time_in_millis\" : 1491443505137,\r\n", " \"running_time_in_nanos\" : 3198063,\r\n", " \"cancellable\" : false,\r\n", " \"parent_task_id\" : \"LXj8U7-rR4el_IoiVAa8Cg:665199\"\r\n", " }\r\n", " }\r\n", " },\r\n", " \"kMWX4wbYTC-jYH-UJHmVmg\" : {\r\n", " \"name\" : \"kMWX4wb\",\r\n", " \"transport_address\" : \"XXX.XXX.XXX.232:9300\",\r\n", " \"host\" : \"XXX.XXX.XXX.232\",\r\n", " \"ip\" : \"XXX.XXX.XXX.232:9300\",\r\n", " \"roles\" : [\r\n", " \"master\",\r\n", " \"data\",\r\n", " \"ingest\"\r\n", " ],\r\n", " \"tasks\" : {\r\n", " \"kMWX4wbYTC-jYH-UJHmVmg:167687\" : {\r\n", " \"node\" : \"kMWX4wbYTC-jYH-UJHmVmg\",\r\n", " \"id\" : 167687,\r\n", " \"type\" : \"netty\",\r\n", " \"action\" : \"cluster:monitor/tasks/lists[n]\",\r\n", " \"start_time_in_millis\" : 1491443505137,\r\n", " \"running_time_in_nanos\" : 4089741,\r\n", " \"cancellable\" : false,\r\n", " \"parent_task_id\" : \"LXj8U7-rR4el_IoiVAa8Cg:665199\"\r\n", " }\r\n", " }\r\n", " },\r\n", " \"LXj8U7-rR4el_IoiVAa8Cg\" : {\r\n", " \"name\" : \"LXj8U7-\",\r\n", " \"transport_address\" : \"XXX.XXX.XXX.231:9300\",\r\n", " \"host\" : \"XXX.XXX.XXX.231\",\r\n", " \"ip\" : \"XXX.XXX.XXX.231:9300\",\r\n", " \"roles\" : [\r\n", " \"master\",\r\n", " \"data\",\r\n", " \"ingest\"\r\n", " ],\r\n", " \"tasks\" : {\r\n", " \"LXj8U7-rR4el_IoiVAa8Cg:665199\" : {\r\n", " \"node\" : \"LXj8U7-rR4el_IoiVAa8Cg\",\r\n", " \"id\" : 665199,\r\n", " \"type\" : \"transport\",\r\n", " \"action\" : \"cluster:monitor/tasks/lists\",\r\n", " \"start_time_in_millis\" : 1491443505133,\r\n", " \"running_time_in_nanos\" : 4885742,\r\n", " \"cancellable\" : false\r\n", " },\r\n", " \"LXj8U7-rR4el_IoiVAa8Cg:665200\" : {\r\n", " \"node\" : \"LXj8U7-rR4el_IoiVAa8Cg\",\r\n", " \"id\" : 665200,\r\n", " \"type\" : \"direct\",\r\n", " \"action\" : \"cluster:monitor/tasks/lists[n]\",\r\n", " \"start_time_in_millis\" : 1491443505135,\r\n", " \"running_time_in_nanos\" : 2679468,\r\n", " \"cancellable\" : false,\r\n", " \"parent_task_id\" : \"LXj8U7-rR4el_IoiVAa8Cg:665199\"\r\n", " }\r\n", " }\r\n", " }\r\n", " }\r\n", "}\r\n" ] } ], "source": [ "!curl -XGET \"http://$ES_HOST/_tasks?pretty\"" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc045e-14ed-11e7-b31a-0242ac12000d", "next": "a6fc0616-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc02a6-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■確認内容**" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc0616-14ed-11e7-b31a-0242ac12000d", "history": [ { "current": "a6fc0616-14ed-11e7-b31a-0242ac12000d", "next": "a6fc07e2-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc045e-14ed-11e7-b31a-0242ac12000d" }, { "current": "a6fc0616-14ed-11e7-b31a-0242ac12000d", "next": "a6fbff22-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc045e-14ed-11e7-b31a-0242ac12000d" }, { "current": "a6fc0616-14ed-11e7-b31a-0242ac12000d", "next": "a6fc07e2-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc045e-14ed-11e7-b31a-0242ac12000d" }, { "current": "a6fc0616-14ed-11e7-b31a-0242ac12000d", "next": "a6fbff22-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc045e-14ed-11e7-b31a-0242ac12000d" } ], "next": "a6fc07e2-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc045e-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "応答の\"nodes\"部分にNodeごとの状態が出力されます。 \n", "その下の\"tasks\"部分に走行しているタスクの種類と状態が出力されます。次を確認してください。\n", "\n", "1. tasksに現れるタスクに想定外の物が無いか？\n", "2. 特にrunning_time_in_nanos（実行時間）の長いものが無いか？\n", "3. 大量のタスクが滞留していないか？" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fc07e2-14ed-11e7-b31a-0242ac12000d", "next": "a6fc09a4-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc0616-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "### インデックスが作成されているか？\n", "日付等のインデックスで、投入したドキュメントが適切に分割・管理されているか確認します。" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc09a4-14ed-11e7-b31a-0242ac12000d", "next": "a6fc0b8e-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc07e2-14ed-11e7-b31a-0242ac12000d" }, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "health status index uuid pri rep docs.count docs.deleted store.size pri.store.size\r\n" ] } ], "source": [ "!curl -XGET http://$ES_HOST/_cat/indices?v" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc0b8e-14ed-11e7-b31a-0242ac12000d", "next": "a6fc0d64-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc09a4-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■確認内容**" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc0d64-14ed-11e7-b31a-0242ac12000d", "next": "a6fc0f30-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc0b8e-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "出力結果から次の内容が確認できれば正常です。\n", "\n", "1. indexの列に表示されるインデックス名が想定どおりか？ \n", "例えば日付インデックスであれば、日付形式の文字列が含まれるか？\n", "2. 各インデックスの\"health\" 列が \"green\" かどうか？\n", " - **green** \n", " ：正常です。\n", " - **yellow** \n", " ：Primary Shardは割り当てられていますが、Replica Shardが割り当てられていません。\n", " - **red** \n", " ：特定のShardがクラスタに割り当てられていません" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fc0f30-14ed-11e7-b31a-0242ac12000d", "next": "a6fc1106-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc0d64-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "### Shard数が設定通りか？\n", "インデックスごとに設定されるShard数が設定した（はずの）値になっているか確認します。" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc1106-14ed-11e7-b31a-0242ac12000d", "next": "a6fc12d2-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc0f30-14ed-11e7-b31a-0242ac12000d" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{ }\r\n" ] } ], "source": [ "!curl -XGET http://$ES_HOST/$INDEX/_settings?pretty" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc12d2-14ed-11e7-b31a-0242ac12000d", "next": "a6fc1494-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc1106-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "インデックスごとに次のような内容が出力されます。" ] }, { "cell_type": "raw", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc1494-14ed-11e7-b31a-0242ac12000d", "next": "a6fc1656-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc12d2-14ed-11e7-b31a-0242ac12000d" } }, "source": [ " \"meteorological-data-2015.02\" : {\n", " \"settings\" : {\n", " \"index\" : {\n", " \"creation_date\" : \"1489559717930\",\n", " \"number_of_shards\" : \"3\",\n", " （略）" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc1656-14ed-11e7-b31a-0242ac12000d", "next": "a6fc1818-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc1494-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■確認内容**" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc1818-14ed-11e7-b31a-0242ac12000d", "next": "a6fc19da-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc1656-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "\"number_of_shards\"の数について次を確認してください。\n", "\n", "- 最初にセットアップしたShard数と一致しているか？\n", "- Shard数を設定したことが無いのであれば、現状のData Node数と比べて小さくないか？" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "lc_cell_meme": { "current": "a6fc19da-14ed-11e7-b31a-0242ac12000d", "next": "a6fc1b9c-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc1818-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "## 症状と対処\n", "診断の章で異常があった場合、本章の内容に従い対処します。" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc1b9c-14ed-11e7-b31a-0242ac12000d", "next": "a6fc1d72-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc19da-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**先頭から順に**症状を確認してください。 \n", "症状に対処したら、いったん[診断](#診断)の章に戻り、コマンドを**全て再実行し直してください。**\n", "\n", "これは、本章の各節にある対処が、その前までの節に症状が出ていないことを前提に記述しているためです。 \n", "※例えば「クラスタ状態が\"green\"でない」節の対処は、その前にある「サーバマシンが稼働していない」節では問題が無いことを前提に記述しています。よってサーバが起動しているかの確認はしていません。" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fc1d72-14ed-11e7-b31a-0242ac12000d", "next": "a6fc1f3e-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc1b9c-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "### サーバマシンが稼働していない\n", "**■項目**：[サーバマシンが稼働しているか？](#サーバマシンが稼働しているか？) \n", "**■症状**：pingの応答がないマシンが存在する" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc1f3e-14ed-11e7-b31a-0242ac12000d", "next": "a6fc2100-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc1d72-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■原因**： \n", "ハードウェアの障害や誤操作でサーバが停止している。 \n", "または、サーバは稼働しているが、通信経路に問題がありpingが到達しない。" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc2100-14ed-11e7-b31a-0242ac12000d", "next": "a6fc22d6-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc1f3e-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■対処**： \n", "- マシンが停止している場合は稼働させてください。 \n", "なんらかのハードウェア障害が発生している場合は、それを解決してください。\n", "- マシンが停止していない場合は、通信経路の状態を確認し、問題を是正してください。" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fc22d6-14ed-11e7-b31a-0242ac12000d", "next": "a6fc24ca-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc2100-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "### プロセスが稼働していない\n", "**■項目**：[プロセスが稼働しているか？](#プロセスが稼働しているか？) \n", "**■症状**：稼働していないElasticsearchプロセスが存在する" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc24ca-14ed-11e7-b31a-0242ac12000d", "next": "a6fc268c-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc22d6-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■原因**： \n", "なんらかのエラー発生、または誤操作によりプロセスがダウンしている。もしくは、そもそも起動していない。" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc268c-14ed-11e7-b31a-0242ac12000d", "next": "a6fc2902-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc24ca-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■対処**： \n", "- 「[エラーログが無いか？](#エラーログが無いか？)」でエラーが発生していない場合は、クラスタを起動してください。 \n", "次のコマンドでクラスタを起動できます。" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc2902-14ed-11e7-b31a-0242ac12000d", "next": "a6fc2af6-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc268c-14ed-11e7-b31a-0242ac12000d" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/usr/local/lib/python2.7/dist-packages/cffi/model.py:525: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'\n", " % self._get_c_name())\n", "\n", "PLAY [all] *********************************************************************\n", "\n", "TASK [setup] *******************************************************************\n", "\u001b[0;32mok: [XXX.XXX.XXX.231]\u001b[0m\n", "\u001b[0;32mok: [XXX.XXX.XXX.232]\u001b[0m\n", "\u001b[0;32mok: [XXX.XXX.XXX.233]\u001b[0m\n", "\u001b[0;32mok: [XXX.XXX.XXX.235]\u001b[0m\n", "\u001b[0;32mok: [XXX.XXX.XXX.234]\u001b[0m\n", "\n", "TASK [start elasticsearch] *****************************************************\n", "\u001b[0;32mok: [XXX.XXX.XXX.231]\u001b[0m\n", "\u001b[0;32mok: [XXX.XXX.XXX.232]\u001b[0m\n", "\u001b[0;32mok: [XXX.XXX.XXX.233]\u001b[0m\n", "\u001b[0;32mok: [XXX.XXX.XXX.235]\u001b[0m\n", "\u001b[0;32mok: [XXX.XXX.XXX.234]\u001b[0m\n", "\n", "PLAY RECAP *********************************************************************\n", "\u001b[0;32mXXX.XXX.XXX.231\u001b[0m : \u001b[0;32mok=2 \u001b[0m changed=0 unreachable=0 failed=0 \n", "\u001b[0;32mXXX.XXX.XXX.232\u001b[0m : \u001b[0;32mok=2 \u001b[0m changed=0 unreachable=0 failed=0 \n", "\u001b[0;32mXXX.XXX.XXX.233\u001b[0m : \u001b[0;32mok=2 \u001b[0m changed=0 unreachable=0 failed=0 \n", "\u001b[0;32mXXX.XXX.XXX.234\u001b[0m : \u001b[0;32mok=2 \u001b[0m changed=0 unreachable=0 failed=0 \n", "\u001b[0;32mXXX.XXX.XXX.235\u001b[0m : \u001b[0;32mok=2 \u001b[0m changed=0 unreachable=0 failed=0 \n", "\n" ] } ], "source": [ "!ansible-playbook playbooks/start_cluster.yml -i ./hosts -u $USER --private-key=$KEYPATH" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc2af6-14ed-11e7-b31a-0242ac12000d", "next": "a6fc2ccc-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc2902-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "- エラーログが出力されている場合は「[エラーが出力されている](#エラーが出力されている)」の対処に進んでください。" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fc2ccc-14ed-11e7-b31a-0242ac12000d", "next": "a6fc2e98-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc2af6-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "### ディスクの使用量に余裕がない\n", "**■項目**：[ディスクの使用量に余裕があるか？](#ディスクの使用量に余裕があるか？) \n", "**■症状**：ディスクの残量が少ない" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc2e98-14ed-11e7-b31a-0242ac12000d", "next": "a6fc3064-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc2ccc-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■原因**： \n", "Elasticsearchのプロセスか、それ以外のプロセスによる使用量の増加。" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc3064-14ed-11e7-b31a-0242ac12000d", "next": "a6fc3226-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc2e98-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■対処**： \n", "どのプロセスがディスクサイズを占有しているのか確認します。" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc3226-14ed-11e7-b31a-0242ac12000d", "next": "a6fc33e8-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc3064-14ed-11e7-b31a-0242ac12000d" }, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/usr/local/lib/python2.7/dist-packages/cffi/model.py:525: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'\n", " % self._get_c_name())\n", "\u001b[0;31mXXX.XXX.XXX.232 | FAILED | rc=1 >>\n", "22\t/root\n", "1\t/mnt\n", "1\t/srv\n", "1\t/media\n", "0\t/proc\n", "23\t/etc\n", "6\t/opt\n", "104\t/var\n", "1983\t/usr\n", "1\t/service\n", "0\t/dev\n", "1\t/lost+found\n", "0\t/sys\n", "73\t/run\n", "223\t/tmp\n", "108\t/boot\n", "1\t/home\n", "2541\t/du: cannot access ‘/proc/13688/task/13688/fd/4’: No such file or directory\n", "du: cannot access ‘/proc/13688/task/13688/fdinfo/4’: No such file or directory\n", "du: cannot access ‘/proc/13688/fd/4’: No such file or directory\n", "du: cannot access ‘/proc/13688/fdinfo/4’: No such file or directory\n", "\u001b[0m\n", "\u001b[0;31mXXX.XXX.XXX.231 | FAILED | rc=1 >>\n", "1\t/mnt\n", "1\t/home\n", "1\t/media\n", "104\t/var\n", "6\t/opt\n", "0\t/proc\n", "108\t/boot\n", "22\t/root\n", "73\t/run\n", "223\t/tmp\n", "0\t/sys\n", "1\t/lost+found\n", "1\t/service\n", "23\t/etc\n", "0\t/dev\n", "1983\t/usr\n", "1\t/srv\n", "2542\t/du: cannot access ‘/proc/16391/task/16391/fd/4’: No such file or directory\n", "du: cannot access ‘/proc/16391/task/16391/fdinfo/4’: No such file or directory\n", "du: cannot access ‘/proc/16391/fd/4’: No such file or directory\n", "du: cannot access ‘/proc/16391/fdinfo/4’: No such file or directory\n", "\u001b[0m\n", "\u001b[0;31mXXX.XXX.XXX.233 | FAILED | rc=1 >>\n", "1\t/mnt\n", "1983\t/usr\n", "108\t/boot\n", "22\t/root\n", "1\t/media\n", "1\t/lost+found\n", "0\t/proc\n", "223\t/tmp\n", "0\t/sys\n", "1\t/home\n", "0\t/dev\n", "73\t/run\n", "103\t/var\n", "1\t/service\n", "6\t/opt\n", "23\t/etc\n", "1\t/srv\n", "2541\t/du: cannot access ‘/proc/12908/task/12908/fd/4’: No such file or directory\n", "du: cannot access ‘/proc/12908/task/12908/fdinfo/4’: No such file or directory\n", "du: cannot access ‘/proc/12908/fd/4’: No such file or directory\n", "du: cannot access ‘/proc/12908/fdinfo/4’: No such file or directory\n", "\u001b[0m\n", "\u001b[0;31mXXX.XXX.XXX.235 | FAILED | rc=1 >>\n", "108\t/boot\n", "104\t/var\n", "1\t/media\n", "1983\t/usr\n", "65\t/run\n", "0\t/proc\n", "0\t/dev\n", "1\t/service\n", "1\t/srv\n", "6\t/opt\n", "1\t/mnt\n", "22\t/root\n", "1\t/home\n", "23\t/etc\n", "1\t/lost+found\n", "0\t/sys\n", "223\t/tmp\n", "2533\t/du: cannot access ‘/proc/2457/task/2457/fd/4’: No such file or directory\n", "du: cannot access ‘/proc/2457/task/2457/fdinfo/4’: No such file or directory\n", "du: cannot access ‘/proc/2457/fd/4’: No such file or directory\n", "du: cannot access ‘/proc/2457/fdinfo/4’: No such file or directory\n", "\u001b[0m\n", "\u001b[0;31mXXX.XXX.XXX.234 | FAILED | rc=1 >>\n", "0\t/dev\n", "65\t/run\n", "105\t/var\n", "1\t/srv\n", "0\t/sys\n", "1\t/media\n", "425\t/tmp\n", "23\t/etc\n", "1\t/mnt\n", "1\t/lost+found\n", "0\t/proc\n", "6\t/opt\n", "108\t/boot\n", "1\t/service\n", "1\t/home\n", "22\t/root\n", "2208\t/usr\n", "2961\t/du: cannot access ‘/proc/3788/task/3788/fd/4’: No such file or directory\n", "du: cannot access ‘/proc/3788/task/3788/fdinfo/4’: No such file or directory\n", "du: cannot access ‘/proc/3788/fd/4’: No such file or directory\n", "du: cannot access ‘/proc/3788/fdinfo/4’: No such file or directory\n", "\u001b[0m\n" ] } ], "source": [ "!ansible all -m shell -a \"sudo du -m --max-depth=1 /\" -i ./hosts -u $USER --private-key=$KEYPATH\n", "# MB単位で、ルートから1階層のみ計測しています。必要に応じてオプションを変更してください。" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc33e8-14ed-11e7-b31a-0242ac12000d", "history": [ { "current": "a6fc33e8-14ed-11e7-b31a-0242ac12000d", "next": "a6fc35aa-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc3226-14ed-11e7-b31a-0242ac12000d" } ], "next": "08ecb21e-182f-11e7-b31a-0242ac12000d", "previous": "a6fc3226-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "- Elasticsearchのデフォルトの設定では、ディスク使用率が85%を越えると、そのノードに新たなシャードを割り当てません。そのため、ディスク使用率が85%を越えないようにしてください。\n", "- Elasticsearch以外が占有している場合は、そちらのデータ削減を検討してください。\n", "- Elasticsearchが占有している場合は、不要になったインデックス等の削除を検討してください。\n", "- 削除できない場合は、Data Nodeを追加してください。 \n", "[01_50_Add_Data_Node.ipynb](01_50_Add_Data_Node.ipynb)を実施してください。\n" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "08ecb21e-182f-11e7-b31a-0242ac12000d", "next": "08ecb66a-182f-11e7-b31a-0242ac12000d", "previous": "a6fc33e8-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "- なおElasticsearchには、ディスク使用率が100%にならないようwater markを設け、状態に応じてShardを割り振る機能があります。\n", " - 詳細はElasticsearch Referenceの[Disk-based Shard Allocation](https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html)を参照してください。\n", " - 例えばディスク使用率が80%を超えた場合、そのNodeには新しいShardを割り当てなくし、 \n", "ディスクの空き領域が50GBあった場合、Shardを再配置するelasticsearch.ymlの設定は次の通りです。\n" ] }, { "cell_type": "raw", "metadata": { "hidden": true, "lc_cell_meme": { "current": "08ecb66a-182f-11e7-b31a-0242ac12000d", "next": "a6fc35aa-14ed-11e7-b31a-0242ac12000d", "previous": "08ecb21e-182f-11e7-b31a-0242ac12000d" } }, "source": [ "cluster.routing.allocation.disk.watermark.low: \"80%\"\n", "cluster.routing.allocation.disk.watermark.high: \"50gb\"\n", "cluster.info.update.interval: \"1m\"" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fc35aa-14ed-11e7-b31a-0242ac12000d", "history": [ { "current": "a6fc35aa-14ed-11e7-b31a-0242ac12000d", "next": "a6fc3776-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc33e8-14ed-11e7-b31a-0242ac12000d" } ], "next": "a6fc3776-14ed-11e7-b31a-0242ac12000d", "previous": "08ecb66a-182f-11e7-b31a-0242ac12000d" } }, "source": [ "### クラスタと通信できない\n", "**■項目**：[クラスタと通信できるか？](#クラスタと通信できるか？) \n", "**■症状**：APIの応答がない、または異常な内容" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc3776-14ed-11e7-b31a-0242ac12000d", "next": "a6fc3938-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc35aa-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■原因**： \n", "稼働しているElasticsearchに外部通信が許可されていない。" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc3938-14ed-11e7-b31a-0242ac12000d", "next": "a6fc3afa-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc3776-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■対処**： \n", "- セキュリティの設定等で、クラスタと外部との通信が阻害されていないか確認してください。 \n", "Elasticsearchは外部との通信用に**9200**番ポートを利用します。 \n", "9200番ポートが許可されているか確認してください。\n", " - 例えば、AWSの場合はセキュリティグループの設定を確認してください。 \n", " 次のコマンドで \"FromPort\"と\"ToPort\"の値域に**9200**を含むブロックを確認してください。 \n", " そのブロックの\"IpRanges\"配下に通信先のサーバが登録されていなければ、外部からの通信が遮断されます。" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc3afa-14ed-11e7-b31a-0242ac12000d", "next": "a6fc3cbc-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc3938-14ed-11e7-b31a-0242ac12000d" }, "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"SecurityGroups\": [\n", " ]\n", "}\n" ] } ], "source": [ "!aws ec2 describe-security-groups\n", "\n", "#VPCのIDがわかっている場合は次のコマンドで表示を絞り込めます。\n", "#VPC_ID='vpc-bd3b81d9'\n", "#!aws ec2 describe-security-groups --filters \"Name=vpc-id,Values=$VPC_ID\"" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fc3cbc-14ed-11e7-b31a-0242ac12000d", "next": "a6fc3e7e-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc3afa-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "### クラスタにNodeが参加していない\n", "**■項目**：[クラスタに全Nodeが参加しているか？](#クラスタに全Nodeが参加しているか？) \n", "**■症状**：\"status\"がgreenでない、または \"number_of_data_nodes\"の数が、起動しているはずのData Node数より少ない" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc3e7e-14ed-11e7-b31a-0242ac12000d", "next": "a6fc4040-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc3cbc-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■原因**： \n", "クラスタ内部の通信が阻害されており、検出できないNodeがある。 \n", "※ここまでの対処で、サーバもプロセスも稼働しており、pingも正常に応答している前提" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc4040-14ed-11e7-b31a-0242ac12000d", "next": "a6fc4216-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc3e7e-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■対処**： \n", "- セキュリティの設定等で、クラスタ内部の通信が阻害されていないか確認してください。 \n", "Elasticsearchはクラスタ内部の通信用に**9300**番ポートを利用します。9300番ポートが許可されているか確認してください。\n", " - 例えばAWSの場合はセキュリティグループの設定を確認してください。 \n", " 次のコマンドで \"FromPort\"と\"ToPort\"の値域に**9300**を含むブロックを確認してください。 \n", " そのブロックの\"IpRanges\"配下に通信先のサーバが登録されていなければ、内部の通信が遮断されます。" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc4216-14ed-11e7-b31a-0242ac12000d", "next": "a6fc43d8-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc4040-14ed-11e7-b31a-0242ac12000d" }, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"SecurityGroups\": [\n", " ]\n", "}\n" ] } ], "source": [ "!aws ec2 describe-security-groups\n", "\n", "#VPCのIDがわかっている場合は次のコマンドで表示を絞り込めます。\n", "#VPC_ID='vpc-bd3b81d9'\n", "#!aws ec2 describe-security-groups --filters \"Name=vpc-id,Values=$VPC_ID\"" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fc43d8-14ed-11e7-b31a-0242ac12000d", "next": "a6fc459a-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc4216-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "### エラーログが出力されている（ディスクフル）\n", "**■項目**：[エラーログが無いか？](#エラーログが無いか？) \n", "**■症状**：ログに次のようなディスクフルを示すメッセージが出力されている" ] }, { "cell_type": "raw", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc459a-14ed-11e7-b31a-0242ac12000d", "next": "a6fc475c-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc43d8-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "[cluster.routing.allocation.decider] [Milan] low disk watermark [15%] exceeded on [DZqnmWIZRpapZY_TPkkMBw][Milan] free: 58.6gb[12.6%], replicas will not be assigned to this node" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc475c-14ed-11e7-b31a-0242ac12000d", "next": "a6fc4928-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc459a-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■原因**： \n", "Elasticsearchのプロセスか、それ以外のプロセスによる使用量の増加。" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc4928-14ed-11e7-b31a-0242ac12000d", "next": "a6fc4aea-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc475c-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■対処**： \n", "適切に「[ディスクの使用量に余裕がない](#ディスクの使用量に余裕がない)」の節での対処を実施している場合は、おそらくこの問題は既に解決しています。 \n", "「[ディスクの使用量に余裕がない](#ディスクの使用量に余裕がない)」の対処後、同様のログが出力されるか確認してください。" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fc4aea-14ed-11e7-b31a-0242ac12000d", "next": "a6fc4cac-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc4928-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "### エラーログが出力されている（OSのメモリマップ設定誤り）\n", "**■項目**：[エラーログが無いか？](#エラーログが無いか？) \n", "**■症状**：ログに次のようなOSのメモリマップ設定誤りを示すメッセージが出力されている" ] }, { "cell_type": "raw", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc4cac-14ed-11e7-b31a-0242ac12000d", "next": "a6fc4e64-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc4aea-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "publish_address {192.168.0.1:9300}, bound_addresses {[::]:9300}\n", "Exception in thread \"main\" java.lang.RuntimeException: bootstrap checks failed\n", "initial heap size [268435456] not equal to maximum heap size [2147483648]; this can cause resize pauses and prevents mlockall from locking the entire heap\n", "max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc4e64-14ed-11e7-b31a-0242ac12000d", "next": "a6fc5026-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc4cac-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■原因**： \n", "カーネルパラメータvm.max_map_countの設定が誤っています。" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc5026-14ed-11e7-b31a-0242ac12000d", "next": "a6fc51f2-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc4e64-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■対処**： \n", "[01_04_Install.ipynbの「Elasticsearchの設定」](01_04_Install.ipynb#Elasticsearchの設定)にある「カーネルパラメータを変更」のコマンドを実行し、 \n", "カーネルパラメータvm.max_map_countを設定してください。" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc51f2-14ed-11e7-b31a-0242ac12000d", "next": "a6fc53aa-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc5026-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "### Slowログが出力されている\n", "**■項目**：[Slowログが無いか？](#Slowログが無いか？) \n", "**■症状**：Slowログが出力されている" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc53aa-14ed-11e7-b31a-0242ac12000d", "next": "a6fc556c-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc51f2-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■原因/対処**： \n", "サーバのスペック不足、サーバ数の不足、ヒープの枯渇、インデックスの作成漏れ等が考えられます。" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc556c-14ed-11e7-b31a-0242ac12000d", "next": "a6fc572e-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc53aa-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "[診断](#診断)結果以外に、**Profile API**でクエリの実行状況を調査できます。 \n", "Profile APIは、通常の検索query内に \n", "**\"profile\": true ** \n", "のパラメータを追加するだけで動作します。 \n", "\n", "例えば2015年1月分の東京のデータを検索するクエリでProfileを有効にするには次のようにします。" ] }, { "cell_type": "code", "execution_count": 58, "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc572e-14ed-11e7-b31a-0242ac12000d", "next": "a6fc5904-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc556c-14ed-11e7-b31a-0242ac12000d" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ " % Total % Received % Xferd Average Speed Time Time Time Current\n", " Dload Upload Total Spent Left Speed\n", "\r", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:30 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:31 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:32 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:33 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:34 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:35 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:36 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:37 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:38 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:39 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:40 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:41 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:42 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:43 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:44 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:45 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:46 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:47 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:48 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:49 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:50 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:51 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:52 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:53 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:54 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:55 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:56 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:57 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:58 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:00:59 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:00 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:01 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:02 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:03 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:04 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:05 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:06 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:07 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:08 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:09 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:10 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:11 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:12 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:13 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:14 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:15 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:16 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:17 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:18 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:19 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:20 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:21 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:22 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:23 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:24 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:25 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:26 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:27 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:28 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:29 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:30 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:31 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:32 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:33 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:34 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:35 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:36 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:37 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:38 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:39 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:40 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:41 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:42 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:43 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:44 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:45 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:46 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:47 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:48 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:49 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:50 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:51 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:52 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:53 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:54 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:55 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:56 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:57 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:58 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:01:59 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:02:00 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:02:01 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:02:02 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:02:03 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:02:04 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:02:05 --:--:-- 0\r", " 0 0 0 0 0 0 0 0 --:--:-- 0:02:06 --:--:-- 0curl: (7) Failed to connect to ec2-54-238-216-212.ap-northeast-1.compute.amazonaws.com port 9200: Connection timed out\n" ] } ], "source": [ "%%bash\n", "curl -XGET \"http://$ES_HOST/meteorological-data-2015.01/_search?pretty\" -H 'Content-Type:application/json' -d @- << EOF\n", "{\n", " \"profile\": true,\n", " \"query\": {\n", " \"term\" : { \"location\" : \"tokyo\" } \n", " }\n", "}\n", "EOF" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc5904-14ed-11e7-b31a-0242ac12000d", "next": "a6fc5ac6-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc572e-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "Profile APIの詳細は、Elasticsearch Referenceの[Profile API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-profile.html)を参照してください。\n", "\n", "ただし、Profileの結果は時間が立つと変化してしまうため注意が必要です。 \n", "これは、データがどのshardに入るのかは、データのハッシュ値によって変化するため、 \n", "クラスタ状況によってはデータの再配置が起きて、Shard間でデータの移動が行われる可能性があるためです。 \n", "（例：ある特定ノードで残容量がwater markを超過したためにshardの移動が発生した。など) \n" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc5ac6-14ed-11e7-b31a-0242ac12000d", "next": "a6fc5c88-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc5904-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**[診断](#診断)結果とProfile結果を踏まえ、以下の節の内容を確認して対処してください。**" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fc5c88-14ed-11e7-b31a-0242ac12000d", "next": "a6fc5e4a-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc5ac6-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "#### クラスタ内のサーバのCPU・メモリが枯渇している\n", "**■項目**：[クラスタ内のサーバのCPU・メモリが枯渇していないか？](#クラスタ内のサーバのCPU・メモリが枯渇していないか？) \n", "**■症状**：CPUかメモリのどちらかが枯渇している" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc5e4a-14ed-11e7-b31a-0242ac12000d", "next": "a6fc6002-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc5c88-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■原因**： \n", "サーバで動作しているElasticsearchかそれ以外のプロセスに対し、サーバのスペックが足りていない。" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc6002-14ed-11e7-b31a-0242ac12000d", "next": "a6fc61d8-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc5e4a-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■対処**： \n", "- 他のプロセスによるCPUやメモリの使用率が大きい場合は、プロセスの停止か別サーバへの移動を検討してください。\n", "- ElasticsearchのCPU使用率が大きい場合は、サーバへのメモリ追加か、サーバ台数の追加を検討してください。\n", " - AWSのインスタンスサイズを変更する場合は、AWS公式ページの[インスタンスのサイズ変更](http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/ec2-instance-resize.html)を参照してください。\n", " - サーバ台数の追加は[01_50_Add_Data_Node.ipynb](01_50_Add_Data_Node.ipynb)を実施してください。\n", "- Elasticsearchのメモリ使用量は、Javaの起動オプションに設定した**\"-Xmx\"**（最大ヒープサイズ）の値を超えることはありません。 \n", "この使用量が想定外であった場合は次を確認してください。\n", " 1. 実際に稼働しているプロセスのオプションを確認します。 \n", " 診断の章の[プロセスが稼働しているか？](#プロセスが稼働しているか？)の節で実施したpsコマンドの出力に、Javaプロセスの引数が出ているはずです。\"-Xmx\" オプションの内容が想定通りか確認してください。\n", " 2. 想定と異なる場合、Elasticsearch用のJava起動オプションの設定が正しいか確認してください。次のコマンドで確認できます。 \n", " 再設定が必要な場合は[01_04_Install.ipynbの「Elasticsearchの設定」](01_04_Install.ipynb#Elasticsearchの設定)を実施してください。" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc61d8-14ed-11e7-b31a-0242ac12000d", "next": "a6fc63a4-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc6002-14ed-11e7-b31a-0242ac12000d" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/usr/local/lib/python2.7/dist-packages/cffi/model.py:525: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'\n", " % self._get_c_name())\n", "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS | rc=0 >>\n", "-Xmx31g\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.233 | SUCCESS | rc=0 >>\n", "-Xmx31g\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.231 | SUCCESS | rc=0 >>\n", "-Xmx31g\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.234 | SUCCESS | rc=0 >>\n", "-Xmx31g\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.235 | SUCCESS | rc=0 >>\n", "-Xmx31g\n", "\u001b[0m\n" ] } ], "source": [ "!ansible all -m shell -a \"sudo grep -e '^-Xmx' /etc/elasticsearch/jvm.options\" -i ./hosts -u $USER --private-key=$KEYPATH" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fc63a4-14ed-11e7-b31a-0242ac12000d", "next": "a6fc6566-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc61d8-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "#### ヒープが枯渇している\n", "**■項目**：[ヒープが枯渇していないか？](#ヒープが枯渇していないか？) \n", "**■症状**：ヒープの使用量が100%近くに達している、またはGC数が大きい" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc6566-14ed-11e7-b31a-0242ac12000d", "next": "a6fc6746-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc63a4-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■原因**： \n", "ヒープサイズの設定漏れ、またはクラスタの負荷状況にサーバ台数が及んでいない。" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc6746-14ed-11e7-b31a-0242ac12000d", "next": "a6fc6908-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc6566-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■対処**： \n", "- ヒープサイズの設定が想定通りか確認してください。\n", " 1. 実際に稼働しているプロセスのオプションを確認します。 \n", " 診断の章の[プロセスが稼働しているか？](#プロセスが稼働しているか？)の節で実施したpsコマンドの出力に、Javaプロセスの引数が出ているはずです。\"-Xmx\" オプションの内容が想定通りか確認してください。\n", " 2. 想定と異なる場合、Elasticsearch用のJava起動オプションの設定が正しいか確認してください。次のコマンドで確認できます。 \n", " 再設定が必要な場合は[01_04_Install.ipynbの「Elasticsearchの設定」](01_04_Install.ipynb#Elasticsearchの設定)を実施してください。\n" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc6908-14ed-11e7-b31a-0242ac12000d", "next": "a6fc6b1a-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc6746-14ed-11e7-b31a-0242ac12000d" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/usr/local/lib/python2.7/dist-packages/cffi/model.py:525: UserWarning: 'point_conversion_form_t' has no values explicitly defined; guessing that it is equivalent to 'unsigned int'\n", " % self._get_c_name())\n", "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS | rc=0 >>\n", "-Xmx31g\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.233 | SUCCESS | rc=0 >>\n", "-Xmx31g\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.231 | SUCCESS | rc=0 >>\n", "-Xmx31g\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.234 | SUCCESS | rc=0 >>\n", "-Xmx31g\n", "\u001b[0m\n", "\u001b[0;32mXXX.XXX.XXX.235 | SUCCESS | rc=0 >>\n", "-Xmx31g\n", "\u001b[0m\n" ] } ], "source": [ "!ansible all -m shell -a \"sudo grep -e '^-Xmx' /etc/elasticsearch/jvm.options\" -i ./hosts -u $USER --private-key=$KEYPATH" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc6b1a-14ed-11e7-b31a-0242ac12000d", "next": "a6fc6d04-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc6908-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "- ヒープサイズが想定通りで、それでもヒープの使用量が100%に近くなってしまっている場合は、Data Nodeの追加を検討してください。 \n", "[01_50_Add_Data_Node.ipynb](01_50_Add_Data_Node.ipynb)を実施してください。" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fc6d04-14ed-11e7-b31a-0242ac12000d", "next": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc6b1a-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "#### 実行中の重いタスクがある\n", "**■項目**：[実行中の重いタスクが無いか？](#実行中の重いタスクが無いか？) \n", "**■症状**：想定外のタスクや実行時間の長いタスクがある、または大量のタスクが滞留している" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d", "next": "a6fc7088-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc6d04-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■原因**： \n", "他のタスクがマシンリソースを圧迫し、クエリの応答を遅くしている。" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc7088-14ed-11e7-b31a-0242ac12000d", "history": [ { "current": "a6fc7088-14ed-11e7-b31a-0242ac12000d", "next": "a6fc724a-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d" }, { "current": "a6fc7088-14ed-11e7-b31a-0242ac12000d", "next": "a6fba2a2-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d" }, { "current": "a6fc7088-14ed-11e7-b31a-0242ac12000d", "next": "a6fc724a-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d" } ], "next": "a6fc6d04-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■対処**： \n", "- 呼び出されているタスクの種類や量が想定外の場合、Elasticsearchを呼び出している側のアプリケーションの挙動を確認してください。\n", "- 他のタスクがindexingであった場合、マシンへの負荷が高まり応答に時間がかかる場合があります。詳しくはElasticsearch Referenceの[Indexing Performance Tips](https://www.elastic.co/guide/en/elasticsearch/guide/2.x/indexing-performance.html)を参照してください。\n", "- タスクがキャンセル処理をサポートしている場合、APIによるキャンセルが可能です。詳しくはElasticsearch Referenceの[Task Cancellation](https://www.elastic.co/guide/en/elasticsearch/reference/current/tasks.html#_task_cancellation)を参照してください。\n" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fc6d04-14ed-11e7-b31a-0242ac12000d", "history": [ { "current": "a6fc6d04-14ed-11e7-b31a-0242ac12000d", "next": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc6b1a-14ed-11e7-b31a-0242ac12000d" } ], "next": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc7088-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "#### インデックスが作成されていない\n", "**■項目**：[インデックスが作成されているか？](#インデックスが作成されているか？) \n", "**■症状**：インデックスが作成されていない" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d", "next": "a6fc7088-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc6d04-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■原因**： \n", "インデックスでデータが管理されていないため、不要な範囲の検索が走行しており、遅くなっている。" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc7088-14ed-11e7-b31a-0242ac12000d", "history": [ { "current": "a6fc7088-14ed-11e7-b31a-0242ac12000d", "next": "a6fc724a-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d" }, { "current": "a6fc7088-14ed-11e7-b31a-0242ac12000d", "next": "a6fba2a2-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d" } ], "next": "a6fc724a-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc6ec6-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■対処**： \n", "- インデックスを設定し、インデックス名が想定通りになっているか（例えば日付インデックスであればインデックス名に日付文字列が入るか）を確認してください。 \n", "インデックスについては[05_Indexing.ipynb](05_Indexing.ipynb)を参照してください。" ] }, { "cell_type": "markdown", "metadata": { "heading_collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fc724a-14ed-11e7-b31a-0242ac12000d", "next": "a6fc7416-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc7088-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "#### Shard数が足りていない\n", "**■項目**：[Shard数が設定通りか？](#Shard数が設定通りか？) \n", "**■症状**：Shard数が設定したはずの数より不足している" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc7416-14ed-11e7-b31a-0242ac12000d", "next": "a6fc75ce-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc724a-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■原因**： \n", "テンプレートによるShard数の設定漏れ。またはData Nodeを増やしたときにShard数も増やす処理を忘れている。 \n", "結果、存在するData Node群での検索処理が十分に並列化できず、遅くなっている。" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc75ce-14ed-11e7-b31a-0242ac12000d", "next": "a6fc7790-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc7416-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "**■対処**： \n", "- 作成済みのインデックスのシャード数は変更することができません。 \n", "シャード数を増やしたインデックスを作成した後、作成したインデックスにデータを移動させてください。(reindex) \n", "次のコマンドを実行して新しいインデックスのシャード数を設定してください。" ] }, { "cell_type": "raw", "metadata": { "collapsed": true, "hidden": true, "lc_cell_meme": { "current": "a6fc7790-14ed-11e7-b31a-0242ac12000d", "next": "a6fc795c-14ed-11e7-b31a-0242ac12000d", "previous": "a6fc75ce-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "%%bash\n", "curl -XPUT \"http://<ホスト名>:<ポート番号>/_template/<テンプレート名>\" -d @- << EOF\n", "{\n", " \"template\" : \"<インデックス名>\",\n", " \"settings\": {\n", " \"number_of_shards\": <シャード数>\n", " }\n", " \"mappings\": {\n", " ...\n", " }\n", "}\n", "EOF" ] }, { "cell_type": "markdown", "metadata": { "hidden": true, "lc_cell_meme": { "current": "a6fc795c-14ed-11e7-b31a-0242ac12000d", "next": null, "previous": "a6fc7790-14ed-11e7-b31a-0242ac12000d" } }, "source": [ "- reindexを実行し、新しいインデックスにデータを移動します。 \n", "reindexについては[05_Indexing.ipynb](05_Indexing.ipynb#reindex-APIを使った投入後データの加工方法)を参考にしてください。 \n", "このreindexの説明ではデータを加工していますが、\"script\"や\"pipeline\"を指定しないようにすることで加工せずにデータの移動のみ行うことができます。" ] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { "display_name": "Python 2", "language": "python", "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.9" }, "latex_envs": { "bibliofile": "biblio.bib", "cite_by": "apalike", "current_citInitial": 1, "eqLabelWithNumbers": true, "eqNumInitial": 0 }, "lc_notebook_meme": { "current": "a6fb9e56-14ed-11e7-b31a-0242ac12000d", "lc_server_signature": { "history": [] } }, "toc": { "colors": { "hover_highlight": "#DAA520", "navigate_num": "#000000", "navigate_text": "#333333", "running_highlight": "#FF0000", "selected_highlight": "#FFD700", "sidebar_border": "#EEEEEE", "wrapper_background": "#FFFFFF" }, "moveMenuLeft": true, "nav_menu": { "height": "117px", "width": "252px" }, "navigate_menu": true, "number_sections": true, "sideBar": true, "threshold": 4, "toc_cell": false, "toc_position": { "height": "840px", "left": "0px", "right": "1168px", "top": "130px", "width": "268px" }, "toc_section_display": "block", "toc_window_display": true, "widenNotebook": false } }, "nbformat": 4, "nbformat_minor": 1 }